• Experiment Tracking & Doc Writing
  • Experiment
    • 1. Use configs: for all pipeline-level hyperparameters
    • 2. Save results: config, logs, prediction, metrics, (panels)
    • 3. Have one script to compare: *.ipynb
  • Weekly Doc Writing
    • Structure

Experiment Tracking & Doc Writing Link to heading

Experiment Link to heading

1. Use configs
2. Save results
3. Have one script to compare

1. Use configs: for all pipeline-level hyperparameters Link to heading

• Base config: captures every optional feature in the pipeline, shared across experiments
• Experiment config: modifies 1–2 items from the base config (overrides)
• Stored config: for each experiment run, make sure it can be traced back later

Recommend using *.yaml for better readability example

- name: '01.SOL'
  description: 'SOL'
  overrides: 
    symbol: "SOL"
    data_file_prefix: "20251104"

- name: '02.BTC'
  description: 'BTC'
  overrides: 
    symbol: "BTC"
    data_file_prefix: "20251104"

2. Save results: config, logs, prediction, metrics, (panels) Link to heading

results
├── (20251105)
│   └── 20251105_030057.00.SOL
├── 01.SOL
│   ├── all_feature_coefficients.csv
│   ├── all_feature_coefficients.parquet
│   ├── config.json
│   ├── config.yaml
│   ├── hyperparameter_history.json
│   ├── metrics.json
│   ├── pipeline_20251105_032010.log
│   └── predictions.parquet
└── 02.BTC
    ├── all_feature_coefficients.csv
    ├── all_feature_coefficients.parquet
    ├── config.json
    ├── config.yaml
    ├── hyperparameter_history.json
    ├── metrics.json
    ├── pipeline_20251105_035457.log
    └── predictions.parquet

3. Have one script to compare: *.ipynb Link to heading

• set up: result dir, start_date, end_date (, baseline)

• tell the difference in configs (experiment names)

• show metrics of graphs & tables

• export to *.html: ipynb + quarto

  quarto render "$notebook_path"
  

• in distinct result dir, we have distinct groups for comparison

  • so the table & graphs only focus on current ablation analysis

Weekly Doc Writing Link to heading

use Typora to write *.md

• toc: content table
• slide content size 16:9 (for each page)
• marp: export to ppt/html/pdf

Structure Link to heading

• Finished jobs
  • track new dev & changes (complement of git)
• Findings
  • any results of interest (whether structured or to-be-analysis)
  • doubts
• Discussion
  • open questions (for GPT prompting and for team-members to know)
• TODOs
  • always track & update that