Experiment Tracking & Doc Writing Link to heading

Experiment Link to heading

  1. Use configs
  2. Save results
  3. Have one script to compare

1. Use configs: for all pipeline-level hyperparameters Link to heading

  • 基础配置: 体现 Pipeline 中所有可选功能, 在实验之间共享
  • 实验配置: 修改 1~2 项基础配置 (覆盖)
  • 存储配置: 每次实验时, 确保能追溯

建议使用 *.yaml 可读性更好 example

- name: '01.SOL'
  description: 'SOL'
  overrides: 
    symbol: "SOL"
    data_file_prefix: "20251104"

- name: '02.BTC'
  description: 'BTC'
  overrides: 
    symbol: "BTC"
    data_file_prefix: "20251104"

2. Save results: config, logs, prediction, metrics, (panels) Link to heading

results
├── (20251105)
│   └── 20251105_030057.00.SOL
├── 01.SOL
│   ├── all_feature_coefficients.csv
│   ├── all_feature_coefficients.parquet
│   ├── config.json
│   ├── config.yaml
│   ├── hyperparameter_history.json
│   ├── metrics.json
│   ├── pipeline_20251105_032010.log
│   └── predictions.parquet
└── 02.BTC
    ├── all_feature_coefficients.csv
    ├── all_feature_coefficients.parquet
    ├── config.json
    ├── config.yaml
    ├── hyperparameter_history.json
    ├── metrics.json
    ├── pipeline_20251105_035457.log
    └── predictions.parquet

3. Have one script to compare: *.ipynb Link to heading

  • set up: result dir, start_date, end_date (, baseline)

  • tell the difference in configs (experiment names)

  • show metrics of graphs & tables

  • export to *.html: ipynb + quarto example

    quarto render "$notebook_path"

  • in distinct result dir, we have distinct groups for comparison

    • so the table & graphs only focus on current ablation analysis

Weekly Doc Writing Link to heading

use Typora to write *.md

  • toc: content table
  • slide content size 16:9 (for each page)
  • marp: export to ppt/html/pdf

Structure Link to heading

  • Finished jobs
    • track new dev & changes (complement of git)
  • Findings
    • any results of interest (whether structured or to-be-analysis)
    • doubts
  • Discussion
    • open questions (for GPT prompting and for team-members to know)
  • TODOs
    • always track & update that

example slide