docs: update activity.md with Session 11 progress

Session 11: CUDA OOM Troubleshooting & Memory Optimization
- Documented root cause investigation (batch_size=256, 9 quantiles)
- Explained memory optimization fix (commit 7a9aff9)
- Listed next steps for resuming work
- Status: Waiting for HF Space rebuild

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

doc/activity.md

@@ -2,6 +2,349 @@
 
 ---
 
+## Session 11: CUDA OOM Troubleshooting & Memory Optimization
+**Date**: 2025-11-17
+**Duration**: ~3 hours
+**Status**: IN PROGRESS - Memory fix committed, awaiting HF Space rebuild
+
+### Objectives
+1. ✓ Recover workflow after unexpected session termination
+2. ✓ Validate multivariate forecasting with smoke test
+3. ✓ Diagnose CUDA OOM error (18GB memory usage on 24GB GPU)
+4. ✓ Implement memory optimization fix
+5. ⏳ Run October 2024 evaluation (pending HF Space rebuild)
+6. ⏳ Calculate MAE metrics D+1 through D+14
+7. ⏳ Document results and complete Day 4
+
+### Problem: CUDA Out of Memory Error
+
+**HF Space Error**:
+```
+CUDA out of memory. Tried to allocate 10.75 GiB.
+GPU 0 has a total capacity of 22.03 GiB of which 3.96 GiB is free.
+Including non-PyTorch memory, this process has 18.06 GiB memory in use.
+```
+
+**Initial Confusion**: Why is 18GB being used for:
+- Model: Chronos-2 (120M params) = ~240MB in bfloat16
+- Data: 25MB parquet file
+- Context: 256h × 615 features
+
+This made no sense - should require <2GB total.
+
+### Root Cause Investigation
+
+Investigated multiple potential causes:
+1. **Historical features in context** - Initially suspected 2,514 features (603+12+1899) was the issue
+2. **User challenge** - Correctly questioned whether historical features should be excluded
+3. **Documentation review** - Confirmed context SHOULD include historical features (for pattern learning)
+4. **Deep dive into defaults** - Found the real culprits
+
+### Root Causes Identified
+
+#### 1. Default batch_size = 256 (not overridden)
+```python
+# predict_df() default parameters
+batch_size: 256  # Processes 256 rows in parallel!
+```
+
+With 256h context × 2,514 features × batch_size 256 → massive memory allocation
+
+#### 2. Default quantile_levels = 9 quantiles
+```python
+quantile_levels: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]  # Computing 9 quantiles
+```
+
+We only use 3 quantiles (0.1, 0.5, 0.9) - the other 6 waste GPU memory
+
+#### 3. Transformer attention memory explosion
+Chronos-2's group attention mechanism creates intermediate tensors proportional to:
+- (sequence_length × num_features)²
+- With batch_size=256 and 9 quantiles, memory explodes exponentially
+
+### The Fix (Commit 7a9aff9)
+
+**Changed**: `src/forecasting/chronos_inference.py` lines 203-213
+
+```python
+# BEFORE (using defaults)
+forecasts_df = pipeline.predict_df(
+    context_data,
+    future_df=future_data,
+    prediction_length=prediction_hours,
+    id_column='border',
+    timestamp_column='timestamp',
+    target='target'
+    # batch_size defaults to 256
+    # quantile_levels defaults to [0.1-0.9] (9 values)
+)
+
+# AFTER (memory optimized)
+forecasts_df = pipeline.predict_df(
+    context_data,
+    future_df=future_data,
+    prediction_length=prediction_hours,
+    id_column='border',
+    timestamp_column='timestamp',
+    target='target',
+    batch_size=32,  # Reduce from 256 → ~87% memory reduction
+    quantile_levels=[0.1, 0.5, 0.9]  # Only compute needed quantiles → ~67% reduction
+)
+```
+
+**Expected Memory Savings**:
+- batch_size: 256 → 32 = ~87% reduction
+- quantiles: 9 → 3 = ~67% reduction
+- **Combined**: ~95% reduction in inference memory usage
+
+**Impact on Quality**:
+- **NONE** - batch_size only affects computation speed, not forecast values
+- **NONE** - we only use 3 quantiles anyway, others were discarded
+
+### Git Activity
+
+```
+7a9aff9 - fix: reduce batch_size to 32 and quantiles to 3 for GPU memory optimization
+  - Comprehensive commit message documenting the fix
+  - No quality impact (batch_size is computational only)
+  - Should resolve CUDA OOM on 24GB L4 GPU
+```
+
+Pushed to GitHub: https://github.com/evgspacdmy/fbmc_chronos2
+
+### Files Modified
+- `src/forecasting/chronos_inference.py` - Added batch_size and quantile_levels parameters
+- `scripts/evaluate_october_2024.py` - Created evaluation script (uses local data)
+
+### Testing Results
+
+**Smoke Test (before fix)**:
+- ✓ Single border (AT_CZ) works fine
+- ✓ Forecast shows variation (mean 287 MW, std 56 MW)
+- ✓ API connection successful
+
+**Full 38-border test (before fix)**:
+- ✗ CUDA OOM on first border
+- Error shows 18GB usage + trying to allocate 10.75GB
+- Returns debug file instead of parquet
+
+**Full 38-border test (after fix)**:
+- ⏳ Waiting for HF Space rebuild with commit 7a9aff9
+- HF Spaces auto-rebuild can take 5-20 minutes
+- May require manual "Factory Rebuild" from Space settings
+
+### Current Status
+
+- [x] Root cause identified (batch_size=256, 9 quantiles)
+- [x] Memory optimization implemented
+- [x] Committed to git (7a9aff9)
+- [x] Pushed to GitHub
+- [ ] HF Space rebuild (in progress)
+- [ ] Smoke test validation (pending rebuild)
+- [ ] Full Oct 1-14, 2024 forecast (pending rebuild)
+- [ ] Calculate MAE D+1 through D+14 (pending forecast)
+- [ ] Document results in activity.md (pending evaluation)
+
+### Next Steps (Resume Here Next Session)
+
+**PRIORITY 1**: Wait for HF Space rebuild or trigger manual Factory Rebuild
+- Go to: https://huggingface.co/spaces/evgueni-p/fbmc-chronos2/settings
+- Click "Factory Rebuild" if auto-rebuild hasn't triggered
+- Wait ~5 minutes for rebuild completion
+
+**PRIORITY 2**: Validate memory fix works
+```bash
+cd /c/Users/evgue/projects/fbmc_chronos2
+.venv/Scripts/python.exe scripts/evaluate_october_2024.py
+```
+
+**PRIORITY 3**: If successful, proceed with original Day 4 plan
+- Calculate MAE metrics for D+1 through D+14
+- Update activity.md with Session 11 results
+- Create HANDOVER_GUIDE.md
+- Archive test scripts
+- Commit and push final results
+
+**PRIORITY 4**: If still OOM, consider alternatives:
+- Further reduce batch_size to 16 or 8
+- Reduce context_hours from 256h to 128h
+- Document as limitation requiring A100 40GB GPU for production
+
+### Key Learnings
+
+1. **Always check default parameters** - Libraries often have defaults optimized for different use cases
+2. **batch_size doesn't affect quality** - It's purely a computational optimization parameter
+3. **Memory usage isn't linear** - Transformer attention creates quadratic memory growth
+4. **Test with realistic data sizes** - Smoke tests (1 border) can hide multi-border issues
+5. **Document assumptions** - User correctly challenged the historical features assumption
+6. **HF Space rebuild delays** - May need manual trigger, not instant after push
+
+### Technical Notes
+
+**Why batch_size=32 vs 256**:
+- batch_size controls parallel processing of rows within a single border forecast
+- Larger = faster but more memory
+- Smaller = slower but less memory
+- **No impact on final forecast values** - same predictions either way
+
+**Context features breakdown**:
+- Full-horizon D+14: 603 features (always available)
+- Partial D+1: 12 features (load forecasts)
+- Historical: 1,899 features (prices, gen, demand)
+- **Total context**: 2,514 features
+- **Future covariates**: 615 features (603 + 12)
+
+**Why historical features in context**:
+- Help model learn patterns from past behavior
+- Not available in future (can't forecast price/demand)
+- But provide context for understanding historical trends
+- Standard practice in time series forecasting with covariates
+
+---
+
+**Status**: [IN PROGRESS] Waiting for HF Space rebuild with memory optimization
+**Timestamp**: 2025-11-17 16:30 UTC
+**Next Action**: Trigger Factory Rebuild or wait for auto-rebuild, then run evaluation
+
+---
+
+## Session 10: CRITICAL FIX - Enable Multivariate Covariate Forecasting
+**Date**: 2025-11-15
+**Duration**: ~2 hours
+**Status**: CRITICAL REGRESSION FIXED - Awaiting HF Space rebuild
+
+### Critical Issue Discovered
+
+**Problem**: HF Space deployment was using **univariate forecasting** (target values only), completely ignoring all 615 collected features!
+
+**Impact**:
+- Weather per zone: IGNORED
+- Generation per zone: IGNORED
+- CNEC outages (200 CNECs): IGNORED
+- LTA allocations: IGNORED
+- Load forecasts: IGNORED
+
+**Root Cause**: When optimizing for batch inference in Session 9, we switched from DataFrame API (`predict_df()`) to tensor API (`predict()`), which doesn't support covariates. The entire covariate-informed forecasting capability was accidentally disabled.
+
+### The Fix (Commit 0b4284f)
+
+**Changes Made**:
+
+1. **Switched to Chronos2Pipeline** - Model that supports covariates
+   ```python
+   # OLD (Session 9)
+   from chronos import ChronosPipeline
+   pipeline = ChronosPipeline.from_pretrained("amazon/chronos-t5-large")
+
+   # NEW (Session 10)
+   from chronos import Chronos2Pipeline
+   pipeline = Chronos2Pipeline.from_pretrained("amazon/chronos-2")
+   ```
+
+2. **Changed inference API** - DataFrame API supports covariates
+   ```python
+   # OLD - Tensor API (univariate only)
+   forecasts = pipeline.predict(
+       inputs=batch_tensor,  # Only target values!
+       prediction_length=168
+   )
+
+   # NEW - DataFrame API (multivariate with covariates)
+   forecasts = pipeline.predict_df(
+       context_data,         # Historical data with ALL features
+       future_df=future_data,  # Future covariates (615 features)
+       prediction_length=168,
+       id_column='border',
+       timestamp_column='timestamp',
+       target='target'
+   )
+   ```
+
+3. **Model configuration updates**:
+   - Model: `amazon/chronos-t5-large` → `amazon/chronos-2`
+   - Dtype: `bfloat16` → `float32` (required for chronos-2)
+
+4. **Removed batch inference** - Reverted to per-border processing to enable covariate support
+   - Per-border processing allows full feature utilization
+   - Chronos-2's group attention mechanism shares information across covariates
+
+**Files Modified**:
+- `src/forecasting/chronos_inference.py` (v1.1.0):
+  - Lines 1-22: Updated imports and docstrings
+  - Lines 31-47: Changed model initialization
+  - Lines 66-70: Updated model loading
+  - Lines 164-252: Complete inference rewrite for covariates
+
+**Expected Impact**:
+- **Significantly improved forecast accuracy** by leveraging all 615 collected features
+- Model now uses Chronos-2's in-context learning with exogenous features
+- Zero-shot multivariate forecasting as originally intended
+
+### Git Activity
+
+```
+0b4284f - feat: enable multivariate covariate forecasting with 615 features
+  - Switch from ChronosPipeline to Chronos2Pipeline
+  - Change from predict() to predict_df() API
+  - Now passes both context_data AND future_data
+  - Enables zero-shot multivariate forecasting capability
+```
+
+Pushed to:
+- GitHub: https://github.com/evgspacdmy/fbmc_chronos2
+- HF Space: https://huggingface.co/spaces/evgueni-p/fbmc-chronos2 (rebuild in progress)
+
+### Current Status
+
+- [x] Code changes complete
+- [x] Committed to git (0b4284f)
+- [x] Pushed to GitHub
+- [ ] HF Space rebuild (in progress)
+- [ ] Smoke test validation
+- [ ] Full Oct 1-14 forecast with covariates
+- [ ] Calculate MAE D+1 through D+14
+
+### Next Steps
+
+1. **PRIORITY 1**: Wait for HF Space rebuild with commit 0b4284f
+2. **PRIORITY 2**: Run smoke test and verify logs show "Using 615 future covariates"
+3. **PRIORITY 3**: Run full Oct 1-14, 2024 forecast with all 38 borders
+4. **PRIORITY 4**: Calculate MAE for D+1 through D+14 (user's explicit request)
+5. **PRIORITY 5**: Compare accuracy vs univariate baseline (Session 9 results)
+6. **PRIORITY 6**: Document final results and handover
+
+### Key Learnings
+
+1. **API mismatch risk**: Tensor API vs DataFrame API have different capabilities
+2. **Always verify feature usage**: Don't assume features are being used without checking
+3. **Regression during optimization**: Speed improvements can accidentally break functionality
+4. **Testing is critical**: Should have validated feature usage in Session 9
+5. **User feedback essential**: User caught the issue immediately
+
+### Technical Notes
+
+**Why Chronos-2 supports multivariate forecasting in zero-shot**:
+- Group attention mechanism shares information across time series AND covariates
+- In-context learning (ICL) handles arbitrary exogenous features
+- No fine-tuning required - works in zero-shot mode
+- Model pre-trained on diverse time series with various covariate patterns
+
+**Feature categories now being used**:
+- Weather: 52 grid points × multiple variables = ~200 features
+- Generation: 13 zones × fuel types = ~100 features
+- CNEC outages: 200 CNECs with weighted binding scores = ~200 features
+- LTA: Long-term allocations per border = ~38 features
+- Load forecasts: Per-zone load predictions = ~77 features
+- **Total**: 615 features actively used in multivariate forecasting
+
+---
+
+**Status**: [IN PROGRESS] Waiting for HF Space rebuild at commit 0b4284f
+**Timestamp**: 2025-11-15 23:20 UTC
+**Next Action**: Monitor rebuild, then test smoke test with covariate logs
+
+---
+
 ## Session 9: Batch Inference Optimization & GPU Memory Management
 **Date**: 2025-11-15
 **Duration**: ~4 hours