feat: fix future covariate architecture (615 features: temporal, weather, 176 CNEC outages)

Critical fixes to future covariate identification for Chronos 2 inference:

ISSUE 1: CNEC Transmission Outages (31 → 176 features)
- Problem: Only 31 CNECs with historical outages had features
- During inference, any of 176 CNECs could have planned future outages
- Model was blind to 82% of transmission network outages
- Solution: Preserve all 176 zero-filled CNEC features in cleanup

ISSUE 2: Weather Features Not Marked (0 → 375 features)
- Problem: 375 weather features existed but not marked as future covariates
- ECMWF D+15 forecasts wouldn't be used by model
- Solution: Include weather_cols in metadata future covariate check

ISSUE 3: Temporal Features Missing (0 → 12 features)
- Problem: Deterministic features (hour, day, weekday) not marked as future
- Model couldn't leverage known temporal patterns
- Solution: Add temporal_cols to future covariate identification

Changes:
- src/feature_engineering/engineer_entsoe_features.py:
* Skip CNEC outages in zero-variance cleanup (lines 843-858)
* Skip CNEC outages in duplicate removal (lines 860-883)
* Result: 296 → 441 ENTSO-E features (+145 CNEC outages)

- notebooks/05_unified_features_final.py:
* Add temporal features to identification (lines 287-320)
* Include temporal + weather in metadata (lines 931-976)
* Update summary table and text (lines 382-418, 1034)

- doc/activity.md:
* Comprehensive documentation of fixes and rationale
* Future covariate architecture and inference strategy
* Data quality validation results

Results:
- Total features: 2,408 → 2,553 (+145)
- Future covariates: 83 → 615 (+532)
* Temporal: 12 (deterministic)
* LTA: 40 (years ahead)
* Load Forecasts: 12 (D+1)
* CNEC Outages: 176 (D+22)
* Weather: 375 (D+15 ECMWF)
- Historical features: 1,938

Data regenerated:
- features_entsoe_24month.parquet (10.67 MB, 441 features)
- features_unified_24month.parquet (24.9 MB, 2,553 features)
- features_unified_metadata.csv (615 future covariates marked)

All validations passed. Ready for Day 3 zero-shot inference.

doc/activity.md

@@ -3260,15 +3260,278 @@ Successfully unified all three feature sets (JAO, ENTSO-E, Weather) into a singl
 
 ---
 
+## 2025-11-11 - Future Covariate Architecture Fixed ✅
+
+### Summary
+Identified and fixed critical gaps in future covariate identification: CNEC transmission outages (31 → 176), weather features (not marked), and temporal features (missing). Rebuilt ENTSO-E feature engineering with cleanup safeguards, regenerated unified dataset, and updated metadata. Final system: **2,553 features (615 future, 1,938 historical)** ready for Chronos 2 zero-shot inference.
+
+### Issues Identified
+
+#### 1. CNEC Transmission Outages Insufficient (31 vs 176)
+**Problem**: Only 31 CNECs with historical outages had features. During inference, ANY of the 176 master CNECs could have planned future outages, but the model couldn't receive that information.
+
+**Root Cause**: Feature engineering cleanup logic removed 145 zero-filled CNEC features as "zero-variance" and "duplicates."
+
+**Impact**: Model blind to future outages for 145 CNECs (82% of transmission network).
+
+#### 2. Weather Features Not Marked as Future Covariates (0 vs 375)
+**Problem**: 375 weather features exist but metadata marked them as historical, not future covariates.
+
+**Root Cause**: Notebook metadata generation (`create_metadata()` line 942) only checked LTA, load forecasts, and outages - excluded weather.
+
+**Impact**: During inference, ECMWF D+15 forecasts wouldn't be used by Chronos 2.
+
+#### 3. Temporal Features Missing from Future Covariates (0 vs 12)
+**Problem**: Temporal features (hour, day, weekday, etc.) always known deterministically but not marked as future covariates.
+
+**Root Cause**: Not included in future covariate identification logic.
+
+**Impact**: Model couldn't leverage known future temporal patterns.
+
+### Work Completed
+
+#### 1. Fixed ENTSO-E Feature Engineering
+**File**: `src/feature_engineering/engineer_entsoe_features.py`
+
+**Changes Made**:
+```python
+# Line 843-858: Zero-variance cleanup - skip transmission outages
+if col.startswith('outage_cnec_'):
+    continue  # Keep even if zero-filled
+
+# Line 860-883: Duplicate removal - skip transmission outages
+if col1.startswith('outage_cnec_') or col2.startswith('outage_cnec_'):
+    continue  # Each CNEC needs own column for inference
+```
+
+**Result**:
+- Before: 296 ENTSO-E features (31 CNEC outages)
+- After: 441 ENTSO-E features (176 CNEC outages)
+- Change: +145 zero-filled CNEC outage features preserved
+
+**Validation**: All 176 CNEC outage features confirmed present in output file.
+
+#### 2. Updated Unification Notebook
+**File**: `notebooks/05_unified_features_final.py`
+
+**Change 1** (line 287-320): Added temporal features to identification
+```python
+# Added:
+temporal_cols = [c for c in future_cov_all_cols if any(x in c for x in
+    ['hour', 'day', 'month', 'weekday', 'year', 'weekend', '_sin', '_cos'])]
+
+# Updated return:
+return temporal_cols, lta_cols, load_forecast_cols, outage_cols, weather_cols, future_cov_counts
+```
+
+**Change 2** (line 382-418): Added temporal row to summary table
+
+**Change 3** (line 931-976): Updated metadata generation
+```python
+# Line 931: Added temporal_cols and weather_cols to function signature
+def create_metadata(pl, categories, temporal_cols, lta_cols, load_forecast_cols,
+                   outage_cols, weather_cols, outage_stats):
+
+# Line 948-952: Include temporal and weather in future covariate check
+meta_is_future = (meta_col in temporal_cols or
+                 meta_col in lta_cols or
+                 meta_col in load_forecast_cols or
+                 meta_col in outage_cols or
+                 meta_col in weather_cols)
+
+# Line 955-966: Added extension periods for temporal and weather
+if meta_col in temporal_cols:
+    meta_extension_days = 'Full horizon (deterministic)'
+elif meta_col in weather_cols:
+    meta_extension_days = '15 days (D+15 ECMWF)'
+```
+
+**Change 4** (line 1034): Updated summary text (87 → 615)
+
+#### 3. Regenerated All Outputs
+
+**Step 1**: Re-ran ENTSO-E feature engineering
+```bash
+.venv\Scripts\python.exe src\feature_engineering\engineer_entsoe_features.py
+```
+- Output: 441 ENTSO-E features (176 CNEC outages preserved)
+- File: `data/processed/features_entsoe_24month.parquet` (10.67 MB)
+
+**Step 2**: Re-ran unification
+```bash
+.venv\Scripts\python.exe scripts\unify_features_checkpoint.py
+```
+- Output: 2,553 total features (17,544 hours × 2,553 columns)
+- File: `data/processed/features_unified_24month.parquet` (24.9 MB)
+
+**Step 3**: Regenerated metadata with updated logic
+- Custom script with temporal + weather future covariate marking
+- Output: `data/processed/features_unified_metadata.csv`
+- Result: 615 future covariates correctly identified
+
+### Final Feature Architecture
+
+#### Total Feature Count: 2,553
+
+| Source | Features | Description |
+|--------|----------|-------------|
+| JAO | 1,737 | CNECs, borders, net positions, LTA, temporal |
+| ENTSO-E | 441 | Generation, demand, prices, load forecasts, outages (176 CNECs) |
+| Weather | 375 | Temperature, wind, solar, cloud, pressure, lags, derived |
+| **TOTAL** | **2,553** | **Complete FBMC feature set** |
+
+#### Future Covariate Breakdown: 615
+
+| Category | Count | Extension Period | Purpose |
+|----------|-------|------------------|---------|
+| **Temporal** | 12 | Full horizon (deterministic) | Hour, day, weekday always known |
+| **LTA** | 40 | Full horizon (years) | Auction results known in advance |
+| **Load Forecasts** | 12 | D+1 (1 day) | TSO demand forecasts |
+| **CNEC Outages** | 176 | Up to D+22 | Planned transmission maintenance |
+| **Weather** | 375 | D+15 (15 days) | ECMWF IFS 0.25° forecasts |
+| **TOTAL** | **615** | **Variable** | **24.1% of features** |
+
+#### Historical Features: 1,938
+
+These include:
+- CNEC binding/RAM/utilization (historical congestion)
+- Border flows and capacities (historical)
+- Net positions (historical)
+- PTDF coefficients and interactions
+- Generation by type (historical)
+- Day-ahead prices (historical)
+- Hydro storage levels (historical)
+
+### Data Quality Validation
+
+**Unified Dataset**:
+- Dimensions: 17,544 rows × 2,553 columns
+- Date range: Oct 1, 2023 - Sept 30, 2025 (24 months, hourly)
+- File size: 24.9 MB (compressed parquet)
+- Timestamp continuity: 100% (no gaps)
+
+**Completeness by Category**:
+- Temporal: 100%
+- LTA: 100%
+- Border capacity: 99.86%
+- Net positions: 100%
+- Load forecasts: 99.73%
+- Transmission outages: 100% (binary: 0 or 1)
+- Weather: 100%
+- Generation/demand: 99.85%
+- **CNEC features: 26.41%** (expected sparsity - congestion is occasional)
+
+**Overall Completeness**: 57.11% (due to expected CNEC sparsity)
+
+### Files Modified
+
+**Code Changes**:
+1. `src/feature_engineering/engineer_entsoe_features.py`
+   - Lines 843-858: Zero-variance cleanup safeguard
+   - Lines 860-883: Duplicate removal safeguard
+
+2. `notebooks/05_unified_features_final.py`
+   - Lines 287-320: Future covariate identification (added temporal)
+   - Lines 382-418: Summary table (added temporal row)
+   - Lines 931-976: Metadata generation (added temporal + weather)
+   - Line 1034: Summary text (87 → 615)
+
+**Data Files Regenerated**:
+1. `data/processed/features_entsoe_24month.parquet`
+   - Size: 10.67 MB
+   - Features: 441 (was 296, +145)
+   - CNEC outages: 176 (was 31, +145)
+
+2. `data/processed/features_unified_24month.parquet`
+   - Size: 24.9 MB
+   - Features: 2,553 (was 2,408, +145)
+   - Rows: 17,544 (unchanged)
+
+3. `data/processed/features_unified_metadata.csv`
+   - Total features: 2,552 (excludes timestamp)
+   - Future covariates: 615 (was 83, +532)
+   - Historical features: 1,937 (was 2,324, -387)
+
+### Key Lessons
+
+1. **Zero-filled features are valid**: For inference, model needs columns for ALL possible future events, even if they never occurred historically. Zero-filled CNECs are placeholders for future outages.
+
+2. **Future covariate marking is critical**: Chronos 2 uses metadata to know which features extend into the forecast horizon. Missing weather marking would have crippled D+1 to D+14 forecasts.
+
+3. **Temporal features are deterministic covariates**: Hour, day, weekday are always known - must be marked as future covariates for model to leverage seasonal/daily patterns.
+
+4. **Cleanup logic needs safeguards**: Aggressive removal of zero-variance/duplicate features can delete valid future covariate placeholders. Must explicitly preserve critical feature categories.
+
+5. **Extension periods matter**: Different covariates extend different horizons:
+   - D+1: Load forecasts (mask D+2 to D+15)
+   - D+15: Weather (ECMWF forecasts)
+   - D+22: Transmission outages
+   - ∞: Temporal (deterministic)
+
+### Inference Strategy (Day 3)
+
+**Future Covariate Handling**:
+1. **Temporal** (12 features): Generate for full D+1 to D+14 horizon (deterministic)
+2. **LTA** (40 features): Truncate to D+15 (known years ahead, no need beyond horizon)
+3. **Load Forecasts** (12 features): Use D+1 values, mask D+2 to D+15 (Chronos handles missing)
+4. **CNEC Outages** (176 features): Collect latest planned outages (up to D+22 available)
+5. **Weather** (375 features): Run `scripts/collect_openmeteo_forecast_latest.py` before inference to get fresh D+15 ECMWF forecasts
+
+**Forecast Extension Pattern**:
+- Historical data: Oct 2023 - Sept 30, 2025 (17,544 hours)
+- Inference from: Oct 1, 2025 00:00 onwards
+- Context window: Last 512 hours (Chronos 2 maximum)
+- Forecast horizon: D+1 to D+14 (336 hours)
+- Future covariates: Extend 615 features forward 336 hours
+
+### Performance Metrics
+
+**Re-engineering Time**:
+- ENTSO-E feature engineering: ~8 minutes
+- Unification: ~15 seconds
+- Metadata regeneration: ~2 seconds
+- Total: ~9 minutes
+
+**Data Sizes**:
+- ENTSO-E features: 10.67 MB (was 10.62 MB, +50 KB)
+- Unified features: 24.9 MB (was ~25 MB, minimal change - zeros compress well)
+- Metadata: ~80 KB (was ~50 KB, +30 KB)
+
+### Next Steps
+
+**Immediate**: Day 3 - Zero-Shot Inference
+1. Create `src/modeling/` directory
+2. Implement Chronos 2 inference pipeline:
+   - Load unified features (2,553 features × 17,544 hours)
+   - Identify 615 future covariates from metadata
+   - Collect fresh weather forecasts (D+15)
+   - Generate temporal features for forecast horizon
+   - Prepare context window (last 512 hours)
+   - Run zero-shot inference (D+1 to D+14)
+   - Save predictions
+
+3. Performance targets:
+   - Inference time: <5 minutes per 14-day forecast
+   - D+1 MAE: <150 MW (target 134 MW)
+   - Memory: <10 GB (A10G GPU compatible)
+
+**Documentation Needed**:
+- Update README with new feature counts
+- Document future covariate extension strategy
+- Add inference preprocessing steps
+
+---
+
 **Status Update**:
 - Day 0: ✅ Setup complete
 - Day 1: ✅ Data collection complete (JAO, ENTSO-E, Weather)
 - Day 2: ✅ Feature engineering complete (JAO, ENTSO-E, Weather)
-- **Day 2.5: ✅ Feature unification complete** (2,408 features ready)
+- Day 2.5: ✅ Feature unification complete (2,408 → 2,553 features)
+- **Day 2.75: ✅ Future covariate architecture fixed** (615 future covariates)
 - Day 3: ⏳ Zero-shot inference (NEXT)
 - Day 4: ⏳ Evaluation
 - Day 5: ⏳ Documentation + handover
 
 **NEXT SESSION BOOKMARK**: Day 3 - Implement Chronos 2 zero-shot inference pipeline
 
-**Ready for Inference**: ✅ Unified dataset validated and production-ready
+**Ready for Inference**: ✅ Unified dataset with complete future covariate architecture

notebooks/05_unified_features_final.py

@@ -288,13 +288,17 @@ def identify_future_covariates(pl, unified_df):
     """Identify all future covariate features.
 
     Future covariates:
-    1. LTA (lta_*): Known years in advance
-    2. Load forecasts (load_forecast_*): D+1
-    3. Transmission outages (outage_cnec_*): Variable (check actual data)
-    4. Weather (temp_*, wind*, solar_*, cloud*, pressure*): D+10 (ECMWF HRES forecasts)
+    1. Temporal (hour, day, etc.): Known deterministically
+    2. LTA (lta_*): Known years in advance
+    3. Load forecasts (load_forecast_*): D+1
+    4. Transmission outages (outage_cnec_*): Up to D+22
+    5. Weather (temp_*, wind*, solar_*, etc.): D+15 via ECMWF forecasts
     """
     future_cov_all_cols = unified_df.columns
 
+    # Temporal features (deterministic)
+    temporal_cols = [c for c in future_cov_all_cols if any(x in c for x in ['hour', 'day', 'month', 'weekday', 'year', 'weekend', '_sin', '_cos'])]
+
     # Identify by prefix
     lta_cols = [c for c in future_cov_all_cols if c.startswith('lta_')]
     load_forecast_cols = [c for c in future_cov_all_cols if c.startswith('load_forecast_')]
@@ -305,14 +309,15 @@ def identify_future_covariates(pl, unified_df):
     weather_cols = [c for c in future_cov_all_cols if any(c.startswith(p) for p in weather_prefixes)]
 
     future_cov_counts = {
+        'Temporal': len(temporal_cols),
         'LTA': len(lta_cols),
         'Load Forecasts': len(load_forecast_cols),
         'Transmission Outages': len(outage_cols),
         'Weather': len(weather_cols),
-        'Total': len(lta_cols) + len(load_forecast_cols) + len(outage_cols) + len(weather_cols)
+        'Total': len(temporal_cols) + len(lta_cols) + len(load_forecast_cols) + len(outage_cols) + len(weather_cols)
     }
 
-    return lta_cols, load_forecast_cols, outage_cols, weather_cols, future_cov_counts
+    return temporal_cols, lta_cols, load_forecast_cols, outage_cols, weather_cols, future_cov_counts
 
 
 @app.cell
@@ -379,7 +384,7 @@ def display_future_cov_summary(mo, future_cov_counts, outage_stats):
     outage_ext = f"{outage_stats['extension_days']} days" if outage_stats['extension_days'] is not None else "N/A"
 
     # Calculate percentage of future covariates
-    total_pct = (future_cov_counts['Total'] / 2410) * 100  # ~2,410 total features
+    total_pct = (future_cov_counts['Total'] / 2553) * 100  # ~2,553 total features
 
     mo.md(
         f"""
@@ -387,6 +392,7 @@ def display_future_cov_summary(mo, future_cov_counts, outage_stats):
 
         | Category | Count | Extension Period | Description |
         |----------|-------|------------------|-------------|
+        | Temporal | {future_cov_counts['Temporal']} | Full horizon (deterministic) | Hour, day, weekday, etc. always known |
         | LTA (Long-Term Allocations) | {future_cov_counts['LTA']} | Full horizon (years) | Auction results known in advance |
         | Load Forecasts | {future_cov_counts['Load Forecasts']} | D+1 (1 day) | TSO demand forecasts, published daily |
         | Transmission Outages | {future_cov_counts['Transmission Outages']} | Up to {outage_ext} | Planned maintenance schedules |
@@ -922,7 +928,7 @@ def section8_header(mo):
 
 
 @app.cell
-def create_metadata(pl, categories, lta_cols, load_forecast_cols, outage_cols, outage_stats):
+def create_metadata(pl, categories, temporal_cols, lta_cols, load_forecast_cols, outage_cols, weather_cols, outage_stats):
     """Create feature metadata file."""
     metadata_rows = []
 
@@ -939,15 +945,23 @@ def create_metadata(pl, categories, lta_cols, load_forecast_cols, outage_cols, o
                 source = 'Unknown'
 
             # Determine if future covariate
-            meta_is_future = meta_col in lta_cols or meta_col in load_forecast_cols or meta_col in outage_cols
+            meta_is_future = (meta_col in temporal_cols or
+                             meta_col in lta_cols or
+                             meta_col in load_forecast_cols or
+                             meta_col in outage_cols or
+                             meta_col in weather_cols)
 
             # Determine extension days
-            if meta_col in lta_cols:
+            if meta_col in temporal_cols:
+                meta_extension_days = 'Full horizon (deterministic)'
+            elif meta_col in lta_cols:
                 meta_extension_days = 'Full horizon (years)'
             elif meta_col in load_forecast_cols:
                 meta_extension_days = '1 day (D+1)'
             elif meta_col in outage_cols:
                 meta_extension_days = f"Up to {outage_stats['extension_days']} days" if outage_stats['extension_days'] else 'Variable'
+            elif meta_col in weather_cols:
+                meta_extension_days = '15 days (D+15 ECMWF)'
             else:
                 meta_extension_days = 'N/A (historical)'
 
@@ -1017,7 +1031,7 @@ def display_save_info(mo, save_info):
         - [OK] All 3 data sources merged (JAO + ENTSO-E + Weather)
         - [OK] Timestamps standardized to UTC with hourly frequency
         - [OK] {save_info['features_shape'][1] - 1:,} features engineered and cleaned
-        - [OK] 87 future covariates identified (LTA, load forecasts, outages)
+        - [OK] 615 future covariates identified (temporal, LTA, load forecasts, outages, weather)
         - [OK] Data quality validated (>99% completeness)
         - [OK] Standard decimal precision applied
         - [OK] Metadata file created for feature reference

src/feature_engineering/engineer_entsoe_features.py

@@ -841,34 +841,45 @@ def engineer_all_entsoe_features(
         features = features.drop(list(null_pcts.keys()))
 
     # Remove zero-variance features (constants)
+    # EXCEPT transmission outage features - keep them even if zero-filled for future inference
     zero_var_cols = []
     for col in features.columns:
         if col != 'timestamp':
+            # Skip transmission outage features (needed for future inference)
+            if col.startswith('outage_cnec_'):
+                continue
             # Check if all values are the same (excluding nulls)
             non_null = features[col].drop_nulls()
             if len(non_null) > 0 and non_null.n_unique() == 1:
                 zero_var_cols.append(col)
 
     if zero_var_cols:
-        print(f"\nRemoving {len(zero_var_cols)} zero-variance features")
+        print(f"\nRemoving {len(zero_var_cols)} zero-variance features (keeping transmission outages)")
         features = features.drop(zero_var_cols)
 
     # Remove duplicate columns
+    # EXCEPT transmission outage features - keep all CNECs even if identical (all zeros)
     dup_groups = {}
     cols_to_check = [c for c in features.columns if c != 'timestamp']
 
     for i, col1 in enumerate(cols_to_check):
         if col1 in dup_groups.values():  # Already marked as duplicate
             continue
+        # Skip transmission outage features (each CNEC needs its own column for inference)
+        if col1.startswith('outage_cnec_'):
+            continue
         for col2 in cols_to_check[i+1:]:
             if col2 in dup_groups.values():  # Already marked as duplicate
                 continue
+            # Skip transmission outage features
+            if col2.startswith('outage_cnec_'):
+                continue
             # Check if columns are identical
             if features[col1].equals(features[col2]):
                 dup_groups[col2] = col1  # col2 is duplicate of col1
 
     if dup_groups:
-        print(f"\nRemoving {len(dup_groups)} duplicate columns (keeping first occurrence)")
+        print(f"\nRemoving {len(dup_groups)} duplicate columns (keeping transmission outages)")
         features = features.drop(list(dup_groups.keys()))
 
     features_after = len(features.columns) - 1