temp_final_summary.txt

================================================================================
JAO DATA STRUCTURE VERIFICATION - FINAL REPORT
================================================================================

QUESTION: What should be the forecast target for "max capacity in a given direction"?

================================================================================
1. JAO DATA TYPES IDENTIFIED
================================================================================

A. DIRECTIONAL FLOW COLUMNS (CZ>PL, PL>CZ format)
   - Total: 132 columns (12 x 11 bidirectional combinations)
   - Source: MaxBEX dataset from JAO
   - Represents: Maximum Bilateral Exchange Capacity (hub-to-hub)
   - Type: Commercial trading capacity (MW)
   - Includes: ALL zone pairs (physical + virtual borders)

   Example values for CZ<->PL:
shape: (5, 3)
┌────────────────────────────────┬────────┬────────┐
│ mtu                            ┆ CZ>PL  ┆ PL>CZ  │
│ ---                            ┆ ---    ┆ ---    │
│ datetime[ns, Europe/Amsterdam] ┆ f64    ┆ f64    │
╞════════════════════════════════╪════════╪════════╡
│ 2023-10-01 02:00:00 CEST       ┆ 2785.0 ┆ 3883.0 │
│ 2023-10-01 03:00:00 CEST       ┆ 2711.0 ┆ 3775.0 │
│ 2023-10-01 04:00:00 CEST       ┆ 2831.0 ┆ 3787.0 │
│ 2023-10-01 05:00:00 CEST       ┆ 2778.0 ┆ 3361.0 │
│ 2023-10-01 06:00:00 CEST       ┆ 2744.0 ┆ 3057.0 │
└────────────────────────────────┴────────┴────────┘

   Statistics (CZ<->PL):
shape: (1, 6)
┌───────────────┬───────────────┬──────────────┬──────────────┬──────────────┬──────────────┐
│ CZ>PL_mean_MW ┆ PL>CZ_mean_MW ┆ CZ>PL_min_MW ┆ PL>CZ_min_MW ┆ CZ>PL_max_MW ┆ PL>CZ_max_MW │
│ ---           ┆ ---           ┆ ---          ┆ ---          ┆ ---          ┆ ---          │
│ f64           ┆ f64           ┆ f64          ┆ f64          ┆ f64          ┆ f64          │
╞═══════════════╪═══════════════╪══════════════╪══════════════╪══════════════╪══════════════╡
│ 3481.789045   ┆ 2697.566404   ┆ 144.0        ┆ 0.0          ┆ 5699.0       ┆ 4631.0       │
└───────────────┴───────────────┴──────────────┴──────────────┴──────────────┴──────────────┘

B. BORDER COLUMNS (border_CZ_PL format)
   - Total: 38 columns
   - Source: LTA (Long-Term Allocations) dataset from JAO
   - Represents: Pre-allocated capacity from long-term contracts (MW)
   - Type: Allocated capacity (reduces available MaxBEX)
   - Includes: ONLY physical borders with direct interconnectors

   Example values for CZ-PL border:
shape: (5, 3)
┌────────────────────────────────┬──────────────┬──────────────┐
│ mtu                            ┆ border_CZ_PL ┆ border_PL_CZ │
│ ---                            ┆ ---          ┆ ---          │
│ datetime[ns, Europe/Amsterdam] ┆ i64          ┆ i64          │
╞════════════════════════════════╪══════════════╪══════════════╡
│ 2023-10-01 02:00:00 CEST       ┆ 0            ┆ 0            │
│ 2023-10-01 03:00:00 CEST       ┆ 0            ┆ 0            │
│ 2023-10-01 04:00:00 CEST       ┆ 0            ┆ 0            │
│ 2023-10-01 05:00:00 CEST       ┆ 0            ┆ 0            │
│ 2023-10-01 06:00:00 CEST       ┆ 0            ┆ 0            │
└────────────────────────────────┴──────────────┴──────────────┘

   Statistics (CZ-PL border):
shape: (1, 4)
┌──────────────────────┬──────────────────────┬───────────────────────┬───────────────────────┐
│ border_CZ_PL_mean_MW ┆ border_PL_CZ_mean_MW ┆ border_CZ_PL_total_MW ┆ border_PL_CZ_total_MW │
│ ---                  ┆ ---                  ┆ ---                   ┆ ---                   │
│ f64                  ┆ f64                  ┆ i64                   ┆ i64                   │
╞══════════════════════╪══════════════════════╪═══════════════════════╪═══════════════════════╡
│ 0.0                  ┆ 9.573358             ┆ 0                     ┆ 167955                │
└──────────────────────┴──────────────────────┴───────────────────────┴───────────────────────┘

================================================================================
2. KEY DIFFERENCES
================================================================================

DIRECTIONAL COLUMNS (CZ>PL):
- MaxBEX = Commercial trading capacity in specific direction
- CZ>PL != PL>CZ (asymmetric, depends on network constraints)
- Avg CZ>PL: 3,482 MW vs Avg PL>CZ: 2,698 MW (significant difference!)
- Calculated by JAO optimization considering ALL network constraints
- THIS IS THE FORECAST TARGET!

BORDER COLUMNS (border_CZ_PL):
- LTA = Long-term allocated capacity (pre-sold)
- Only exists for 38 physical borders (not all 132 zone pairs)
- Much smaller values (avg border_CZ_PL: 0 MW, border_PL_CZ: 9.6 MW)
- Acts as INPUT/CONSTRAINT to MaxBEX calculation
- NOT a capacity forecast target

================================================================================
3. RELATIONSHIP BETWEEN MaxBEX AND LTA
================================================================================

From JAO documentation:
  MaxBEX (available capacity) = Optimized capacity - LTA allocations

  LTA reduces available MaxBEX because capacity is pre-sold in:
  - Yearly auctions
  - Monthly auctions
  - Other long-term contracts

================================================================================
4. VERIFICATION: PHYSICAL vs VIRTUAL BORDERS
================================================================================

Physical borders (with LTA): 38
Total MaxBEX pairs: 132 (12 x 11)
Virtual borders: 94 (zone pairs without physical interconnectors)

================================================================================
5. FINAL ANSWER
================================================================================

TARGET FOR FORECASTING "Max Capacity in a Given Direction":

  USE: Directional columns (CZ>PL, PL>CZ, DE>FR, etc.)
  - These are MaxBEX values = commercial trading capacity
  - Represents actual available capacity in that specific direction
  - Accounts for network constraints, LTA allocations, and physics
  - 132 total targets (all zone-pair combinations)

  DO NOT USE: border_ columns (border_CZ_PL, border_PL_CZ, etc.)
  - These are LTA values = pre-allocated capacity
  - Should be used as INPUT FEATURES (future covariates)
  - Only 38 physical borders (incomplete coverage)
  - Much smaller values (often near zero)

================================================================================
6. CURRENT IMPLEMENTATION STATUS
================================================================================

[OK] The change from border_* to directional columns was CORRECT!

Before: Using border_CZ_PL (LTA allocations) as targets
  - WRONG: Forecasting pre-allocated capacity (not meaningful)
  - Only 38 borders covered
  - Very low values (mostly zeros)

After: Using CZ>PL directional columns (MaxBEX) as targets
  - CORRECT: Forecasting commercial trading capacity
  - All 132 zone pairs covered
  - Represents actual "max capacity in given direction"
  - Values match expected capacity ranges (hundreds to thousands of MW)

================================================================================
END OF REPORT
================================================================================