Initial commit: CodeFlow AI - NL to SQL Generator

README.md

@@ -0,0 +1,420 @@
+---
+title: CodeFlow AI - Natural Language to Production ETL
+emoji: 🚀
+colorFrom: blue
+colorTo: purple
+sdk: gradio
+sdk_version: 4.8.0
+app_file: app.py
+pinned: true
+tags:
+  - mcp-in-action-track-enterprise
+  - etl
+  - sql
+  - llamaindex
+  - modal
+  - data-engineering
+  - natural-language-processing
+license: mit
+---
+
+# 🚀 CodeFlow AI - Advanced Natural Language to SQL Engine
+
+Transform plain English into production-ready, optimized SQL queries using Claude AI!
+
+## 🏆 Hackathon Submission
+
+**Track:** MCP in Action (Enterprise Category)
+**Sponsors:**
+- 🎯 **LlamaIndex** (PRIMARY) - RAG-powered template matching
+- ⚡ **Modal** (SECONDARY) - Serverless SQL testing
+  
+**Built for:** MCP 1st Birthday Hackathon
+
+## ✨ Features
+
+### Core Capabilities
+- 🧠 **Advanced NL Understanding**: Powered by Claude 3 Opus for superior natural language comprehension
+- 🔍 **Schema-Aware Validation**: Never hallucinates tables or columns
+- ✨ **Automatic Optimization**: Applies SQL best practices automatically
+- 📊 **Complex Query Support**: CTEs, Window Functions, Subqueries, Analytics
+- ⚠️ **Intelligent Warnings**: Identifies potential issues and suggests improvements
+- 🎯 **Multi-Dialect Support**: PostgreSQL, MySQL, SQLite, SQL Server
+
+### 🎯 Sponsor Integrations
+
+#### LlamaIndex RAG (PRIMARY SPONSOR)
+- **Similar Pattern Suggestions**: Retrieves relevant SQL patterns from template library
+- **15+ ETL Templates**: Pre-built patterns for common data transformations
+- **Context-Aware Learning**: Vector similarity search for pattern matching
+- **Production-Ready Examples**: Customer aggregations, window functions, cohort analysis, and more
+
+#### Modal Serverless Testing (SECONDARY SPONSOR)
+- **Sandboxed Execution**: Test SQL queries in isolated environments
+- **DuckDB Integration**: Fast in-memory SQL execution
+- **Result Preview**: See query results before production deployment
+- **Error Detection**: Catch syntax and runtime errors early
+
+### Production Export
+- **dbt Model Generation**: Export to production-ready dbt models
+- **Automatic Documentation**: Schema.yml with column descriptions
+- **Test Generation**: Basic data quality tests included
+- **ZIP Download**: Complete dbt project structure
+
+### Advanced SQL Features
+- **Common Table Expressions (CTEs)**: For complex, multi-step queries
+- **Window Functions**: Rankings, running totals, moving averages
+- **Analytical Queries**: Cohort analysis, trends, comparisons
+- **Optimized Joins**: Proper JOIN syntax, EXISTS/NOT EXISTS
+- **NULL Handling**: Correct IS NULL/IS NOT NULL usage
+- **Date Range Optimization**: Efficient date filtering
+
+## 🎯 What Makes CodeFlow AI Unique?
+
+Unlike generic ChatGPT prompts, CodeFlow AI provides:
+
+1. **Schema-Aware Generation** - Never hallucinates tables or columns
+2. **RAG-Powered Learning** (LlamaIndex) - Learns from organizational SQL patterns
+3. **Automated Testing** (Modal) - Serverless execution with result preview
+4. **Production Export** (dbt) - One-click export to production-ready dbt models
+5. **Advanced Validation** - Multi-layer validation with optimization suggestions
+6. **Template Library** - 15+ pre-built ETL patterns for common scenarios
+
+This makes CodeFlow AI **significantly better** than just asking ChatGPT for SQL!
+
+## 🚀 Quick Start
+
+### Prerequisites
+- Python 3.8+
+- Anthropic API key
+- OpenAI API key (for LlamaIndex embeddings)
+- Modal account (optional, for serverless testing)
+
+### Installation
+
+1. Clone the repository:
+```bash
+git clone <repository-url>
+cd codeflow-ai
+```
+
+2. Install dependencies:
+```bash
+pip install -r requirements.txt
+```
+
+3. Set up your API keys:
+```bash
+# Create .env file
+cat > .env << EOF
+ANTHROPIC_API_KEY=your_anthropic_key_here
+OPENAI_API_KEY=your_openai_key_here
+EOF
+```
+
+4. Initialize the database (optional):
+```bash
+python init_database.py
+```
+
+5. Run the application:
+```bash
+python app.py
+```
+
+6. Open your browser to: `http://127.0.0.1:7860`
+
+## 📖 Usage Examples
+
+### Simple Queries
+```
+"Show all customers"
+→ SELECT * FROM customers;
+```
+
+### Aggregations
+```
+"Count orders by customer"
+→ SELECT customer_id, COUNT(*) as order_count
+  FROM orders
+  GROUP BY customer_id;
+```
+
+### Advanced Analytics
+```
+"Find top 5 customers by revenue with running totals"
+→ WITH customer_revenue AS (
+    SELECT c.id, c.name, SUM(o.amount) as total_revenue
+    FROM customers c
+    JOIN orders o ON c.id = o.customer_id
+    GROUP BY c.id, c.name
+  )
+  SELECT 
+    id, name, total_revenue,
+    ROW_NUMBER() OVER (ORDER BY total_revenue DESC) as rank,
+    SUM(total_revenue) OVER (ORDER BY total_revenue DESC) as running_total
+  FROM customer_revenue
+  ORDER BY total_revenue DESC
+  LIMIT 5;
+```
+
+### Complex Conditions
+```
+"Customers who ordered in 2023 but not in 2024"
+→ SELECT c.*
+  FROM customers c
+  WHERE EXISTS (
+    SELECT 1 FROM orders o
+    WHERE o.customer_id = c.id
+    AND o.order_date >= '2023-01-01'
+    AND o.order_date < '2024-01-01'
+  )
+  AND NOT EXISTS (
+    SELECT 1 FROM orders o
+    WHERE o.customer_id = c.id
+    AND o.order_date >= '2024-01-01'
+  );
+```
+
+## 🏗️ Architecture
+
+### Core Components
+
+1. **nl_parser.py**: Advanced Natural Language to SQL Engine
+   - Claude 3 Opus integration
+   - Comprehensive prompt engineering
+   - JSON-structured output with metadata
+
+2. **schema_analyzer.py**: Database Schema Analysis
+   - Extracts table and column information
+   - Provides schema context to the AI
+
+3. **sql_validator.py**: SQL Validation and Optimization
+   - Schema validation
+   - Anti-pattern detection
+   - Optimization suggestions
+
+4. **app.py**: Gradio Web Interface
+   - User-friendly UI
+   - Real-time query generation
+   - Metadata and analysis display
+
+### Sponsor Integration Modules
+
+5. **rag/template_store.py**: LlamaIndex RAG (PRIMARY SPONSOR)
+   - Vector store with ChromaDB
+   - 15+ ETL template library
+   - Semantic similarity search
+   - Context-aware pattern matching
+
+6. **testing/test_runner.py**: Modal Testing (SECONDARY SPONSOR)
+   - Serverless SQL execution
+   - DuckDB integration
+   - Sandboxed query testing
+   - Result preview
+
+7. **export/dbt_exporter.py**: Production Export
+   - dbt model generation
+   - Schema documentation
+   - Test generation
+   - ZIP packaging
+
+## 🔧 Configuration
+
+### Environment Variables
+
+Create a `.env` file in the project root:
+
+```env
+ANTHROPIC_API_KEY=your_anthropic_api_key_here
+```
+
+### SQL Dialect Support
+
+The system supports multiple SQL dialects:
+- **PostgreSQL** (default)
+- **MySQL**
+- **SQLite**
+- **SQL Server**
+
+Select your dialect in the UI dropdown.
+
+## 📊 Query Types
+
+The system automatically detects and optimizes different query types:
+
+- **Simple**: Basic SELECT statements
+- **Aggregate**: GROUP BY with aggregate functions
+- **Join**: Multiple table queries
+- **Window**: Analytical functions with OVER clause
+- **CTE**: Common Table Expressions (WITH clause)
+- **Analytical**: Complex multi-step analysis
+
+## ✅ SQL Best Practices
+
+The engine automatically applies these best practices:
+
+### NULL Handling
+❌ Bad: `WHERE column = NULL`
+✅ Good: `WHERE column IS NULL`
+
+### Date Ranges
+❌ Bad: `WHERE date BETWEEN '2024-01-01' AND '2024-12-31'`
+✅ Good: `WHERE date >= '2024-01-01' AND date < '2025-01-01'`
+
+### Subqueries
+❌ Bad: `WHERE id NOT IN (SELECT ...)`
+✅ Good: `WHERE NOT EXISTS (SELECT 1 FROM ... WHERE ...)`
+
+### Joins
+❌ Bad: `FROM table1, table2 WHERE table1.id = table2.id`
+✅ Good: `FROM table1 JOIN table2 ON table1.id = table2.id`
+
+### CTEs for Readability
+❌ Bad: Nested subqueries
+✅ Good: 
+```sql
+WITH step1 AS (...),
+     step2 AS (...)
+SELECT * FROM step2;
+```
+
+## 🧪 Testing
+
+Test your API connection:
+```bash
+python test_api.py
+```
+
+This will:
+- Verify your API key
+- Test available models
+- Confirm connectivity
+
+## 📝 Sample Database Schema
+
+The default sample database includes:
+
+### Customers Table
+- id (INTEGER, PRIMARY KEY)
+- name (TEXT, NOT NULL)
+- email (TEXT)
+- country (TEXT)
+- created_at (TIMESTAMP)
+
+### Orders Table
+- id (INTEGER, PRIMARY KEY)
+- customer_id (INTEGER)
+- order_date (DATE)
+- total_amount (DECIMAL)
+- status (TEXT)
+
+### Products Table
+- id (INTEGER, PRIMARY KEY)
+- name (TEXT, NOT NULL)
+- price (DECIMAL)
+- category (TEXT)
+- stock_quantity (INTEGER)
+
+## 🎓 Advanced Examples
+
+### 1. Moving Averages
+```
+"Show daily order count with 7-day moving average"
+```
+
+### 2. Cohort Analysis
+```
+"Calculate customer cohorts based on first purchase month"
+```
+
+### 3. Year-over-Year Comparison
+```
+"Show monthly revenue trends with year-over-year comparison"
+```
+
+### 4. Top N per Group
+```
+"Find the 3 most popular products in each category"
+```
+
+### 5. Missing Data Analysis
+```
+"Get customers who have never placed an order"
+```
+
+## 🔒 Security
+
+- Never exposes raw API keys in the UI
+- Schema validation prevents SQL injection patterns
+- All queries are generated, not concatenated from user input
+
+## 🐛 Troubleshooting
+
+### API Key Issues
+If you see 404 model errors:
+1. Run `python test_api.py` to check available models
+2. Ensure your API key is valid
+3. Check if you have access to Claude 3 models
+
+### Schema Not Loading
+1. Verify `sample.db` exists
+2. Run `python init_database.py` to recreate it
+3. Check file permissions
+
+### Gradio Errors
+1. Ensure Gradio >= 4.44.1: `pip install gradio>=4.44.1`
+2. Clear browser cache
+3. Try a different browser
+
+## 📚 Resources
+
+- [Anthropic Claude API Documentation](https://docs.anthropic.com/)
+- [SQL Best Practices](https://www.sqlstyle.guide/)
+- [Window Functions Guide](https://www.postgresql.org/docs/current/tutorial-window.html)
+
+## 🤝 Contributing
+
+Contributions are welcome! Please ensure:
+- Code follows Python PEP 8 style
+- Add tests for new features
+- Update documentation
+
+## 📄 License
+
+MIT License
+
+## 🙏 Acknowledgments
+
+- Built with [Anthropic Claude AI](https://www.anthropic.com/)
+- RAG powered by [LlamaIndex](https://www.llamaindex.ai/) ⭐ PRIMARY SPONSOR
+- Serverless testing with [Modal](https://modal.com/) ⚡ SECONDARY SPONSOR
+- UI powered by [Gradio](https://gradio.app/)
+- Database support via [SQLAlchemy](https://www.sqlalchemy.org/)
+
+## 🏆 Hackathon Success Criteria
+
+✅ **Natural language → SQL**: Complete
+✅ **LlamaIndex RAG with templates**: Complete (15 templates)
+✅ **Modal testing integration**: Complete (with DuckDB fallback)
+✅ **dbt export functionality**: Complete
+✅ **README with hackathon tags**: Complete
+✅ **Works on Hugging Face Space**: Ready to deploy
+
+## 📹 Demo & Social
+
+- **Demo Video**: [Coming Soon - Will be added before submission]
+- **LinkedIn Post**: [Coming Soon - Will be shared]
+- **Twitter/X Post**: [Coming Soon - Will be shared]
+
+## 📞 Contact & Support
+
+For questions about this hackathon submission:
+- **Track**: MCP in Action (Enterprise Category)
+- **Built for**: MCP 1st Birthday Hackathon
+- **Sponsors**: LlamaIndex (PRIMARY), Modal (SECONDARY)
+
+---
+
+**🏆 MCP 1st Birthday Hackathon Submission**
+Made with ❤️ for the data engineering community

app.py

@@ -0,0 +1,459 @@
+import gradio as gr
+import os
+from dotenv import load_dotenv
+from nl_parser import NaturalLanguageParser
+from schema_analyzer import SchemaAnalyzer
+from sql_validator import SQLValidator
+from rag.template_store import get_template_store
+from testing.test_runner import get_test_runner
+from export.dbt_exporter import get_dbt_exporter
+
+# Load environment variables
+load_dotenv()
+
+# Initialize components
+print("Initializing CodeFlow AI - Advanced NL-to-SQL Engine...")
+try:
+    parser = NaturalLanguageParser()
+    print("✓ Natural Language Parser ready")
+except Exception as e:
+    print(f"✗ Error initializing parser: {e}")
+    print("Make sure your ANTHROPIC_API_KEY is set in .env file!")
+    exit(1)
+
+schema_analyzer = SchemaAnalyzer()
+print("✓ Schema Analyzer ready")
+
+# Get database schema
+DB_SCHEMA = schema_analyzer.get_schema()
+print(f"✓ Loaded schema with {len(DB_SCHEMA)} tables")
+
+# Initialize SQL Validator
+validator = SQLValidator(DB_SCHEMA)
+print("✓ SQL Validator ready")
+
+# Initialize RAG Template Store (PRIMARY SPONSOR - LlamaIndex)
+try:
+    template_store = get_template_store()
+    print("✓ RAG Template Store ready (LlamaIndex)")
+except Exception as e:
+    print(f"⚠️ Warning: RAG Template Store initialization failed: {e}")
+    template_store = None
+
+# Initialize Test Runner (SECONDARY SPONSOR - Modal)
+try:
+    test_runner = get_test_runner()
+    print("✓ SQL Test Runner ready (Modal/DuckDB)")
+except Exception as e:
+    print(f"⚠️ Warning: Test Runner initialization failed: {e}")
+    test_runner = None
+
+# Initialize dbt Exporter
+try:
+    dbt_exporter = get_dbt_exporter()
+    print("✓ dbt Exporter ready")
+except Exception as e:
+    print(f"⚠️ Warning: dbt Exporter initialization failed: {e}")
+    dbt_exporter = None
+
+def generate_sql(description, dialect):
+    """Main function to generate SQL with advanced analysis"""
+    
+    if not description.strip():
+        return "⚠️ Please enter a description of your query.", "", ""
+    
+    try:
+        # Generate SQL using enhanced parser
+        result = parser.generate_sql(description, DB_SCHEMA, dialect)
+        
+        # Extract components
+        sql = result.get("sql", "-- No SQL generated")
+        explanation = result.get("explanation", "")
+        query_type = result.get("query_type", "unknown")
+        warnings = result.get("warnings", [])
+        optimizations = result.get("optimizations", [])
+        
+        # Validate SQL using sql_validator (NEW: Integration)
+        validation_result = validator.validate(sql)
+        validator_suggestions = validator.suggest_optimizations(sql)
+        
+        # Find similar patterns using RAG (NEW: PRIMARY SPONSOR - LlamaIndex)
+        similar_patterns = []
+        if template_store and template_store.index is not None:
+            try:
+                similar_patterns = template_store.find_similar(description, top_k=3)
+            except Exception as e:
+                print(f"⚠️ RAG query failed: {e}")
+        
+        # Build metadata display
+        metadata_parts = []
+        
+        # Query type
+        metadata_parts.append(f"**Query Type:** {query_type.title()}")
+        
+        # Explanation
+        if explanation:
+            metadata_parts.append(f"\n**Explanation:**\n{explanation}")
+        
+        # Optimizations applied
+        if optimizations:
+            metadata_parts.append("\n**✨ Optimizations Applied:**")
+            for opt in optimizations:
+                metadata_parts.append(f"  • {opt}")
+        
+        # Validator suggestions (NEW)
+        if validator_suggestions:
+            metadata_parts.append("\n**💡 Additional Optimization Suggestions:**")
+            for suggestion in validator_suggestions:
+                metadata_parts.append(f"  • {suggestion}")
+        
+        # Validation errors (NEW)
+        if validation_result.get("errors"):
+            metadata_parts.append("\n**❌ Validation Errors:**")
+            for error in validation_result["errors"]:
+                metadata_parts.append(f"  • {error}")
+        
+        # Validation warnings (NEW)
+        if validation_result.get("warnings"):
+            metadata_parts.append("\n**⚠️ Validation Warnings:**")
+            for warn in validation_result["warnings"]:
+                metadata_parts.append(f"  • {warn}")
+        
+        # Original warnings from parser
+        if warnings:
+            metadata_parts.append("\n**⚠️ Parser Warnings:**")
+            for warn in warnings:
+                metadata_parts.append(f"  • {warn}")
+        
+        # Similar patterns from RAG (NEW: PRIMARY SPONSOR)
+        if similar_patterns:
+            metadata_parts.append("\n**🔍 Similar Patterns (LlamaIndex RAG):**")
+            for i, pattern in enumerate(similar_patterns, 1):
+                template_name = pattern.get("template_name", "Unknown")
+                metadata_parts.append(f"  {i}. **{template_name}**")
+                if pattern.get("excerpt"):
+                    # Show first 2 lines of excerpt
+                    excerpt_lines = pattern["excerpt"].split('\n')[:2]
+                    for line in excerpt_lines:
+                        if line.strip():
+                            metadata_parts.append(f"     `{line.strip()}`")
+        
+        metadata_text = "\n".join(metadata_parts)
+        
+        # Format SQL with header
+        formatted_sql = f"""-- CodeFlow AI - Advanced NL-to-SQL Engine
+-- Dialect: {dialect}
+-- Query Type: {query_type.title()}
+
+{sql}"""
+        
+        return formatted_sql, metadata_text, ""
+    
+    except Exception as e:
+        error_msg = f"-- Error generating SQL\n-- {str(e)}\n\n-- Please try rephrasing your request."
+        metadata_msg = f"**❌ Error:**\n{str(e)}"
+        return error_msg, metadata_msg, ""
+
+def test_sql_query(sql_code):
+    """Test SQL query execution (SECONDARY SPONSOR - Modal)"""
+    
+    if not sql_code or not sql_code.strip():
+        return "⚠️ No SQL to test. Generate SQL first."
+    
+    # Extract just the SQL (remove comments)
+    sql_lines = []
+    for line in sql_code.split('\n'):
+        stripped = line.strip()
+        if stripped and not stripped.startswith('--'):
+            sql_lines.append(line)
+    
+    clean_sql = '\n'.join(sql_lines).strip()
+    
+    if not clean_sql:
+        return "⚠️ No executable SQL found."
+    
+    if not test_runner:
+        return "⚠️ Test runner not available. Install Modal or DuckDB."
+    
+    try:
+        # Test the SQL
+        result = test_runner.test_sql(clean_sql)
+        
+        # Format results
+        if result["success"]:
+            output_parts = []
+            output_parts.append(f"**✅ Query Executed Successfully**")
+            output_parts.append(f"**Execution Method:** {result.get('execution_method', 'Unknown')}")
+            output_parts.append(f"**Rows Returned:** {result['row_count']}")
+            
+            if result.get("columns"):
+                output_parts.append(f"\n**Columns:** {', '.join(result['columns'])}")
+            
+            if result.get("rows"):
+                output_parts.append(f"\n**Sample Results (first 10 rows):**")
+                output_parts.append("```")
+                for i, row in enumerate(result['rows'][:10], 1):
+                    output_parts.append(f"{i}. {row}")
+                output_parts.append("```")
+                
+                if result['row_count'] > 10:
+                    output_parts.append(f"\n_(Showing 10 of {result['row_count']} rows)_")
+            
+            return "\n".join(output_parts)
+        else:
+            return f"**❌ Query Execution Failed**\n\n**Error:** {result.get('error', 'Unknown error')}"
+    
+    except Exception as e:
+        return f"**❌ Test Error:**\n{str(e)}"
+
+def export_to_dbt(sql_code, model_name, description):
+    """Export SQL as dbt model with downloadable files"""
+    
+    if not sql_code or not sql_code.strip():
+        return "⚠️ No SQL to export. Generate SQL first.", None
+    
+    if not model_name or not model_name.strip():
+        return "⚠️ Please provide a model name.", None
+    
+    if not dbt_exporter:
+        return "⚠️ dbt Exporter not available.", None
+    
+    try:
+        # Clean model name (replace spaces with underscores, lowercase)
+        clean_model_name = model_name.strip().lower().replace(' ', '_').replace('-', '_')
+        clean_model_name = ''.join(c for c in clean_model_name if c.isalnum() or c == '_')
+        
+        if not clean_model_name:
+            return "⚠️ Invalid model name. Use letters, numbers, and underscores only.", None
+        
+        # Export to dbt
+        files = dbt_exporter.export_model(
+            sql=sql_code,
+            model_name=clean_model_name,
+            description=description or f"Generated model: {clean_model_name}",
+            materialization="table",
+            schema="analytics",
+            tags="codeflow_generated"
+        )
+        
+        # Create ZIP file
+        zip_bytes = dbt_exporter.create_zip(files, clean_model_name)
+        
+        # Create success message
+        message = f"""**✅ dbt Model Exported Successfully!**
+
+**Model Name:** `{clean_model_name}`
+**Files Generated:**
+- `models/{clean_model_name}.sql` - dbt model
+- `models/schema.yml` - Documentation and tests
+- `README.md` - Usage instructions
+
+**Next Steps:**
+1. Download the ZIP file below
+2. Extract to your dbt project's `models/` directory
+3. Run: `dbt run --select {clean_model_name}`
+4. Test: `dbt test --select {clean_model_name}`
+
+The model is configured as a table in the `analytics` schema.
+You can modify the configuration at the top of the model file.
+"""
+        
+        return message, zip_bytes
+        
+    except Exception as e:
+        return f"**❌ Export Error:**\n{str(e)}", None
+
+def create_app():
+    """Create Gradio interface"""
+    
+    with gr.Blocks(title="CodeFlow AI - Advanced NL-to-SQL") as app:
+        gr.Markdown("""
+        # 🚀 CodeFlow AI - Advanced Natural Language to SQL Engine
+        
+        Transform plain English into **production-ready, optimized SQL** instantly!
+        
+        **Advanced Features:**
+        - ✨ Automatic query optimization
+        - 🔍 Schema-aware validation
+        - 📊 Support for CTEs, window functions, and complex analytics
+        - ⚠️ Intelligent warnings and suggestions
+        - 🎯 Multiple SQL dialect support
+        - 🔍 **RAG-powered similar patterns** (LlamaIndex)
+        - 💡 **Advanced optimization suggestions**
+        """)
+        
+        with gr.Row():
+            with gr.Column(scale=1):
+                gr.Markdown("### 📝 Input")
+                
+                description = gr.Textbox(
+                    label="Describe your data transformation",
+                    placeholder="Example: Find top 5 customers by total order value with running totals",
+                    lines=6
+                )
+                
+                dialect = gr.Dropdown(
+                    choices=["PostgreSQL", "MySQL", "SQLite", "SQL Server"],
+                    value="PostgreSQL",
+                    label="SQL Dialect"
+                )
+                
+                with gr.Row():
+                    generate_btn = gr.Button("✨ Generate SQL", variant="primary", scale=2)
+                    clear_btn = gr.Button("🗑️ Clear", scale=1)
+            
+            with gr.Column(scale=2):
+                gr.Markdown("### 💻 Generated SQL")
+                
+                output = gr.Code(
+                    label="SQL Output",
+                    language="sql",
+                    lines=15
+                )
+                
+                # Test Query button (SECONDARY SPONSOR - Modal)
+                with gr.Row():
+                    test_btn = gr.Button("🧪 Test Query (Modal)", variant="secondary")
+                
+                gr.Markdown("### 📊 Query Analysis")
+                
+                metadata = gr.Markdown(
+                    value="",
+                    label="Analysis"
+                )
+                
+                # Test results section
+                with gr.Accordion("🧪 Test Results", open=False):
+                    test_results = gr.Markdown(
+                        value="Click 'Test Query' to execute the SQL and see results.",
+                        label="Test Output"
+                    )
+                
+                # dbt Export section
+                gr.Markdown("### 📦 Export to dbt")
+                with gr.Row():
+                    with gr.Column(scale=2):
+                        model_name_input = gr.Textbox(
+                            label="Model Name",
+                            placeholder="customer_analytics",
+                            value="my_model"
+                        )
+                    with gr.Column(scale=3):
+                        model_desc_input = gr.Textbox(
+                            label="Model Description",
+                            placeholder="Describe what this model does...",
+                            value=""
+                        )
+                
+                export_btn = gr.Button("📦 Export to dbt", variant="secondary")
+                
+                with gr.Accordion("📦 Export Status", open=False):
+                    export_status = gr.Markdown(
+                        value="Configure model details above and click 'Export to dbt' to generate production-ready dbt files.",
+                        label="Export Output"
+                    )
+                    download_file = gr.File(label="Download dbt Model", visible=False)
+        
+        # Example queries
+        gr.Markdown("### 💡 Try These Advanced Examples")
+        gr.Examples(
+            examples=[
+                ["Find top 10 customers by total order value with their rank"],
+                ["Show monthly revenue trends with year-over-year comparison"],
+                ["Get customers who have never placed an order"],
+                ["Calculate running total of sales by date for each product"],
+                ["Find the 3 most popular products in each category"],
+                ["Show customers with above-average order values"],
+                ["List orders with customer details, handling missing customers gracefully"],
+                ["Calculate customer cohorts based on first purchase month"],
+                ["Find products that were ordered in 2023 but not in 2024"],
+                ["Show daily order count with 7-day moving average"]
+            ],
+            inputs=[description]
+        )
+        
+        # Event handlers
+        generate_btn.click(
+            fn=generate_sql,
+            inputs=[description, dialect],
+            outputs=[output, metadata, gr.Textbox(visible=False)]
+        )
+        
+        test_btn.click(
+            fn=test_sql_query,
+            inputs=[output],
+            outputs=[test_results]
+        )
+        
+        def handle_export(sql_code, model_name, description):
+            """Handle dbt export and show/hide download button"""
+            message, zip_bytes = export_to_dbt(sql_code, model_name, description)
+            if zip_bytes:
+                # Save to temp file for download
+                import tempfile
+                with tempfile.NamedTemporaryFile(delete=False, suffix='.zip', mode='wb') as f:
+                    f.write(zip_bytes)
+                    temp_path = f.name
+                return message, gr.File(value=temp_path, visible=True)
+            else:
+                return message, gr.File(visible=False)
+        
+        export_btn.click(
+            fn=handle_export,
+            inputs=[output, model_name_input, model_desc_input],
+            outputs=[export_status, download_file]
+        )
+        
+        clear_btn.click(
+            fn=lambda: ("", "", "", "Click 'Test Query' to execute the SQL and see results.",
+                       "Configure model details above and click 'Export to dbt' to generate production-ready dbt files.",
+                       gr.File(visible=False)),
+            outputs=[description, output, metadata, test_results, export_status, download_file]
+        )
+        
+        # Schema display
+        with gr.Accordion("📋 Database Schema", open=False):
+            schema_md = "## Available Tables and Columns\n\n"
+            for table, columns in DB_SCHEMA.items():
+                schema_md += f"### 📊 {table}\n\n"
+                schema_md += "| Column | Type | Constraints |\n"
+                schema_md += "|--------|------|-------------|\n"
+                for col in columns:
+                    constraints = []
+                    if col.get('primary_key'):
+                        constraints.append("PRIMARY KEY")
+                    if not col.get('nullable', True):
+                        constraints.append("NOT NULL")
+                    constraint_str = ", ".join(constraints) if constraints else "-"
+                    schema_md += f"| {col['name']} | {col['type']} | {constraint_str} |\n"
+                schema_md += "\n"
+            
+            gr.Markdown(schema_md)
+        
+        gr.Markdown("---")
+        gr.Markdown("""
+        **🏆 CodeFlow AI - Advanced NL-to-SQL Engine**
+        
+        **Core Features:**
+        - 🧠 Powered by Claude 3 Opus (Anthropic)
+        - 🔍 **LlamaIndex RAG** - Similar pattern suggestions from template library
+        - ✨ Automatic query optimization with validation
+        - 📊 Complex query support (CTEs, Window Functions, Subqueries)
+        - ⚠️ Intelligent error handling and warnings
+        
+        **Sponsor Integrations:**
+        - 🎯 **LlamaIndex** - RAG-powered template matching (PRIMARY SPONSOR)
+        - ⚡ **Modal** - Serverless SQL testing (coming soon)
+        
+        Built for: MCP 1st Birthday Hackathon
+        """)
+    
+    return app
+
+if __name__ == "__main__":
+    print("\n" + "="*50)
+    print("Starting CodeFlow AI - Advanced NL-to-SQL Engine...")
+    print("="*50 + "\n")
+    
+    app = create_app()
+    app.launch(share=False)

export/__init__.py

@@ -0,0 +1,4 @@
+"""
+Export module for CodeFlow AI
+Provides dbt model generation and other export formats
+"""

export/dbt_exporter.py

@@ -0,0 +1,244 @@
+"""
+dbt Exporter for CodeFlow AI
+Generates production-ready dbt models from SQL queries
+"""
+
+import io
+import zipfile
+from datetime import datetime
+from typing import Dict, Optional
+from jinja2 import Template
+
+
+class DBTExporter:
+    """
+    Export SQL queries as dbt models
+    Generates model.sql and schema.yml files
+    """
+    
+    def __init__(self):
+        # Use {0}, {1} placeholders instead of Jinja {{ }} to avoid conflict with dbt
+        self.model_template = """{{{{
+    config(
+        materialized='{materialization}',
+        schema='{schema}',
+        tags=['{tags}']
+    )
+}}}}
+
+/*
+    Model: {model_name}
+    Description: {description}
+    Generated: {generated_date}
+    Generated by: CodeFlow AI
+*/
+
+{sql_query}
+"""
+        
+        self.schema_template = """version: 2
+
+models:
+  - name: {{ model_name }}
+    description: {{ description }}
+    columns:
+      {% for column in columns %}
+      - name: {{ column.name }}
+        description: {{ column.description }}
+        {% if column.tests %}
+        tests:
+          {% for test in column.tests %}
+          - {{ test }}
+          {% endfor %}
+        {% endif %}
+      {% endfor %}
+"""
+    
+    def export_model(
+        self,
+        sql: str,
+        model_name: str,
+        description: str = "",
+        materialization: str = "table",
+        schema: str = "analytics",
+        tags: str = "codeflow_generated",
+        columns: Optional[list] = None
+    ) -> Dict[str, str]:
+        """
+        Export SQL as dbt model
+        
+        Args:
+            sql: SQL query to export
+            model_name: Name for the dbt model
+            description: Model description
+            materialization: dbt materialization (table, view, incremental)
+            schema: Target schema name
+            tags: Model tags
+            columns: List of column definitions
+            
+        Returns:
+            Dict with model.sql and schema.yml content
+        """
+        
+        # Clean SQL (remove CodeFlow comments)
+        clean_sql = self._clean_sql(sql)
+        
+        # Generate model.sql using format() instead of Jinja2 to avoid dbt {{ }} conflict
+        model_content = self.model_template.format(
+            materialization=materialization,
+            schema=schema,
+            tags=tags,
+            model_name=model_name,
+            description=description or f"Generated model: {model_name}",
+            generated_date=datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
+            sql_query=clean_sql
+        )
+        
+        # Generate schema.yml
+        if not columns:
+            # Auto-detect columns from SQL (basic)
+            columns = self._detect_columns(clean_sql)
+        
+        schema_template = Template(self.schema_template)
+        schema_content = schema_template.render(
+            model_name=model_name,
+            description=description or f"Generated model: {model_name}",
+            columns=columns
+        )
+        
+        return {
+            "model.sql": model_content,
+            "schema.yml": schema_content
+        }
+    
+    def _clean_sql(self, sql: str) -> str:
+        """Remove CodeFlow-specific comments from SQL"""
+        lines = []
+        for line in sql.split('\n'):
+            stripped = line.strip()
+            # Skip CodeFlow header comments
+            if stripped.startswith('-- CodeFlow'):
+                continue
+            if stripped.startswith('-- Dialect:'):
+                continue
+            if stripped.startswith('-- Query Type:'):
+                continue
+            lines.append(line)
+        
+        return '\n'.join(lines).strip()
+    
+    def _detect_columns(self, sql: str) -> list:
+        """
+        Basic column detection from SQL
+        Looks for SELECT clause columns
+        """
+        columns = []
+        
+        # Find SELECT clause (very basic parser)
+        sql_upper = sql.upper()
+        if 'SELECT' not in sql_upper:
+            return columns
+        
+        # Extract between SELECT and FROM
+        select_idx = sql_upper.find('SELECT')
+        from_idx = sql_upper.find('FROM', select_idx)
+        
+        if from_idx == -1:
+            from_idx = len(sql)
+        
+        select_clause = sql[select_idx + 6:from_idx].strip()
+        
+        # Split by comma (basic - doesn't handle nested commas)
+        parts = select_clause.split(',')
+        
+        for part in parts[:10]:  # Limit to first 10 columns
+            part = part.strip()
+            if not part:
+                continue
+            
+            # Extract column name (after AS if present)
+            if ' AS ' in part.upper():
+                col_name = part.upper().split(' AS ')[-1].strip()
+            elif ' ' in part:
+                # Take last word
+                col_name = part.split()[-1].strip()
+            else:
+                col_name = part
+            
+            # Clean column name
+            col_name = col_name.replace('`', '').replace('"', '').replace("'", "")
+            
+            if col_name and col_name != '*':
+                columns.append({
+                    "name": col_name.lower(),
+                    "description": f"Column: {col_name}",
+                    "tests": ["not_null"] if "id" in col_name.lower() else []
+                })
+        
+        return columns
+    
+    def create_zip(self, files: Dict[str, str], model_name: str) -> bytes:
+        """
+        Create a ZIP file containing the dbt files
+        
+        Args:
+            files: Dict of filename -> content
+            model_name: Model name for directory structure
+            
+        Returns:
+            ZIP file as bytes
+        """
+        zip_buffer = io.BytesIO()
+        
+        with zipfile.ZipFile(zip_buffer, 'w', zipfile.ZIP_DEFLATED) as zip_file:
+            # Create models directory structure
+            for filename, content in files.items():
+                if filename.endswith('.sql'):
+                    path = f"models/{model_name}.sql"
+                elif filename.endswith('.yml'):
+                    path = f"models/schema.yml"
+                else:
+                    path = f"models/{filename}"
+                
+                zip_file.writestr(path, content)
+            
+            # Add README
+            readme = f"""# dbt Model: {model_name}
+
+Generated by CodeFlow AI on {datetime.now().strftime("%Y-%m-%d")}
+
+## Files Included
+
+- `models/{model_name}.sql` - The dbt model
+- `models/schema.yml` - Model and column documentation
+
+## Usage
+
+1. Copy these files to your dbt project's `models/` directory
+2. Run `dbt run --select {model_name}` to execute
+3. Run `dbt test --select {model_name}` to run tests
+
+## Configuration
+
+You can modify the model configuration at the top of {model_name}.sql:
+- materialization: table, view, or incremental
+- schema: target schema name
+- tags: for organizing models
+
+For more information, visit: https://docs.getdbt.com/
+"""
+            zip_file.writestr("README.md", readme)
+        
+        zip_buffer.seek(0)
+        return zip_buffer.getvalue()
+
+
+# Singleton instance
+_dbt_exporter = None
+
+def get_dbt_exporter() -> DBTExporter:
+    """Get or create the global dbt exporter instance"""
+    global _dbt_exporter
+    if _dbt_exporter is None:
+        _dbt_exporter = DBTExporter()
+    return _dbt_exporter

nl_parser.py

@@ -0,0 +1,189 @@
+import anthropic
+import json
+import os
+import re
+from dotenv import load_dotenv
+
+load_dotenv()
+
+class NaturalLanguageParser:
+    """Advanced Natural Language to SQL Engine using Claude"""
+    
+    def __init__(self):
+        api_key = os.getenv("ANTHROPIC_API_KEY")
+        if not api_key:
+            raise ValueError("ANTHROPIC_API_KEY not found in .env file!")
+        self.client = anthropic.Anthropic(api_key=api_key)
+    
+    def generate_sql(self, description, schema, dialect="PostgreSQL"):
+        """
+        Generate complete, optimized SQL directly from natural language.
+        This is the main engine that handles all SQL generation logic.
+        """
+        
+        schema_text = self._format_schema(schema)
+        
+        prompt = f"""You are an advanced Natural-Language-to-SQL Engine.
+Your job is to convert user instructions into correct, executable, efficient SQL queries, based solely on the provided database schema.
+
+🔥 Core Responsibilities
+
+1. NEVER hallucinate tables or columns. Only use what exists in the provided schema.
+2. Follow the SQL dialect: {dialect}
+3. Validate user intent and generate the best query, even if their natural language is unclear.
+4. Fix common SQL mistakes:
+   - Use IS NULL / IS NOT NULL (never = NULL)
+   - Use >= and < for date ranges instead of BETWEEN
+   - Proper JOIN syntax
+   - Correct aggregate/grouping logic
+5. Optimize the SQL:
+   - Use proper filters
+   - Use EXISTS/NOT EXISTS for subqueries
+   - Use CTEs for clarity in complex queries
+   - Avoid unnecessary computation
+6. For complex analysis (cohorts, trends, rankings, window functions):
+   - Use Common Table Expressions (CTEs)
+   - Use window functions when appropriate
+
+🗃️ Available Database Schema:
+{schema_text}
+
+👤 User Request: {description}
+
+🎯 SQL Construction Rules:
+
+WHERE clauses:
+- Use IS NULL / IS NOT NULL, not = 'NULL'
+- For date ranges use >= and < instead of BETWEEN
+
+Aggregations:
+- GROUP BY all non-aggregated fields
+- Avoid GROUP BY unnecessary columns
+
+Joins:
+- Prefer explicit JOIN syntax
+- Use LEFT JOIN for "missing data" queries
+- Always specify join conditions clearly
+
+Subqueries:
+- Prefer EXISTS/NOT EXISTS for performance
+- Use CTEs for readability in complex queries
+
+Window Functions (use when user asks for):
+- Top N per group
+- Rankings
+- Running totals
+- Comparisons to averages
+- Rolling windows
+
+📋 Output Format:
+
+Return a JSON object with this exact structure:
+{{
+    "sql": "the complete SQL query here",
+    "explanation": "brief explanation of what the query does",
+    "query_type": "simple|aggregate|join|window|cte|analytical",
+    "warnings": ["any warnings about schema limitations or assumptions"],
+    "optimizations": ["list of optimizations applied"]
+}}
+
+CRITICAL RULES FOR JSON OUTPUT:
+- Return ONLY valid JSON (no markdown, no code blocks, no extra text)
+- Escape ALL special characters in the SQL string:
+  * Newlines must be \\n
+  * Quotes must be \\"
+  * Backslashes must be \\\\
+- The "sql" field must be a single-line string with \\n for line breaks
+- Always end SQL with semicolon
+- If the request is impossible with the given schema, set "sql" to "-- ERROR: <explanation>" and explain in "warnings"
+
+Generate the SQL query now:"""
+
+        try:
+            response = self.client.messages.create(
+                model="claude-3-opus-20240229",
+                max_tokens=4000,
+                messages=[{"role": "user", "content": prompt}]
+            )
+            
+            content = response.content[0].text.strip()
+            
+            # Remove markdown code blocks if present
+            content = content.replace("```json", "").replace("```", "").strip()
+            
+            # Try to parse JSON
+            try:
+                result = json.loads(content)
+                
+                # Ensure all required fields exist
+                if "sql" not in result or not result["sql"]:
+                    result["sql"] = "-- ERROR: No SQL generated"
+                if "explanation" not in result:
+                    result["explanation"] = "SQL query generated"
+                if "query_type" not in result:
+                    result["query_type"] = "select"
+                if "warnings" not in result:
+                    result["warnings"] = []
+                if "optimizations" not in result:
+                    result["optimizations"] = []
+                    
+                return result
+                
+            except json.JSONDecodeError as e:
+                # If JSON parsing fails, try to extract components manually using regex
+                print(f"JSON parsing failed: {e}, attempting manual extraction...")
+                
+                # Try to extract SQL between "sql": " and next quote
+                sql_match = re.search(r'"sql"\s*:\s*"((?:[^"\\]|\\.|\\n)*)"', content, re.DOTALL)
+                if sql_match:
+                    sql = sql_match.group(1)
+                    # Unescape the JSON string
+                    sql = sql.replace('\\n', '\n').replace('\\"', '"').replace('\\\\', '\\')
+                else:
+                    # Try alternative format or use entire content
+                    sql = content
+                
+                # Try to extract explanation
+                expl_match = re.search(r'"explanation"\s*:\s*"((?:[^"\\]|\\.)*)"', content, re.DOTALL)
+                explanation = expl_match.group(1) if expl_match else "SQL generated (with parsing issues)"
+                
+                # Try to extract query type
+                type_match = re.search(r'"query_type"\s*:\s*"([^"]*)"', content)
+                query_type = type_match.group(1) if type_match else "select"
+                
+                result = {
+                    "sql": sql,
+                    "explanation": explanation,
+                    "query_type": query_type,
+                    "warnings": ["JSON parsing issue - SQL may need review"],
+                    "optimizations": []
+                }
+                
+                return result
+        
+        except Exception as e:
+            return {
+                "sql": f"-- ERROR: {str(e)}",
+                "explanation": "An error occurred during SQL generation",
+                "query_type": "error",
+                "warnings": [str(e)],
+                "optimizations": []
+            }
+    
+    def _format_schema(self, schema):
+        """Format schema for prompt"""
+        lines = ["Tables and Columns:"]
+        for table, columns in schema.items():
+            lines.append(f"\n📊 {table}")
+            for col in columns:
+                nullable = "NULL" if col.get('nullable', True) else "NOT NULL"
+                pk = " (PRIMARY KEY)" if col.get('primary_key', False) else ""
+                lines.append(f"   - {col['name']}: {col['type']} {nullable}{pk}")
+        return "\n".join(lines)
+    
+    # Legacy method for backward compatibility
+    def parse(self, description, schema):
+        """Legacy method - redirects to generate_sql"""
+        result = self.generate_sql(description, schema)
+        # Return just the SQL for backward compatibility
+        return {"raw_sql": result.get("sql", ""), "metadata": result}

rag/__init__.py

@@ -0,0 +1,4 @@
+"""
+RAG (Retrieval-Augmented Generation) module for CodeFlow AI
+Uses LlamaIndex to provide similar ETL pattern suggestions
+"""

rag/sample_templates/01_customer_aggregation.sql

@@ -0,0 +1,23 @@
+-- Template: Customer Order Aggregation
+-- Pattern: Aggregate metrics by customer dimension
+-- Use Case: Calculate total order value, count, average per customer
+
+SELECT 
+    c.customer_id,
+    c.name,
+    c.email,
+    COUNT(o.order_id) as total_orders,
+    SUM(o.total_amount) as total_spent,
+    AVG(o.total_amount) as avg_order_value,
+    MIN(o.order_date) as first_order_date,
+    MAX(o.order_date) as last_order_date
+FROM customers c
+LEFT JOIN orders o ON c.customer_id = o.customer_id
+GROUP BY c.customer_id, c.name, c.email
+ORDER BY total_spent DESC;
+
+-- Key Concepts:
+-- - LEFT JOIN to include customers with no orders
+-- - Multiple aggregations (COUNT, SUM, AVG, MIN, MAX)
+-- - GROUP BY all non-aggregated columns
+-- - ORDER BY to rank results

rag/sample_templates/02_date_filtering.sql

@@ -0,0 +1,23 @@
+-- Template: Date Range Filtering
+-- Pattern: Filter records within specific date ranges
+-- Use Case: Get orders for specific time periods
+
+SELECT 
+    order_id,
+    customer_id,
+    order_date,
+    total_amount,
+    status
+FROM orders
+WHERE order_date >= '2024-01-01' 
+  AND order_date < '2024-04-01'  -- Use < instead of <= for exclusive end
+ORDER BY order_date DESC;
+
+-- Key Concepts:
+-- - Use >= and < for date ranges (more precise than BETWEEN)
+-- - BETWEEN is inclusive on both ends, which can cause issues
+-- - Always use ISO format YYYY-MM-DD for dates
+-- - Consider timezone implications in production
+
+-- Alternative with BETWEEN (less preferred):
+-- WHERE order_date BETWEEN '2024-01-01' AND '2024-03-31'

rag/sample_templates/03_window_functions.sql

@@ -0,0 +1,43 @@
+-- Template: Window Functions for Ranking
+-- Pattern: Rank records within partitions
+-- Use Case: Top N per category, rankings, row numbers
+
+-- Example 1: Rank customers by total orders in each city
+SELECT 
+    customer_id,
+    name,
+    city,
+    total_orders,
+    ROW_NUMBER() OVER (PARTITION BY city ORDER BY total_orders DESC) as row_num,
+    RANK() OVER (PARTITION BY city ORDER BY total_orders DESC) as rank,
+    DENSE_RANK() OVER (PARTITION BY city ORDER BY total_orders DESC) as dense_rank
+FROM (
+    SELECT 
+        c.customer_id,
+        c.name,
+        c.city,
+        COUNT(o.order_id) as total_orders
+    FROM customers c
+    LEFT JOIN orders o ON c.customer_id = o.customer_id
+    GROUP BY c.customer_id, c.name, c.city
+) customer_metrics;
+
+-- Example 2: Top 3 customers per city
+SELECT *
+FROM (
+    SELECT 
+        customer_id,
+        name,
+        city,
+        total_orders,
+        ROW_NUMBER() OVER (PARTITION BY city ORDER BY total_orders DESC) as rn
+    FROM customer_metrics
+) ranked
+WHERE rn <= 3;
+
+-- Key Concepts:
+-- - ROW_NUMBER(): Unique sequential number (1,2,3,4...)
+-- - RANK(): Same rank for ties, skips next rank (1,2,2,4...)
+-- - DENSE_RANK(): Same rank for ties, no gaps (1,2,2,3...)
+-- - PARTITION BY: Restart ranking within each group
+-- - ORDER BY: Defines ranking order within partition

rag/sample_templates/04_running_totals.sql

@@ -0,0 +1,31 @@
+-- Template: Running Totals and Cumulative Sums
+-- Pattern: Calculate cumulative values over ordered records
+-- Use Case: Running balance, cumulative revenue, YTD totals
+
+SELECT 
+    order_date,
+    customer_id,
+    total_amount,
+    SUM(total_amount) OVER (
+        ORDER BY order_date 
+        ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
+    ) as running_total,
+    SUM(total_amount) OVER (
+        PARTITION BY customer_id 
+        ORDER BY order_date
+    ) as customer_running_total,
+    AVG(total_amount) OVER (
+        ORDER BY order_date 
+        ROWS BETWEEN 6 PRECEDING AND CURRENT ROW
+    ) as moving_avg_7day
+FROM orders
+ORDER BY order_date;
+
+-- Key Concepts:
+-- - SUM() OVER (): Running total across all rows
+-- - PARTITION BY: Separate running totals per customer
+-- - ROWS BETWEEN: Define window frame
+--   * UNBOUNDED PRECEDING: From start
+--   * CURRENT ROW: Up to current row
+--   * N PRECEDING: Previous N rows
+-- - Useful for trends, YTD calculations, moving averages

rag/sample_templates/05_top_n_per_group.sql

@@ -0,0 +1,37 @@
+-- Template: Top N Records Per Group
+-- Pattern: Get top performers within each category/partition
+-- Use Case: Best sellers per category, top customers per region
+
+WITH ranked_products AS (
+    SELECT 
+        p.product_id,
+        p.name,
+        p.category,
+        p.price,
+        COUNT(oi.order_id) as times_ordered,
+        SUM(oi.quantity) as total_quantity_sold,
+        SUM(oi.quantity * oi.unit_price) as total_revenue,
+        ROW_NUMBER() OVER (
+            PARTITION BY p.category 
+            ORDER BY SUM(oi.quantity * oi.unit_price) DESC
+        ) as revenue_rank
+    FROM products p
+    LEFT JOIN order_items oi ON p.product_id = oi.product_id
+    GROUP BY p.product_id, p.name, p.category, p.price
+)
+SELECT 
+    category,
+    product_id,
+    name,
+    total_revenue,
+    total_quantity_sold,
+    revenue_rank
+FROM ranked_products
+WHERE revenue_rank <= 3
+ORDER BY category, revenue_rank;
+
+-- Key Concepts:
+-- - CTE for better readability
+-- - PARTITION BY category to rank within each group
+-- - Filter WHERE rank <= N to get top N
+-- - Useful for identifying best performers in each segment

rag/sample_templates/06_left_join_missing.sql

@@ -0,0 +1,43 @@
+-- Template: Finding Missing Data with LEFT JOIN
+-- Pattern: Identify records that don't have matching records in related table
+-- Use Case: Customers without orders, products never sold, orphaned records
+
+-- Example 1: Customers who have never placed an order
+SELECT 
+    c.customer_id,
+    c.name,
+    c.email,
+    c.registration_date
+FROM customers c
+LEFT JOIN orders o ON c.customer_id = o.customer_id
+WHERE o.order_id IS NULL
+ORDER BY c.registration_date DESC;
+
+-- Example 2: Products that have never been ordered
+SELECT 
+    p.product_id,
+    p.name,
+    p.category,
+    p.price,
+    p.stock_quantity
+FROM products p
+LEFT JOIN order_items oi ON p.product_id = oi.product_id
+WHERE oi.order_item_id IS NULL;
+
+-- Key Concepts:
+-- - LEFT JOIN keeps all records from left table
+-- - WHERE column IS NULL finds unmatched records
+-- - Must check a column from right table that cannot be NULL (like primary key)
+-- - Alternative: NOT EXISTS (often faster for large datasets)
+
+-- Alternative using NOT EXISTS:
+SELECT 
+    c.customer_id,
+    c.name,
+    c.email
+FROM customers c
+WHERE NOT EXISTS (
+    SELECT 1 
+    FROM orders o 
+    WHERE o.customer_id = c.customer_id
+);

rag/sample_templates/07_exists_not_exists.sql

@@ -0,0 +1,47 @@
+-- Template: EXISTS and NOT EXISTS Set Operations
+-- Pattern: Efficient subquery filtering for existence checks
+-- Use Case: Find records that match/don't match conditions in other tables
+
+-- Example 1: Customers who placed orders in 2024
+SELECT 
+    c.customer_id,
+    c.name,
+    c.email
+FROM customers c
+WHERE EXISTS (
+    SELECT 1 
+    FROM orders o 
+    WHERE o.customer_id = c.customer_id
+      AND o.order_date >= '2024-01-01'
+      AND o.order_date < '2025-01-01'
+);
+
+-- Example 2: Products ordered in 2023 but not in 2024
+SELECT 
+    p.product_id,
+    p.name,
+    p.category
+FROM products p
+WHERE EXISTS (
+    SELECT 1 
+    FROM order_items oi
+    JOIN orders o ON oi.order_id = o.order_id
+    WHERE oi.product_id = p.product_id
+      AND o.order_date >= '2023-01-01'
+      AND o.order_date < '2024-01-01'
+)
+AND NOT EXISTS (
+    SELECT 1 
+    FROM order_items oi
+    JOIN orders o ON oi.order_id = o.order_id
+    WHERE oi.product_id = p.product_id
+      AND o.order_date >= '2024-01-01'
+      AND o.order_date < '2025-01-01'
+);
+
+-- Key Concepts:
+-- - EXISTS returns TRUE if subquery returns any rows
+-- - More efficient than IN for large datasets
+-- - Better NULL handling than IN/NOT IN
+-- - Short-circuits: stops at first match
+-- - Use SELECT 1 (constant) instead of SELECT * for efficiency

rag/sample_templates/08_cte_multi_step.sql

@@ -0,0 +1,55 @@
+-- Template: Multi-Step CTEs for Complex Analysis
+-- Pattern: Break complex queries into logical steps
+-- Use Case: Complex calculations, data transformation pipelines
+
+WITH 
+-- Step 1: Calculate customer metrics
+customer_metrics AS (
+    SELECT 
+        c.customer_id,
+        c.name,
+        COUNT(o.order_id) as total_orders,
+        SUM(o.total_amount) as total_spent,
+        MIN(o.order_date) as first_order_date,
+        MAX(o.order_date) as last_order_date
+    FROM customers c
+    LEFT JOIN orders o ON c.customer_id = o.customer_id
+    GROUP BY c.customer_id, c.name
+),
+-- Step 2: Calculate overall averages
+overall_stats AS (
+    SELECT 
+        AVG(total_spent) as avg_customer_value,
+        AVG(total_orders) as avg_orders_per_customer
+    FROM customer_metrics
+),
+-- Step 3: Classify customers
+customer_segments AS (
+    SELECT 
+        cm.*,
+        os.avg_customer_value,
+        CASE 
+            WHEN cm.total_spent > os.avg_customer_value * 2 THEN 'VIP'
+            WHEN cm.total_spent > os.avg_customer_value THEN 'High Value'
+            WHEN cm.total_spent > 0 THEN 'Regular'
+            ELSE 'Inactive'
+        END as segment
+    FROM customer_metrics cm
+    CROSS JOIN overall_stats os
+)
+-- Final output
+SELECT 
+    segment,
+    COUNT(*) as customer_count,
+    AVG(total_spent) as avg_spent,
+    AVG(total_orders) as avg_orders
+FROM customer_segments
+GROUP BY segment
+ORDER BY avg_spent DESC;
+
+-- Key Concepts:
+-- - WITH clause defines multiple CTEs (Common Table Expressions)
+-- - Each CTE builds on previous ones
+-- - Better readability than nested subqueries
+-- - Can reference previous CTEs in later ones
+-- - Improves maintainability and debugging

rag/sample_templates/09_cohort_analysis.sql

@@ -0,0 +1,40 @@
+-- Template: Cohort Analysis
+-- Pattern: Group customers by acquisition period and track behavior
+-- Use Case: First purchase cohorts, retention analysis, lifetime value
+
+WITH first_purchases AS (
+    SELECT 
+        customer_id,
+        MIN(order_date) as cohort_month,
+        DATE_TRUNC('month', MIN(order_date)) as cohort_date
+    FROM orders
+    GROUP BY customer_id
+),
+customer_cohorts AS (
+    SELECT 
+        o.customer_id,
+        o.order_date,
+        o.total_amount,
+        fp.cohort_date,
+        DATE_TRUNC('month', o.order_date) as order_month,
+        EXTRACT(YEAR FROM AGE(o.order_date, fp.cohort_date)) * 12 + 
+        EXTRACT(MONTH FROM AGE(o.order_date, fp.cohort_date)) as months_since_first
+    FROM orders o
+    JOIN first_purchases fp ON o.customer_id = fp.customer_id
+)
+SELECT 
+    cohort_date,
+    months_since_first,
+    COUNT(DISTINCT customer_id) as active_customers,
+    SUM(total_amount) as cohort_revenue,
+    AVG(total_amount) as avg_order_value
+FROM customer_cohorts
+GROUP BY cohort_date, months_since_first
+ORDER BY cohort_date, months_since_first;
+
+-- Key Concepts:
+-- - Cohorts: Groups defined by shared characteristic (e.g., signup month)
+-- - Track behavior over time relative to cohort start
+-- - DATE_TRUNC for period grouping
+-- - AGE function for time calculations
+-- - Useful for retention and LTV analysis

rag/sample_templates/10_yoy_comparison.sql

@@ -0,0 +1,37 @@
+-- Template: Year-over-Year Comparison
+-- Pattern: Compare metrics across time periods
+-- Use Case: Revenue growth, seasonal trends, period comparisons
+
+WITH monthly_revenue AS (
+    SELECT 
+        DATE_TRUNC('month', order_date) as month,
+        EXTRACT(YEAR FROM order_date) as year,
+        EXTRACT(MONTH FROM order_date) as month_num,
+        SUM(total_amount) as revenue,
+        COUNT(DISTINCT customer_id) as unique_customers
+    FROM orders
+    GROUP BY DATE_TRUNC('month', order_date), 
+             EXTRACT(YEAR FROM order_date),
+             EXTRACT(MONTH FROM order_date)
+)
+SELECT 
+    curr.month as current_month,
+    curr.revenue as current_revenue,
+    prev.revenue as previous_year_revenue,
+    curr.revenue - prev.revenue as revenue_change,
+    ROUND(((curr.revenue - prev.revenue) / prev.revenue * 100), 2) as growth_pct,
+    curr.unique_customers as current_customers,
+    prev.unique_customers as previous_year_customers
+FROM monthly_revenue curr
+LEFT JOIN monthly_revenue prev 
+    ON curr.month_num = prev.month_num 
+    AND curr.year = prev.year + 1
+WHERE prev.revenue IS NOT NULL
+ORDER BY curr.month DESC;
+
+-- Key Concepts:
+-- - DATE_TRUNC for period aggregation
+-- - Self-join to compare same periods across years
+-- - Percentage change calculation
+-- - EXTRACT for getting year/month components
+-- - Useful for trend analysis and forecasting

rag/sample_templates/11_moving_average.sql

@@ -0,0 +1,36 @@
+-- Template: Moving Average (Rolling Average)
+-- Pattern: Calculate average over sliding time window
+-- Use Case: Smooth out fluctuations, identify trends, 7-day MA, 30-day MA
+
+SELECT 
+    order_date,
+    daily_revenue,
+    AVG(daily_revenue) OVER (
+        ORDER BY order_date 
+        ROWS BETWEEN 6 PRECEDING AND CURRENT ROW
+    ) as moving_avg_7day,
+    AVG(daily_revenue) OVER (
+        ORDER BY order_date 
+        ROWS BETWEEN 29 PRECEDING AND CURRENT ROW
+    ) as moving_avg_30day,
+    daily_order_count,
+    AVG(daily_order_count) OVER (
+        ORDER BY order_date 
+        ROWS BETWEEN 6 PRECEDING AND CURRENT ROW
+    ) as avg_orders_7day
+FROM (
+    SELECT 
+        DATE(order_date) as order_date,
+        SUM(total_amount) as daily_revenue,
+        COUNT(*) as daily_order_count
+    FROM orders
+    GROUP BY DATE(order_date)
+) daily_stats
+ORDER BY order_date;
+
+-- Key Concepts:
+-- - Window function with ROWS BETWEEN for sliding window
+-- - N PRECEDING means previous N rows
+-- - CURRENT ROW includes the current row
+-- - For 7-day MA: use 6 PRECEDING (includes current = 7 total)
+-- - Useful for smoothing volatile metrics

rag/sample_templates/12_self_join.sql

@@ -0,0 +1,41 @@
+-- Template: Self-Join for Hierarchical or Sequential Data
+-- Pattern: Join table to itself to find relationships within same table
+-- Use Case: Manager-employee, sequential orders, referrals, product recommendations
+
+-- Example 1: Find customers who ordered same product
+SELECT DISTINCT
+    c1.customer_id as customer1,
+    c1.name as customer1_name,
+    c2.customer_id as customer2,
+    c2.name as customer2_name,
+    oi1.product_id,
+    p.name as product_name
+FROM order_items oi1
+JOIN order_items oi2 ON oi1.product_id = oi2.product_id 
+    AND oi1.order_id < oi2.order_id  -- Avoid duplicates
+JOIN orders o1 ON oi1.order_id = o1.order_id
+JOIN orders o2 ON oi2.order_id = o2.order_id
+JOIN customers c1 ON o1.customer_id = c1.customer_id
+JOIN customers c2 ON o2.customer_id = c2.customer_id
+JOIN products p ON oi1.product_id = p.product_id
+WHERE c1.customer_id != c2.customer_id;
+
+-- Example 2: Find sequential orders by same customer
+SELECT 
+    c.name,
+    o1.order_id as first_order,
+    o1.order_date as first_date,
+    o2.order_id as next_order,
+    o2.order_date as next_date,
+    o2.order_date - o1.order_date as days_between
+FROM orders o1
+JOIN orders o2 ON o1.customer_id = o2.customer_id 
+    AND o2.order_date > o1.order_date
+JOIN customers c ON o1.customer_id = c.customer_id
+ORDER BY c.name, o1.order_date;
+
+-- Key Concepts:
+-- - Join table to itself with different aliases
+-- - Use inequality (< or !=) to avoid matching same row
+-- - Useful for finding patterns, sequences, or relationships
+-- - Can be combined with window functions for "next" or "previous" record

rag/sample_templates/13_union_dedupe.sql

@@ -0,0 +1,47 @@
+-- Template: UNION vs UNION ALL for Deduplication
+-- Pattern: Combine results from multiple queries
+-- Use Case: Merge data from different sources, combine segments
+
+-- UNION ALL: Keeps all rows including duplicates (FASTER)
+SELECT customer_id, name, 'Active' as status
+FROM customers
+WHERE last_order_date >= CURRENT_DATE - INTERVAL '90 days'
+UNION ALL
+SELECT customer_id, name, 'VIP' as status
+FROM customers
+WHERE total_lifetime_value > 10000;
+
+-- UNION: Removes duplicates (SLOWER but cleaner)
+SELECT customer_id, name, email
+FROM customers
+WHERE city = 'New York'
+UNION
+SELECT customer_id, name, email
+FROM customers
+WHERE total_lifetime_value > 5000;
+
+-- Example: Combine current and archived orders
+SELECT 
+    order_id,
+    customer_id,
+    order_date,
+    total_amount,
+    'current' as source
+FROM orders
+WHERE order_date >= '2024-01-01'
+UNION ALL
+SELECT 
+    order_id,
+    customer_id,
+    order_date,
+    total_amount,
+    'archived' as source
+FROM archived_orders
+WHERE order_date < '2024-01-01';
+
+-- Key Concepts:
+-- - UNION removes duplicates (requires sort/compare - slower)
+-- - UNION ALL keeps all rows (faster, no deduplication)
+-- - All SELECT statements must have same number and type of columns
+-- - Use UNION ALL when you know there are no duplicates
+-- - Add source identifier column to track origin

rag/sample_templates/14_pivoting.sql

@@ -0,0 +1,35 @@
+-- Template: Pivoting Data (CASE with Aggregation)
+-- Pattern: Transform rows into columns
+-- Use Case: Crosstab reports, monthly summaries, category breakdowns
+
+-- Example 1: Revenue by month in columns
+SELECT 
+    EXTRACT(YEAR FROM order_date) as year,
+    SUM(CASE WHEN EXTRACT(MONTH FROM order_date) = 1 THEN total_amount ELSE 0 END) as jan,
+    SUM(CASE WHEN EXTRACT(MONTH FROM order_date) = 2 THEN total_amount ELSE 0 END) as feb,
+    SUM(CASE WHEN EXTRACT(MONTH FROM order_date) = 3 THEN total_amount ELSE 0 END) as mar,
+    SUM(CASE WHEN EXTRACT(MONTH FROM order_date) = 4 THEN total_amount ELSE 0 END) as apr,
+    SUM(CASE WHEN EXTRACT(MONTH FROM order_date) = 5 THEN total_amount ELSE 0 END) as may,
+    SUM(CASE WHEN EXTRACT(MONTH FROM order_date) = 6 THEN total_amount ELSE 0 END) as jun,
+    SUM(total_amount) as total_year
+FROM orders
+GROUP BY EXTRACT(YEAR FROM order_date)
+ORDER BY year;
+
+-- Example 2: Customer count by status and city
+SELECT 
+    city,
+    SUM(CASE WHEN status = 'active' THEN 1 ELSE 0 END) as active_customers,
+    SUM(CASE WHEN status = 'inactive' THEN 1 ELSE 0 END) as inactive_customers,
+    SUM(CASE WHEN status = 'vip' THEN 1 ELSE 0 END) as vip_customers,
+    COUNT(*) as total_customers
+FROM customers
+GROUP BY city
+ORDER BY total_customers DESC;
+
+-- Key Concepts:
+-- - CASE WHEN inside aggregate function
+-- - Each CASE creates a new column
+-- - SUM with 0/1 for counts, SUM with value for totals
+-- - Alternative: Use FILTER (WHERE) in PostgreSQL
+-- - Useful for summary reports and dashboards

rag/sample_templates/15_subquery_optimization.sql

@@ -0,0 +1,49 @@
+-- Template: Subquery Optimization - NOT IN vs NOT EXISTS
+-- Pattern: Efficient ways to exclude records based on subquery
+-- Use Case: Find records that don't match conditions, anti-joins
+
+-- ❌ SLOWER: NOT IN with subquery (issues with NULL values)
+SELECT customer_id, name
+FROM customers
+WHERE customer_id NOT IN (
+    SELECT customer_id 
+    FROM orders
+    WHERE order_date >= '2024-01-01'
+);
+-- Problem: If orders.customer_id has ANY NULL, entire query returns empty!
+
+-- ✅ BETTER: NOT EXISTS (faster and NULL-safe)
+SELECT c.customer_id, c.name
+FROM customers c
+WHERE NOT EXISTS (
+    SELECT 1 
+    FROM orders o
+    WHERE o.customer_id = c.customer_id
+      AND o.order_date >= '2024-01-01'
+);
+
+-- ✅ ALTERNATIVE: LEFT JOIN with NULL check
+SELECT c.customer_id, c.name
+FROM customers c
+LEFT JOIN orders o ON c.customer_id = o.customer_id
+    AND o.order_date >= '2024-01-01'
+WHERE o.order_id IS NULL;
+
+-- Example: Products NOT ordered in specific date range
+SELECT p.product_id, p.name, p.category
+FROM products p
+WHERE NOT EXISTS (
+    SELECT 1
+    FROM order_items oi
+    JOIN orders o ON oi.order_id = o.order_id
+    WHERE oi.product_id = p.product_id
+      AND o.order_date >= '2024-01-01'
+      AND o.order_date < '2024-04-01'
+);
+
+-- Key Concepts:
+-- - NOT IN fails if subquery returns NULL
+-- - NOT EXISTS is NULL-safe and often faster
+-- - LEFT JOIN with IS NULL also works well
+-- - NOT EXISTS can short-circuit (stops at first match)
+-- - Use SELECT 1 in EXISTS for efficiency

rag/template_store.py

@@ -0,0 +1,204 @@
+"""
+RAG Template Store using LlamaIndex
+PRIMARY SPONSOR INTEGRATION - LlamaIndex
+Provides similar pattern suggestions from ETL template library
+"""
+
+import os
+from pathlib import Path
+from typing import List, Dict
+from llama_index.core import VectorStoreIndex, Document, StorageContext
+from llama_index.core.node_parser import SimpleNodeParser
+from llama_index.embeddings.openai import OpenAIEmbedding
+from llama_index.core import Settings
+import chromadb
+from llama_index.vector_stores.chroma import ChromaVectorStore
+
+
+class TemplateStore:
+    """
+    RAG-powered template store using LlamaIndex
+    Finds similar SQL patterns from template library
+    """
+    
+    def __init__(self, templates_dir: str = None):
+        """Initialize the template store with LlamaIndex"""
+        
+        if templates_dir is None:
+            # Default to sample_templates directory
+            current_dir = Path(__file__).parent
+            templates_dir = current_dir / "sample_templates"
+        else:
+            templates_dir = Path(templates_dir)
+        
+        self.templates_dir = templates_dir
+        
+        # Initialize OpenAI embeddings
+        try:
+            Settings.embed_model = OpenAIEmbedding(
+                model="text-embedding-3-small",
+                api_key=os.getenv("OPENAI_API_KEY")
+            )
+        except Exception as e:
+            print(f"⚠️ Warning: Could not initialize OpenAI embeddings: {e}")
+            print("   RAG features will be limited. Set OPENAI_API_KEY in .env")
+            self.index = None
+            return
+        
+        # Initialize ChromaDB
+        try:
+            chroma_client = chromadb.EphemeralClient()
+            chroma_collection = chroma_client.create_collection("sql_templates")
+            vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
+            storage_context = StorageContext.from_defaults(vector_store=vector_store)
+            
+            # Load templates
+            documents = self._load_templates()
+            
+            if not documents:
+                print("⚠️ Warning: No templates found")
+                self.index = None
+                return
+            
+            # Create index
+            self.index = VectorStoreIndex.from_documents(
+                documents,
+                storage_context=storage_context
+            )
+            
+            print(f"✓ RAG Template Store initialized with {len(documents)} templates")
+            
+        except Exception as e:
+            print(f"⚠️ Warning: Could not initialize RAG: {e}")
+            self.index = None
+    
+    def _load_templates(self) -> List[Document]:
+        """Load all SQL templates from directory"""
+        documents = []
+        
+        if not self.templates_dir.exists():
+            print(f"⚠️ Warning: Templates directory not found: {self.templates_dir}")
+            return documents
+        
+        # Load all .sql files
+        for sql_file in sorted(self.templates_dir.glob("*.sql")):
+            try:
+                with open(sql_file, 'r', encoding='utf-8') as f:
+                    content = f.read()
+                
+                # Extract template name from filename
+                template_name = sql_file.stem.replace('_', ' ').title()
+                
+                # Create document with metadata
+                doc = Document(
+                    text=content,
+                    metadata={
+                        "filename": sql_file.name,
+                        "template_name": template_name,
+                        "path": str(sql_file)
+                    }
+                )
+                documents.append(doc)
+                
+            except Exception as e:
+                print(f"⚠️ Warning: Could not load template {sql_file}: {e}")
+        
+        return documents
+    
+    def find_similar(self, query: str, top_k: int = 3) -> List[Dict]:
+        """
+        Find similar templates based on natural language query
+        
+        Args:
+            query: Natural language description of desired pattern
+            top_k: Number of similar templates to return
+            
+        Returns:
+            List of dicts with template info and similarity scores
+        """
+        if self.index is None:
+            return []
+        
+        try:
+            # Query the index
+            retriever = self.index.as_retriever(similarity_top_k=top_k)
+            nodes = retriever.retrieve(query)
+            
+            results = []
+            for node in nodes:
+                # Extract template information
+                template_info = {
+                    "template_name": node.metadata.get("template_name", "Unknown"),
+                    "filename": node.metadata.get("filename", ""),
+                    "content": node.text,
+                    "score": node.score if hasattr(node, 'score') else 0.0,
+                    "excerpt": self._extract_excerpt(node.text)
+                }
+                results.append(template_info)
+            
+            return results
+            
+        except Exception as e:
+            print(f"⚠️ Warning: Error finding similar templates: {e}")
+            return []
+    
+    def _extract_excerpt(self, content: str, max_lines: int = 5) -> str:
+        """Extract a short excerpt from template"""
+        lines = content.split('\n')
+        
+        # Skip comment lines and find actual SQL
+        sql_lines = []
+        for line in lines:
+            stripped = line.strip()
+            if stripped and not stripped.startswith('--'):
+                sql_lines.append(line)
+                if len(sql_lines) >= max_lines:
+                    break
+        
+        if sql_lines:
+            excerpt = '\n'.join(sql_lines[:max_lines])
+            if len(sql_lines) > max_lines:
+                excerpt += "\n..."
+            return excerpt
+        else:
+            # Return first few non-empty lines if no SQL found
+            non_empty = [l for l in lines if l.strip()][:max_lines]
+            return '\n'.join(non_empty)
+    
+    def get_all_templates(self) -> List[Dict]:
+        """Get information about all available templates"""
+        templates = []
+        
+        if not self.templates_dir.exists():
+            return templates
+        
+        for sql_file in sorted(self.templates_dir.glob("*.sql")):
+            try:
+                with open(sql_file, 'r', encoding='utf-8') as f:
+                    content = f.read()
+                
+                # Extract first comment line as description
+                first_line = content.split('\n')[0].strip()
+                description = first_line.replace('--', '').strip() if first_line.startswith('--') else ""
+                
+                templates.append({
+                    "filename": sql_file.name,
+                    "template_name": sql_file.stem.replace('_', ' ').title(),
+                    "description": description,
+                    "path": str(sql_file)
+                })
+            except Exception as e:
+                print(f"⚠️ Warning: Error reading template {sql_file}: {e}")
+        
+        return templates
+
+
+# Singleton instance
+_template_store = None
+
+def get_template_store() -> TemplateStore:
+    """Get or create the global template store instance"""
+    global _template_store
+    if _template_store is None:
+        _template_store = TemplateStore()
+    return _template_store

requirements.txt

@@ -0,0 +1,28 @@
+# Core UI
+gradio>=4.44.1
+python-dotenv==1.0.0
+
+# AI/ML
+anthropic>=0.39.0
+
+# PRIMARY SPONSOR - LlamaIndex RAG
+llama-index==0.9.14
+llama-index-embeddings-openai
+chromadb==0.4.18
+openai
+
+# SECONDARY SPONSOR - Modal Testing
+modal>=1.2.0
+
+# Database
+sqlalchemy==2.0.23
+pandas==2.1.3
+duckdb==0.9.2
+
+# SQL Processing
+sqlparse==0.4.4
+jinja2==3.1.2
+pyyaml==6.0.1
+
+# Optional
+plotly==5.18.0

schema_analyzer.py

@@ -0,0 +1,32 @@
+import sqlite3
+
+class SchemaAnalyzer:
+    """Analyzes database schema"""
+    
+    def __init__(self, db_path='sample.db'):
+        self.db_path = db_path
+    
+    def get_schema(self):
+        """Get complete database schema"""
+        conn = sqlite3.connect(self.db_path)
+        cursor = conn.cursor()
+        
+        # Get all tables
+        cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
+        tables = [row[0] for row in cursor.fetchall()]
+        
+        schema = {}
+        for table in tables:
+            cursor.execute(f"PRAGMA table_info({table})")
+            columns = []
+            for row in cursor.fetchall():
+                columns.append({
+                    "name": row[1],
+                    "type": row[2],
+                    "nullable": not row[3],
+                    "primary_key": bool(row[5])
+                })
+            schema[table] = columns
+        
+        conn.close()
+        return schema

sql_validator.py

@@ -0,0 +1,171 @@
+"""
+SQL Query Validator and Optimizer
+Provides validation, optimization suggestions, and query analysis
+"""
+
+import re
+from typing import Dict, List, Tuple
+
+class SQLValidator:
+    """Validates and analyzes SQL queries"""
+    
+    def __init__(self, schema: Dict):
+        self.schema = schema
+        self.table_names = set(schema.keys())
+        self.column_map = {}
+        
+        # Build column map for quick lookup
+        for table, columns in schema.items():
+            for col in columns:
+                col_name = col['name'].lower()
+                if col_name not in self.column_map:
+                    self.column_map[col_name] = []
+                self.column_map[col_name].append(table)
+    
+    def validate(self, sql: str) -> Dict[str, any]:
+        """
+        Validate SQL query against schema
+        Returns dict with validation results
+        """
+        results = {
+            "valid": True,
+            "errors": [],
+            "warnings": [],
+            "suggestions": [],
+            "query_type": self._detect_query_type(sql)
+        }
+        
+        # Check for common SQL anti-patterns
+        self._check_null_comparison(sql, results)
+        self._check_select_star(sql, results)
+        self._check_missing_where(sql, results)
+        self._check_implicit_joins(sql, results)
+        self._check_table_names(sql, results)
+        
+        # If we found errors, mark as invalid
+        if results["errors"]:
+            results["valid"] = False
+        
+        return results
+    
+    def _detect_query_type(self, sql: str) -> str:
+        """Detect the type of query"""
+        sql_upper = sql.upper()
+        
+        if "WITH" in sql_upper and "AS" in sql_upper:
+            return "cte"
+        elif any(func in sql_upper for func in ["ROW_NUMBER(", "RANK(", "DENSE_RANK(", "PARTITION BY"]):
+            return "window"
+        elif "GROUP BY" in sql_upper or any(func in sql_upper for func in ["COUNT(", "SUM(", "AVG(", "MAX(", "MIN("]):
+            return "aggregate"
+        elif "JOIN" in sql_upper:
+            return "join"
+        elif "UNION" in sql_upper:
+            return "union"
+        elif "EXISTS" in sql_upper or "IN (SELECT" in sql_upper:
+            return "subquery"
+        else:
+            return "simple"
+    
+    def _check_null_comparison(self, sql: str, results: Dict):
+        """Check for = NULL instead of IS NULL"""
+        if re.search(r"=\s*NULL|!=\s*NULL|<>\s*NULL", sql, re.IGNORECASE):
+            results["errors"].append(
+                "Use IS NULL or IS NOT NULL instead of = NULL or != NULL"
+            )
+    
+    def _check_select_star(self, sql: str, results: Dict):
+        """Warn about SELECT *"""
+        if re.search(r"SELECT\s+\*", sql, re.IGNORECASE):
+            results["warnings"].append(
+                "Consider specifying column names instead of SELECT * for better performance"
+            )
+    
+    def _check_missing_where(self, sql: str, results: Dict):
+        """Check for queries without WHERE clause on large tables"""
+        sql_upper = sql.upper()
+        if "DELETE" in sql_upper or "UPDATE" in sql_upper:
+            if "WHERE" not in sql_upper:
+                results["errors"].append(
+                    "DELETE or UPDATE without WHERE clause will affect all rows"
+                )
+    
+    def _check_implicit_joins(self, sql: str, results: Dict):
+        """Check for implicit joins (comma-separated tables)"""
+        # Look for FROM table1, table2 pattern
+        if re.search(r"FROM\s+\w+\s*,\s*\w+", sql, re.IGNORECASE):
+            results["suggestions"].append(
+                "Consider using explicit JOIN syntax instead of comma-separated tables"
+            )
+    
+    def _check_table_names(self, sql: str, results: Dict):
+        """Check if referenced tables exist in schema"""
+        # Extract table names from FROM and JOIN clauses
+        from_pattern = r"FROM\s+(\w+)"
+        join_pattern = r"JOIN\s+(\w+)"
+        
+        tables_in_query = set()
+        
+        for match in re.finditer(from_pattern, sql, re.IGNORECASE):
+            tables_in_query.add(match.group(1).lower())
+        
+        for match in re.finditer(join_pattern, sql, re.IGNORECASE):
+            tables_in_query.add(match.group(1).lower())
+        
+        # Check against schema
+        schema_tables = set(t.lower() for t in self.table_names)
+        invalid_tables = tables_in_query - schema_tables
+        
+        if invalid_tables:
+            results["errors"].append(
+                f"Table(s) not found in schema: {', '.join(invalid_tables)}"
+            )
+    
+    def suggest_optimizations(self, sql: str) -> List[str]:
+        """Suggest optimizations for the query"""
+        suggestions = []
+        sql_upper = sql.upper()
+        
+        # Check for NOT IN with subquery
+        if "NOT IN (SELECT" in sql_upper:
+            suggestions.append(
+                "Consider using NOT EXISTS instead of NOT IN for better NULL handling"
+            )
+        
+        # Check for multiple subqueries
+        subquery_count = sql_upper.count("(SELECT")
+        if subquery_count > 2:
+            suggestions.append(
+                "Consider using CTEs (WITH clause) for better readability with multiple subqueries"
+            )
+        
+        # Check for DISTINCT
+        if "DISTINCT" in sql_upper and "GROUP BY" not in sql_upper:
+            suggestions.append(
+                "DISTINCT can be expensive. Consider if GROUP BY might be more appropriate"
+            )
+        
+        # Check for ORDER BY in subquery
+        if re.search(r"\(SELECT.*ORDER BY.*\)", sql, re.IGNORECASE):
+            suggestions.append(
+                "ORDER BY in subquery may be ignored. Apply ORDER BY to outer query"
+            )
+        
+        return suggestions
+
+    def format_sql(self, sql: str) -> str:
+        """Basic SQL formatting for readability"""
+        # This is a simple formatter - for production use a proper SQL formatter
+        formatted = sql.strip()
+        
+        # Add newlines before major keywords
+        keywords = ['SELECT', 'FROM', 'WHERE', 'GROUP BY', 'HAVING', 'ORDER BY', 'LIMIT']
+        for keyword in keywords:
+            formatted = re.sub(
+                f'\\b{keyword}\\b',
+                f'\n{keyword}',
+                formatted,
+                flags=re.IGNORECASE
+            )
+        
+        return formatted.strip()

testing/__init__.py

@@ -0,0 +1,4 @@
+"""
+Testing module for CodeFlow AI
+Uses Modal for serverless SQL execution
+"""

testing/test_runner.py

@@ -0,0 +1,245 @@
+"""
+SQL Test Runner using Modal (SECONDARY SPONSOR)
+Provides serverless SQL execution for testing queries
+"""
+
+import os
+from typing import Dict, Any
+
+# Try to import Modal, but don't fail if not available
+try:
+    import modal
+    MODAL_AVAILABLE = True
+except ImportError:
+    MODAL_AVAILABLE = False
+    print("⚠️ Modal not available. Install with: pip install modal")
+
+# Try to import DuckDB for local fallback
+try:
+    import duckdb
+    DUCKDB_AVAILABLE = True
+except ImportError:
+    DUCKDB_AVAILABLE = False
+    print("⚠️ DuckDB not available for local testing fallback")
+
+
+class SQLTestRunner:
+    """
+    Test runner for SQL queries
+    Uses Modal for serverless execution, with DuckDB fallback
+    """
+    
+    def __init__(self):
+        self.modal_available = MODAL_AVAILABLE
+        self.duckdb_available = DUCKDB_AVAILABLE
+        
+        # Initialize Modal app if available
+        if self.modal_available:
+            try:
+                self.app = modal.App("codeflow-sql-tester")
+                self._setup_modal()
+            except Exception as e:
+                print(f"⚠️ Modal setup failed: {e}")
+                self.modal_available = False
+    
+    def _setup_modal(self):
+        """Setup Modal function for SQL execution"""
+        if not self.modal_available:
+            return
+        
+        try:
+            # Define Modal image with DuckDB
+            image = modal.Image.debian_slim().pip_install("duckdb==0.9.2")
+            
+            # Define Modal function
+            @self.app.function(image=image, timeout=60)
+            def execute_sql_modal(sql: str, sample_data: dict = None) -> dict:
+                """Execute SQL in Modal sandbox"""
+                import duckdb
+                
+                try:
+                    # Create in-memory DuckDB database
+                    conn = duckdb.connect(':memory:')
+                    
+                    # If sample data provided, create tables
+                    if sample_data:
+                        for table_name, data in sample_data.items():
+                            conn.execute(f"CREATE TABLE {table_name} AS SELECT * FROM ?", [data])
+                    
+                    # Execute the SQL
+                    result = conn.execute(sql).fetchall()
+                    columns = [desc[0] for desc in conn.description] if conn.description else []
+                    
+                    conn.close()
+                    
+                    return {
+                        "success": True,
+                        "rows": result[:100],  # Limit to 100 rows
+                        "row_count": len(result),
+                        "columns": columns,
+                        "error": None
+                    }
+                    
+                except Exception as e:
+                    return {
+                        "success": False,
+                        "rows": [],
+                        "row_count": 0,
+                        "columns": [],
+                        "error": str(e)
+                    }
+            
+            self.execute_sql_modal = execute_sql_modal
+            
+        except Exception as e:
+            print(f"⚠️ Modal function setup failed: {e}")
+            self.modal_available = False
+    
+    def test_sql(self, sql: str, sample_data: Dict = None) -> Dict[str, Any]:
+        """
+        Test SQL query execution
+        
+        Args:
+            sql: SQL query to test
+            sample_data: Optional sample data as dict of table_name -> rows
+            
+        Returns:
+            Dict with test results
+        """
+        
+        # Try Modal first if available
+        if self.modal_available:
+            try:
+                result = self.execute_sql_modal.remote(sql, sample_data)
+                result["execution_method"] = "Modal (Serverless)"
+                return result
+            except Exception as e:
+                print(f"⚠️ Modal execution failed, falling back to local: {e}")
+        
+        # Fallback to local DuckDB
+        if self.duckdb_available:
+            return self._test_sql_local(sql, sample_data)
+        
+        # No execution method available
+        return {
+            "success": False,
+            "rows": [],
+            "row_count": 0,
+            "columns": [],
+            "error": "No SQL execution method available. Install Modal or DuckDB.",
+            "execution_method": "None"
+        }
+    
+    def _test_sql_local(self, sql: str, sample_data: Dict = None) -> Dict[str, Any]:
+        """Execute SQL locally using DuckDB"""
+        try:
+            conn = duckdb.connect(':memory:')
+            
+            # Create sample tables if provided
+            if sample_data:
+                for table_name, data in sample_data.items():
+                    conn.execute(f"CREATE TABLE {table_name} AS SELECT * FROM ?", [data])
+            
+            # Execute SQL
+            result = conn.execute(sql).fetchall()
+            columns = [desc[0] for desc in conn.description] if conn.description else []
+            
+            conn.close()
+            
+            return {
+                "success": True,
+                "rows": result[:100],  # Limit to 100 rows
+                "row_count": len(result),
+                "columns": columns,
+                "error": None,
+                "execution_method": "DuckDB (Local)"
+            }
+            
+        except Exception as e:
+            return {
+                "success": False,
+                "rows": [],
+                "row_count": 0,
+                "columns": [],
+                "error": str(e),
+                "execution_method": "DuckDB (Local)"
+            }
+    
+    def validate_syntax(self, sql: str) -> Dict[str, Any]:
+        """
+        Quick syntax validation without execution
+        
+        Args:
+            sql: SQL query to validate
+            
+        Returns:
+            Dict with validation results
+        """
+        if not self.duckdb_available:
+            return {
+                "valid": None,
+                "error": "DuckDB not available for syntax validation"
+            }
+        
+        try:
+            conn = duckdb.connect(':memory:')
+            # Try to prepare the query (doesn't execute, just validates syntax)
+            conn.execute(f"EXPLAIN {sql}")
+            conn.close()
+            
+            return {
+                "valid": True,
+                "error": None
+            }
+            
+        except Exception as e:
+            return {
+                "valid": False,
+                "error": str(e)
+            }
+    
+    def get_execution_plan(self, sql: str) -> Dict[str, Any]:
+        """
+        Get query execution plan
+        
+        Args:
+            sql: SQL query
+            
+        Returns:
+            Dict with execution plan
+        """
+        if not self.duckdb_available:
+            return {
+                "success": False,
+                "plan": None,
+                "error": "DuckDB not available"
+            }
+        
+        try:
+            conn = duckdb.connect(':memory:')
+            plan = conn.execute(f"EXPLAIN {sql}").fetchall()
+            conn.close()
+            
+            return {
+                "success": True,
+                "plan": "\n".join([str(row[0]) for row in plan]),
+                "error": None
+            }
+            
+        except Exception as e:
+            return {
+                "success": False,
+                "plan": None,
+                "error": str(e)
+            }
+
+
+# Singleton instance
+_test_runner = None
+
+def get_test_runner() -> SQLTestRunner:
+    """Get or create the global test runner instance"""
+    global _test_runner
+    if _test_runner is None:
+        _test_runner = SQLTestRunner()
+    return _test_runner