Spaces:
Running
Running
| title: BirdScope AI - MCP Multi-Agent System | |
| emoji: 🦅 | |
| colorFrom: green | |
| colorTo: blue | |
| sdk: gradio | |
| python_version: 3.11 | |
| app_file: app.py | |
| pinned: false | |
| # 🦅 BirdScope AI - Multi-Agent Bird Identification System | |
| **AI-powered bird identification with specialized MCP agents** | |
| Built for the [MCP 1st Birthday Hackathon](https://huggingface.co/MCP-1st-Birthday) | |
| --- | |
| ## 🎯 Overview | |
| BirdScope AI is a production-ready multi-agent system that combines **Modal GPU classification** with **Nuthatch species database** to provide comprehensive bird identification and exploration. Users can upload photos, search species, explore taxonomic families, and access rich multimedia content (images, audio recordings, conservation data). | |
| **Two Agent Modes:** | |
| 1. **Specialized Subagents (3 Specialists)** - Router orchestrates image identifier, species explorer, and taxonomy specialist | |
| 2. **Audio Finder Agent** - Specialized agent for discovering bird audio recordings | |
| --- | |
| ## ✨ Features | |
| - 🔍 **Image Classification**: Upload bird photos for instant GPU-powered identification | |
| - 📸 **Reference Images**: High-quality Unsplash photos for each species | |
| - 🎵 **Audio Recordings**: Bird calls and songs from xeno-canto.org | |
| - 🌍 **Conservation Data**: IUCN status and taxonomic information | |
| - 🧠 **Multi-Agent Architecture**: Specialized agents with focused tool subsets | |
| - 🔄 **Dual Streaming**: Separate outputs for chat responses and tool execution logs | |
| - 🤖 **Multi-Provider**: OpenAI (GPT-4), Anthropic (Claude), HuggingFace (Qwen) | |
| --- | |
| ## 🚀 Quick Start (For Users) | |
| ### Option 1: OpenAI (Recommended) | |
| 1. Get your OpenAI API key from [platform.openai.com/api-keys](https://platform.openai.com/api-keys) | |
| 2. Select **OpenAI** as provider in the sidebar | |
| 3. Enter your API key | |
| 4. Model used: `gpt-4o-mini` | |
| ### Option 2: Anthropic (Claude) | |
| 1. Get your Anthropic API key from [console.anthropic.com/settings/keys](https://console.anthropic.com/settings/keys) | |
| 2. Select **Anthropic** as provider | |
| 3. Enter your API key | |
| 4. Model used: `claude-sonnet-4-5` | |
| ### Option 3: HuggingFace | |
| ⚠️ **Note**: HuggingFace Inference API has limited function calling support. OpenAI or Anthropic recommended for full functionality. | |
| --- | |
| ## 🛠️ Environment Setup (For Developers) | |
| ### Prerequisites | |
| - Python 3.11+ | |
| - Modal account (for GPU classifier) | |
| - Nuthatch API key | |
| - LLM API key (OpenAI, Anthropic, or HuggingFace) | |
| --- | |
| ### 🏠 Local Development Setup | |
| #### Step 1: Clone and Install | |
| ```bash | |
| cd ~/Desktop/hackathon/hackathon_draft | |
| # Create virtual environment | |
| python3.11 -m venv .venv | |
| source .venv/bin/activate # On Windows: .venv\Scripts\activate | |
| # Install dependencies | |
| pip install -r requirements.txt | |
| ``` | |
| #### Step 2: Configure Environment Variables | |
| Create a `.env` file from the example: | |
| ```bash | |
| cp .env.example .env | |
| ``` | |
| Edit `.env` with your API keys: | |
| ```bash | |
| # ================================================ | |
| # REQUIRED: Modal Bird Classifier (GPU) | |
| # ================================================ | |
| MODAL_MCP_URL=https://your-modal-app--mcp-server.modal.run/mcp | |
| BIRD_CLASSIFIER_API_KEY=your-modal-api-key-here | |
| # ================================================ | |
| # REQUIRED: Nuthatch Species Database | |
| # ================================================ | |
| NUTHATCH_API_KEY=your-nuthatch-api-key-here | |
| NUTHATCH_BASE_URL=https://nuthatch.lastelm.software/v2 # Default, can omit | |
| # Nuthatch Transport Mode (STDIO or HTTP) | |
| NUTHATCH_USE_STDIO=true # Recommended for local development | |
| # Only needed if NUTHATCH_USE_STDIO=false: | |
| # NUTHATCH_MCP_URL=http://localhost:8001/mcp | |
| # NUTHATCH_MCP_AUTH_KEY=your-auth-key-here | |
| # ================================================ | |
| # LLM Provider (Choose ONE) | |
| # ================================================ | |
| # OpenAI (Recommended) | |
| OPENAI_API_KEY=sk-your-openai-key-here | |
| DEFAULT_OPENAI_MODEL=gpt-4o-mini | |
| OPENAI_TEMPERATURE=0.0 | |
| # OR Anthropic | |
| # ANTHROPIC_API_KEY=sk-ant-your-anthropic-key-here | |
| # DEFAULT_ANTHROPIC_MODEL=claude-sonnet-4-5-20250929 | |
| # ANTHROPIC_TEMPERATURE=0.0 | |
| # OR HuggingFace (Limited function calling support) | |
| # HF_API_KEY=hf_your-huggingface-token-here | |
| # DEFAULT_HF_MODEL=Qwen/Qwen2.5-Coder-32B-Instruct | |
| # HF_TEMPERATURE=0.1 | |
| ``` | |
| #### Step 3: Understanding Nuthatch Transport Modes | |
| **STDIO Mode (Recommended for Local):** | |
| - Nuthatch MCP server runs as subprocess | |
| - Automatically started by the app | |
| - No separate server process needed | |
| - Set `NUTHATCH_USE_STDIO=true` | |
| **HTTP Mode (Alternative for Local):** | |
| - Nuthatch MCP server runs as separate HTTP server | |
| - Useful for debugging or multiple clients | |
| - Requires running server in separate terminal | |
| To use HTTP mode: | |
| ```bash | |
| # Terminal 1: Run Nuthatch MCP server | |
| python nuthatch_tools.py --http --port 8001 | |
| # Terminal 2: Run the app | |
| # Set in .env: | |
| # NUTHATCH_USE_STDIO=false | |
| # NUTHATCH_MCP_URL=http://localhost:8001/mcp | |
| python app.py | |
| ``` | |
| #### Step 4: Run the App | |
| ```bash | |
| # With STDIO mode (default, easiest): | |
| python app.py | |
| # Or using Gradio CLI: | |
| gradio app.py | |
| ``` | |
| App will be available at: `http://127.0.0.1:7860` | |
| --- | |
| ### ☁️ HuggingFace Spaces Deployment | |
| #### Step 1: Create a New Space | |
| 1. Go to [huggingface.co/new-space](https://huggingface.co/new-space) | |
| 2. Choose: | |
| - **SDK**: Gradio | |
| - **Hardware**: CPU Basic (free) or CPU Upgrade (faster) | |
| - **Visibility**: Public or Private | |
| #### Step 2: Upload Your Code | |
| **Option A: Using `upload_to_space.py` (Recommended)** | |
| ```bash | |
| # 1. Install HuggingFace CLI | |
| pip install huggingface_hub | |
| # 2. Login | |
| huggingface-cli login | |
| # 3. Update upload_to_space.py with your Space name | |
| # Edit line with repo_id: | |
| # repo_id="YOUR-USERNAME/YOUR-SPACE-NAME" | |
| # 4. Upload | |
| python upload_to_space.py | |
| ``` | |
| **Option B: Using Git** | |
| ```bash | |
| git remote add hf-space https://huggingface.co/spaces/YOUR-USERNAME/YOUR-SPACE-NAME | |
| git push hf-space main | |
| ``` | |
| #### Step 3: Configure Secrets in HuggingFace Spaces | |
| ⚠️ **CRITICAL**: Spaces use **Secrets**, not `.env` files! | |
| Go to your Space → **Settings** → **Variables and secrets** | |
| **Add these secrets:** | |
| ```bash | |
| # REQUIRED: Modal Bird Classifier | |
| MODAL_MCP_URL = https://your-modal-app--mcp-server.modal.run/mcp | |
| BIRD_CLASSIFIER_API_KEY = your-modal-api-key-here | |
| # REQUIRED: Nuthatch Species Database | |
| NUTHATCH_API_KEY = your-nuthatch-api-key-here | |
| NUTHATCH_BASE_URL = https://nuthatch.lastelm.software/v2 # Optional | |
| NUTHATCH_USE_STDIO = true # MUST be "true" for Spaces | |
| # OPTIONAL: Backend-provided LLM keys (users can provide their own) | |
| # Only add if you want to provide default keys: | |
| # OPENAI_API_KEY = sk-your-key-here | |
| # ANTHROPIC_API_KEY = sk-ant-your-key-here | |
| ``` | |
| **Important Notes:** | |
| - ✅ **ALWAYS** use `NUTHATCH_USE_STDIO=true` on Spaces (subprocess mode) | |
| - ✅ HTTP mode not supported on Spaces (port binding restrictions) | |
| - ✅ Users can provide their own LLM keys via the UI | |
| - ✅ Environment variables from Spaces **do not** auto-inherit to subprocesses | |
| - The app explicitly passes `NUTHATCH_API_KEY` and `NUTHATCH_BASE_URL` to the subprocess (see `mcp_clients.py`) | |
| #### Step 4: Verify Deployment | |
| 1. Wait for Space to build (2-5 minutes) | |
| 2. Check **Logs** tab for errors | |
| 3. Try the app - upload a bird photo or ask about species | |
| --- | |
| ## 📁 Project Structure | |
| ``` | |
| hackathon_draft/ | |
| ├── app.py # Main Gradio app | |
| ├── upload_to_space.py # HF Spaces upload script | |
| ├── requirements.txt # Python dependencies | |
| ├── .env.example # Environment template | |
| ├── langgraph_agent/ | |
| │ ├── __init__.py | |
| │ ├── agents.py # Agent factory (single/multi-agent) | |
| │ ├── config.py # Configuration loader | |
| │ ├── mcp_clients.py # MCP client setup | |
| │ ├── subagent_config.py # Agent mode definitions | |
| │ ├── prompts.py # System prompts | |
| │ └── structured_output.py # Response formatting | |
| ├── nuthatch_tools.py # Nuthatch MCP server | |
| └── agent_cache.py # Session-based agent caching | |
| ``` | |
| --- | |
| ## 🏗️ Architecture | |
| ### MCP Servers | |
| **1. Modal Bird Classifier (GPU)** | |
| - Hosted on Modal (serverless GPU) | |
| - ResNet50 trained on 555 bird species | |
| - Tools: `classify_from_url`, `classify_from_base64` | |
| - Transport: Streamable HTTP | |
| **2. Nuthatch Species Database** | |
| - Species reference API (1000+ birds) | |
| - Tools: `search_birds`, `get_bird_info`, `get_bird_images`, `get_bird_audio`, `search_by_family`, `filter_by_status`, `get_all_families` | |
| - Transport: **STDIO** (subprocess on Spaces), STDIO or HTTP (local) | |
| - Data sources: Unsplash (images), xeno-canto (audio) | |
| ### Agent Modes | |
| **Mode 1: Specialized Subagents (3 Specialists)** | |
| - **Router** orchestrates 3 specialized agents: | |
| 1. **Image Identifier**: classify images, show reference photos | |
| 2. **Species Explorer**: search by name, provide multimedia | |
| 3. **Taxonomy Specialist**: conservation status, family search | |
| - Each specialist has focused tool subset | |
| **Mode 2: Audio Finder Agent** | |
| - Single specialized agent for finding bird audio | |
| - Tools: `search_birds`, `get_bird_info`, `get_bird_audio` | |
| - Optimized workflow for xeno-canto recordings | |
| ### Tech Stack | |
| - **Frontend**: Gradio 6.0 with custom CSS (cloud/sky theme) | |
| - **Agent Framework**: LangGraph with streaming | |
| - **MCP Integration**: FastMCP client library | |
| - **LLM Support**: OpenAI, Anthropic, HuggingFace | |
| - **Session Management**: In-memory agent caching | |
| - **Output Parsing**: LlamaIndex Pydantic + regex (optimized) | |
| --- | |
| ## 🎨 Special Features | |
| ### Dual Streaming Output | |
| - **Chat Panel**: LLM responses with markdown rendering | |
| - **Tool Log Panel**: Real-time tool execution traces (inputs/outputs) | |
| ### Dynamic Examples | |
| - Examples change based on selected agent mode | |
| - Photo examples always visible | |
| - Text examples adapt to Audio Finder vs Multi-Agent | |
| ### Structured Output | |
| - Automatic image/audio URL extraction | |
| - Markdown formatting for media | |
| - xeno-canto audio links (browser-friendly) | |
| --- | |
| ## 📝 API Key Sources | |
| | Service | Get Key From | Purpose | | |
| |---------|-------------|---------| | |
| | **Modal** | [modal.com](https://modal.com) | GPU bird classifier | | |
| | **Nuthatch** | [nuthatch.lastelm.software](https://nuthatch.lastelm.software) | Species database | | |
| | **OpenAI** | [platform.openai.com/api-keys](https://platform.openai.com/api-keys) | LLM (recommended) | | |
| | **Anthropic** | [console.anthropic.com/settings/keys](https://console.anthropic.com/settings/keys) | LLM (Claude) | | |
| | **HuggingFace** | [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens) | LLM (limited support) | | |
| --- | |
| ## 🐛 Troubleshooting | |
| ### Space stuck on "Building" | |
| - Check **Logs** tab for errors | |
| - Verify all required secrets are set | |
| - Try Factory Reboot (Settings → Factory Reboot) | |
| ### "Invalid API key" errors | |
| - Ensure secrets are set correctly (no quotes needed) | |
| - Check secret names match exactly (case-sensitive) | |
| ### HuggingFace provider fails with "function calling not support" | |
| - HuggingFace Inference API has limited tool calling | |
| - Use OpenAI or Anthropic instead | |
| ### Nuthatch server not starting (local) | |
| - Check `NUTHATCH_API_KEY` is set in `.env` | |
| - Verify API key is valid | |
| - Try STDIO mode: `NUTHATCH_USE_STDIO=true` | |
| ### Audio links broken | |
| - Check AUDIO_FINDER_PROMPT is working | |
| - Verify xeno-canto URLs include `/download` | |
| - Check structured output parsing logs | |
| --- | |
| ## 📚 Documentation | |
| For detailed implementation docs, see: | |
| - `project_docs/implementation/phase_5_final.md` - Complete agent architecture | |
| - `project_docs/commands_guide/git_spaces_cheatsheet.md` - Deployment guide | |
| --- | |
| ## 🏆 Credits | |
| - **Bird Species Data**: [Nuthatch API](https://nuthatch.lastelm.software) by Last Elm Software | |
| - **Bird Audio**: [xeno-canto.org](https://xeno-canto.org) - Community bird recordings | |
| - **Reference Images**: [Unsplash](https://unsplash.com) + curated collections | |
| - **MCP Protocol**: [Anthropic Model Context Protocol](https://github.com/anthropics/mcp) | |
| - **Hackathon**: [HuggingFace MCP-1st-Birthday](https://huggingface.co/MCP-1st-Birthday) | |
| --- | |
| ## 📄 License | |
| MIT License - Built for educational and research purposes | |