docs/dev/agents_config-README.md

Agent Configuration Guide

Complete guide for adding, removing, and modifying agents/subagents in BirdScope AI

---

📋 Table of Contents

1. Architecture Overview
2. Key Files Reference
3. Adding a New Subagent
4. Removing a Subagent
5. Modifying Existing Subagents
6. App.py Integration Points
7. Testing Your Changes

---

Architecture Overview

BirdScope AI uses a LangGraph supervisor pattern with specialized subagents:

User Request
    ↓
Supervisor (Router)
    ↓
┌───────────────┬──────────────────┬─────────────────┐
│ image_identifier │ taxonomy_specialist │ (other agents) │
└───────────────┴──────────────────┴─────────────────┘

Key Concepts:

• Supervisor: LLM-based router that delegates tasks to specialists
• Subagents: Specialized agents with filtered tool access and focused prompts
• Modes: Different agent configurations (e.g., "Supervisor (Multi-Agent)")
• Tool Filtering: Each subagent only has access to relevant tools

---

Key Files Reference

Core Agent Files

File	Purpose	What to Change
langgraph_agent/subagent_supervisor.py	Creates supervisor workflow	Add/remove agents from supervisor list
langgraph_agent/subagent_config.py	Defines subagent configurations	Add/remove/modify subagent definitions and modes
langgraph_agent/subagent_factory.py	Builds subagent instances	(Usually no changes needed)
langgraph_agent/prompts.py	System prompts for agents	Add provider-specific prompts

UI Integration

File	Purpose	What to Change
app.py	Gradio UI and agent orchestration	Update mode dropdown, default values, examples

---

Adding a New Subagent

Step 1: Define Subagent Configuration

File: langgraph_agent/subagent_config.py

Add your subagent to get_subagent_definitions():

@staticmethod
def get_subagent_definitions(provider: str = "openai") -> Dict[str, Dict]:
    return {
        # ... existing agents ...

        "my_new_agent": {
            "name": "My New Specialist",
            "description": "Expert at specific bird-related tasks",
            "tools": [
                "tool_name_1",
                "tool_name_2",
                "tool_name_3"
            ],
            "prompt": get_prompt("my_new_agent", provider) or """You are a My New Specialist.

**Your Role:**
1. Primary responsibility
2. Secondary responsibility
3. When to use specific tools

**Response Style:**
- How to format responses
- What to emphasize

**When to defer:**
- Task type 1 -> other_agent_name
- Task type 2 -> another_agent_name
""",
            "temperature": AgentConfig.OPENAI_TEMPERATURE,
        }
    }

Step 2: Create System Prompts

File: langgraph_agent/prompts.py

Add prompts for your new agent:

# Default prompt (used by OpenAI/Anthropic)
MY_NEW_AGENT_PROMPT = """Detailed prompt for your agent..."""

# HuggingFace-optimized prompt (more explicit, step-by-step)
MY_NEW_AGENT_PROMPT_HF = """Simplified, step-by-step prompt..."""

# Add to PROMPTS dictionary
PROMPTS = {
    # ... existing prompts ...
    "my_new_agent": {
        "default": MY_NEW_AGENT_PROMPT,
        "huggingface": MY_NEW_AGENT_PROMPT_HF,
    },
}

Step 3: Add to Supervisor Workflow

File: langgraph_agent/subagent_supervisor.py

async def create_supervisor_workflow(all_tools, llm, provider="openai"):
    # Create existing agents
    image_agent = await SubAgentFactory.create_subagent(
        "image_identifier", all_tools, llm, provider=provider
    )
    # ... other agents ...

    # Add your new agent
    my_new_agent = await SubAgentFactory.create_subagent(
        "my_new_agent", all_tools, llm, provider=provider
    )

    # Add to supervisor list
    workflow = create_supervisor(
        [image_agent, taxonomy_agent, my_new_agent],  # Add here
        model=llm,
        prompt=SubAgentConfig.get_router_prompt(provider=provider)
    )

Step 4: Update Router Prompts

File: langgraph_agent/subagent_config.py

Update get_router_prompt():

return """You are BirdScope AI Supervisor...

**Your Team:**
- **image_identifier**: Identifies birds from photos...
- **taxonomy_specialist**: Conservation status, families...
- **my_new_agent**: Specific tasks for my new agent  # Add this

**Routing Guidelines:**
1. **Image uploads/URLs** → image_identifier
2. **Conservation queries** → taxonomy_specialist
3. **New task type** → my_new_agent  # Add this

Also update prompts.py for HuggingFace router:

ROUTER_PROMPT_HF = """...
**Specialists:**
- image_identifier: ...
- taxonomy_specialist: ...
- my_new_agent: New task handling  # Add this

**Routing Rules:**
...
6. "New task keyword" → my_new_agent  # Add this
"""

Step 5: Update Mode Definition

File: langgraph_agent/subagent_config.py

Update get_mode_definitions():

return {
    "Supervisor (Multi-Agent)": {  # Current mode name
        "description": "Router orchestrates 3 specialized agents",
        "subagents": ["image_identifier", "taxonomy_specialist", "my_new_agent"],  # Add here
        "use_router": True
    },
}

Step 6: Integrate with app.py

See App.py Integration Points below.

---

Removing a Subagent

Example: Removing species_explorer from the supervisor

Step 1: Remove from Supervisor Workflow

File: langgraph_agent/subagent_supervisor.py

async def create_supervisor_workflow(all_tools, llm, provider="openai"):
    # Remove agent creation
    # species_agent = await SubAgentFactory.create_subagent(...)  # DELETE

    # Remove from supervisor list
    workflow = create_supervisor(
        [image_agent, taxonomy_agent],  # Remove species_agent
        model=llm,
        prompt=SubAgentConfig.get_router_prompt(provider=provider)
    )

Step 2: Update Mode Definition

File: langgraph_agent/subagent_config.py

return {
    "Supervisor (Multi-Agent)": {  # Update count if needed
        "description": "Router orchestrates 2 specialized agents",
        "subagents": ["image_identifier", "taxonomy_specialist"],  # Remove agent
        "use_router": True
    },
}

Step 3: Update Router Prompts

File: langgraph_agent/subagent_config.py (default router)

return """You are BirdScope AI Supervisor...

**Your Team:**
- **image_identifier**: ...
- **taxonomy_specialist**: ...
# Remove species_explorer reference

**Routing Guidelines:**
# Remove routing rules for deleted agent
# Reassign its responsibilities to other agents

File: langgraph_agent/prompts.py (HuggingFace router)

ROUTER_PROMPT_HF = """...
**Specialists:**
- image_identifier: ...
- taxonomy_specialist: ...
# Remove deleted agent

**Routing Rules:**
# Remove routing rules
# Reassign to remaining agents
"""

Step 4: Update "When to defer" Sections

File: langgraph_agent/subagent_config.py

Update remaining subagents' prompts:

"image_identifier": {
    # ...
    "prompt": """...
    **When to defer:**
    - For family/taxonomy queries -> taxonomy_specialist
    # Remove references to deleted agent
    """,
}

Step 5: Update app.py References

See App.py Integration Points below.

---

Modifying Existing Subagents

Changing Tool Access

File: langgraph_agent/subagent_config.py

"image_identifier": {
    "tools": [
        "classify_from_url",
        "classify_from_base64",
        "get_bird_info",
        "new_tool_name"  # Add new tool
    ],
}

Updating Prompts

File: langgraph_agent/subagent_config.py or langgraph_agent/prompts.py

# For inline prompts (in subagent_config.py)
"image_identifier": {
    "prompt": get_prompt("image_identifier", provider) or """Updated prompt..."""
}

# For dedicated prompts (in prompts.py)
IMAGE_IDENTIFIER_PROMPT = """Updated comprehensive prompt..."""

Changing Temperature

File: langgraph_agent/subagent_config.py

"species_explorer": {
    "temperature": 0.2,  # More creative (was 0.1)
}

---

App.py Integration Points

When you change agent modes, you MUST update these sections in app.py:

1. Mode Dropdown Choices

Location: app.py ~line 1486-1491

agent_mode = gr.Dropdown(
    choices=[
        "Supervisor (Multi-Agent)"  # Update mode name here
    ],
    value="Supervisor (Multi-Agent)",  # Update default here
    show_label=False,
    container=False
)

2. Initial Session Status HTML

Location: app.py ~line 1560

session_status = gr.HTML(
    value=create_config_html(
        provider_choice="OpenAI",
        agent_mode_choice="Supervisor (Multi-Agent)",  # Update here
        hf_key_input="",
        openai_key_input="",
        anthropic_key_input=""
    )
)

3. Health Check Config HTML

Location: app.py ~line 1654

config_html = create_config_html(
    provider_choice=provider_str,
    agent_mode_choice="Supervisor (Multi-Agent)",  # Update here
    hf_key_input=hf_key_value,
    openai_key_input=openai_key_input,
    anthropic_key_input=anthropic_key_input
)

4. Example Loading Logic Comments

Location: app.py ~line 1033

else:  # Supervisor (Multi-Agent)  # Update comment
    samples = [[text] for text in MULTI_AGENT_TEXT_EXAMPLES]

5. (Optional) Add Mode-Specific Examples

Location: app.py ~line 30-40 (add new example list)

# Text-only examples for Specialized Subagents mode
MULTI_AGENT_TEXT_EXAMPLES = [
    "Tell me about Northern Cardinals - show me images and audio",
    "What birds are in the Cardinalidae family?",
    "Show me species with endangered status",
    "Find me audio recordings for Snow Goose",
    "Get me bird call samples for any two species"
]

# Add examples for your new agent mode
MY_NEW_AGENT_EXAMPLES = [
    "Example query 1 for new mode",
    "Example query 2 for new mode",
    "Example query 3 for new mode"
]

Location: app.py ~line 1027-1041 (update conditional logic)

The function includes a placeholder for future modes. Uncomment and customize:

def update_text_examples_for_mode(mode):
    """Return appropriate text example dataset based on agent mode."""
    print(f"[DEBUG] Updating text examples for mode: {mode}")

    # Placeholder for future mode-specific examples
    if mode == "My New Agent Mode":  # UNCOMMENT and update mode name
        samples = [[text] for text in MY_NEW_AGENT_EXAMPLES]
        print(f"[DEBUG] New mode text samples: {len(samples)} examples")
    # elif mode == "Another Mode":  # Add more modes as needed
    #     samples = [[text] for text in ANOTHER_MODE_EXAMPLES]
    else:  # Default: Specialized Subagents

    # Default: Supervisor (Multi-Agent)
    samples = [[text] for text in MULTI_AGENT_TEXT_EXAMPLES]
    print(f"[DEBUG] Multi-agent text samples: {len(samples)} examples")

    return gr.Dataset(samples=samples)

Why keep the conditional? Even with only one mode, we maintain the placeholder structure to make it easy to add new modes later without refactoring the entire function.

---

Testing Your Changes

1. Local Testing

# Run the app locally
python app.py
# or
gradio app.py

2. Check for Errors

Common errors to watch for:

Unknown mode: Old Mode Name. Available: ['Supervisor (Multi-Agent)']

→ Fix: Update app.py mode references

ValueError: Unknown subagent: species_explorer

→ Fix: Remove references to deleted subagent in supervisor or mode definitions

3. Test Agent Routing

Try queries that should route to different agents:

# Test image_identifier routing
"What bird is this? [upload image]"

# Test taxonomy_specialist routing
"Show me endangered bird families"

# Test your new agent
"Query specific to new agent capability"

4. Check Tool Access

Verify agents only use their assigned tools:

# In terminal, watch for:
[SUBAGENT]: Creating Image Identification Specialist
  • Tools: classify_from_url, classify_from_base64, get_bird_info, get_bird_images

5. Verify Provider-Specific Prompts

Test with different LLM providers:

# OpenAI should use default prompts
# HuggingFace should use _HF prompts

---

Quick Reference Checklist

Adding a new subagent:

• Define in subagent_config.py → get_subagent_definitions()
• Create prompts in prompts.py (default + HF versions)
• Add to PROMPTS dictionary
• Create agent in subagent_supervisor.py
• Add to supervisor list
• Update router prompts (default + HF)
• Update mode definition
• Update app.py mode references (5 locations)
• Test locally

Removing a subagent:

• Remove from subagent_supervisor.py workflow
• Update mode definition count and list
• Update router prompts (remove references)
• Update "When to defer" in remaining agents
• Update app.py mode references (5 locations)
• Test locally

Modifying a subagent:

• Update tools list in subagent_config.py
• Update prompts if needed
• Update router if responsibilities changed
• Test locally

---

Troubleshooting

Error: "Unknown mode"

Cause: Mode name mismatch between subagent_config.py and app.py

Fix: Search for all occurrences in app.py and update:

grep -n "Old Mode Name" app.py

Error: "Unknown subagent"

Cause: Subagent referenced in supervisor but not defined in config

Fix: Either define the subagent or remove references

Agent Not Using Expected Tools

Cause: Tool name mismatch or tool not available

Fix: Check MCP server is providing the tool:

print([tool.name for tool in all_tools])

---

Best Practices

1. Always update both default and HuggingFace prompts - HF models need more explicit instructions
2. Keep tool lists minimal - Only give agents tools they truly need
3. Update router prompts - Supervisor needs to know when to use your agent
4. Test routing logic - Verify supervisor correctly delegates tasks
5. Document agent responsibilities - Clear "Your Role" section in prompts
6. Use provider-specific prompts - Optimize for OpenAI vs Anthropic vs HuggingFace
7. Keep "When to defer" up to date - Agents should know their boundaries

---

Example: Recent Changes

Evolution of the supervisor mode:

1. Removed species_explorer - Streamlined from 3 to 2 specialists
2. Added generalist (audio finder) - Back to 3 specialists with audio capabilities
3. Renamed mode - "Specialized Subagents (3 Specialists)" → "Supervisor (Multi-Agent)"

Files changed:

1. subagent_supervisor.py - Added/removed agents from supervisor list
2. subagent_config.py - Updated mode definition and router prompts
3. prompts.py - Updated HuggingFace router prompt, removed pirate personality
4. subagent_config.py - Updated "When to defer" sections
5. app.py - Updated all 5 mode references throughout

Current architecture: Single unified "Supervisor (Multi-Agent)" mode with 3 specialists (image_identifier, taxonomy_specialist, generalist)

---

Questions? Check the LangGraph documentation: https://langchain-ai.github.io/langgraph/