When server_combinations are missing from config.yaml, the models settings are ignored, defaulting to openai and gpt40

[ct-parker]

With the following metacoder configuration, the models parameters are completely ignored. This is caused by omitting the server_combinations parameter. This is unexpected behavior.

Secondly, as with Issue #18 if the OpenAI calls fail for any reason, the code gets caught in a try/fail/repeat loop. This should be documented as a separate bug. It likely has a similar solution as in the cited issue.

Expected behavior could be one (or more) of the following:

1. Produce an error message like "No server_combinations configuration specified in metacoder input file "tests/input/literature_mcp_eval_config.yaml". This is a required setting. Please read the documentation for how to create this file."
2. Produce the above error message and also include a script that can independently validate the YAML file and report any errors.
3. Use all provided information to establish reasonable default parameters (e.g., the models configuration) instead of assuming openai and gpt40.

name: pubmed tools evals
description: |
  Evaluations for multiple pubmed MCPs

coders:
  goose: {}

models:
  claude-sonnet:
    provider: anthropic
    name: claude-sonnet-4-20250514

servers:
  mcp-simple-pubmed:
    name: pubmed
    command: uvx
    args: [mcp-simple-pubmed]
    env:
      PUBMED_EMAIL: ctparker@lbl.gov

# When server_combinations are missing, the models parameter is ignored, defaulting to openai + gpt40
#server_combinations:
#  - [mcp-simple-pubmed]

cases:
- name: PMID_28027860_Full_Text
  metrics: [CorrectnessMetric]
  input: "What is the first sentence of section 2 in PMID: 28027860?"
  expected_output: |
    Even though many of NFLE's core features have been clarified in the last two decades, some critical issues remain controversial."
  threshold: 0.9

The above metacoder configuration results in the following config.yaml in eval_workdir/claude-sonnet_goose_PMID_28027860_Full_Text_no_servers/claude-sonnet_goose_PMID_28027860_Full_Text/.config/goose/:

GOOSE_MODEL: gpt-4o
GOOSE_PROVIDER: openai
extensions:
  developer:
    bundled: true
    display_name: Developer
    enabled: true
    name: developer
    timeout: 300
    type: builtin

If Option 3 (see above) were implemented, this would look like:

GOOSE_MODEL: claude-sonnet-4-20250514
GOOSE_PROVIDER: anthropic
extensions:
  developer:
    bundled: true
    display_name: Developer
    enabled: true
    name: developer
    timeout: 300
    type: builtin

[github-actions]

Summary: Configuration Defaults Issue - Missing server_combinations Silently Ignores Model Settings

This issue reports a critical configuration handling bug where missing server_combinations configuration causes the specified models settings to be completely ignored, resulting in unexpected fallback to OpenAI/GPT-4o defaults.

🔍 Core Problem Analysis

Root Cause: When server_combinations is omitted from the YAML config, the evaluation logic in src/metacoder/evals/runner.py:307-315 correctly generates default server combinations, but there appears to be a disconnect between:

1. The model configuration parsing (which reads the user's claude-sonnet model config)
2. The actual model selection used by the coder (which defaults to gpt-4o + openai)

Expected vs Actual Behavior:

• User specifies: claude-sonnet-4-20250514 with anthropic provider
• System generates: gpt-4o with openai provider when server_combinations is missing

📊 Related Issues Context

This is part of a broader pattern of configuration and error handling issues:

• Issue Model eval gets caught in a loop if the endpoint returns HTTP errors. #18: HTTP error infinite loops (same evaluation context, OpenAI API failures)
• Issue Divide by zero error in metacoder eval summary. #15: Division by zero when no valid coder combinations are available
• Issue Coders should not rely on default encoding from operating system. #14: Windows encoding issues with UTF-8 content

All suggest the need for more robust configuration validation and error handling.

🎯 Proposed Solutions (from issue description)

The issue author suggests three excellent approaches:

1. Better Error Messages: Explicit validation with helpful error messages pointing to documentation
2. Configuration Validation Script: Independent YAML validator for pre-flight checks
3. Smart Defaults: Use all provided configuration data instead of hardcoded fallbacks ⭐ (Recommended)

🔧 Technical Context

Configuration Structure (from eval_model.py:98-101):

server_combinations: List[List[str]] | None = Field(
    None,
    description="Server combinations to evaluate (None = test each individually)",
)

Default Logic (from runner.py:307-315):

• When server_combinations is None, system correctly generates [[]] (no servers) + individual server combinations
• However, model selection logic appears to bypass user-specified models

🧪 Test Case Details

The issue involves evaluating PMID:28027860 (research on Nocturnal Frontal Lobe Epilepsy) using mcp-simple-pubmed for literature queries. This represents a realistic scientific literature evaluation workflow.

🏷️ Recommended Actions

1. Immediate: Investigate model selection logic in coder initialization
2. Short-term: Implement configuration validation as suggested in option 1
3. Long-term: Add comprehensive YAML validation tooling (option 2)

This issue affects the reliability of metacoder evaluations and could lead to incorrect performance assessments when users believe they're testing one model but actually testing another.

---

Status: Requires investigation of model configuration flow between evaluation dataset and coder initialization. Related to issues #15, #18 regarding configuration robustness.