MCP Server Testing

When your AI agent depends on external MCP servers you do not control, those servers can change their tool definitions at any time — rename parameters, remove tools, add required fields. Your agent tests pass today and fail tomorrow. EvalView solves this with MCP contract testing.

The problem with external MCP servers

MCP servers introduce a clean tool contract, but that does not remove regression risk. When a server you depend on changes its interface, your agent breaks silently. Your tests still pass because your code did not change — but the server did.

MCP contract testing

EvalView captures a snapshot of a server's tool definitions and diffs against it on every CI run. If the interface changed, you know immediately — before running your full test suite.

# 1. Snapshot a server's tool contract
evalview mcp snapshot "npx:@modelcontextprotocol/server-github" --name server-github

# 2. Check for interface drift
evalview mcp check server-github

# 3. Use in CI to abort before testing against broken interfaces
evalview run tests/ --contracts --fail-on "REGRESSION,CONTRACT_DRIFT"

What contract drift looks like

CONTRACT_DRIFT - 2 breaking change(s)
  REMOVED: create_pull_request - tool no longer available
  CHANGED: list_issues - new required parameter 'owner'

Supported transport types

• stdio — npx:@modelcontextprotocol/server-filesystem /tmp
• HTTP — http://localhost:8080
• Command — stdio:python my_server.py

Agent behavior testing for MCP

Beyond contract testing, EvalView also tests the agent that uses MCP servers:

• wrong tool selection — did the agent pick the right MCP tool?
• tool argument drift — did the agent pass the correct parameters?
• forbidden tool usage — did the agent use a dangerous tool in a sensitive scenario?
• clarification turns — did the agent ask for missing info before calling a tool?
• behavior changes after server updates — did a server update break your agent flow?

CLI reference

evalview mcp snapshot <endpoint> --name <name>   # Capture contract
evalview mcp check <name>                         # Check for drift
evalview mcp list                                  # List contracts
evalview mcp show <name>                          # Show details
evalview mcp delete <name>                        # Remove contract

CI/CD Integration Guide | Tool-Calling Agent Testing | Back to EvalView homepage