Welcome to FlowZap, the App to diagram with Speed, Clarity and Control.

MCP Architecture Patterns for Production-Grade Agents

2/28/2026

Tags: MCP, AI Agents, Architecture Patterns, Production, System Design, Scalability, Security

Jules Kovac

Jules Kovac

Business Analyst, Founder

MCP Architecture Patterns for Production-Grade Agents

 

The Production Reality of MCP

If you're shipping agents on MCP in production, the day‑2 pains may make you feel like you're losing control: token bills are spiking, remote servers are flaking out, and security is asking how to lock this thing down. Typical pain points include:

  • Your MCP server works in dev and silently dies in prod.
  • Your token invoice looks like a down payment on a house.
  • Your "simple" agent setup turned into a distributed system with 14 failure modes.

That's the real story of the Model Context Protocol: it's not some abstract spec, it's the plumbing between your agents and the messy, real-world tools and data they need to touch.

 

Six Patterns for Production-Grade Agents

This article maps those pains to six MCP architecture patterns you can actually use to ship and scale agents without losing control. These six patterns are a practical field guide, not an official spec — each one maps to a well-documented, real-world engineering pattern with production implementations to back it up.

 

1. Direct Connect – "Ship It Tonight"

Tagline: You, one agent, one MCP server, no drama.

Direct Connect is the "monolith of MCP" – your host app talks straight to the MCP server over stdio or HTTP, no extra hops. It's perfect when you just want to see something work and don't care (yet) about governance slides.

Best when:

  • You're building an MVP or hackathon demo.
  • Single team, single trust boundary, everything runs in "your" infra.
  • You want the lowest possible latency and easiest debugging.

Avoid when:

  • You're exposing tools across teams or tenants.
  • Security wants audit logs, access policies, and someone says "SOX."

Dimension snapshot: Security (⭐☆☆☆), Scalability (⭐☆☆☆), Cost efficiency (⭐⭐⭐☆), Debuggability (⭐⭐⭐⭐)

 

See what it looks like in FlowZap:

Host { # Host Application
  n1: circle label="User sends prompt"
  n2: rectangle label="Agent builds JSON-RPC request"
  n3: rectangle label="Send request via stdio"
  n4: rectangle label="Receive JSON-RPC result"
  n5: rectangle label="Agent responds to user"

  n1.handle(right) -> n2.handle(left)
  n2.handle(right) -> n3.handle(left)
  n3.handle(bottom) -> MCPServer.n6.handle(top) [label="JSON-RPC request"]
  n4.handle(right) -> n5.handle(left)
}

MCPServer { # MCP Server
  n6: rectangle label="Parse incoming request"
  n7: rectangle label="Execute tool or resource"
  n8: rectangle label="Build JSON-RPC response"

  n6.handle(right) -> n7.handle(left)
  n7.handle(right) -> n8.handle(left)
  n8.handle(top) -> Host.n4.handle(bottom) [label="JSON-RPC response"]
}

 

2. Gateway Proxy – "Make Security Happy"

Tagline: Put a bouncer in front of your tools.

Gateway Proxy drops an API gateway between your agent and MCP servers to handle auth, rate limits, and auditing. Your agent still thinks it's calling tools normally; the gateway quietly enforces OAuth 2.0, SAML, SSO, tool-level rate limiting, and team-based quota enforcement before the request ever hits an MCP server. This is not theoretical — products like MintMCP Gateway, Gravitee MCP Proxy, Kong, and Azure APIM all implement this exact pattern.

The real-world case for this is stark: without gateway-level controls, a single agent stuck in a retry loop can exhaust API budgets in hours. Gateways with token-based quotas, burst allowances, and per-tool granularity are the standard prevention.

Best when:

  • Consistent auth (OAuth/JWT/API keys) needed across all tools.
  • Request logs required for compliance (SOC2, GDPR) or incident response.
  • Multiple teams or clients share the same MCP estate.

Avoid when:

  • Ultra-latency-sensitive and every millisecond matters — the gateway adds a network hop.
  • Not enough traffic to justify the added complexity.

Dimension snapshot: Security (⭐⭐⭐⭐), Scalability (⭐⭐⭐⭐), Cost efficiency (⭐⭐☆☆), Debuggability (⭐⭐⭐☆)

 

FlowZap Code

Host { # Host Application
  n1: circle label="User sends prompt"
  n2: rectangle label="Agent builds tool call"
  n3: rectangle label="Send request to gateway"
  n4: rectangle label="Receive gateway response"
  n5: rectangle label="Agent responds to user"

  n1.handle(right) -> n2.handle(left)
  n2.handle(right) -> n3.handle(left)
  n3.handle(bottom) -> Gateway.n6.handle(top) [label="Tool request"]
  n4.handle(right) -> n5.handle(left)
}

Gateway { # MCP Gateway
  n6: rectangle label="Receive and log request"
  n7: diamond label="Authorized?"
  n8: rectangle label="Forward to MCP server"
  n9: rectangle label="Receive MCP response"
  n10: rectangle label="Log response and return"

  n6.handle(right) -> n7.handle(left)
  n7.handle(right) -> n8.handle(left) [label="Yes"]
  n7.handle(top) -> Host.n4.handle(left) [label="401 Unauthorized"]
  n8.handle(bottom) -> MCPServer.n11.handle(top) [label="Forwarded request"]
  n9.handle(right) -> n10.handle(left)
  n10.handle(top) -> Host.n4.handle(bottom) [label="Authorized response"]
}

MCPServer { # MCP Server
  n11: rectangle label="Execute tool"
  n12: rectangle label="Return result"

  n11.handle(right) -> n12.handle(left)
  n12.handle(top) -> Gateway.n9.handle(bottom) [label="Tool result"]
}

 

3. Tool Router – "Stop Feeding the LLM a Phone Book"

Tagline: 50 tools, 1 agent, sane token usage.

The Tool Router pattern puts a routing brain in front of your tools so the LLM only "sees" the subset it actually needs. This is a documented, serious problem: complete tool schema definitions loaded into context can consume 40% of available tokens before the user even sends their first message. Writer.com solved this by building a semantic "search meta-tool" that uses vector embeddings and cosine similarity to match user intent to the right tools dynamically. Speakeasy achieved a 96% reduction in input tokens and 90% reduction in total token consumption using dynamic toolsets.

The Semantic MCP Router approach offers two discovery paths: a curated "top 20" default toolset pre-loaded into context, and a deep semantic search path for specialized tools. This dual-track model keeps the fast path fast and the long tail accessible without bloating every single prompt.

Best when:

  • Beyond five tools and prompts are bloating like crazy.
  • Different use cases need different tool slices (billing vs. analytics vs. ops).
  • Context size reduction is needed without dumbing down the agent.

Avoid when:

  • A tiny app with a couple of tools.
  • The team can't yet support routing logic, metrics, and fallbacks.

Dimension snapshot: Security (⭐⭐⭐☆), Scalability (⭐⭐⭐⭐), Cost efficiency (⭐⭐⭐⭐), Debuggability (⭐⭐☆☆)

 

FlowZap Code

Host { # Host Application
  n1: circle label="User sends prompt"
  n2: rectangle label="Agent extracts intent"
  n3: rectangle label="Send intent to router"
  n4: rectangle label="Receive routed result"
  n5: rectangle label="Agent responds to user"

  n1.handle(right) -> n2.handle(left)
  n2.handle(right) -> n3.handle(left)
  n3.handle(bottom) -> Router.n6.handle(top) [label="Intent + tool request"]
  n4.handle(right) -> n5.handle(left)
}

Router { # Tool Router
  n6: rectangle label="Receive intent"
  n7: rectangle label="Semantic match via embeddings"
  n8: diamond label="Which MCP server?"
  n9: rectangle label="Forward to Server A"
  n10: rectangle label="Forward to Server B"
  n11: rectangle label="Normalize and return result"

  n6.handle(right) -> n7.handle(left)
  n7.handle(right) -> n8.handle(left)
  n8.handle(bottom) -> n9.handle(top) [label="Route A"]
  n8.handle(right) -> n10.handle(left) [label="Route B"]
  n9.handle(bottom) -> ServerA.n12.handle(top) [label="Call Server A"]
  n10.handle(bottom) -> ServerB.n14.handle(top) [label="Call Server B"]
  n11.handle(top) -> Host.n4.handle(bottom) [label="Final result"]
}

ServerA { # MCP Server A
  n12: rectangle label="Execute tool A"
  n13: rectangle label="Return A result"

  n12.handle(right) -> n13.handle(left)
  n13.handle(top) -> Router.n11.handle(bottom) [label="Result A"]
}

ServerB { # MCP Server B
  n14: rectangle label="Execute tool B"
  n15: rectangle label="Return B result"

  n14.handle(right) -> n15.handle(left)
  n15.handle(top) -> Router.n11.handle(left) [label="Result B"]
}

 

4. Agent Mesh – "Squad of Agents, One Brain"

Tagline: Many specialists, shared context, controlled chaos.

Agent Mesh is what happens when you stop pretending one agent can do everything. Multiple agents communicate through a shared context broker backed by MCP, enabling coordinated tool access and state synchronization. Microsoft's Azure implementation uses persistent session state via Cosmos DB (with in-memory fallback), supporting dynamic pattern swapping and traceable multi-agent interactions.

The key architectural choice here is choreography vs. orchestration. In orchestrated setups, a Manager agent coordinates all interactions, maintains a task ledger, and can dynamically re-plan based on intermediate findings. In choreography, agents communicate peer-to-peer through structured JSON-RPC exchanges via MCP, with any agent able to request help from any other. Both approaches rely on shared memory so all agents access the same state store for consistent context.

The risk is real: agents can ping-pong tasks between each other indefinitely without proper termination conditions and observability.

Best when:

  • Distinct roles exist: planner, coder, reviewer, operator.
  • Shared state (tasks, resources, workflows) is needed instead of isolated silos.
  • Workloads naturally decompose into parallelizable subtasks.

Avoid when:

  • A single, well-tooled agent is enough.
  • Observability and tracing aren't in place yet (debugging will be painful).

Dimension snapshot: Security (⭐⭐⭐☆), Scalability (⭐⭐⭐⭐), Cost efficiency (⭐⭐☆☆), Debuggability (⭐⭐☆☆)

 

FlowZap Code

Orchestrator { # Orchestrator Agent
  n1: circle label="Complex task received"
  n2: rectangle label="Decompose into subtasks"
  n3: rectangle label="Assign subtask to Agent B"
  n4: rectangle label="Receive subtask result"
  n5: rectangle label="Request shared context"
  n6: rectangle label="Compile final response"

  n1.handle(right) -> n2.handle(left)
  n2.handle(right) -> n3.handle(left)
  n3.handle(bottom) -> Worker.n7.handle(top) [label="Subtask assignment"]
  n4.handle(right) -> n5.handle(left)
  n5.handle(bottom) -> Broker.n11.handle(top) [label="Context request"]
  n6.handle(left) -> n2.handle(bottom) [label="Next iteration"]
}

Worker { # Worker Agent
  n7: rectangle label="Receive subtask"
  n8: rectangle label="Fetch shared context"
  n9: rectangle label="Call MCP tool"
  n10: rectangle label="Return result to orchestrator"

  n7.handle(right) -> n8.handle(left)
  n8.handle(bottom) -> Broker.n11.handle(left) [label="Context request"]
  n8.handle(right) -> n9.handle(left)
  n9.handle(bottom) -> MCPServer.n13.handle(top) [label="MCP tool call"]
  n10.handle(top) -> Orchestrator.n4.handle(bottom) [label="Subtask result"]
}

Broker { # Context Broker
  n11: rectangle label="Resolve context request"
  n12: rectangle label="Return shared state"

  n11.handle(right) -> n12.handle(left)
  n12.handle(top) -> Orchestrator.n6.handle(bottom) [label="Context to orchestrator"]
  n12.handle(left) -> Worker.n9.handle(top) [label="Context to worker"]
}

MCPServer { # MCP Server
  n13: rectangle label="Execute tool"
  n14: rectangle label="Return tool output"

  n13.handle(right) -> n14.handle(left)
  n14.handle(top) -> Worker.n10.handle(bottom) [label="Tool output"]
}

 

5. Circuit Breaker – "No More Zombie Calls"

Tagline: If a tool is dying, stop hammering it.

Circuit Breaker wraps MCP calls with health-aware gates using three states: Closed (normal operation, requests pass through), Open (failures detected, requests fail fast), and Half-Open (testing if the service has recovered). This is classic distributed-systems hygiene applied directly to MCP tool calls.

Without circuit breakers, the failure cascade is predictable: Tool A fails → retries pile up → resources exhausted → other tools slow down → system overload → everything fails. With circuit breakers: Tool A fails → circuit opens → fast fail → other tools unaffected → system stable → recovery when ready.

This pattern has real MCP implementations. IBM's mcp-context-forge has a full feature request for circuit breakers with half-open state recovery, failure thresholds, and fast failure protection. The MCP Go SDK includes a production-ready error recovery example implementing circuit breakers alongside retry with exponential backoff and bulkhead isolation. Octopus.com documented a complete Langchain + Python implementation using the pybreaker library for MCP tool calls.

Best when:

  • Relying on flaky third-party APIs or legacy databases.
  • Agents have been observed freezing because a single MCP server went unresponsive.
  • Graceful degradation is preferred over all-or-nothing behavior.

Avoid when:

  • Everything is local, fast, and rock-solid (e.g., stdio to a local process).
  • No plan exists for what to do on "fast fail" (fallback tools, user messaging, etc.).

Dimension snapshot: Security (⭐⭐⭐☆), Scalability (⭐⭐⭐⭐), Cost efficiency (⭐⭐⭐☆), Debuggability (⭐⭐⭐⭐)

 

FlowZap Code

Host { # Host Application
  n1: circle label="User sends prompt"
  n2: rectangle label="Agent prepares MCP call"
  n3: rectangle label="Pass call to circuit breaker"
  n4: rectangle label="Receive result or error"
  n5: rectangle label="Agent responds to user"

  n1.handle(right) -> n2.handle(left)
  n2.handle(right) -> n3.handle(left)
  n3.handle(bottom) -> CB.n6.handle(top) [label="MCP tool call"]
  n4.handle(right) -> n5.handle(left)
}

CB { # Circuit Breaker
  n6: rectangle label="Check circuit state"
  n7: diamond label="Circuit open?"
  n8: rectangle label="Forward to MCP server"
  n9: rectangle label="Fast-fail with error"
  n10: diamond label="Call succeeded?"
  n11: rectangle label="Record success"
  n12: rectangle label="Record failure and check threshold"

  n6.handle(right) -> n7.handle(left)
  n7.handle(right) -> n8.handle(left) [label="Closed"]
  n7.handle(bottom) -> n9.handle(top) [label="Open"]
  n8.handle(bottom) -> MCPServer.n13.handle(top) [label="Forward request"]
  n9.handle(top) -> Host.n4.handle(bottom) [label="CircuitOpenError"]
  n10.handle(right) -> n11.handle(left) [label="Yes"]
  n10.handle(bottom) -> n12.handle(top) [label="No"]
  n11.handle(top) -> Host.n4.handle(left) [label="Return result"]
  n12.handle(top) -> Host.n4.handle(right) [label="Return error"]
}

MCPServer { # MCP Server
  n13: rectangle label="Attempt tool execution"
  n14: rectangle label="Return result or error"

  n13.handle(right) -> n14.handle(left)
  n14.handle(top) -> CB.n10.handle(bottom) [label="Execution outcome"]
}

 

6. Context Proxy – "Cut Your LLM Bill in Half"

Tagline: Cache the boring stuff, pay for the smart stuff.

Context Proxy is a caching and compression layer that sits between the agent and MCP servers, intercepting redundant context requests before they hit the wire. This treats context like an actual managed resource with TTLs, invalidation hooks, and hit-rate monitoring — not as a magic infinite stream of tokens.

The evidence for this pattern is strong. The Token Optimizer MCP server combines Brotli compression with persistent SQLite-based caching to achieve up to 95%+ token reduction. The mcp-context-proxy project on GitHub acts as a transparent MCP proxy that compresses large tool responses using an external LLM before passing them to resource-constrained local models. Effective strategies include prompt-level caching (reuse complete prompt-response pairs), partial context caching (reuse static system prompts), and semantic caching (match near-duplicate requests via embeddings).

Cache invalidation is the hard part. Time-based expiration works for slowly changing data, event-based invalidation handles data updates, and hybrid approaches balance freshness with efficiency. Define staleness tolerance based on actual application requirements.

Best when:

  • Agents keep asking for the same docs, schemas, or repo slices.
  • Retrieval operates over relatively static data (knowledge bases, specs).
  • The invoice screams "context bloat" more than "model size".

Avoid when:

  • Data is real-time and staleness is dangerous (trading, critical ops).
  • No clear strategy for invalidation and freshness.

Dimension snapshot: Security (⭐⭐⭐☆), Scalability (⭐⭐⭐⭐), Cost efficiency (⭐⭐⭐⭐), Debuggability (⭐⭐⭐☆)

 

FlowZap Code

Host { # Host Application
  n1: circle label="User sends prompt"
  n2: rectangle label="Agent requests context"
  n3: rectangle label="Receive context"
  n4: rectangle label="Agent responds to user"

  n1.handle(right) -> n2.handle(left)
  n2.handle(bottom) -> Proxy.n5.handle(top) [label="Context request"]
  n3.handle(right) -> n4.handle(left)
}

Proxy { # Context Proxy
  n5: rectangle label="Receive context request"
  n6: rectangle label="Check cache with TTL"
  n7: diamond label="Cache hit?"
  n8: rectangle label="Return cached context"
  n9: rectangle label="Fetch fresh from MCP server"
  n10: rectangle label="Compress and cache response"

  n5.handle(right) -> n6.handle(left)
  n6.handle(right) -> n7.handle(left)
  n7.handle(right) -> n8.handle(left) [label="Hit"]
  n7.handle(bottom) -> n9.handle(top) [label="Miss"]
  n8.handle(top) -> Host.n3.handle(bottom) [label="Cached context"]
  n9.handle(bottom) -> MCPServer.n11.handle(top) [label="Fetch request"]
  n10.handle(top) -> Host.n3.handle(left) [label="Fresh context"]
}

MCPServer { # MCP Server
  n11: rectangle label="Fetch full context"
  n12: rectangle label="Return fresh data"

  n11.handle(right) -> n12.handle(left)
  n12.handle(top) -> Proxy.n10.handle(bottom) [label="Fresh data"]
}

 

How to Actually Use These Patterns

If you're wondering "which one do I pick?", use this ladder:

  1. Start with Direct Connect to get something working.
  2. Add Gateway Proxy once real users and security arrive.
  3. Introduce Tool Router when you hit >5 tools and token pain.
  4. Layer in Circuit Breaker as soon as anything remote can fail (it will).
  5. Reach for Context Proxy the first time finance slacks you about LLM costs.
  6. Only then consider Agent Mesh if a single agent truly can't keep up.

 

Inspirations

Back to all Blog articles