MCP servers in production

Introduction

The first MCP server I wrote took an afternoon: one file, a handful of functions, a client configuration pointing at a local path, and a model that could suddenly reach something real. As a demo it convinced, because the argument that a language model can call a well-described function is not a hard one to win any more.

Then I tried to hand it to somebody else.

Nothing about the protocol changed; the questions did. Who is the caller, and on whose behalf is it acting? What is it allowed to do, and how do I change that answer without a redeploy? What happened last Tuesday at 14:06, when somebody says the assistant did the wrong thing? Who gets woken up when it stops responding? A demo answers whether a model can call a function; a service answers all of that, and the protocol is the easy third of the work.

What follows is what I build into an MCP server before it has users, in the order I would do it, plus the parts I got wrong.

One framing runs through it. An MCP server is not a new kind of program. It is a small service with an unusual consumer: a program that decides what to call by reading prose and will do the wrong thing at machine speed if the interface lets it. Everything below follows from taking that consumer seriously without treating it as a person. “Host” means whatever the person is using; “server” means my process, the one with the tools in it.

The shape of the protocol

Host, client, server

The host owns the conversation: model, interface, consent prompt, transcript. The client is the connection inside the host, one per configured server, and the place where identity is attached. The server exposes capabilities and does the work.

The person is not a participant in the protocol. The server never sees the user, only what the host asserts about the session, which is why authentication does not settle authorisation: a valid token proves the host may talk to you, not that this call is one the person would approve. Design as if the caller is a competent, well-meaning program that may have been talked into something by content it read earlier in the session, and most of the interesting failure modes stop being interesting.

Tools, resources and prompts

The three capability types differ in who decides to use them. Tools are model-controlled: the model picks one and supplies arguments. Resources are application-controlled: the host decides what to fetch and when to attach it. Prompts are user-selected templates, and the model gets no vote.

Read that as a trust classification rather than a naming convention. Only tools have side effects, so only tools need the full machinery: scopes, per-call authorisation, idempotency, and an audit line with a decision in it. Resources still need care, because fetching one is itself a disclosure, and an action filed under the wrong kind makes the protocol’s classification disagree with the policy’s.

Capability negotiation and protocol versioning

A session opens with a handshake. The host sends an initialize request naming the protocol version it wants; the server replies with the version it will use, its identity and its capabilities. Both sides declare optional features: roots, sampling or elicitation on the host side, subscribed resources or tool-list notifications on the server side.

Treat the negotiated result as configuration. Log it, attach it to every call in the audit record, and fail the connection loudly when the versions cannot be reconciled rather than degrading into a request that half-works. A server that quietly falls back to older behaviour fails randomly six weeks later, when a host is upgraded and the advertised tool list shifts underneath it.

The server stays deliberately boring

The server is a thin adapter, and the thinness is a decision. It validates, authorises, translates a call into the backend’s own API, shapes the result and stops: no business rules, no second copy of customer data, no endpoint the backend does not already have.

Logic that lives only in the server is logic only one consumer exercises, which is how one rule ends up with two implementations and one of them wrong. Everything the server touches inherits its obligations: an on-call path, a patching cadence, a data-protection review. And the boundary between server and backend is expensive to move once hosts are configured and tool descriptions are living inside prompts, which is the shape of the work we do on the architecture and technical advisory side: not the diagram, the cost of being wrong about it.

Where the boundary sits is a blast-radius question, with the same arithmetic as any other deployable, because a server is a process with a pipeline, a dashboard and a rotation slot. One server per domain beats one per action, for the reasons in Micro, Mono, Modu: draw the boundary where ownership already is, or the architecture describes how teams wish they talked.

Transport is a deployment decision

stdio: one process per user, owned by the host

With stdio the host launches the server as a child process and speaks JSON-RPC over standard input and output. The lifetime is the session, there is no listener and no network surface, and the credential can be handed to the process at startup. It is single-user by construction: the process runs with the rights of whoever launched it.

Two failures are tedious to debug. Any library that writes to standard output corrupts the protocol stream, so stdout is the transport, everything human-readable goes to stderr, and a test asserts it. The other is the orphaned child: a server holding a database connection or a file lock can outlive the session that owned it, so exit when the parent goes and make backends treat that connection as disposable.

Streamable HTTP: one service, many callers

With HTTP the server is a long-lived service behind an endpoint: a POST carries a request, an optional stream carries server-initiated messages, and a session identifier ties a sequence of requests together.

Credentials belong to the caller, not the process. The host obtains a token for the person, presents it as a bearer credential, and the server validates it and exchanges it for a narrower credential for the backend. The moment a second person connects, a process-wide credential makes the server a confused deputy.

State is the second question. Per-session state in memory means reconnects need affinity, and a restart silently drops sessions that hosts still believe are alive. Either accept sticky routing and accept that every deploy interrupts every session, or keep session state in a store the server can re-read.

Third, token lifetime. The server advertises how a caller should be authorised, the host does the dance with the authorisation server, and the token that arrives is short-lived, audience-bound, and exchanged for something that can do only what the tool needs.

The trade, and where authentication sits

The table is a decision aid, not a rule. I have wanted per-tool scopes on a local server whose tools could touch production data, and one shared read-only credential where every caller was entitled to the same read; the transport says which questions to ask, and what the tools can reach decides the answers.

Authentication sits at the boundary and nowhere else. Credentials never appear in tool arguments, resource contents, error messages or logs, because a tool that accepts a token as a parameter is a token the model can see, echo, and be persuaded to pass somewhere else.

Trust boundaries and authorisation

Every tool call is a privileged action

The model is not a user. It is a program that produces text which the host turns into calls, and that text can be influenced by anything the session has read. The question is never whether the person asked for this, but whether this call, right now, is inside what the session’s identity is entitled to do. So authorise at the server, per call, and not once per connection: a session outlives a token and can contain a hundred calls with different consequences.

Per-tool scopes, not per-noun scopes

Scope names should describe the action and its blast radius, not the noun: orders:read, orders:export, orders:refund. One orders scope is a decision every future tool on that noun inherits, granted for the weakest of them. Default deny, so a tool absent from the policy is never advertised, and advertise only what the caller can use, so a compromised session has a smaller haystack. The test for granularity is reversibility: one scope per irreversible action, one for reads, shared scopes for reversible writes.

The confused deputy, in three disguises

The classic form is a server that authenticates to a backend with a privileged credential while serving less privileged callers. The MCP twist is that the caller is often a session rather than a person, and a session can change subject halfway through when a token is refreshed.

The second disguise is a cache: keyed by path rather than by tenant and subject, it will eventually hand one caller’s data to another. The third is the convenience tool, whether it fetches any URL, runs any query or reads any file, because it turns the server’s network position and credentials into the caller’s capabilities.

The defence is the same in all three: every backend call carries the caller’s identity all the way down, and no process-wide credential is broader than the caller’s rights. Where that is impossible, the tool needs a scope only a small group holds.

Prompt injection is an authorisation failure

Most writing about injection treats it as a prompting problem: delimit the untrusted content, tell the model to ignore instructions inside it, add a classifier. Those are mitigations, not boundaries, and they fail when the content was written by somebody who wants them to fail.

Treat it as an authorisation problem and the design changes. If every call is authorised against the session’s scopes, injected instructions can only cause what the session could already do, so the question becomes blast radius.

Three consequences. Keep read-untrusted-content and act-with-broad-privilege out of one session without a boundary between them, so fetch-and-summarise gets a read scope and no write. Make irreversible tools require a confirmation the model cannot manufacture. And review the read side too, because exfiltration is a side effect even when nothing is written. Whether retrieved content is any good is a separate discipline, the one in evaluating RAG systems: bad retrieval is how the wrong document ends up shaping the call.

Irreversible actions need a person

Sort the tools into three classes: reads need a scope, reversible writes need a scope, an idempotency key and a way to undo, and irreversible actions need all of that plus a human being. The confirmation belongs to the host, as a prompt the person sees with the arguments rendered, not as a boolean on the tool: a tool that takes confirmed: true is a tool the model can confirm to itself. Where the host cannot render a confirmation, do not expose the tool.

Secrets, least privilege and egress

Hold the smallest credential that can do the job, obtained per caller where possible, rotated without a restart and never logged: a user token in, a downscoped backend token out, with an audience that names the backend. Egress deserves its own allowlist for the same reason, because the interesting incident is not a stolen key but a tool that can reach an internal metadata endpoint or an address the model supplied. Pin backends by hostname and treat a URL in a tool argument as untrusted input.

The call path, with the gates marked

sequenceDiagram
    autonumber
    actor U as User
    participant H as Host (model, consent)
    participant C as MCP client
    participant S as MCP server
    participant P as Policy
    participant B as Backend API
    participant A as Audit log
    U->>H: asks for a result
    H->>H: model proposes a tool call
    H->>C: call tool with session identity
    C->>S: tools/call with bearer token
    S->>A: record intent: subject, tool, argument digest
    S->>P: authorise scope for this call
    P-->>S: allow or deny
    Note over S,B: credential exchanged for a downscoped backend token
    S->>B: request scoped to the caller
    B-->>S: result or backend error
    S->>A: record outcome, latency, error class
    S-->>C: result or structured error
    C-->>H: tool result
    H-->>U: answer, with confirmation where required

Read the diagram as three gates rather than a pipeline. The host decides whether the call is offered at all, and consent lives there. The server decides whether it is allowed, and policy lives there. The audit record decides whether the call can be explained afterwards, and that is where trust in the whole arrangement actually lives.

Write one audit line per call before and after the work, and make the two linkable: subject, session, tool and version, a digest of the arguments rather than the arguments, the policy decision and the rule behind it, the error class, latency, and the correlation identifier that ties the call to the backend request. A digest still proves two calls were identical without copying the personal data, and denials matter as much as successes, because a denial is a misconfigured scope or somebody probing.

Designing the tool contract

Descriptions are the interface

The model chooses tools by reading their descriptions, so a description is a prompt that lives in your repository and should be reviewed like code. State what the tool does, when to use it, when not to, the units, the formats and the edge of its range. A description that says “manage records” will be called to manage records.

Schema and hints belong with it: the input schema is the machine-readable half, and annotations such as read-only or idempotent hints tell the host which calls can be retried. Keep them consistent, because a hint that contradicts the schema is worse than no hint.

{
  "name": "orders_export",
  "description": "Export one bounded page of orders for a customer and date range. Read-only. Call orders_search first to confirm the customer id.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "customer_id": { "type": "string", "pattern": "^cus_[A-Za-z0-9]{8,32}$" },
      "from": { "type": "string", "format": "date" },
      "to": { "type": "string", "format": "date" },
      "page_size": {
        "type": "integer",
        "minimum": 1,
        "maximum": 500,
        "default": 100
      }
    },
    "required": ["customer_id", "from", "to"],
    "additionalProperties": false
  },
  "annotations": {
    "readOnlyHint": true,
    "destructiveHint": false,
    "idempotentHint": true
  }
}

Idempotency, retries and timeouts

A call can time out after the backend succeeded, so every write takes an idempotency key from the host, and the server returns the first result for a repeat. Without that, a retry is a duplicate, and the retry will happen, because the host cannot tell slow from failed.

Keep the timeouts in one place: the server’s budget is shorter than the host’s, so it answers before the host gives up, retries happen at one layer rather than two, and non-idempotent calls are never retried automatically. When a tool cannot answer quickly, return a handle and a status tool, and propagate cancellation down to the backend.

Validation and the error taxonomy

Schema validation is necessary and not sufficient: check ranges, lengths and semantics, such as an identifier that exists and belongs to the caller. Order matters, so validate the shape, authorise, then check existence, and an unauthorised caller cannot use error messages to learn which identifiers are real.

Then give errors a small vocabulary the model can act on. Invalid input, which it can fix. Forbidden, which it cannot. Rate limited or unavailable, which means wait. Internal, which means stop. The model sees the class and a safe message, the operator sees the class, the correlation id and the detail, and the taxonomy is defined once for every tool on the server.

def refund_payment(ctx, payment_id, amount_minor, idempotency_key):
    require_scope(ctx, "payments:refund")              # per call, per session
    validate(payment_id, amount_minor, idempotency_key)
    if not ctx.policy.allows(ctx.subject, payment_id):  # ownership, not just scope
        raise ToolError("forbidden")
    try:
        return ctx.backend.refund(payment_id, amount_minor,
                                  idempotency_key=idempotency_key,
                                  token=ctx.downscoped_token,
                                  timeout=ctx.budget.remaining())
    except BackendUnavailable as exc:
        ctx.log.warning("refund backend unavailable", error=exc.classify())
        raise ToolError("unavailable", retry_after_ms=ctx.budget.backoff())

Size, pagination and deprecation

Cap what a tool can return. A tool that can return forty megabytes will eventually return forty megabytes into a context window, and the cost lands on the conversation rather than the tool. Page with an opaque cursor, bound the page size, truncate with an explicit flag and a continuation cursor, and move anything genuinely large to a resource the host fetches on purpose.

Version the server rather than each tool, unless a tool has its own consumers. Additive change first, because old callers ignore fields they do not know. A renamed argument becomes a new tool name, both advertised for the deprecation window with the old one marked deprecated in its description, then removed when the call graph shows nobody is left.

Operability

Logs with an identifier that survives the whole call

Generate a correlation identifier at the host, echo it through the server, put it on the backend request and store it on the audit line, with the tool name, the policy decision, the outcome class and the duration. A support question then becomes a query rather than an afternoon:

id=9f2c1a47
jq -c --arg id "$id" 'select(.correlation_id == $id)
  | {ts, layer, tool, decision, outcome, ms}' \
  logs/host.jsonl logs/mcp.jsonl logs/backend.jsonl | sort

Metrics per tool, and what a call actually costs

Count calls by tool and outcome, not just by server, because one bad tool hides inside a healthy average. Histogram the latency, split errors by class, track concurrency and payload bytes out, and measure the backend’s own latency separately or a slow tool gets blamed forever.

Cost is the metric teams forget. A call is not just a request, because its result lands in a context window and stays there for the rest of the conversation, so the expensive tool is the one that returns something large and unsummarised. Measure bytes and tokens per tool, and make result size part of the contract review rather than a surprise on an invoice.

Tracing, limits and backpressure

One span per call, a child span per backend request, with tool name, scope decision, backend and retry count as attributes. Sample successes and keep every denial and every error, because those are what somebody will ask about.

Limit per caller and per tool, not per server. One agent in a loop should throttle itself rather than starve everybody else, and a limit on the fan-out protects the backend the server just turned into a dependency. Shed load with a retry hint before the backend queues, cap concurrency per backend credential, and treat the limit as something the model can read: a tool that reports itself rate limited tells the model to wait rather than retry immediately.

The kill switch

Two switches, and they are not the same. One removes a tool from the advertised list so the model stops choosing it. The other refuses it at execution time so a valid caller still cannot use it. During an incident the second is the one that matters, because the first takes effect only for sessions that re-read the list.

Check the switch before authorisation, make it changeable without a redeploy, and rehearse it. The first time I pulled one, the tool was already in the model’s context and kept being called for the rest of the session; the switch that worked was the one at execution time.

Failure modes and testing

Partial failure and retry storms

The failure that bites is not a crash but a call that succeeded and was reported as a failure: the backend wrote, the server died before answering, the host retried. Idempotency keys handle it, plus a read tool that answers whether the work already happened. Test it by killing the server between the backend call and the response, because that window is invisible in unit tests.

A retry storm is the same problem at scale. An error that says only that it failed invites the model to try again; a class that says invalid, forbidden or unavailable lets it do the right thing instead. Cap retries per call, back off with jitter, and break the circuit per backend rather than per server.

Contract tests and replayed traces

Snapshot the advertised surface: tool names, argument schemas, description hashes, annotations. Descriptions are behavioural, so a description diff should fail the build the same way a signature change does, with a reviewed golden file behind it. If the model’s behaviour changes because somebody clarified a sentence, that is a release.

Record real calls with arguments redacted and replay them against a new version, asserting the same result shape and the same authorisation decisions. That catches the failure no unit test sees: a policy edit that quietly widens access to a scope nobody was watching.

Canarying a tool, and load-testing the fan-out

Advertise a new tool to a small set of hosts first, with execution denied for everybody else, and watch the error classes and the confirmation rate before widening. A tool that works but is constantly refused by the person in front of it is not ready.

Then load the server the way a host does, which is in parallel. One session can produce several concurrent calls and sessions arrive together, so the limit is usually the backend’s connection pool or the per-credential concurrency cap rather than the server’s processor. Sequential benchmarks say the server is fine right up until it is not.

Rollout and lifecycle

Tool descriptions are reviewed code

A tool description is an interface the model reads, so it lives in the repository, has an owner and goes through review. That is the same discipline as any other AI-augmented delivery work, because the model is given a written, reviewed, testable contract rather than a vague permission, and it is where the work on the AI-augmented engineering side actually lands.

Which raises the question of what done means for a tool. Not that the call works from my machine. Done is: the contract reviewed, the scopes in policy, the audit lines verified end to end, the dashboard carrying the outcome classes, the kill switch rehearsed, the rollback tested, the owner named. That is the same definition of done we apply to anything else we ship, and a tool that satisfies only the first clause is a demo with a version number.

The registry, pinning and staged rollout

Keep a register of servers: version, owner, tools, scopes, backend, data classification. Hosts pin a version, and nothing floats to latest, because the tool list is part of what the model sees and a silent change to it is a behaviour change. Bumping a pin is a reviewed change with a rollback plan.

Roll out in stages, each with an exit criterion: shadow mode, where the server records what it would have done without acting, then a canary group, then everyone. A stage without a criterion is a wait, and the rollout has to be able to stop between stages rather than mid-flight.

Ownership, deprecation and what the model can see

Every server has a named owner and a path into whatever handles incidents. Every tool has a deprecation route: announce, mark it deprecated in the description so the model prefers the replacement, watch the call graph empty, remove, then delete the endpoint.

Review the advertised surface as an information disclosure question in its own right, because tool descriptions leak internal endpoints, resource trees leak paths and prompt templates leak process. Diffing the advertised list between versions is a cheap control.

What I would do differently next time

Write the policy file before the second tool. Retrofitting per-tool scopes onto a server that already shipped a dozen tools means a migration with users on it, and the old broad scope survives until the last caller moves.

Log the digest, not the arguments, from the first day, because adding redaction to a log that already holds customer data turns observability work into a data-protection exercise.

Refuse to add a tool until the backend has an idempotency key. The alternative is building on a retry hazard and discovering it as duplicate work in somebody’s system.

Review the descriptions earlier. The failures I have seen most are not a dangerous tool being chosen; they are the wrong tool being chosen, because one description said “manage” and another said “update”.

Test the kill switch in anger, not in a runbook, because a switch nobody has pulled is a switch nobody knows the shape of.

And keep the server smaller. Every convenience endpoint added for the agent became a second API with its own contract, its own edge cases and its own owner, which is exactly the thing I said at the start not to build.

Closing

None of this is exotic. It is service design, applied to a service whose consumer reads prose. The protocol supplies the vocabulary, and everything that makes a server safe to leave running comes from outside it: a policy, a contract, a log line, an owner.

The two halves of this are the same work. Getting guardrails and review into how a model uses tools, and getting a boundary, a transport and an authorisation model right before there are users to migrate, are the two sides of AI-augmented engineering and architecture and technical advisory respectively. The demo is the easy part; the service is the part with an owner, a scope and a line in a log.