A passing local lab is a useful starting point. A customer-facing service introduces identities, concurrency, infrastructure failures, and a compatibility promise. The next implementation phase should add those dimensions one at a time, preserving the tests already earned.
This chapter is a build sequence and acceptance plan. The remote components described here are not implemented by the companion lab.
Milestone 1: replace fixture identity with a real resource boundary
Choose the authorization server and target customer hosts. Implement protected-resource metadata and the supported client authorization flow using established libraries. Put token validation before domain dispatch. Derive tenant membership from trusted identity and application records, not tool input or a self-reported client name.
Acceptance requires a successful least-privilege connection plus denials for expired tokens, wrong issuer, wrong audience, missing scope, revoked access, and wrong-tenant objects. Assert absence of backend effects, not only an error response. Test key rotation and refresh failures with the real identity infrastructure.
OAuth client registration, protected-resource discovery, and business membership are different responsibilities. Follow the versioned authorization requirements; a custom login screen is not evidence that they are satisfied.
Milestone 2: put the domain state and operation ledger in a database
Replace the map with persistent storage. For a database-local note, enforce a unique operation identity, compare a canonical payload hash, and commit the note and recorded result atomically. Avoid a check-then-insert race without a uniqueness constraint.
The companion’s durability extension implements the database-local core on SQLite: concurrent MCP workers, worker replacement, atomic note/ledger commit, and abrupt process-crash tests. It deliberately has no ledger expiry and does not establish multi-host operation or external side-effect guarantees. Use it as the worked starting point for this milestone.
The conceptual production schema is:
operations(
tenant_id, actor_id, tool_name, operation_key,
payload_hash, result_id, created_at, expires_at,
UNIQUE(tenant_id, actor_id, tool_name, operation_key)
)
notes(id, tenant_id, incident_id, author_id, text, created_at)Acceptance tests: concurrent identical requests through different workers; changed-payload conflict; crash before commit; crash after commit but before response; retry after worker replacement; revocation before replay; and expired-ledger behavior. The expected outcome must include the number of business effects.
For an external side effect, add a durable workflow/outbox and reconciliation. A successful enqueue is not the same as the external operation succeeding. Document which component owns each state transition.
Milestone 3: mount and protect the HTTP endpoint
Use the SDK’s HTTP handler behind TLS, validated Host/Origin policy, request/body limits, timeouts, and authentication middleware. Ensure proxies forward required headers consistently and handle streaming without unwanted buffering. Keep diagnostic routes separate from the MCP endpoint.
Acceptance tests: malformed envelopes, header/body disagreement, unsupported revisions, invalid origins, oversized payloads, client disconnects, backend timeouts, and rolling restarts. Verify responses with an actual network client as well as in-process tests. A deployment may pass handler tests while failing because a proxy strips a header or buffers a stream.
Milestone 4: establish customer fairness and supportability
Implement per-tenant admission limits and global protection, with explicit behavior on overload. Apply downstream budgets so one tool cannot fan out without limit. Provide connection diagnostics that identify protocol, authentication, permission, or backend failures without disclosing secrets.
Use the prior chapter’s 300-customer model as a starting hypothesis, then replace its inputs with measurements. Test steady load, correlated bursts, slow dependencies, and a noisy tenant. Measure accepted/rejected work, latency distribution, queue depth, resource use, and the effect on other tenants.
Do not call a load test representative merely because it starts 300 clients. The operation mix, per-customer activity, data sizes, dependency latency, and retry behavior determine whether it tests the intended system.
Milestone 5: evaluate the customer workflow
Now add model-driven consumption. Use a fixed corpus containing easy investigations, ambiguous service names, conflicting timestamps, truncated data, inaccessible objects, and untrusted instructions embedded in source content.
Compare a direct-tools baseline with selective discovery and, only if justified, constrained code composition. Record correctness, evidence coverage, authority violations, cost, and latency. Run multiple trials because model behavior varies. Preserve prompts, schemas, model versions, and evaluation criteria so improvements can be interpreted.
A model that successfully calls every endpoint may still recommend the wrong rollback. A system that refuses every request may pass denial checks while being useless. Keep positive controls and evaluate whole-task outcomes.
Milestone 6: ship a compatibility promise, not an assumption
Pick a small supported-host matrix. Test its actual versions and authentication paths. Add legacy protocol support only when required by customers, with a documented retirement policy and separate coverage.
Release to internal users, then a small customer cohort, then expand against predefined success and rollback criteria. Version breaking business semantics even when the JSON shape is unchanged. Provide an owner, changelog, support route, and emergency disable mechanism for each consequential tool.
The MCP feature lifecycle policy governs protocol evolution. Your product needs its own policy for tool contracts and client support; the standards body will not maintain those promises for you.
A compact incident runbook
| Symptom | First evidence to inspect | Avoid this reflex |
|---|---|---|
| Customer cannot connect | Endpoint, protocol era, auth discovery, issuer and audience | Reinstall everything |
| Tool exists for one user only | Catalog authorization context and scopes | Share the working user’s token |
| Calls fail after a deploy | Schema/semantic changes, required headers, supported matrix | Blame the model immediately |
| Duplicate writes | Operation ledger, retries, commit timing, downstream receipt | Retry the whole workflow |
| Correct calls, wrong answer | Object selection, time range, truncation, model evaluation | Count HTTP 200s as success |
| One tenant slows everyone | Admission limits, pools, queues, downstream fan-out | Increase global retries |
| Private data appears elsewhere | Source provenance, output destinations, logs and host policy | Assume read-only access prevented disclosure |
Practice decisions before reading the answer
These exercises distinguish producing a reference document from learning to use it. Write your prediction or proposed design first. Record any help received. The answer sketches are comparison points, not proof of competence.
Exercise A: the interface choice
Your team controls a nightly reconciliation job and its backend. Another team wants customers to explore the same data through their own assistants. Should both callers use MCP?
Answer sketch: keep the deterministic job on the existing API/SDK unless there is a separate benefit. Add an MCP adapter for the interactive cross-host use case, sharing domain authorization and semantics. The shared implementation matters more than forcing every caller onto one interface.
Exercise B: discovery is not permission
Alice discovered add_note, then her write permission was revoked. The host still has a cached catalog. Should the server execute the call?
Answer sketch: no. Discovery describes a capability; current authorization controls execution. Cache invalidation improves UX, but server enforcement is the security boundary. The same reasoning applies to replayed operation keys.
Exercise C: timeout after commit
The server committed a customer-visible action, but the response was lost. The host changes the JSON-RPC ID and retries. How many effects occur?
Answer sketch: the protocol ID does not answer the question. The application needs a stable operation key and durable replay/reconciliation semantics. If these are absent, the outcome is uncertain; do not promise exactly once.
Exercise D: the large catalog
A provider offers 1,500 operations. A code-execution surface cuts initial schema tokens sharply. Should you adopt it?
Answer sketch: test end-to-end tasks and security controls, including nested operations. Initial token count excludes discovery, execution, output, errors, and retries. A small curated surface may remain better for a narrow product workflow.
Exercise E: organizational ownership
Five teams publish MCP tools with different approval, logging, and error conventions. A gateway can unify the endpoint. Does that solve the platform problem?
Answer sketch: only part of it. Establish minimum contracts, owners, identity propagation, data-flow policy, semantic versioning, and incident responsibility. A central endpoint can increase shared blast radius if governance and isolation do not accompany it.
A useful transfer check
Take a fresh domain—expense approvals, feature flags, or customer support. In one page, specify the user workflow, why MCP helps, why a direct API might still be preferable, identities, two tools, bounded outputs, failure semantics, and rollout evidence. Then have a domain owner or operational reviewer challenge it.
That exercise tests selection and integration of ideas. Repeating the incident lab proves familiarity with the lab; applying the reasoning to a new domain begins to test transferable engineering judgment.