“Hundreds of customers” is a commercial count, not a capacity specification. Three hundred customers can generate a quiet service or an intense burst during a shared incident. Start with the contract and workload before choosing infrastructure.
This chapter is a reference design for the incident product, not a report of a deployed service. Protocol facts are cited; numerical inputs and architecture choices are explicit assumptions.
Keep one domain service behind several interfaces
Customer host / custom client
|
v
TLS ingress + request limits + Host/Origin policy
|
v
Token verification -> trusted principal -> tenant routing
|
v
MCP adapter: schema + protocol + result shaping
|
v
Domain service: object authorization + business invariants
|
+--> database: incidents, notes, operation ledger
+--> downstream API: separately scoped credentials
+--> durable jobs/outbox: explicit operation handles
Direct API and CLI -----------------> same domain serviceDo not duplicate business authorization in each adapter. An MCP-only check can be bypassed by another interface; a direct-API-only check can be omitted by a new MCP handler. Put domain invariants at the point where data is read or changed, and keep protocol validation in the adapter.
Define the tool as a product contract
For add_note, state the affected object, required authority, maximum text size, visibility, external effects, replay behavior, and returned identifier. “Add a note” is not enough if some notes notify customers and others remain internal.
For reads, specify filters, time semantics, ordering, pagination, field projection, freshness, and truncation. A bounded first page must not masquerade as a complete result set. A stable output schema helps clients compose results, but any change in meaning still needs review.
Descriptions and annotations help hosts choose and display tools. They are not authorization enforcement. A tool marked read-only must actually be read-only; a tool marked idempotent must implement the promised behavior over its documented lifetime.
Our starting catalog is deliberately small: list incidents, read one incident, list deployments, and add an internal note. We can add aggregate tools after measuring repeated workflows. We do not start with arbitrary SQL, arbitrary URLs, or unrestricted shell execution.
Authenticate the request, authorize the object
For protected HTTP access, the MCP server acts as a resource server. Use established token-verification middleware or introspection. Validate issuer, audience, expiry, and applicable scopes; decoding a JWT is not validation. The authorization specification defines discovery and token requirements.
Construct a trusted principal from the verified identity and current tenant membership. A model-supplied tenant_id is a requested target, never proof of membership. Even with notes:write, Alice must be allowed to modify the requested incident.
Prefer queries constrained by tenant and object together. Avoid fetching an arbitrary object and leaking its existence through distinguishable error messages before policy is applied. Database row policies can provide defense in depth, but do not replace careful credential and application design.
If the MCP service calls another provider, obtain an appropriate downstream credential through the supported delegation flow. Preserve the distinction between incoming authority and downstream authority. The official security guidance forbids token passthrough and discusses confused-deputy conditions and state-handle protection.
Approval belongs to a trusted flow
A write tool argument such as confirmed: true proves only that the caller supplied a boolean. It does not prove that a human reviewed the operation.
If policy requires approval, mint an approval record through a trusted UI or application service. Bind it to principal, tenant, target, normalized operation payload, expiry, and relevant object version. At execution, verify the binding and recheck authorization. If the object changed materially since approval, require a new decision.
Not every internal note requires a human click. Approval policy should be proportionate to consequence. The important point is that when approval is required, it cannot be manufactured by the same untrusted arguments proposing the effect.
Make write replay explicit
An operation key identifies a business intent across retries. It is separate from JSON-RPC IDs and trace IDs.
For a database-local write, use a unique key such as (tenant, actor, tool, operation_key). Store a canonical payload hash and the resulting business identifier. In one transaction, authorize against current state, establish the operation record, perform the mutation, and persist its result. A repeated key with the same payload returns the recorded result; a changed payload conflicts.
first attempt --> authorize --> claim key --> mutate + save result --> commit
retry --> authorize --> find key --> same payload? return result
\--> changed payload? conflictThis is a design sketch, not complete database code. Concurrent attempts need a uniqueness constraint and correct transaction handling. Replays must not bypass revoked permission. Define retention long enough for supported retries; deleting the ledger too early can allow an old request to execute again.
External effects are harder. A database transaction cannot atomically commit with an unrelated API by wishful thinking. Use the downstream service’s idempotency contract where available, plus an outbox or durable workflow and reconciliation. Preserve an “outcome unknown” state after an ambiguous timeout. Avoid promising exactly-once execution across boundaries you cannot control.
The baseline lab demonstrates a smaller property: one synchronous critical section with an in-memory map. Restart destroys its replay history. The durable extension commits a note and operation record atomically in SQLite, with concurrent-worker and process-crash tests. Its guarantee is limited to database-local effects on one host; external effects, operational retention and customer-scale behavior remain separate work.
Workload arithmetic for a hypothetical 300-customer service
Assume 300 organizations, 20 configured users per organization, 5% active concurrently, one workflow per active user per minute, and six MCP calls per workflow.
configured users = 300 × 20 = 6,000
concurrently active = 6,000 × 0.05 = 300
workflows per second = 300 / 60 = 5
MCP calls per second = 5 × 6 = 30
fivefold burst = 30 × 5 = 150
in-flight calls ≈ request rate × mean time
at 0.8 seconds mean = 150 × 0.8 = 120
at 3 seconds mean = 150 × 3 = 450The concurrency estimate uses a stable-window approximation based on Little’s Law. It is not a queue-tail bound. Add discovery, authentication, retries, subscription streams, and background jobs separately if they are not included in the six calls.
If each MCP call makes three downstream calls, the burst becomes 450 downstream requests per second before retries. A provider quota may be the first bottleneck. A 50 KB mean response at 150 calls per second is about 7.5 MB/s of payload egress, excluding framing and retransmission. The model’s token cost depends on what the host actually includes, not just bytes the server returns.
These inputs are teaching assumptions. Measure them for your product, segment them by tool, and include correlated incidents where many customers become active together.
Fairness and backpressure
Use per-tenant and global concurrency limits, bounded queues, request deadlines, response-size caps, and downstream budgets. A large customer should not consume every database connection. Expensive searches and simple lookups may need separate pools or weighted admission.
Return an understandable overload response before queues become unbounded. Provide retry guidance where appropriate, and account for client retries in capacity planning. Avoid synchronized retries by using backoff and jitter in controlled clients.
Scale stateless request workers independently of durable state. If you offer long-lived subscriptions, account for connection count, fan-out, idle timeouts, and reconnect storms. Removing protocol sessions does not eliminate those resource costs.
Observability that answers customer questions
Record a workflow/trace identifier, authenticated principal, tenant, client version, protocol revision, tool and schema version, policy outcome, duration, downstream status, operation key reference, and result category. Avoid raw tokens, full sensitive payloads, or arbitrary tracing baggage in logs.
The trace-context proposal adopted by MCP documents propagation conventions. Propagation is not an audit system: the server must create trustworthy records of policy decisions and effects.
Separate reliability from task usefulness. Suggested starting objectives are an availability target for eligible calls, latency targets by tool class, and zero tolerated cross-tenant disclosure. Choose actual numbers with customer expectations and backend limits; do not copy a universal 99.9% target. Track unauthorized attempts and correct denials separately from server failures so “securely rejected” does not look like an outage.
Compatibility and ownership are part of the service
Maintain a tested matrix of client product/version, transport, protocol revision, auth flow, required primitives, and extensions. Adding a field can be syntactically compatible while changing model tool selection. Run behavioral evaluations on important workflows alongside schema/contract tests.
Version breaking tool semantics deliberately. Support a migration period, communicate changes, canary by tenant or client cohort, and retain a rollback path. Avoid silently changing units, default filters, or external effects under the same tool name.
Assign owners: platform engineering for transport/runtime, identity/security for token and consent infrastructure, domain teams for permissions and invariants, and product/support for onboarding and compatibility promises. The provider remains accountable for the end-to-end service even when these owners are different teams.
At L7 depth, defend how the pieces compose under failure. At L8 depth, decide whether the integration surface is worth maintaining, which customer workflows justify it, and who will own it after the launch team moves on.