On a Tuesday afternoon, a developer asks the team’s coding agent for a test environment for a feature branch. The agent calls the MCP tool provision_test_environment. Behind that tool, your server creates a namespace, applies role bindings, deploys six services, and seeds a database. That takes about four minutes on a good day.
The client gives up at 60 seconds. The agent decides the call failed and retries. The second call times out too, so it tries a third time. Soon three environments are running. Two belong to the right developer. The third got a different user’s permissions, because the server resolved “current user” from a shared session cache another request had just updated. Nobody can say which agent made which call, because every agent uses the same API token.
If you build MCP servers that wrap databases, ticketing, CI/CD, data pipelines, or ERP, some version of this scene is probably on your backlog. Here’s why it happens and how to design long-running tools so retries are harmless, restarts don’t lose work, and every call has a verified caller.
Why long-running tools break
The Model Context Protocol gives agents a clean way to discover and call tools. Most tool implementations, though, are written as quick functions: take arguments, do the work, return a result. That holds up for “look up ticket 4711.” It falls apart for “run the quarterly revenue report” or “bulk-update 40,000 customer records.”
Timeouts and retries cause duplicates. Clients and agent frameworks set timeouts, and agents are trained to be persistent. After a timeout, the agent can’t tell whether the work never started, is still running, or finished. Retrying is reasonable from its point of view. If your tool isn’t idempotent, each retry is a new execution.
In-flight work lives in memory. If the MCP server restarts during a deploy or crash, whatever it was halfway through is gone. Maybe the namespace exists but the services don’t. There’s no record of where it stopped, so there’s no clean way to resume or roll back.
Shared credentials erase the caller. Many teams start with one API token shared by every agent. Your logs then show one identity doing everything. You can’t rate limit a misbehaving agent without throttling all of them, or answer “who asked for this?” during an incident.
User identity gets lost or confused. The agent acts for a human, but that identity often travels as a loosely trusted header or session lookup. When that link is weak, the tool runs with the wrong person’s permissions, as with the third environment.
There’s no audit trail. Logs are scattered across the agent, the MCP server, and the downstream system, with nothing tying an invocation to its inputs, caller, and outcome.
Why the usual workarounds fall short
The first instinct is to raise the timeout. That helps until the next tool takes longer, and it does nothing about restarts.
The second is telling the agent not to retry in the tool description. Models follow instructions most of the time, which isn’t something provisioning should depend on. Clients and frameworks also have retry behavior a prompt can’t control.
The third is a dedupe check: “if an environment with this branch name exists, return it.” That races when two calls arrive close together, and a half-built environment looks like an existing one.
For identity, teams add per-agent API keys. Static keys get copied into config files, rarely rotate, and say nothing about which human the agent acts for.
The general approach
The fix is to treat tool calls like any other distributed operation that can fail halfway.
Split long work into “start” and “check.” Instead of one provision_test_environment tool that blocks, expose start_provision_test_environment, which returns a job ID right away, and get_job_status, which returns state, progress, and the result once done. The agent can poll, report progress, or come back later. Timeouts stop mattering because no call does the slow part.
Make “start” idempotent. Derive an idempotency key, for example from the user, the branch, and the request intent. If a start call arrives with a key you’ve seen, return the existing job. The cleanest version uses the key as the job ID, so the uniqueness check and job creation are one operation.
Back each job with a durable workflow. The job should be a sequence of steps whose progress is persisted after each one. If the process restarts, the job resumes from the last completed step instead of starting over or disappearing. Each step should also be idempotent on its own, using deterministic names (a namespace called env-<job-id>) so a repeated step finds its own earlier work.
Give every caller its own workload identity. Each agent and MCP server should have a cryptographic identity, such as a SPIFFE ID, issued and rotated automatically. Require mutual TLS so both sides prove who they are. Now logs, rate limits, and access policies work per agent.
Propagate verified end-user identity. Workload identity says which software is calling; you also need the human. Pass a verified OAuth token through the call chain and have the tool enforce permissions from that token, not from a session cache or a free-text argument.
Apply per-tool access policies. A support agent might read tickets but should never call bulk_update_records. Express this as policy keyed on workload identity, enforced before tool code runs.
Record every invocation in a tamper-evident history. For each job, keep the caller’s workload identity, the end user, the inputs, each step’s outcome, and the final result. If that history is signed or otherwise protected against edits, you can hand it to an auditor.
Protect downstream systems. Your MCP server is a new, enthusiastic client of systems built for human-paced traffic. Put timeouts, retries with backoff, and circuit breakers between your activities and those systems, and rate limit per caller identity.
Implementing it with Dapr Workflow
You can build this by hand with a job table, a queue, and careful code, or lean on a workflow engine. Dapr, the CNCF graduated runtime, includes Dapr Workflow, which stores an event-sourced history of each workflow in a state store. After a crash, it replays that history, and completed activities are not re-executed. Orchestrator code must be deterministic, so all I/O goes into activities.
A Python sketch of the provisioning job:
import dapr.ext.workflow as wf
from datetime import timedelta
wfr = wf.WorkflowRuntime()
retry = wf.RetryPolicy(first_retry_interval=timedelta(seconds=2), max_number_of_attempts=5, backoff_coefficient=2, max_retry_interval=timedelta(minutes=1))
@wfr.workflow(name=”provision_test_environment”)
def provision_env(ctx: wf.DaprWorkflowContext, data: dict):
ns = yield ctx.call_activity(create_namespace, input=data, retry_policy=retry)
yield ctx.call_activity(apply_user_permissions, input=data, retry_policy=retry)
results = yield wf.when_all([ctx.call_activity(deploy_service, input=s) for s in data[“services”]])
return {“namespace”: ns, “services”: results}
@wfr.activity(name=”create_namespace”)
def create_namespace(ctx: wf.WorkflowActivityContext, data: dict): …
# apply_user_permissions and deploy_service are defined like create_namespace
Your start_provision_test_environment tool schedules this workflow using the idempotency key as the workflow instance ID, then returns that ID to the agent. get_job_status reports the instance’s state. If an instance with that ID already exists, the start tool catches the conflict and returns the existing ID, so the agent gets the same job back, not a third environment. If the server restarts during the service deploys, the namespace and permissions steps are already in the history and won’t run again.
apply_user_permissions reads the user from data, which the start tool fills from the verified token, never from agent arguments.
On the security side, Dapr’s Sentry service acts as a certificate authority that issues SPIFFE-based identities, and sidecars use mTLS with each other by default. Access-control policies can restrict which app IDs may call your MCP server at all; per-tool rules still need enforcement inside the server, keyed on the caller’s verified identity. Resiliency policies, defined in YAML, add timeouts, retries, and circuit breakers to calls into downstream components, and OpenTelemetry tracing ties an invocation together across services.
If you’d rather not operate these pieces yourself, one option is Diagrid‘s Catalyst, which is built on Dapr. In Catalyst, every agent and MCP server gets a cryptographic identity, mTLS is on by default, and verified user identity (OAuth) is propagated through agent calls. It also offers opt-in workflow history signing (which requires mTLS), relevant to the tamper-evident audit requirement. As with any vendor, test these claims against your own threat model and environment before relying on them.
A rollout checklist
Start with the tools that are slow, expensive, or hard to undo.
- ☐ List your MCP tools and mark any that can take longer than your clients’ timeout, or that create, delete, or bulk-modify anything.
- ☐ Convert those to a “start job” plus “get job status” pair, with progress reporting.
- ☐ Define an idempotency key for each start tool and use it as the job ID.
- ☐ Move job execution into a durable workflow, with I/O in idempotent activities and deterministic resource names.
- ☐ Kill a server mid-job in a test environment and confirm the job resumes without repeating finished steps.
- ☐ Replace shared tokens with per-agent workload identities and enforce mTLS.
- ☐ Pass verified end-user tokens through to the tool and enforce permissions from them only.
- ☐ Write per-tool access policies keyed on caller identity.
- ☐ Add timeouts, retries, circuit breakers, and per-caller rate limits for each downstream system.
- ☐ Store a per-invocation history with caller, user, inputs, steps, and result, and protect it against modification.
- ☐ Simulate an agent retry storm and check that exactly one environment, report, or update results.
Conclusion
Agents will retry and servers will restart. That’s normal distributed-systems behavior arriving through a new interface. If you design long-running MCP tools as idempotent jobs on top of a durable workflow engine, and you tie every call to a verified workload and a verified human, the Tuesday-afternoon scene turns into one environment, built for the right person, with a record of exactly how it happened. Whether you run open-source Dapr yourself or use a managed platform offering durable execution for AI agents, the principles are the same. Adopt them before your agents find the gaps for you.
