Stateful vs Stateless Agent Architecture Tradeoffs

Stateful vs. stateless agent architecture is a choice about task horizon and continuity. Stateless designs fit work that ends with the request, while stateful designs fit work that must survive it. For per-user agents in the second category, Maritime provides a concrete implementation of persistent, VM-isolated execution with snapshot-restore scheduling.

Why the stateful vs. stateless question is actually a question about task horizon

Every major LLM API is stateless by construction. The model has no built-in memory of prior requests, so what appears to be persistent chat memory is the surrounding application resending earlier messages. A stateful agent is therefore a compound system, with a frozen model wrapped in a runtime that owns memory, identity, and orchestration.

A pipeline has a clear endpoint within an invocation, whereas a relationship accumulates context across many invocations. Both can need a great deal of reasoning and tool use, so how complex they are does not tell them apart. The choice depends on whether the agent must keep information from finished work to use in future work.

The information at stake extends well beyond conversation history. State includes tool results, the status of a workflow, decisions made along the way, and user preferences. A workflow may need to keep its progress without keeping a conversational transcript, while a personalized assistant may need preferences after a conversation ends. Treating all of this as chat memory obscures the actual decision, which is what must survive and for how long.

Where stateless agents are the correct and sufficient design

Stateless architecture is the right design when the full relevant context fits in one request and nothing needs a persistent identity. This works for summarization, extraction, classification, and one-shot tool calls. High-throughput work, including moderation queues and embedding generation, also benefits from independent requests. These are not lesser agent workloads, but production tasks whose boundaries make continuity unnecessary.

Those boundaries produce structural scaling advantages. A request can go to whichever worker is free, without sticky sessions or session data that must stay synchronized across servers. There is no requirement for a shared session cache or a state database on the hot path. Concurrency is easier to manage because independent requests cannot overwrite one another’s agent state. Commodity workers can therefore scale horizontally without coordinating a continuing relationship.

Independent calls are reproducible and cacheable in evaluation harnesses, while a zero-retention runtime can avoid writing conversation data to disk. That narrower retention surface matters for zero-retention contracts and EU data-residency requirements. An agent that retains no prior interactions also has no accumulated conversation store for an attacker to exfiltrate, though current inputs and tool access remain separate concerns. Statelessness stops being sufficient when the next request depends on a correction, preference, or completed step that the request does not carry.

Where stateless agents quietly break under real workloads

Beyond that boundary, failures often look like ordinary interactions. An agent asks the same qualifying questions because it has no record of earlier answers. A subject-matter expert corrects it, but the correction lasts only while it remains in the supplied context. Work that spans days or weeks loses continuity between sessions. Each response can make sense on its own while the workflow as a whole makes no progress that lasts.

Resending the entire history postpones this problem. As the conversation grows, so does the request payload and the amount of context the model must process. Eventually, the client or application must decide what to summarize or discard. Those choices decide which corrections and intermediate decisions the agent can still use. The system has acquired a state-management problem even if its agent interface is still described as stateless.

Without a retained log of attempted steps, an agent cannot recognize a failed path from a previous invocation and branch differently. It restarts without the evidence needed to distinguish a retry from a first attempt. Repetition is therefore not necessarily a reasoning failure, but a missing record.

Token costs can compound this loss of continuity. Prefix caching reuses the KV cache for a stable prompt prefix, but recognizing that shared prefix requires state somewhere in the runtime. Rebuilding the prefix on every call forfeits that reuse. Stateless model calls and stateful caching are compatible, but a design that discards both history and reusable execution context gives up separate forms of continuity.

What a stateful agent actually requires to work in production

A stateful interface moves responsibility for continuity behind the API boundary. The client sends a new message with an identifier, and the runtime loads the relevant state, processes the message, and writes updated state afterward. This keeps client payloads the same size as the conversation grows. It also gives the server control over how history is reranked, summarized, or reorganized for each turn.

Memory has distinct roles within this runtime. Working memory holds the active context, including the message buffer, tool traces, and system prompt, and is re-derived for each request. Episodic memory records past interactions in time order, typically in a vector store or key-value store. Semantic memory keeps facts, preferences, and profiles that have been brought together in a structured store or knowledge graph. These tiers separate what the agent needs now from what it may need later.

The scope of that memory must match the task. Session-scoped state fits a conversation or support ticket that ends and is discarded. User-scoped state preserves a profile across sessions, while workflow-scoped state tracks progress through audits, migrations, or onboarding programs. Durable logs of attempted steps and user corrections then give recovery logic evidence for choosing a different path.

Production operation also requires concurrency control, retention policies, permissions, observability, and write-back safeguards. Parallel overwrites can leave stale state, partial updates can leave inconsistent state, and retries can lose progress when write-back is not idempotent. Accumulated context can also introduce prompt drift. A common hybrid therefore keeps compute stateless while externalizing session state, workflow checkpoints, and long-term memory into dedicated stores. The model remains stateless, but the runtime owns continuity.

Why VM-level isolation is the right substrate for stateful agents that handle untrusted inputs

Once an agent preserves files and executes code or tool calls across sessions, the boundary around its environment becomes a security requirement. Persistent state gives a compromise something that can outlive the current request. In a multi-tenant deployment, that boundary must also separate one user’s agent from another user’s data.

VM-level isolation gives each workload its own kernel under hardware virtualization. Unlike a shared container boundary, the guest kernel is separated from the host and adjacent workloads by a VM boundary. This provides containment when an agent runs untrusted code or handles untrusted tool outputs. Firecracker makes that model practical through microVMs that boot in well under a second with minimal memory overhead and hardware-level isolation at container-equivalent infrastructure density.

MCP tooling adds a reason to take that containment seriously. Tool metadata, including descriptions and parameters, can carry prompt injection into the agent’s context. When injected content is written into persistent state, its influence can extend into later sessions. VM isolation limits the execution blast radius, but it does not turn malicious instructions into trustworthy data.

The same concern applies when agents receive inputs from agents outside their originating system. As agents interact more, they have more chances to process material that the runtime does not control. For agents with persistent workspaces and tool access, VM-level boundaries provide the execution substrate without replacing permissions or safeguards on what enters memory.

How snapshot-restore scheduling makes per-agent VM isolation economically viable

A VM per agent looks expensive if every VM must run continuously. That assumption does not match agents that spend most of their time waiting for a user, a tool call, or another process. The relevant compute demand is active concurrency. Snapshot-restore scheduling separates the cost of preserving an environment from the cost of running it.

The lifecycle starts by booting a microVM and capturing a snapshot that can be restored on demand. Between periods of work, the agent stays suspended and uses no running compute. Its idle cost is storage, which is very small per tenant compared with keeping a machine active. Across a large fleet, usage-driven compute therefore costs substantially less than always-on compute. On infrastructure such as Maritime, keeping idle agents as snapshots cuts the extra cost of persistent isolation.

For tenant-facing agents, that model calls for a short idle timeout, automatic resume, and a TTL sized to the longest expected absence. State held in VM RAM is checkpointed to durable storage on suspend. When the agent resumes, it returns to the state it left. With sub-second wake latency, this pattern works for conversations with gaps of minutes or hours.

The remaining economic choice is whether to build or rent the isolation infrastructure. Below a sustained utilization threshold, pre-built infrastructure costs less than building and operating it. Above that threshold, owning the infrastructure can pay off. The costs must therefore be compared using actual concurrency and the cost of running the infrastructure, with no assumption that every provisioned agent runs all the time.

How Maritime is built around the per-user agent model this architecture requires

Maritime is built so that each user has an agent running in its own persistent Firecracker micro-VM. Each environment runs on bare metal and has its own kernel, persistent disk, and the right to install packages. Files, credentials, browser sessions, and memory survive sleep and redeploy, so continuity includes the execution environment as well as stored messages.

That persistent filesystem can keep logs across sessions of steps the agent has tried, user corrections, and how far a workflow has got. Recovery logic can use those records to detect a prior failure and branch differently on retry. Maritime supplies the isolated, persistent environment that makes this possible without reconstructing a workspace on top of stateless execution. The application still has to decide what to record and how to use it safely.

Snapshot-restore scheduling supplies the corresponding lifecycle. Wake latency is under one second, and idle agents incur only storage costs. Deployment goes from code to a live API endpoint in a single push, with isolation and scheduling handled by the platform. This makes Maritime a direct fit for developers whose product gives each user an enduring agent environment.

The alternatives remain valid for different requirements. Commodity stateless workers work well for independent requests, while stateless compute with dedicated stores can keep application memory without giving every agent a persistent machine. Building and operating microVM infrastructure gives a team responsibility for the isolation and scheduling layer itself. The distinction is not whether an approach can store data, but whether durable execution environments are part of the product’s core unit of deployment.

A decision framework for choosing architecture based on task horizon and continuity requirements

The two starting axes are how far the task extends and what must survive between sessions. If one prompt and one response complete the work, stateless execution is correct and sufficient. If continuity ends with a single conversation, session-scoped state with stateless compute fits the boundary. If work continues across days or weeks, durable state and workflow checkpoints become part of the architecture.

Then identify what owns that continuity. Preferences and profiles call for user-scoped state, while resumable progress calls for workflow-scoped state with explicit checkpoints and idempotent write-back. Work that needs no persistent identity, including classification and zero-retention processing, is still stateless by design. These choices define the information that must persist without requiring every layer to be stateful.

Apply isolation as a separate requirement. Agents that execute code, install packages, or handle untrusted tool outputs should run within VM-level boundaries whether their tasks are stateful or stateless. For persistent, multi-tenant agents, one user’s environment must also be kept from reaching another’s data. Evaluate their cost using active concurrency and snapshot storage. Choose stateless when the task is the request and stateful when the task is the relationship, with VM-level isolation and snapshot-restore economics when that relationship involves real user data, persistent files, or untrusted code.