AWS Infinity: How a Rust Runtime Fits 75,000 AI Agents in 8GB
Infinity is addressing one of the core bottlenecks in modern AI agentic systems: waiting. Large language model agents are designed to interact with the world via tools—calling APIs, running code, or querying databases. However, in traditional agent harnesses, every time an agent waits for a tool to return a result, the entire process holds memory, threads, and an active stack. For systems needing thousands of concurrent agents, this quickly becomes resource-intensive and unsustainable, preventing efficient deployment on cost-optimized serverless platforms.
AWS has tackled this challenge with Infinity, an open-source runtime written entirely in Rust, introduced via the Hydro project. The foundational innovation of Infinity is its use of stackless coroutines, mirroring how modern programming languages like Rust implement asynchronous functions. Instead of keeping the entire state of a waiting agent in active memory, Infinity suspends the agent at the moment it makes a tool call, saves only the necessary conversation history, and only resumes the agent when the tool result message arrives. This model fundamentally shifts how memory is consumed, transforming a multi-threaded, blocking process into an extremely lightweight, message-driven state machine.
The Power of Stackless Coroutines in Agents
In conventional systems, holding an active process for an agent requires maintaining its full call stack in memory, even when idle. In contrast, Rust's async fn compiler transforms functions into state machines, which only require storing the relevant values at each await point. Infinity leverages this mechanism: when an agent calls a tool, its conversation history (the necessary continuation) is persisted, and the agent is suspended. When the tool returns a message, the runtime reloads this minimal state and resumes the agent precisely where it left off.
This approach allows the runtime to treat every tool call not as a blocking operation, but as a suspension point. This design is crucial for achieving high density. AWS benchmarked 80,000 agents running through 3.2 million completions on a single thread of an AMD EPYC 9R14, demonstrating that idle agents occupy about 100 KB of memory. This figure is what allows the theoretical capacity of 75,000 agents to fit within the 8 GB memory capacity of a Raspberry Pi 5, making large-scale, low-cost deployment possible.
Serverless Integration and Event Handling
One of the most compelling aspects of Infinity is its compatibility with serverless architectures, particularly AWS Lambda. Because an agent can be suspended while waiting for a tool call that might take days, the serverless function does not need to remain active for the duration of the wait. The Infinity reference deployment utilizes an SQS FIFO queue as the input queue; each message delivery invokes a short-lived AWS Lambda function that processes one slice of the agent's work and then exits. Conversation histories are managed externally using Aurora DSQL and DynamoDB for state persistence, maintaining the stateless nature of the execution slice.
The runtime utilizes a Reactive Agent Protocol (RAP) and synthetic tool calls to manage asynchronous events. Instead of building a separate, complex event system, synthetic tool calls allow the runtime to present asynchronous events—like user messages or tool results—to the model within the structure already understood by standard LLM APIs (OpenAI, Anthropic). This design ensures message integrity while maintaining a highly decoupled, scalable architecture.
Implementation Details and Open Questions
Infinity is an early research release, not a fully managed service, and AWS is transparent about its current limitations and areas of open research. The core technology is available open-source under the Apache 2.0 license as part of the Hydro project, built using 21 Rust crates.
Despite its impressive metrics, several engineering challenges remain: 1. Scheduling Policies: The runtime currently processes threads in the order messages arrive (first come, first served). Prioritizing critical, user-facing threads over background events is an open research question. 2. Event Authenticity: For agents exposed to the internet, the runtime needs a robust way to establish the provenance and authenticity of events to ensure actions are traceable and accountable. 3. Compaction for Long Histories: While conversation history is stored, studying how to apply compaction (summarizing long histories) efficiently across large, complex agent trees is ongoing research. 4. Model Training Bias: LLMs are primarily trained on synchronous tool traces, and it is not fully understood how interleaved, asynchronous events affect agent reasoning, which is a vital area for future research.
Conclusion: A Framework for Scalable AI
AWS Infinity represents a major step in transitioning AI agent development from monolithic, memory-intensive processes to highly lightweight, scalable, and serverless architectures. By leveraging Rust's stackless coroutines, it solves the fundamental problem of agent latency and resource bloat. While the 75,000 agent figure is based on a controlled, in-memory, simulated benchmark, the architecture proves a viable paradigm for enterprise-scale AI operations. As a research project, it serves as a crucial blueprint for the future of high-throughput, resource-efficient AI systems running on cloud infrastructure, and its open-source nature encourages community feedback and further development, especially around its open research questions.
Sources
- https://aws.amazon.com/blogs/opensource/introducing-infinity-a-runtime-for-extreme-agent-concurrency/
- https://agentopsguide.com/aws-infinity-agent-runtime/
- https://www.datastudios.org/post/aws-infinity-open-source-rust-runtime-75000-idle-ai-agents