Agent Framework Gets a Real Memory Boundary

Long-term memory is where many agent demos quietly stop being architecture and start being a pile of callbacks. The model receives a transcript, maybe a vector search result, maybe a user profile summary, and the application code is left to decide what should be remembered, when it should be extracted, and whether the next session should see it.

Microsoft's September 4, 2026 preview integration for Microsoft Agent Framework and Azure Cosmos DB is useful because it makes that boundary explicit. The new Python package, agent-framework-azure-cosmos-memory, adds CosmosMemoryContextProvider, a context provider that participates in the agent lifecycle, stores turns, extracts durable memories, and retrieves relevant facts, summaries, and profiles in later runs.

This is not a new model announcement. It is more interesting than that. It is a storage and lifecycle decision for engineers trying to make agents behave consistently after the process restarts, the user opens a new conversation, or the workload moves from a toy notebook to a multi-tenant service.

What Happened

On September 4, 2026, the Microsoft Agent Framework team announced a preview Azure Cosmos DB memory integration for Python agents. The package exposes CosmosMemoryContextProvider, which plugs into an Agent Framework agent through the existing context_providers collection. Microsoft is clear about the preview status: the package is Python-only today, and APIs may change before general availability.

The integration builds on the Azure Cosmos DB Agent Memory Toolkit, which Microsoft had already published as a public preview Python SDK. The toolkit stores raw conversation turns and derived memory in Azure Cosmos DB, then makes that memory searchable with semantic retrieval. The PyPI page for azure-cosmos-agent-memory lists the latest toolkit release as August 17, 2026 and requires Python 3.11 or later.

The current Microsoft Learn documentation separates two Cosmos-backed context patterns. If you need exact transcript persistence, use CosmosChatHistoryProvider for .NET or CosmosHistoryProvider for Python. If you need extracted long-term knowledge, use CosmosMemoryContextProvider, currently for Python. That distinction matters. Transcript storage and semantic memory are not the same feature, even if teams often blur them under the word "memory."

Microsoft also published a deployable sample, foundry-cosmos-memory, that provisions a Foundry account and project, chat and embedding model deployments, Cosmos DB for NoSQL, RBAC, and a Foundry Agent Service prompt agent. The sample makes a useful design point: conversation identity and memory identity are separate. A new Agent Framework session can start a clean conversation while the same stable user_id still lets the agent recall previously extracted memories for that user.

Why It Matters

Most agent memory failures are not retrieval failures. They are ownership failures.

A team adds RAG for documents and calls it memory. Another team stores every chat message forever and calls it memory. A third extracts user preferences into a profile table with no deletion policy, no confidence threshold, and no tenant isolation story. Each version works in a demo. Each becomes uncomfortable when the system has real users, regulated data, and agents that can take actions.

The Microsoft integration is worth paying attention to because it pushes memory behind a framework extension point instead of treating it as a tool the model may or may not call. In Agent Framework, a ContextProvider runs around each invocation. Before the model runs, it can add relevant context. After the run completes, it can observe the conversation and update storage. That is exactly where long-term memory belongs.

Putting memory in the lifecycle has several practical benefits.

First, recall becomes consistent. The agent does not need to decide, in natural language, whether to call a "remember" or "search memory" tool. The provider can retrieve relevant memory before every run and inject it into context according to application policy.

Second, extraction can move out of the response path. The provider stores the turn, then the memory toolkit can extract facts, produce summaries, and update user profiles in the background. For production systems, that means the memory pipeline can evolve separately from the latency budget of the user-facing response.

Third, identity is no longer implicit in a chat thread. A stable user_id can carry memory across sessions. Without that stable ID, Microsoft says the provider scopes memory to the current session. That fallback is safer than accidentally leaking cross-session data, but production applications should derive user, tenant, and session identifiers from authenticated application identity, not request payloads.

Finally, this gives engineers a clearer place to discuss policy. Which memory types are retained? What gets a time-to-live? Which regions are allowed? What is redacted before persistence? Can extracted memories influence authorization decisions? Microsoft Learn explicitly says not to use extracted memories directly for authorization decisions. That warning is important because LLM-derived memory is evidence, not authority.

How It Works

The architecture has three layers.

Agent Framework owns the agent loop. It calls context providers before and after runs. Your application still owns the authenticated user, tenant, and session model.

CosmosMemoryContextProvider adapts that lifecycle into the Cosmos-backed memory system. Before a run, it searches for relevant memories and adds them to the model context. After a run, it stores the new conversation turns and starts memory processing.

The Azure Cosmos DB Agent Memory Toolkit owns the storage and processing pipeline. It can keep raw turns and derived memory such as facts, procedural knowledge, episodic memories, thread summaries, and user profiles. Its public API includes sync and async clients, CosmosMemoryClient and AsyncCosmosMemoryClient, plus processor backends.

The default processor runs in-process. That is the right shape for prototypes, local development, and low-throughput applications because it avoids extra infrastructure. For fleet or higher-throughput use, the toolkit supports a DurableFunctionProcessor marker. In that mode, the SDK writes turns, and a sibling Azure Function app picks them up from the Cosmos DB change feed. There is no SDK-to-Function HTTP call in the current design.

Cosmos DB is doing more than acting as a blob store for transcripts. The supporting documentation describes vector, full-text, and hybrid retrieval from the same database. That matters because agent memory retrieval is not always pure nearest-neighbor search. A user preference, a procedural rule, and a recent episodic event may need different retrieval behavior, filtering, and confidence thresholds.

The provider also exposes operational controls that experienced teams will care about: top_k, min_confidence, selected memory types, custom extraction prompts through Prompty templates, extraction cadence, deduplication cadence, and an explicit flush() path so pending background extraction can complete before shutdown.

The security model is also visible in the examples. The Microsoft Agent Framework post and README use DefaultAzureCredential, so local development can use az login, while production can use managed identity. The deployable sample disables Cosmos DB local authentication and uses Microsoft Entra ID instead of account keys or connection strings. That is the right default for agent memory because the storage layer may contain sensitive user facts, not just disposable chat logs.

Example

The minimal shape is small, which is the point. You create the memory provider, attach it to the agent, and set a stable memory identity in the session state.

import os

from agent_framework import Agent
from agent_framework.foundry import FoundryChatClient
from agent_framework_azure_cosmos_memory import CosmosMemoryContextProvider
from azure.identity.aio import DefaultAzureCredential


async def build_agent():
    credential = DefaultAzureCredential()

    memory = CosmosMemoryContextProvider(
        cosmos_endpoint=os.environ["COSMOS_ENDPOINT"],
        cosmos_database=os.getenv("COSMOS_DATABASE", "ai_memory"),
        foundry_endpoint=os.environ["FOUNDRY_ENDPOINT"],
        embedding_model=os.getenv("EMBEDDING_MODEL", "text-embedding-3-large"),
        chat_model=os.getenv("CHAT_MODEL", "gpt-4o-mini"),
        credential=credential,
        top_k=5,
        min_confidence=0.7,
        memory_types=["fact", "procedural"],
    )

    agent = Agent(
        client=FoundryChatClient(
            project_endpoint=os.environ["FOUNDRY_ENDPOINT"],
            model=os.getenv("CHAT_MODEL", "gpt-4o-mini"),
            credential=credential,
        ),
        instructions="You are a support agent with scoped long-term memory.",
        context_providers=[memory],
    )

    return agent, memory, credential

A production request handler should not accept user_id from the prompt. It should derive it from the authenticated principal and tenant context.

async def run_turn(agent, memory, principal, text):
    session = agent.create_session()

    provider_state = session.state.setdefault(memory.source_id, {})
    provider_state["user_id"] = principal.user_id

    result = await agent.run(text, session=session)
    return result.text

That design gives you a clean separation:

• The session is the conversation container.
• The authenticated user ID defines memory scope for cross-session recall.
• Tenant boundaries stay in your application, resource, and partitioning design.
• The provider decides what memory enters the model context.
• The toolkit stores turns and derives higher-value memory asynchronously.
• Cosmos DB enforces the persistence and retrieval substrate.

For a support agent, this means a user can start a new conversation and still have the agent remember durable preferences or prior issue summaries. For an internal developer agent, it means the system can remember team-specific procedural rules such as "always open a change request before touching production Terraform" without stuffing every old conversation into the context window.

The hard part is policy, not plumbing. You still need retention rules, deletion flows, regional placement, redaction, and tests that prove memory isolation across users and tenants. You should also test false memory extraction. If the model incorrectly extracts "the user approved destructive changes" from an ambiguous conversation, that memory should not become operational authority.

My Take

This is the direction agent frameworks need to move. Memory should be a runtime concern with explicit lifecycle hooks, identity scope, and storage policy. It should not be a magical chat feature or a casual vector-search sidecar.

The most important design choice here is that memory is not exposed primarily as another model tool. Tool-based memory can work, but it asks the model to participate in its own persistence discipline. That is the wrong default for production. The framework should retrieve policy-approved memory before the run, persist candidate material after the run, and let the application define identity and retention boundaries.

I also like the split between conversation history and long-term semantic memory. Transcript persistence is useful for resuming a session exactly. Semantic memory is useful for extracting stable facts, preferences, procedures, and summaries. Mixing them leads to wasteful context construction and weak compliance controls. A full transcript may need short retention. A derived preference may need a user-facing correction path. An episodic memory may need a default TTL. These are different data products.

The preview limitations are real. Python-only support means .NET teams can persist history but cannot use the same long-term semantic provider yet. The package is pre-GA, so API shape may shift. And Cosmos DB plus Foundry plus embeddings is still an Azure-centered stack; teams committed to another cloud or a self-hosted vector database will not adopt it blindly.

But the pattern is portable. Every serious agent runtime needs a memory interface that can answer four questions: who is this memory for, what type of memory is it, how was it derived, and when should it be forgotten? The Cosmos integration gives Microsoft Agent Framework a concrete answer for Azure teams and a useful reference design for everyone else.

My bias would be to deploy this conservatively. Start with low-risk facts and procedural preferences. Set confidence thresholds. Keep episodic memories on a TTL. Redact before persistence. Keep extracted memories out of authorization decisions. Add an admin and user inspection path before memory becomes part of a critical workflow. Agents that remember are more useful, but they also create a new class of stale, inferred, and sensitive state.

Conclusion

Microsoft's September 4 preview of CosmosMemoryContextProvider is a meaningful agent engineering release because it treats memory as part of the agent lifecycle. The provider retrieves relevant memory before a run, stores turns after a run, and delegates extraction and retrieval to a Cosmos-backed toolkit that supports derived memory and hybrid search.

The practical lesson is not "use Cosmos for everything." It is that long-term agent memory needs a real boundary. Put it behind a framework extension point. Scope it by authenticated identity. Distinguish transcripts from extracted knowledge. Process memory outside the critical response path when possible. Treat every remembered fact as data that needs governance.

That is less glamorous than a model launch, but it is the kind of plumbing that determines whether agents can safely operate across sessions, users, and teams.

Sources

• Native memory for Microsoft Agent Framework with Azure Cosmos DB
• Microsoft Learn: Azure Cosmos DB context providers for Agent Framework
• Microsoft Agent Framework: Azure Cosmos DB Memory README
• Azure Cosmos DB Agent Memory Toolkit
• PyPI: azure-cosmos-agent-memory
• Foundry Cosmos Memory sample