Credential Management: The Hidden Production Bottleneck for Agentic AI on Kubernetes

Agentic AI has moved from experimental curiosity to a production imperative. Organizations are deploying AI agents that don’t just answer questions but take actions: querying databases, updating records, orchestrating workflows, and provisioning infrastructure. These systems are no longer confined to innovation labs and are increasingly embedded in core business operations. The question is no longer whether to deploy agents, but whether organizations are prepared to operate them responsibly and securely.

The answer, increasingly, is Kubernetes. As agents transition from standalone demos to workflow-centric systems handling thousands of concurrent operations, the industry is converging on agentic architectures where agents operate as coordinated services within Kubernetes clusters — bringing horizontal scaling, self-healing, and GitOps-driven deployment. 

But this convergence has exposed a critical gap that many organizations only discover after moving beyond the proof-of-concept stage: credential management. While building agents has become increasingly accessible, operating them securely at scale remains a formidable challenge. In many environments, this gap becomes visible only after the first serious security incident.

The Credential Problem Is Worse Than We Thought

Recent security research demonstrates the vulnerability. In a documented ethical hack against a live AI agent deployment, a researcher harvested multiple production credentials in minutes using basic tools such as cat, grep, and find. No advanced exploits were required. No sophisticated malware. Just ordinary access to the runtime environment.

The default approach is Kubernetes Secrets: base64-encoded values stored within the cluster. For development, this works. For production, it breaks down quickly. Kubernetes Secrets lack dynamic rotation and are duplicated across environments, leading to configuration drift. 

Open agent frameworks compound the problem — in projects such as OpenClaw, credentials are frequently embedded directly in configuration files, prompts, or agent memory. Once that happens, every agent becomes a potential breach point, and secrets propagate through systems in ways that are difficult to track or control.

Why “Just Use a Secrets Manager” Isn’t Enough

Secrets manager platforms address many of these gaps — dynamic rotation, centralized management, fine-grained access policies, and full audit trails. On paper, these platforms should solve the problem. In practice, integration is often incomplete, inconsistent, or postponed entirely in fast-moving projects, creating architectures that are fragile, difficult to govern, and nearly impossible to audit holistically.

We need a solution that integrates secrets management into the agent workflow itself, not one that depends on each agent being individually coded to comply. Credential management must become part of the platform, not a responsibility pushed onto individual developers.

The Model Context Protocol Changes the Equation

The Model Context Protocol (MCP), first proposed by Anthropic and now a Linux Foundation open-source project, has emerged as the standardization layer for how agents interact with external platforms and services. It uses a client-server design in which MCP client interact with language models while MCP server communicate with tools.

While MCP solves the interface problem, it also creates a natural integration point for solving the credential problem. An MCP server tied to a secrets manager can broker just-in-time, short-lived credentials — requesting scoped tokens or leases on behalf of the agent and passing them to downstream tool calls. In some designs, the MCP server acts as a proxy, executing the tool action itself so raw secrets never reach the agent. Either way, agents can now access external tools in a safer, scalable, and auditable way without hardcoding or long-term credentials.

Instead of trusting every agent to behave correctly, the platform enforces security by design.

Identity: The Missing Foundation

Still, two underlying problems remain: identity and authorization. How do we know an AI agent is the entity it claims to be? The agent cannot simply provide a hardcoded token, as this creates a “bottom turtle” dilemma — who issued that token, who trusts the issuer, and where trust ultimately begins. 

This is where the Secure Production Identity Framework for Everyone (SPIFFE) becomes relevant. SPIFFE provides standardized workload identity through verifiable credentials issued after attestation. These credentials can take the form of X.509 certificates or JWTs (or JSON Web) Tokens for machine-to-machine authentication.

AI agents running in Kubernetes already meet the prerequisites. A SPIFFE Runtime Environment (SPIRE) agent on each node verifies workloads and issues credentials to trusted entities. Agents can then authenticate to downstream systems using mutual TLS or token-based authentication.

SPIFFE provides a root of trust that most agent deployments currently lack.

Authorization Still Matters

Identity alone is not enough. Knowing who an agent is does not determine what it should be allowed to do.

Secrets managers can centralize this logic. By mapping SPIFFE identities to role-based policies, they enforce granular access control and eliminate the need for agents to manage their own permissions.

Converging Security Secrets Managers

Secrets managers are increasingly adopting SPIFFE and modern identity standards, making them a natural control point for securing agent-based AI architectures. Combined with MCP, they form an enforcement layer where authentication, authorization, rotation, and auditing converge. 

Rather than scattering security logic across hundreds of agents, organizations can manage trust centrally and consistently. This reflects a shift toward treating secrets management as core infrastructure.

The Path Forward

For platform engineers, site reliability engineers, and DevOps teams responsible for operationalizing AI workloads, credential management is where many agentic initiatives stall.

These agents provision infrastructure, modify production data, and trigger automated workflows. A single exposed credential is no longer a localized incident. It becomes systemic risk.

By integrating external secrets management with Kubernetes through MCP and grounding identity in SPIFFE, organizations can move beyond fragile prototypes and build agentic AI systems that are secure, auditable, and scalable.

A Hard Truth

The agent isn’t the problem. As long as teams continue embedding credentials in prompts, configuration files, or agent memory — whether through convenience or speed — agentic AI will remain one breach away from failure. The technology is ready. But technology alone is not enough. What determines resilience is a platform-centric approach where agent identity is continuously enforced — not configured once and forgotten. Automated credential rotation, workload attestation, policy-as-code, and drift detection aren’t optional features. In mature environments, compliance is a continuous property of the platform, not a periodic audit.

Rod Anam, Lead Site Reliability Engineer at Kyndryl, is a co-author of this article.