Secrets and Configuration: Vault, Env, Key Rotation in Practice

Why Secrets Become a Headache for Engineers

Every application, whether a cloud microservice or an enterprise CRM, handles sensitive data: API keys, database passwords, SSL certificates. Sooner or later the team faces the same problems: secrets leaking into the repository, difficulty rotating them without downtime, and a missing audit trail. These risks directly impact operational security and product reputation, especially in cloud infrastructure.

Cloud development at ESK Solutions is initially built with strict configuration protection requirements in mind. Next, we'll examine how to structure secrets management without compromise using modern tools and practices.

Centralized Storage and Vault

Scattered .env files and hard-coded values in the code are the main source of leaks. The only way to take control is to implement a centralized secrets management service. The most mature solution remains HashiCorp Vault: it not only stores static secrets but also generates dynamic credentials, encrypts data on the fly, and access policies allow granular permissions down to the role of a specific application.

• Dynamic secrets. Instead of a single long-lived password, Vault creates temporary logins for each application session. Upon expiration, access is automatically revoked—so even if such a token leaks, it becomes irrelevant.
• Encryption as a service. The transit engine mechanism allows encrypting/decrypting data through an API without worrying about keys and algorithms. Especially convenient for sensitive fields in CRM systems where personal data needs protection.
• Integration with orchestrators. Vault Agent or sidecar containers deliver secrets directly into Kubernetes pods without requiring code changes. This is the foundation of secure SaaS development where configuration isolation between tenants is critical.

Example architecture: an application requests a temporary client certificate from Vault, using authentication via a Kubernetes Service Account. This approach eliminates storing long-term credentials anywhere outside Vault.

Environment Variables: Not Evil, But a Tool with Caveats

Many teams out of habit load secrets through env variables—it's fast and familiar. However, in cloud deployments, this approach quickly becomes problematic: variables are inherited by child processes, end up in memory dumps and logs, and version-controlling .env files is not a great idea.

The right compromise: use environment variables only as transport for runtime, where they are populated not from a file but from a storage like Vault or directly from Kubernetes secrets. For web applications, this means that the container at startup receives values via an entrypoint script that queries Vault and exports them to the environment of only the current process, preventing leaks into journald or child processes.

Another important point—masking. Even if secrets come through the environment, the logging platform should automatically filter sensitive patterns: card numbers, tokens, private keys. A tool like Vault Audit Logs helps track which application and when accessed a specific secret.

Key Rotation Without Service Downtime

Manually replacing certificates and passwords in the middle of a workday is the surest way to trigger an incident. Automated rotation, configured via the central storage API, solves this seamlessly. Main scenarios:

• Database credential rotation. Vault can act as an intermediary between the application and the DBMS: create new login-password pairs on a schedule and update them in the application via lease renewal. Upon connection failure, the application automatically gets fresh credentials—downtime is eliminated.

The critical requirement is "zero-downtime rotation." This is achieved by having the old and new secrets coexist during a grace period. For example, when changing a DB master key, the old password remains valid for an additional 10 minutes until all pods restart with the new value.

Audit and access monitoring

It's not enough to distribute secrets—you need to understand who uses them and when. Vault audit devices allow detailed logging of every request: authentication, secret request, lease renewal. These logs are streamed to a SIEM system, where alerts are configured for anomalies: mass requests from an unknown host, attempts to access forbidden paths, unusual activity times.

For cloud deployments, you can add an observability layer through metrics: how many dynamic secrets are active at a given moment, average response time to requests, number of policy-denied requests. This information helps both operations and information security. In ESK Solutions cloud development projects, we often implement ready-made Grafana dashboards for monitoring Vault, giving the DevSecOps team full transparency.

Frequently asked questions

What to choose: store secrets in a .env file or switch to Vault immediately?

A .env file is acceptable only during local development and should never be committed to a shared repository or production environment. For test and production environments, a central Vault is the only solution. Even in small projects, it can be deployed in a minimal configuration, providing instant protection against leaks and simplified rotation.

How often should you rotate keys?

There is no universal rule, but good practice is: certificates every 24–72 hours, DB passwords daily, external API access keys according to provider requirements (often every 30–90 days). Automation through Vault allows this without human intervention, so you can set a more aggressive policy without increasing the team's workload.

Can you do without Vault, using only Kubernetes secrets?

Kubernetes secrets solve the basic task of delivering values to pods, but they do not provide dynamic secrets, encryption as a service, detailed audit, or a unified management window for multi-cluster environments. In large cloud projects, Vault complements Kubernetes Secrets: the vault manages the lifecycle, and the orchestrator only transports ready values. This combination gives maximum security with minimal overhead.

How to organize Vault backup and what happens if the master unseal keys are lost?

Unseal keys are the "master keys to all safes," and their loss is catastrophic. Practice recommends using Auto Unseal with cloud KMS (AWS KMS, GCP Cloud KMS), as well as storing backup key shares with security officers. Vault data backup is done through the standard snapshot mechanism; they are encrypted with the same key as the storage, so it is safe to store such copies in a separate cloud bucket.

Conclusion

Operational security begins with discipline in secret management. Centralization through Vault, automatic rotation, and transparent audit turn configuration data from a source of problems into a controllable asset. Whether it's a cloud SaaS, CRM, or high-load web portal, the rules are the same: no hardcoded secrets, every token's life is finite, and access is logged. The ESK Solutions team helps establish these processes at the start of the project, so you don't have to put out fires later due to a leaked production key.