Many organizations have well-thought-out, documented processes for managing secrets that work well enough as long as requests are infrequent. In the context of app development, “secrets” might refer to any information that is used by both machines and humans to verify access to restricted environments such as credentials, keys, tokens, and certificates.
When every person on the dev team and any app, software, microservice and container needs to be granted permission to perform a given action, we can understand how important proper “management” can be. As your organization grows and the number of applications, components, cloud endpoints and systems you support increases, the process of managing secrets becomes increasingly complex and time-consuming.
If your team is among the majority of organizations that have begun to adopt a DevSecOps approach, much of this burden falls upon you. It’s easy to become quickly overwhelmed when supporting hundreds of containers and developers to ensure that secrets are managed and distributed securely.
Accessing Secrets for Machines
When developers need secrets for their applications, they must make a request through a defined process, such as creating a ticket or sending an email. The DevSecOps team then provides the requested secrets or updates the developer’s access to the central repository where the secrets are stored.
While this process frequently ensures the security of sensitive information, it can also introduce challenges and complexities in distributing and managing secrets, such as the overhead needed for an infrastructure team to manage thousands of credentials with requests coming in nonstop anytime a change is required.
In an attempt to preserve an organization’s security, engineers who might not be familiar with a specific application end up handling a high volume of identical tasks on a daily basis. And as a rule of thumb, any time there’s a manual process combined with an overwhelming workload, you’re running a high risk of errors compromising the security of sensitive information and your organization’s efficiency.
These seemingly minor errors can have severe consequences in high-stakes deployments of new features. For example, an extra space or a typo can cause a deployment to fail, leading to a slower time-to-market and potentially impacting business operations.
Additionally, as snafus occur and process friction rises, you risk losing the trust of development teams. They never know ahead of time if their secret will work correctly in a production-level environment and may worry about its potential impact on end users. In this case, the secrets are impossible to validate. Your secrets may have worked in all lower testing environments and even an inactive production environment… but when it’s time to deploy changes to production, a failure still occurs.
Guiding Principles
In the aforementioned scenario, the manual process for managing secrets directly opposes the core principles of effective secret management. As a DevSecOps engineer, it’s essential to strive for automation in all aspects of the job, including the distribution of secrets.
To address this challenge, it’s crucial to implement the Zero Trust principle. This means that all access to secrets must be verified and authenticated before granting access… or, as we like to call it, “guilty until proven innocent.” On top of Zero Trust, other fundamental principles, like Just-in-Time credentials and continuous monitoring, are also key to success.
In the fast-paced world of DevOps, machines access information much more quickly than humans do, making automation of secret generation, rotation, and management essential to reducing the risk of human error. Also, real-time monitoring, alerting, and reporting capabilities are essential to quickly detect and respond to any potential incidents or issues related to secrets.
Let’s go back to our example and start to unpack the steps needed to align with industry best practices for secrets management.
For starters, the manual generation of secrets must stop and be taken off the plate of the DevSecOps team. There are a couple ways of doing this, which your organization should consider based on its container infrastructure and security requirements.
Deploy-Time Secrets
These are like access keys or passwords, encrypted and bundled with the application code and stored in the code repository. They are embedded within the code and stored within version control systems like Git.
The idea here is to put control in the hands of developers. They can access a tool to request and generate a production secret without ever seeing the value and by simply using a reference to the value.
Some key advantages of managing secrets for deploy-time injection:
- Solid Security – Parameterized values are stored in the source code, so developers only have access to encrypted values.
- Improved Control – Secrets can be updated or revoked without impacting code.
- Scalability – Secrets are centrally managed, and developers can begin self-servicing the secrets with the right solutions implemented.
Runtime Secrets
Runtime secrets, on the other hand, are encrypted, stored and accessed separately from the code and managed by a secure secret management system. These are used at runtime, like user credentials or API keys.
Some key advantages of managing secrets for runtime injection:
- Just-in-Time Access – This minimizes the risk of stand privileges that are easily exploitable.
- Revoke Access at Any Time – Ensure that sensitive data is only accessed by authorized parties or applications, and that secrets are never hard coded.
- Dynamic – Your secrets can be updated without performing a redeployment of the application.
Using dynamic, parameterized secrets certainly makes sense, whether they’re used upon deployment or in runtime. However, no matter what approach you take, proper use of secrets does involve added layers of complexity. Any time you adopt new tools and processes, and a higher level of developer ownership, you’re necessarily looking at more to manage. You’re also likely to compromise visibility, as monitoring and revocation becomes harder.
There are several effective options available for the transformation of deploy-time and runtime secret management. Akeyless, a SaaS platform, offers centralized management and automation for Just-in-Time credentials, secret rotation and access management. Unlike the open-source HashiCorp Vault, Akeyless offers a more scalable approach while providing a Zero Trust level of security for your secrets as they are not visible / accessible to third-party cloud service providers.
To Summarize
The main difference between deploy-time and runtime secrets is when they are used and how they are stored. Deploy-time secrets are used during the deployment process, while runtime secrets are used while the application runs.
Starting out, your organization might opt for deploy-time secret loading, which means a developer will be able to self-service the deployment of secrets as configuration to relevant containers. In our example, this could be a practical first step as they continue the transformation of secret management.
Then, eventually, DevSecOps teams are often best off transitioning to runtime secrets by identifying all secrets currently stored in code, migrating them to a secret management system, and updating the code to retrieve secrets at runtime.
This process can be complex and time-consuming, but it’s essential to ensure the security and privacy of sensitive information in today’s fast-paced, dynamic development environments.