Stenio Ferreira | December 30, 2016
Managing secrets using Hashicorp Vault
The complexity of managing secrets
Managing secrets throughout the lifecycle of an application is not a trivial task. If you’re too restrictive with access, implementations can become too complex. But if you have a lax approach, you’ll increase the risk of a security breach. Other concerns should also be addressed while managing the secrets:
- How to handle external and internal threats (like hackers and disgruntled employees)
- How to securely share secret management responsibility among a set of stakeholders
- How to allow seamless access to both humans and services
- How to audit who accessed a secret at a certain date and time
Hashicorp Vault is an open source secret management and distribution tool that proposes an answer to these and other questions. In this blog post, I’ll introduce the technology and provide a walkthrough—through a Proof of Concept example—describing how to install, initialize, and access the secrets using token authentication. This walkthrough provides an example closer to enterprise reality than the one found in the official tutorial, while not going into details of more complex implementations.
Hashicorp Vault is developed and maintained by our friends from Hashicorp, famous for making delightful and popular tools like Vagrant, Consul, and Terraform. Hashicorp Vault follows the same guiding principles as the company’s other solutions, resulting in a product that is simple to use due to its modularity, yet powerful and flexible in how it can be implemented. There are many ways of describing Hashicorp Vault’s features, but Seth Vargo, one of its maintainers, summarizes it in this interview by citing four main pillars:
- Mitigation of internal and external threats: Allowing Vault to be sealed/unsealed in response to crises, and supporting a variety of security backends.
- Dynamic secrets: Vault is able to generate credentials automatically for different backends.
- Lease renewal/revoke access to secrets: Vault allows you to specify TTL rules for a secret, enabling fine control over permissions.
- Auditing: Once enabled, every secret request will be logged, with the output directed to a file or to syslog.
Vault offers modular plug-in for three main areas: encrypted secret storage, authentication controls, and audit logs:
- Secret storage: This is the solution that will “host” the secrets. Available backends include AWS S3, Consul, Generic (file storage), among others.
- Authentication controls: The authentication mechanism used. Available backends are AWS EC2, LDAP, Github, Tokens, Username/password, and others.
- Audit logs: Where logs are sent. Available File or syslog. The specific mix of backends will depend on your project’s requirements and constraints, and more than one backend can be enabled. Additional information on each backend, including API request commands to access them, can be found in the official documentation.
Accessing secrets using token authentication and response wrapping
One possible scenario while using Vault is when you have a set of clients that are previously defined and need access to secrets. For example, in order for Mary the developer to access a secret using her Github account, a Vault manager would need to enable the GitHub backend, and based on the associated policy, Mary would only need to authenticate to Vault using her GitHub credentials. A similar scenario would take place for sample-app-service, which has an AWS IAM-based service account and requires access to certain secrets.
However, recently I worked on a client project that had a different requirement. Using a combination of Jenkins, Chef, and Microsoft’s Hyper-V manager, a json file was used to describe VMs that should be dynamically created in the Hyper-V host. Given the dynamic nature of these VMs, and the client’s preference for non-cloud solutions, it would be less than ideal to have to hardcode VM names in advance, in order to create Vault access credentials.
In this section, I’ll describe a proof of concept (PoC) using Vault’s token authentication backend to address the above scenario. Three participants will work in this PoC:
- A VM where Vault is installed
- A VM representing the client that wants access to a secret
- My workstation, which will be acting as a Trusted Entity. The Trusted Entity will be responsible for interacting with most of the Vault API, and passing the secrets to the end consumer. In the real world, this role can be played by Jenkins, Terraform, or another CI/Orchestration tool.
One critical problem that this PoC will address is how to pass the access token from the Trusted Entity to the client VM, while lowering the risk that it could be intercepted along the way. This will be accomplished by using Cubbyhole’s response wrapping—a feature that allows a Vault response to be associated with a single use, time-limited token, stored within Vault. The Trusted Entity will only receive this token, and when the client VM uses it to retrieve the access token, it will expire—ensuring that the access token was only exposed once.
Here's a sequence diagram of what we plan to accomplish:
Once both are installed, open the cmd prompt and execute:
This will start the Hashicorp Vault VM and the Client VM that needs access to the secrets.
Before you can execute instructions against Vault, like creating secrets and access tokens, you need to initialize it. Run the following in the command prompt to connect to Vault using Vagrant:
Once inside Vault, execute the following to initialize Vault with a given configuration file:
This will output five unseal tokens, which need to be used to “unseal” or “seal” the Vault. Secrets can only be accessed when Vault is “unsealed,” and the configuration used for initialization defaults to minimum of three tokens to allow unseal.
In a real world scenario, these tokens would be distributed among key stakeholders, guaranteeing distributed responsibility for secrets management. They should be stored safely in the same way as personal passwords.
There will also be an additional token in the output –this is the root access token, with full permission to Vault. It should only be used for initial configuration, while recurring operations should be done by policy-constrained tokens.
With the tokens available, execute this command three times, entering one of the provided unseal tokens:
Before performing any operations on an unsealed Vault, you need to first login. This can be accomplished by executing:
As a best practice, we will enable auditing log:
You can also check the default secrets folders by executing:
This will return /cubbyhole, /secrets and /sys, with descriptions of their intended use. For this PoC, we’re going to use all three of them.
The git repository includes three access policies, with the idea of mapping an enterprise’s environment secrets to different folders. First we create the policies by executing:
Now that we have the policies in place, let’s add some sample secrets to each of the mapped environments:
Create an access token associated with a policy
Here’s where we’ll create the token that the Node VM will use to communicate with Hashicorp Vault. Since this will be created by the Trusted Entity (i.e. Jenkins), we’ll use the wrapper pattern to prevent anyone other than the Node VM itself and Vault from knowing the access token. By passing the -wrap-ttl parameter during token creation, Vault returns a Cubbyhole wrapped token, with a specified, single use time-to-live. The idea is: once it’s used, it returns the access id token and lease, and can’t be used again.
This creates an access token and lease with “production” policy applied to it, and stores this information on the Cubbyhole.
If additional control is desired, a few other parameters can be used with token-create:
- explicit_max_ttl: Sets the time after which this token can never be renewed.
- num_uses: Number of times this token can be used. Default is unlimited.
- renewable: If token is renewable or not. Default is true.
- ttl: Time token is valid. Default is 720hs.
Send single-use token to VM
You can take different approaches to sending the single-use token to the Client VM. In our PoC, we’ll do a simple remote copy by copying the “wrapping token” value from the above output, opening another terminal window and issuing:
Client VM gets access token
Now that the Node VM has the single access token. Open a third console window and execute:
If returns permission is denied, token is either expired or has been compromised. Trusted Entity should be notified to contact Vault to revoke that token and create a new one.
If successful, it will return the output:
In the background, this has assigned the following json to the ‘RESPONSE’ variable:
To parse this json response, you can use regex or the jq tool command line json processor.
This will store the access token in the ACCESS_TOKEN variable in memory.
Client VM communicating with Vault
Now anytime Client wants to communicate with Vault, it should use the value in the $ACCESS_TOKEN environment variable, for example:
This will return the json.
This token has ttl and will expire in 720hs. In order to keep alive, at regular intervals the Client VM must issue.
Lease renewal is only possible if token still valid. If expired or revoked, notify Trusted Entity and go back to the instructions described in the “Create an access token associated with a policy” section.
Congratulations! You now have a functioning Hashicorp Vault implementation using token authentication.
Where to go from here
- Distribute unseal tokens securely using Keybase or PGP, as documented here.
- Put vault behind a SSL web server proxy, as described here.
- Explore the different Authentication and Secret Storage backends.
- Support High Availability mode by integrating Vault with Consul, Zookeeper or etcd.
- Enable dynamic secrets, such as the AWS example described in the official documentation.
Looking for more engineering tips?
Our engineers have a whole lot to say about custom software. They’re in the trenches every day, building, breaking, re-building, and sharing their hard-won wisdom along the way. Find their latest and greatest discoveries on Slalom’s new software engineering blog.
Stenio Ferreira is a solution architect in Slalom’s Chicago office. He’s passionate about helping companies introduce DevOps best practices to their internal processes, like creating CI and CD pipelines, describing infrastructure as code, and executing cloud migrations. Follow him on Twitter: @stenio123.
Stenio Ferreira is no longer with Slalom.