Why I’m still not using Hashicorp Vault inside Kubernetes

I have now, for the 3rd time, explored using Vault for secrets management inside Kubernetes.

Up until recently Kubernetes dashboard did not have the ability to edit any values inside the cluster. This made secret management particularly difficult because to get the secret you had to do something like this:

kubectl get secrets mysecret -o yaml

Then you had to run

echo "c29tZXNlY3JldAo=" | base64 -d

Just to see the secret.

If you want to edit the secret you have to do like this

echo "agreatsecret" | base64

Then run

kubectl edit secret mysecret

and manually update the data table with the base64 encoded secret value

And then there is of course all kinds of other types of secrets all of which are base64 encoded.

Vault seemed like a great tool to add user interface for application secrets.

Vault Helm chart

In August 2019 Hashicorp released a helm chart for vault. Yet on the date of this article you cannot find it in the helm hub.

Installation of the chart is easy enough

helm install vault --name=vault --set='server.dev.enabled=true'

So now I’m supposed to do something different in production which is like to manually unseal it. Ok whatever that means. I’ll just use dev mode for now.

Cool so now I can run

kubectl port-forward svc/vault-vault-internal 8200:8200

Cool dashboard. Great how do I login…?

Oh well actually the root key was lost because it was generated during installation. I guess I can generate a root key? Gosh just for development!? How do I disable this login thingy?

Ok can’t figure that out so I’ll just create some secrets… On the command line? Oh on the command line inside the pod.

kubectl exec -it vault-vault-0 -- /bin/sh

Sadly bash is not loved. Good old shell will have to do

vault kv put internal/database/config username="db-readonly-username" password="db-secret-password"

Nice so now I have a database secret and password which I still cannot view via UI. Maybe I’ll figure out the UI later.

Getting the secrets into your pod

Ok here’s basically where I took a step back and said to myself “Wait a minute. What am I actually gaining with Vault?” There are apparently capabilities in Vault to autorotate keys but that implies that for example you update the database password in the database and update the database password in Vault which is pretty cool if automated. However, in the example given above though we are just setting a static username and password. No rotations.

This tutorial describes even more steps I must take

• Create a service account
• Create a policy for each and every service that is going to access the key I just created
• Create an auth token that Vault is going to use? Idk
• Update my application to read the secret from the file system rather than environment variables going against 12 factor app ideals.

All this for what? To store a static username and password that I can only see by executing shell commands? Oh, also we will know which pod accessed the secret every time it’s accessed 😐. But how does that help exactly if every pod is using the same secret? The point of doing this is to have some kind of audit trail if a security breach occurred, but its pretty useless if everything is sharing a secret with a key-value pair type secret.

Is Vault trying to entice me to create an even more complex secrets management solution?

Ok recently there was a Vault Backend added to kubernets-external-secrets library. That means that the external secrets manager would sync a key-value pair to a Kubernetes secret which could then be mapped into an environment variable. That’s definitely better but again, if its a shared secret do I actually get any benefit by using Vault? Ok maybe I can use the secrets rotator which might be useful when I get to that point. If that’s the case though do I gain any benefit now by using Vault without secrets rotation?

The simple answer is no. I gain no benefit by using Vault in Kubernetes without also using secrets rotation.

Yes I know you can say I need to do the more complicated thing, but realistically this is already extremely complicated.

If you make security hard, developers are going to skip it.

Unless of course you are a towering corporation that is fully willing and able to make developers’ lives miserable.

Also, where’s the infrastructure as code? Why can’t I generate secrets automatically? Do I have to do all these manual steps every time I prop up a new cluster?

A more direct approach?

Suppose instead of using this mapping of environment variables to secrets and k8s secrets to vault secrets or the sidecar injection of a secret file, I just fetch the secrets directly from my app using a library like node-vault. Well then I still have to watch for changes, or regularly poll the secret to see if its updated, or implement a webhook when changes are made. Then I have to exit the process in the container or re-connect the database when I do detect a change.

Sigh.

Summary

I really like the things that Vault is trying to do, but I think Vault in its present state is for like military grade applications with enough bureaucracy available to spend long hours on implementing obscure security features. And while that might sound sexy to you as a solutions architect or whatever you are, it is not sexy for the average developer in your company that will spend a very long time trying to figure it out without much success.

And of course if your next solution is “We’ll build an abstraction to make it easier for those stupid developers,” well therein lies the main problem. Unless you are a developer on the Vault team, building an abstraction that is able to accomplish the goals of Vault and be easy for the average developer is going to be very time consuming and may not yield as much benefits as you would like. Even so, whatever you build will have to be supported by your internal team and, like most innovations, will eventually become technical debt you have to rip out later on.

Since, I already have basic secrets management that supports environment variables natively in Kubernetes which I can update through a UI easily via Kubernetes Dashboard, there’s really no reason for me to use Vault in the recommended manner.

TLDR

At this stage of Vault’s evolution, in my opinion, Vault provides less value for basic secrets management than the Kubernetes dashboard does due to its complexity. And the extra layers of security that it provides for complex secrets management is just not worth the effort or maintenance at the early stages of any project.

Sadly the best time to implement a secrets management solution is at the early stages of a project. Changing it later on will be a huge undertaking.