Kubernetes helm gotchas that will drive you mad

Like all 1st generation package managers for an ecosystem, they usually are riddled with bugs and problems. helm is no different. The only issue is, like at one point we had no choice other than NPM, right now we have no other choice than helm. It’s been 2 years since we have been warned to think twice before using helm and we still are running into the same problems caused by the the helm team’s attitude towards developers.

One day a Yarn for Kubernetes will come along and the helm developers will finally have to start listening to the community like the NPM developers had to start listening to their community, but that day is yet to be seen.

Some great tricks for templating workarounds have been written to solve the Kubernetes code re-use problem. Kustomize is even touted an escape from HELLm. helm is supposed to be more than just a templating tool, its supposed to be the missing package manager for Kubernetes. Unless you want to write and maintain all your Kubernetes software installs from scratch, you’re stuck with helm.

So in order to save you from head-desking-hair-pullers, here are 6totally un-obvious gotchas you had better be aware of.

1. Resources with a pre-install hook will not be installed at all if you are adding those resources to an existing release

In NPM there is a concept of a “pre-install” hook which means it will run before the package is installed. If you add a dependency that has a pre-install hook, that dependency will run the pre-install hook when you run npm install or yarn install. So with NPM you actually have to run npm install or yarn install in order for those hooks to run. I won’t even get into the other stupid things that npm install does but in this behavior it is similar across most package managers.

helm is no different. If you don’t run helm install then the resources with pre-install hooks will not be installed before everything else is installed. The problem though is that they will not be installed at all if they were not already installed when you ran helm upgrade . Imagine adding a NodeJS package as a dependency and that package had files in it which would only be installed when you ran yarn install and those files must be installed for the package to work. This is basically what will happen if you add a dependency to your Chart.yaml and that dependency has resources with a pre-install hook. You will never be able to install that resource unless you delete your existing helm release and re-install everything. The reason why is that you cannot run helm install on a release that has already been created. The only other workaround to this problem as of now is to run helm upgrade with the --debug -dry-run options, find the rendered resource which was not installed, then copy paste the output into a file and manually install it with kubectl apply -f .

You also could open a bug report on the repository that has the pre-install hook and ask them to add pre-upgrade hook to the resource. It really does make you wonder though, if every place that has a pre-install hook should also have pre-upgrade hook to address this problem what’s really the difference between the two?

2. helm 2 charts that don’t work with helm 3

helm is in the middle still of doing a major upgrade from version 2 to version 3. One of the biggest changes is a separation of CRD’s to a separate directory so that the CRDs are installed before the resources. Overall I think this was an important update and the right way to go. You still may come across helm repositories that still have the crd’s inside the templates directory. If you do and you are using helm 3. There is just no way you’re going to make it work. There used to be a practice of running pre-install hook for CRD’s but that proved to be a problem when running helm upgrade so they kind of abandoned the approach and separated into different phases. CRD phase and Regular Templates phase.

If you already have helm 2 resources installed in a cluster then you can use the helm 2to3 plugin but this still doesn’t fix the fact that your chart dependencies are still using version 2 layouts. You will have to remove those dependencies all together after migrating to version 3 until those dependencies become version 3 compliant. If you still need to use those dependencies you will have to just copy paste the source code into your own chart and make the changes yourself.

3. Some helm charts have CRD’s that the chart depends on in a separate helm chart.

One of the best examples of this is the Istio helm chart. When helm had a lot of problems with CRD’s they separated the CRD’s into 2 separate helm charts. This was a major pain because you couldn’t install any system from scratch in a single command that had istio as a dependency. You had to first run the istio crds install, tap your fingers and wait for all the crds to install. Then you had to install the chart that contained istio. To this day they still have 2 separate helm charts even though helm 3 has addressed the CRD’s issue.

Yes sure they have “documented it” and you should “READ THE DOCUMENTATION” but its really painful and inconvenient to have to do this when you are managing a lot of environments for a lot of applications.

4. Overridden sub-chart values cannot be used in parent chart templates if the sub-chart name contains a hyphen or underscore.

Say what!? So imagine someone has a chart called pizza-store and you have another chart called mall

Suppose in my mall chart I have a template that will show a list of all the stores in the mall. The pizza-store has a values property called storeName . So in the parent value.yaml file I can change the store name of the sub-chart by specifying like this:

values.yaml :

pizza-store:
  storeName: "Biggies NY Slices"

Now suppose I have a template for a configmap in my parent chart that will contain all the store names of the stores in the mall, I would want to do something like this.

configmap.yaml :

data:
  pizza-store-name: {{ .Values.pizza-store.storeName }}

But I cannot do that because Go Templates don’t allow - to be in the variable name. The most frustrating thing here is that because the hyphen is actually part of the name of a sub-chart nothing can be done to fix the issue. This will actually happen a lot more often than you think when you start composing charts that have - in the chart name. The only known workaround to this issue is to have 2 configurations with the same value:

values.yaml :

pizza-store:
  storeName: "Biggies NY Slices"
pizzaStore:
  storeName: "Biggies NY Slices"

So its definitely not a good practice to have - or _ in your chart names even though helm allows you to do this. The best alternative is to use some kind of camel case which charts names that contain multiple words. Personally I very much dislike camel case in folder names but it’s really all we’ve got.

Update:

Some of my better informed colleagues pointed out that another solution to this issue is to use index like so:

data:
  pizza-store-name: {{ index (index .Values "pizza-store") "storeName" }}

5. Dependencies that were removed will still be installed if you don’t delete the .tgz archive

If you add a dependency to your Chart.yaml then run

helm dep up

To download the tgz file

Remove the dependency. Then run

helm dep up

The .tgz archive will not be deleted from your charts directory. When you run

helm install
helm upgrade

That dependency which is not in the Chart.yaml or Chart.lock will still be installed because there is still a .tgz file in your charts directory.

6. Regardless of what bugs you find in helm, they will almost always be labeled as a feature.

We make jokes about “that’s not a bug, that’s a feature” but the jokes are based in truth. Rather than acknowledge the problems, for some reason the Kubernetes community as a whole tends to just dismiss every issue that is found as “working as intended” and that includes the helm developers as well. The only exception to this is the Minikube project which I have found has an exceptional group of developers that take responsibility for bugs and don’t try to dismiss every bug as a feature.

If you browse the helm open issues today you will find that most issues have been labeled “question/support” and if you look at the closed issues, wow are you in for a surprise!

So before you go and think of opening a bug report on the helm repo, do yourself a favor and do anything except that. You will be a happier person because of it. Just figure out the workaround and pray for the day when helm is a long forgotten memory.