At JFrog, we rely on Kubernetes and Helm to orchestrate our systems and keep our workloads running and up-to-date. Our JFrog Cloud services had initially been deployed with Helm v2 and Tillerless plugin for enhanced security, but we have now successfully migrated our many thousands of releases to Helm v3.

Like many SaaS service providers, JFrog Cloud runs with many Kubernetes clusters in different regions, across AWS, Azure and Google cloud providers.

We learned some important lessons along the way that I’m happy to share.

Why Migrate to Helm v3

While the first version of Helm v3 was released in November, 2019, Helm v2 continued to be supported with updated versions for another year. But with the final version of Helm 2.17.0 in November, 2020, Helm v3 is now the only standard supported by the Helm developer community.

Helm v3 offers some major improvements, most notably the removal of Tiller. This in-cluster server to interface with the Helm v2 client required administrator privileges to perform its duties, which was considered to be a security risk in shared K8s clusters. This could be overcome with the Tillerless plugin, but Helm v3 makes this no longer necessary.

In addition, Helm v3 offers some new features and greater stability. It’s now also the only version that will receive future updates for effectiveness and security.

Strategies for Migration

To make migration of your clusters from Helm v2 to v3 much easier, the Helm developer community has created the helm-2to3 plugin for use with the helm3 client. A Helm blog post provides some good information on how to use it.

Installing the plugin is simple:

$ helm3 plugin install https://github.com/helm/helm-2to3

But how you go about the task next will likely differ depending on the number of releases you need to migrate.

Manual Migration

If you’re only needing to migrate a few releases, you can do so one-by-one through the client on the command line.

First, you can use the helm list command to list all deployed or failed releases for the current namespace:

$ helm2 list
NAME        REVISION    UPDATED                     STATUS      CHART               APP VERSION NAMESPACE
postgres    1           Thu Nov 14 15:01:00 2019    DEPLOYED    postgresql-7.1.0    11.5.0      postgres

You can then perform the conversion of each named release through the migration plugin. We suggest using the --dry-run flag first to be certain the conversion will run cleanly.

$ helm3 2to3 convert --dry-run postgres
$ helm3 2to3 convert postgres

You can repeat this procedure for all your releases, and you are done!

Automating Migration for the Enterprise

To migrate any more than a handful of Helm v2 releases to v3, you’ll want to automate the process with a shell script.

Your script will need a list of all the releases that need conversion. You can generate a list using the Helm v2 client, in this case to a file named releases.log.

$ helm2 tiller run -- helm2 ls --max=0 | sed -n '1!p' | awk 'FNR > 1 { print $1 }' > releases.log

That produces fast results for a relatively small number of releases, up to around 200. Much more than that, however, it will take the Helm client ages to fetch them all. Additionally, I have encountered Kubernetes API limits with AWS EKS clusters.

JFrog Cloud services run thousands of Helm releases per Kubernetes cluster, so an alternative, faster method was needed. As all Tiller secrets are labeled we discovered we can use the kubectl command to fetch them into a releases.log file:

$ kubectl -n kube-system get secret -l OWNER=TILLER,STATUS=DEPLOYED | awk 'FNR > 1 { print $1 }' | cut -f1 -d"." > releases.log

Using our list of Helm v2 releases in releases.log, we can automate the migration step with a bash script:

#!/bin/bash
# Get releases list
RELEASES=$(cat releases.log )
for r in $RELEASES
do
  echo
  echo "Processing release $r"
  helm3 2to3 convert --tiller-out-cluster --release-versions-max=5 \
   --delete-v2-releases --dry-run 2>&1 | tee -a convert.log
done

In this script, you may want or need to change these flags:

--tiller-out-cluster is used if you do not have Tiller running in the Kubenetes cluster; it should be removed if Tiller is installed.

--delete-v2-releases deletes Helm v2 releases when they get migrated to Helm v3

--dry-run is used to test that migration script works first. Omit when performing the actual migration.

If you choose to omit the flag --delete-v2-releases and keep Helm 2 releases, you can cleanup them later with:

$ helm3 2to3 cleanup --tiller-out-cluster --releases-cleanup --skip-confirmation <release_name>

The script captures migration results in a file, convert.log that you should review for any migration issues that may have been encountered.

In our migration of JFrog Cloud services, not all releases were on the same chart version -- they used the charts that were valid at the time they were first deployed. So some older releases that were migrated failed to be upgraded with Helm v3.

The problem was that some Helm v3 labels and annotations had not been added to migrated Kubernetes objects. This was easy to fix by adding them to the Helm upgrade step when a check showed that they weren’t present:

$ kubectl -n ${NAMESPACE} label deployment -l "app.kubernetes.io/instance=${RELEASE}" "app.kubernetes.io/managed-by=Helm"

$ kubectl -n ${NAMESPACE} annotate deployment -l "app.kubernetes.io/instance=${RELEASE}" "meta.helm.sh/release-name=${RELEASE}" "meta.helm.sh/release-namespace=${NAMESPACE}"

Happy Migrating :-)

That’s all it takes to migrate your releases to Helm v3! The process is simple, but keep in mind that it won’t necessarily be fast. When there are thousands of releases -- as there will be in most enterprise-scale organizations -- the migration process really takes time to complete.

Using these steps, you can create an automation tool that can help you migrate a large number of releases running in Kubernetes from Helm v2 to Helm v3, and keep your Kubernetes infrastructure up-to-date.