Accessing Azure Key Vault Secrets in Azure Kubernetes with Secrets Store CSI Driver

Published in

Dzero Labs

10 min readNov 28, 2020

One bad day away from Kubernetes despair

Most days, I really enjoy working with Kubernetes. I really do. But far too often, I find myself in a situation whereby I am thoroughly lost in a deep black hole. The black hole of, “Hey, I want to try this Kubernetes-native thingamaroo and the vendor’s docs make it look soooo easy to follow.” And then it turns out that nothing f*cking works, and you’re stuck troubleshooting for days, obsessing over the fact that this seemingly simple thing refuses to work. WORK, GODDAMMIT!! Then your mind is flooded with thoughts of, “This is it. THIS is the one problem that I will never ever ever be able to solve.” And suddenly, the heavens open up, and the angels shine a guiding light on the solution. By a stroke of luck, I happen to glean over the obscure log message or GitHub issue post that gives some glimmer of a hint as to why the f*ck this stupid example from the vendor’s GitHub repo isn’t working. At that point, all of the colour returns to my face, and my heart goes from palpitating, to beating normally.

Has that ever happened to you? Because it has happened to me countless times, and it’s especially true in the wonderful world of Kubernetes.

The Trigger: Kubernetes Secrets

Now, you may be wondering what on earth brought on this mad rant. Fair question. It all started when I decided to explore better alternatives for managing secrets in Kubernetes in Azure this week. Kubernetes does have a Secrets resource for storing passwords, SSH keys, certificates, and all that sensitive stuff; however, as Gaurav Agarwal points out, “the problem with Kubernetes secrets is they store sensitive information as a base64 string. Anyone can decode the base64 string to get the original token from the Secret manifest.” In an Enterprise environment, that won’t do.

I nixed the idea of HashCorp Vault, because various articles had pointed out how much of a pain in the ass it is to set up. Plus, I was working on AKS, so I wanted to see if I could find something that could leverage secrets stored in Azure Key Vault.

I was originally going to dive deep into Azure Key Vault to Kubernetes, but then a serendipitious Google search led me to Azure Key Vault Provider for Secrets Store CSI Driver, by Microsoft itself. Hard to pass up.

Azure Key Vault Provider for Secrets Store CSI Driver maps a Kubernetes resource called SecretProviderClass to an Azure Key Vault, and lets you select which of that Key Vault’s secrets, keys, and/or certificates you’d like to expose. You can then mount a volume on a Pod to access those secrets by referencing the SecretProviderClass. To make sure that you don’t have some nefarious character accessing your Key Vault, you use an Azure Managed Identity, and assign it permissions to access your Key Vault on Kubernetes’ behalf. To use the Azure Managed Identity to authenticate your Key Vault, you also need AAD Pod Identity installed on your cluster.

You basically end up with a setup like this:

Seems relatively simple, right? Well, getting this to work properly took me two days of non-stop work and poring over GitHub repos, blog posts, GitHub issues, StackOverflow posts, YouTube videos, and whatever else I could get my hands on until…I finally cracked the nut.

The thing is, on the most part, the instructions and videos were actually pretty decent. And yet, I could not get this blasted thing to work. And I finally landed on the problem!

Actually, there were two problems. First, the Helm charts used to install the Azure Pod Identity CRDs didn’t match up with the non-helm deployment YAMLs. I went the pure YAML route initially, only to end up with some broken shit in my cluster. Once I solved that problem, I kept getting stuck on an error whereby the logs were claiming that a value in one of my YAML manifests didn’t exist. It was then that I realized that the version is of the Azure Pod Identity YAML manifests that I was using had the wrong case for a bunch of the fields. It took me two hours to chase down the problem.

Wut. The. F*ck.

Skeletons in the Closet

That was when I realized the brittle nature of this Kubernetes stuff. Actually, it’s not just Kubernetes, but Infrastructure-as-Code (IaC) in general. The problem is, I might have solved a problem for setting up thing-a-ma-whatchit-x using a set of APIs and CLI commands one day, but then a day later, some breaking change may be harmlessly introduced without much fanfare, causing my beautiful house of cards setup to topple. That actually happened to me this past August, and it took me a good couple of hours to realize that a set of Kubernetes CRDs that I’d installed in my cluster on Monday stopped working on Tuesday, when I decided to nuke and recreate my cluster before a demo. It turned out that I was referencing the latest package (rather than a versioned package) for my installs.

Which leads me to my point. IaC is great and all, but it all goes to hell because it’s not treated more respect.

To make IaC less brittle, it should follow these principles:

1- Repeatable

I should not be scared of creating and destroying my infrastructure many times over, because I know that I’ll get the same results each time

2- Ephemeral

It should be all or nothing. None of this maintaining state crap, where you’re accumulating layers upon layers of changes. If you’re making changes to your infrastructure, you nuke the old infra and re-create it with the new stuff. Don’t add on to the old infra, because you have no idea what Franken-infra you’re creating.

3- Simple

The minute you start getting overly-elaborate with your infrastructure, where you find yourself trying to jam a square peg in a round hole, STOP. For example, if you find yourself manipulating Terraform state files, it’s time to back up, buddy, and start from scratch.

4- API-Based

Your infrastructure should be designed and documented like an API. This means proper versioning and error-checking, so you don’t get weird-ass error messages from the ether. For example, when I encountered that case-sensitivity issue with my YAML, that should’ve been caught when I tried to apply it to the cluster, by some sort of validation. Not AFTER the fact. Similarly, I should, by default, be forced to specify a particular version of a CRD that I’m installing to my cluster, rather than have the Helm charts default to the latest.

A little bit of standardization goes a long way. Much better than documenting steps that can soon become outdated.

I’ll still give you a solution: Azure Key Vault Access via Azure Key Vault Provider for Secret Store with Azure Pod Identity

For what it’s worth, I still want to post a solution here, in case it helps someone down the road, even if Microsoft ends up introducing breaking changes to subsequent updates to Azure Key Vault Provider and AAD Pod Identity, which will inevitably happen.

Assumptions:
1. You have an Kubernetes cluster running in Azure (AKS)
2. You’ve installed Helm 3 on your local machine
3. You’ve created your AKS cluster using a Service Principal

You can also see the source code in the accompanying GitHub repo here.

NOTE: If you run into a pickle when installing AAD Pod Identity and/or Secrets Store CSI Driver for Azure, check out some alternate ways to install the CRDs here.

1- Set up Env Vars & Login to Azure

Fill out the values in the file below, and save it locally.

Don’t run this on its own. It will be used in the next few steps.

If you’re wondering what your Azure Subscription ID is, you can get it like this:

az login -u <username> -p <password>
export AZ_SUBSCRIPTION_ID=$(az account show --query id -o tsv)

Now that you’ve got all the info to fill out your environment vars above, let’s login to Azure and make sure your Kubernetes cluster is added to your kubectl file:

. ./0-env_vars.shaz login -u $AZ_USERNAME -p $AZ_PASSWORDaz aks get-credentials --resource-group=$RESOURCE_GROUP --name=$AKS_CLUSTER_NAME --overwrite-existing

2- Install Secrets Store CSI Driver for Azure v0.0.10 in Kubernetes

Use Helm 3 to install version 0.0.10 of the Secrets Store CSI Driver:

helm repo add --insecure-skip-tls-verify csi-secrets-store-provider-azure https://raw.githubusercontent.com/Azure/secrets-store-csi-driver-provider-azure/master/chartshelm repo updatekubectl create ns csi-driver# Helm chart v0.0.6 installs Secrets Store CSI Driver for Azure v0.0.10
helm install csi-secrets-store-provider-azure csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --version 0.0.6 -n csi-driver

3- Install Azure Pod Identity v1.7.1 in Kubernetes

This allows us to use an Azure Managed Identity to authenticate from k8s to the Azure Key Vault.

Use Helm 3 to install version 1.7.1 of the AAD Pod Identity:

helm repo add --insecure-skip-tls-verify aad-pod-identity https://raw.githubusercontent.com/Azure/aad-pod-identity/master/chartshelm repo updatekubectl create ns aad-pod-id# Helm chart v3.0.0 installs AAD Pod Identity v1.7.1
helm install aad-pod-identity aad-pod-identity/aad-pod-identity --version 3.0.0 -n aad-pod-id

4- Create a Key Vault in Azure

The script below will do the following:

Create a Resource Group in Azure
Create a Key Vault in the Resource Group
Grant the given user ID permissions on the keys and secrets in the Key Vault
Create 2 secrets in the Key Vault

To run the script:

./1-keyvault_creation.sh

5- Create the Managed Identity & Set Permissions

Create an Azure Managed Identity. We need to give it the following permissions:

Permission to access our Key Vault
Permission to access the secrets and keys in our Key Vault

Since the cluster was created with a Service Principal, we must also grant the Service Principal Managed Identity Operator and Virtual Machine Contributor roles.

To run the script:

./2-aks_setup.sh

6- Configure the AzureIdentity & AzureIdentity Binding Resources

The AzureIdentity resource was created as part of the Azure Pod Identity installation in Step 3. It references the Azure Managed Identity created via az identity create in Step 5.

The AzureIdentityBinding resource was created as part of the Azure Pod Identity installation in Step 3. It references the Azure Managed Identity created via az identity create in Step 5.

It serves as a glue to bind the AzureIdentity to a Pod. This is done via the selector field. This field is refernced as a label in a Pod’s definition.

You’ll need to configure your own AzureIdentity and AzureIdentityBinding resources as follows:

AzureIdentity

The resourceID field references the id field of the Managed Identity created above. Get the id by running the command below:

az identity show --resource-group $RESOURCE_GROUP --name $AZ_ID_NAME --query id -o tsv

The clientID field references the clientId field of the Managed Identity created above. Get the clientId by running the command below:

az identity show --resource-group $RESOURCE_GROUP --name $AZ_ID_NAME --query clientId -o tsv

AzureIdentityBinding

The azureIdentity field refers to the name given to the AzureIdentity resource.

The selector field can be whatever you want; however, the same value must be referenced in your Pod definition as a label. More on that later.

We won’t be applying this manifest to Kubernetes just yet.

7- Configure the SecretProviderClass

The SecretProviderClass resource was created as part of the Secrets Store CSI Driver installation.

It points your Key Vault, and allows you to specify the keys, secrets, and certificates that you want to expose.

The usePodIdentity field must be true so that SecretProviderClass can use the Azure Managed Identity (created in Step 5 and referenced in the AzureIdentity spec) to give Kubernetes access to the Key Vault.

The useVMManagedIdentity and userAsignedIdentityID must be false and ””, respectively.

Additional Fields:

keyvaultName is the name of the Key Vault being accessed.
resourceGroup is the name in which the Key Vault resides
subscriptionId is the Key Vault’s subscription ID
tenantId is the Azure Tenant ID
objects lists the secrets/keys/certificates from the specified Key Vault in keyvaultName to be made available to the Pod.

Again, we won’t be applying this manifest to Kubernetes just yet.

8- Configure the Deployment

Our Deployment references the Key Vault secrets by mounting a volume containing the secrets. The volume definition points to the SecretProviderClass defined in Step 7, which gives us access to the secrets that we exposed there. But how do we ensure that we have access to the secrets referenced in the Key Vault? That’s where the label aadpodidbinding: pod-id-binding comes into play. You may recall that we defined pod-id-binding in the selector field of the AzureIdentityBinding YAML (see Step 6).

We won’t be applying this manifest to Kubernetes just yet.

9- Putting it all together

NOW, we can apply all of the YAML files to Kubernetes. You can use the script below.

Run the script using the command below, making sure that the Kubernetes manifests are in the same directory as the above script:

./3-run_sample.sh

You’ll notice that if all goes well, you should see a few things:

When you run kubectl get AzureAssignedIdentities, you’ll see a reference to our 2048-game Pod.
When you run kubectl -n az-keyvault-demo exec -it $(kubectl -n az-keyvault-demo get pods -o jsonpath=’{.items[0].metadata.name}’) -- ls /mnt/secrets-store, you’ll see a listing of the secret names (not values, because they’re secret!) that you exposed in Step 7

Conclusion

IaC is cool, but it’s a real bitch if it doesn’t adhere to software engineering principles. Remember:

Keep your IaC code simple
Make your infrastructure ephemeral
Make sure that your infrastructure creation is repeatable
Treat your IaC code like an API — well-designed and well-documented

If you stick to these principles, you’ll save yourself and others many headaches many times over.

Thanks for putting up with my rants. Here are some penguins for you to enjoy.

Peace out.