Just-in-time Nomad: A Beginner’s Guide to Understanding HashiCorp Nomad
With the popularity of microservices on the rise, having a way to manage these microservices, from deploying them to making sure that the right ones can talk to each other, has become crucial in today’s distributed environment.
The most popular solution out there is Kubernetes. Heck, it’s so popular that most of the major Public Cloud providers offer some sort of managed service for spinning up and maintaining Kubernetes clusters. And while Kubernetes appears to be the container orchestration tool of choice for many organizations, it’s not the only one out there, as I discussed in my intro to Kubernetes post.
HashiCorp Nomad is one such product. If you asked me six months ago to even consider looking at Nomad, I would’ve told you, “Meh. Why bother? The world has chosen Kubernetes.” Nomad felt like the container orchestration Betamax to Kubernetes’ VHS. But recent circumstances have compelled me to dig a little deeper into Nomad, and what I’ve discovered is that Nomad is a fascinating product that definitely warrants attention.
My goal with this post is to share my learnings with you about Nomad, to give you a high-level overview of the product and its capabilities. Coming from a Kubernetes world, I’ve been trying to grasp Nomad in the best way I can – by drawing parallels between Kubernetes and Nomad. And while the two products are not straight 1:1 equivalents, it has helped me to wrap my head around how Nomad works.
I’m not here to convince you to ditch Kubernetes for Nomad. Instead, I hope to share what I’ve learned with you so that you’re at least aware of the fact that yes, there’s more to container orchestration than Kubernetes, and it’s an alternative worth understanding.
Let’s get started!
High-Level Architecture: One Binary to Rule Them All
Nomad architecture is pretty freaking elegant. The coolest thing about it is that all you need to get started is a single binary.
The same binary that is used to set up a Nomad cluster can also be used to run Nomad locally on your machine. Let that sink in. 🤯
But wait…there’s more! The binary also includes the following:
- Nomad CLI
- Nomad management console
When you’re running the Nomad binary locally on your machine, you can start up Nomad by running this command in your terminal:
nomad agent -dev
And once it’s up and running, you can hit up the management console by running this command in a separate terminal window:
Which gives opens up a lovely management console running on localhost (
Go on…try it…you know you want to! You can download the binary here.
That’s waaaay less painful than trying to run Kubernetes locally. To do that, you can use MiniKube, microk8s, or kind. I tried MiniKube once, and never got it working after messing around with it for a couple of hours. After that, I just spun up a cluster in Google cloud whenever I needed one. Maybe you’ve had better luck than me!
Beyond Local Dev
Unlike Kubernetes, Nomad doesn’t have a Cloud-managed offering. While HashiCorp does have a Cloud-managed offering called HashiCorp Cloud Platform (HCP), it’s currently only available for Consul, Vault, and Terraform. Kubernetes definitely has the edge on that one, with all major Cloud providers offering some sort of managed Kubernetes solution.
The good news is that setting up Nomad clusters is relatively straightforward. Compare that to building your own Kubernetes cluster from scratch, which can best be described as an exercise in self-flagellation and should be avoided unless you really just want to understand the nitty-grittiest of Kubernetes.
Now, let’s get back to the Nomad binary, shall we? As I mentioned before, the same binary that you can use to run Nomad locally is actually used in running a Nomad cluster.
A Nomad cluster is made up of Servers and Clients. To set up a Nomad Server, you run the Nomad binary in Server mode. To set up a Nomad Client, you run the Nomad binary in…wait for it…Client mode.
Servers are machines that are responsible for cluster state and scheduling jobs. They are similar in concept to the Kubernetes Master node. HashiCorp recommends running 3 or 5 Servers per cluster (region in Nomad speak) to ensure high availability. One of the servers is elected leader. If the leader goes kaput, one of the other servers is elected as the new leader.
Servers run the Nomad binary in server mode.
Clients are machines that run jobs, and are responsible for lifecycle management. They’re similar to Worker nodes in Kubernetes.
Here’s where it gets interesting. While Kubernetes is used to orchestrate containerized workloads, Nomad can run all sorts of workloads (known as tasks), which include but are not limited to:
This is made possible by the Task Driver. The Task Driver is similar to the Container Runtime in Kubernetes, except that it can run more than just containers. And, as of Nomad 0.9, the Task Driver is pluggable, so whatever workload you can dream up (as long as it follows the specification, of course), Nomad would be able to handle it.
Clients run the Nomad binary in client mode.
So we’ve got a decent feel for the Nomad architecture. Let’s get into the good stuff — deploying applications!
Applications are defined in a file called the job file. This file is written in HCL — HashiCorp Configuration Language. Nomad also speaks JSON, but most people write job files in HCL. The job file is similar to a Kubernetes manifest (written in YAML), except that whereas you can define individual YAML files to describe a Kubernetes application, you define your entire Nomad application in a single job file.
Some fun facts about job files:
- Job files contain all the things you need to define your application. This includes, tasks, ingress configuration, what client to run on, etc.
- A task is the thing that you’re running (e.g. containerized app, Java app, binary, virtualized app).
- A task is the smallest unit of deployment in Nomad
- The smallest unit of deployment in Kubernetes is the Pod
- A Task Group represents a set of tasks that go together
- When defining a task, you specify: number of instances, task driver (e.g. docker vs Java vs ?), image, CPU resources (in MHz), and memory resources
- Nomad Task != Kubernetes Pod
- Nomad Task ~= individual container in a Kubernetes Pod
- Nomad Task Group ~= Kubernetes Pod
If you’ve followed my Kubernetes work before, you know that I’m a huge fan using the Dockerized 2048 app for my examples. I now have a Nomad example for you as well!
For the sake of comparison, I’ve also included a Kubernetes deployment manifest:
Getting a quick example up running locally with Nomad is so simple, that I decided to throw in a short little tutorial. You can find the jobspec code here.
1- Start up Nomad in Dev mode
Open up a new terminal window, and run this command:
nomad agent -dev -bind 0.0.0.0 -log-level INFO
2- Deploy the Nomad jobspec
Open up another terminal window, and run the following command (assuming your file is called
2048-game.nomad and is located in the current working directory):
nomad job run 2048-game.nomad
You’ll see output that looks something like this:
3- Open up the Nomad UI to take a peek at your job deployment.
Run the following command in a new terminal window:
This will open up a new browser window at
4- Run the app
We exposed port
80 in our jobspec file, so we should be able to reach our app at
For a more detailed Nomad tutorial, check out this Medium post by Russ Parsloe.
Nomad != Kubernetes
One thing that I learned early on in my exploration of Nomad, is that by itself Nomad != Kubernetes.
Nomad is actually not meant to run all by itself, unless you’re running it locally. When you’re talking about running a Nomad cluster (like in Pre-Prod and Prod environments), you actually need a setup that includes running Nomad, Vault, and Consul. All of which are…wait for it…HashiCorp tools! Which means that they play super nice together. ❤️
The diagram below illustrates what it looks like to set up Nomad with Consul. You have:
- Consul Servers (similar setup to Nomad Servers), running on separate machines
- Consul Clients running as agents on the Nomad Client machines
As with Nomad, the Consul binary can be set up to run in either Client mode or Server mode.
Don’t they play so well together? 😄
So, when you put it all together, you get:
Nomad + Consul + Vault = Opinionated Kubernetes
I say “Opinionated Kubernetes” because by nature, Kubernetes very much embodies the choose-your-own-adventure mentality when it comes to things like secrets management and service mesh.
Maintaining a Nomad cluster and keeping it up-to-date is a relatively painless process. Relative to Kubernetes, that is. This is because:
- Updating Nomad is a matter of replacing the binary and restarting the service.
- Nomad strives to be backwards compatible for at least 1 point release. This means Nomad v0.10 hosts will work with v0.9 hosts.
- Supports both in-place updates (update binary on existing hosts) and rolling updates (add new hosts with new binary, and remove old hosts with old binary)
For more on Nomad upgrades, check out the Nomad docs.
Compared to maintaining your own self-hosted Kubernetes cluster (don’t do it unless you have a really good reason), this is a walk in the park. Needless to say, maintaining self-hosted Kubernetes clusters involves (among other things) keeping the various Kubernetes components up-to-date. This, on top of also keeping up with security patches on your underlying servers — something you also have to do with Nomad, FYI.
Nomad & Kubernetes Side-by-Side
There’s so much else to talk about, but rather than bore you to death with a wall of text, I have put together a handy-dandy table comparing Kubernetes and Nomad. While things aren’t always 1:1 comparisons, this should at least give you a pretty good idea of what’s up.
Since Nomad terms differ from Kubernetes terms, I also put together a terminology decoder ring below.
Worth A Look
So…which product is better? Nomad over Kubernetes? Honestly, it’s a personal/enterprise choice. It depends on:
- Available expertise. Do you have more in-house Kubernetes or Nomad experts?
- Appetite for managing Nomad clusters. Remember, it also includes having to keep your VMs up-to-date and patched, update the binaries…all that good stuff!
Nomad is definitely a serious contender in the (not just) container orchestration space, and has a lot going for it:
- Relatively easy to set up and upgrade a self-hosted Nomad cluster (installation of single binary on each server), compared to Kubernetes (installation of multiple services)
- Nice integration with Vault for secrets management
- Nice integration with Consul for service mesh
- Not just for containerized workloads!
- Relatively easy to pick up, coming from a Kubernetes background
On the cons side, we have:
- No managed Nomad solution
- Not as much expertise out there compared to Kubernetes
- Not as much flexibility, as you’re “stuck” using Vault and Consul
- Can be pricey for meeting Enterprise needs, since you have likely have to invest in Enterprise offerings of Nomad, Vault, and Consul
- Lacks in parity with Kubernetes offerings, specifically around Container Storage Interface (CSI) integrations
Kubernetes certainly gets a lot more press in the container orchestration space compared to Nomad, so you likely won’t find as many folks versed in Nomad compared to Kubernetes. But, the Hashi community gets some serious love and fandom, so you certainly won’t see a lack of enthusiasm. Plus, SREs I’ve worked with who came from Kubernetes-land and were thrust into Nomad-land have all told me how easy Nomad was to pick up. That’s definitely nothing to snort at!
A final note: I want to make sure that the info here is accurate, so if you find any boo-boos in this post, please send me a quick note to let me know, so that I can make corrections as needed!
And now, please enjoy this lovely picture of my recently departed pet rat and forever furry friend, Susie:
Peace, love, and code.
More from Just-in-Time Nomad
Just-in-Time Nomad: Running Traefik on HashiQube
Deploying ThoughtWorks Tech Radar to Nomad using the Traefik Load Balancer
Just-in-Time Nomad: Configuring HashiCorp Nomad/Vault integration on HashiQube
A step-by-step guide to configuring HashiCorp Nomad and Vault to allow Nomad to pull Vault secrets
Just-in-Time Nomad: Running the OpenTelemetry Collector on Hashicorp Nomad with HashiQube
An in-depth look into the Nomad OTel Collector jobspec using Traefik as a load balancer and pulling API keys from Vault
References & Further Reading
- A Kubernetes User’s Guide to HashiCorp Nomad
- Nomad Reference Architecture VM with Consul
- Upgrading Nomad
- Get Started: Nomad for Kubernetes Practitioners (with Demo)
- Conductor: Why We Migrated from Kubernetes to Nomad
- Nomad vs Kubernetes without the complexity
- Why you should take a look at Nomad before jumping on Kubernetes
- To self host or to not self host Kubernetes cluster(s)
- HashiCorp Nomad — From Zero to WOW!