Let’s Learn About the OTel Operator’s Target Allocator!
Ever heard of the Target Allocator (TA)? Don’t fret if you haven’t. You’re not alone. I certainly hadn’t. That is, not until Iris Dyrmishi mentioned it at one of our recent OTel Q&A sessions, after which, I had to look it up. And it sent my head spinning. Initially, I was SUPER. Duper. Confused.
It took me a lot of Googling and asking TONS of questions to finally wrap my head around it, so I thought I’d share some of my learnings while it was all still fresh in my mind.
First things first: what the heck is the Target Allocator?
The Target Allocator is part of the OTel Operator. The OTel Operator is a Kubernetes Operator that:
- Manages the OpenTelemetry Collector
- Injects and configures auto-instrumentation into your pods.
To help facilitate the above capabilities, the Operator ships with two Custom Resources (CRs): one for managing the Collector, and one for managing auto-instrumentation for your applications.
💥 NOTE: You may recall that I played around a bit with the OTel Collector CR when I created the OTel Operator Kratix promise.
Okay…back to the Target Allocator.
Target Allocator Demystified
As I mentioned earlier, the Target Allocator is part of the OTel Operator, and more specifically, it is an optional component of the OTel Collector Custom Resource (CR).
In a nutshell, the TA is a mechanism for decoupling the service discovery and metric collection functions of Prometheus such that they can be scaled independently. The OTel Collector manages Prometheus metrics without needing to install Prometheus. The TA manages the configuration of the Collector’s Prometheus Receiver. If you follow my work, y’all might remember that ditching Prometheus from the Observability stack is my Observability dream (sorry, Prometheus folks).
The TA serves two functions:
- Even distribution of Prometheus targets among a pool of OTel Collectors
- Discovery of Prometheus Custom Resources
Even Distribution of Prometheus Targets
The Target Allocator’s first job is todiscover targets to scrape and OTel Collectors to allocate targets to. Then it can distribute the targets it discovers among the Collectors. The Collectors in turn query the Target Allocator for Metrics endpoints to scrape, and then the Collectors’ Prometheus Receivers scrape the Metrics targets.
This means that the OTel Collectors collect the Metrics instead of a Prometheus scraper.
✨Target: An endpoint that supplies metrics for Prometheus to store
✨ Scrape: The action of collecting metrics through an HTTP request from a targeted instance, parsing the response, and ingesting the collected samples to storage.
Discovery of Prometheus Custom Resources
The Target Allocator’s second job is the discovery of Prometheus CRs, namely the ServiceMonitor and PodMonitor. The ServiceMonitor and the PodMonitor don’t do any scraping themselves; their purpose is to inform the Target Allocator (or Prometheus, if that’s your jam) to add a new job to their scrape configuration. The Target Allocator then adds the job to the OTel Collector’s Prometheus Receiver’s scrape configuration.
In order for this setup to work in our Prometheus-free world, however, the ServiceMonitor and PodMonitor must be installed in your Kubernetes cluster. These CRs are normally installed via the Prometheus Operator.
Wait…But I just said that we can skip the whole Prometheus thing with this setup, so what gives?
Well, this is where a little hack comes in. Although you normally get these CRs by installing the Prometheus Operator, you can actually install them sans Operator. You can do this by going into the Prometheus Operator’s Releases page, grabbing a copy of the latest bundle.yaml file, and pulling out all that pesky YAML except the ServiceMonitor and PodMonitor YAML definitions. Ta-da! 🪄✨
Final thoughts
I have to admit that the OTel Operator repo docs on the Target Allocator were a bit confusing, so after writing this blog post, I also wanted to make sure that the docs were a bit clearer for newbs such as myself. Check out my PR to update the docs, which has just been merged! 🌈
I hope y’all learned something new and cool with this! There’s obviously a LOT more to dig into with respect to the Target Allocator, but hopefully this gives you an idea of what it’s all about, and who knows…maybe you’ll be inspired to check out the Target Allocator now! If you’d like to learn more about the OTel Operator, you should check out #otel-operator channel in the CNCF Slack. The folks on there are super helpful and responsive.
Before we part ways, be sure to check out the OTel Q&A sessions…they’re suuuuper informative and fun! If videos aren’t your jam, you can check out the blog post summaries of the OTel Q&A sessions via the OTel Blog.
Now, please enjoy this photo of my dear rat Mookie, who is generally VERY difficult to photograph, because she’s always on the move. Shout-out to my husband for snapping this rare pic of her sitting still. 💙
Peace, love, and code! ✌️💜👩💻
PS: Massive shoutout to my co-worker Jacob Aronoff, who is also one of the OTel Operator maintainers, for helping me wrap my head around this topic, and answering all of my questions!
Want to learn more about OpenTelemetry? Check out my other OTel content here:
