If you are using Prometheus to collect metrics from your server, and you don’t have a static set of servers, then you should set up automated discovery.
There are many ways you can set up automated discovery in Prometheus. However, one of my preferred vendor-agnostic ways of doing this is to use DNS-SRV records.
How does it work?
Let’s say you are using Terraform to manage your infrastructure.
You will be creating a DNS-SRV record for each of your servers. As you create or destroy servers, these records will be created and destroyed.
Then, on the Prometheus end, you will configure it to look up these DNS-SRV records and determine the set of targets on the fly.
In Terraform, using DNSimple (referral) only as an example:
The three fields in the value are weight, port, and target (i.e. hostname or IP address).
Once you have applied the Terraform changes, you can now tell Prometheus to use DNS-SRV discovery in the
Once you have restarted Prometheus with this configuration, it will automatically detect new servers you add to the fleet or remove from it.
comments powered by Disqus