Setting Up Prometheus DNS-SRV Discovery with Terraform and DNSimple

If you are using Prometheus to collect metrics from your server, and you don’t have a static set of servers, then you should set up automated discovery.

There are many ways you can set up automated discovery in Prometheus. However, one of my preferred vendor-agnostic ways of doing this is to use DNS-SRV records.

How does it work?

Let’s say you are using Terraform to manage your infrastructure.

You will be creating a DNS-SRV record for each of your servers. As you create or destroy servers, these records will be created and destroyed.

Then, on the Prometheus end, you will configure it to look up these DNS-SRV records and determine the set of targets on the fly.

In Terraform, using DNSimple (referral) only as an example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

resource "dnsimple_zone_record" "SRV_node_prometheus" {
  for_each = toset([
    # Hostnames or IP addresses...
  ])

  zone_name = "example.com"
  name      = "_prometheus_node._tcp"
  value     = "1 9100 ${each.key}"
  type      = "SRV"
  ttl       = 300
  priority  = 10
}

The three fields in the value are weight, port, and target (i.e. hostname or IP address).

Once you have applied the Terraform changes, you can now tell Prometheus to use DNS-SRV discovery in the prometheus.yml file:

1
2
3
4
5

scrape_configs:
  - job_name: node
    dns_sd_configs:
      - names:
        - _prometheus_node._tcp.example.com

Once you have restarted Prometheus with this configuration, it will automatically detect new servers you add to the fleet or remove from it.

This post is 82nd of my #100DaysToOffload challenge. Want to get involved? Find out more at 100daystooffload.com.