Prometheus metrics are being worked on by me, and will hopefully soon be merged in Uncloud.

The uncloudd metrics endpoint is exposed into the cluster network, so this makes it possible for a Prometheus service running inside the cluster to scrape it. For this it just needs to be able to access the API to get all machines in the cluster. This can be done by mounting the uncloud socket inside the container, and using uc with structured output.

This will need another container inside the Prometheus service that will get the machine data from the cluster every few minutes and write a prometheus.yml with file service discovery.

Later this can be extended to probe for other services as well, but that might need labels, so Prometheus can see what services need to be scraped, and labels are not supported for any else than volumes.

When running multiple cluster, we are likely to have a control-cluster. So in that control cluster we run a prometheus that also scrapes the local cluster and scrapes all remote prometheus servers as well. How those are discovered is still an open question, but we might settle on a fixed DNS name for that prometheus server in each cluster.