Prometheus is an open-source systems monitoring and alerting toolkit with an active ecosystem. See the overview.
See the comparison page.
The main Prometheus server runs standalone and has no external dependencies.
Yes, run identical Prometheus servers on two or more separate machines. Identical alerts will be deduplicated by the Alertmanager.
For high availability of the Alertmanager, you can run multiple instances in a Mesh cluster and configure the Prometheus servers to send notifications to each of them.
There are in fact various ways to scale and federate Prometheus. Read Scaling and Federating Prometheus on the Robust Perception blog to get started.
Most Prometheus components are written in Go. Some are also written in Java, Python, and Ruby.
All repositories in the Prometheus GitHub organization that have reached version 1.0.0 broadly follow semantic versioning. Breaking changes are indicated by increments of the major version. Exceptions are possible for experimental components, which are clearly marked as such in announcements.
Even repositories that have not yet reached version 1.0.0 are, in general, quite
stable. We aim for a proper release process and an eventual 1.0.0 release for
each repository. In any case, breaking changes will be pointed out in release
notes (marked by [CHANGE]) or communicated clearly for components that do not
have formal releases yet.
Pulling over HTTP offers a number of advantages:
Overall, we believe that pulling is slightly better than pushing, but it should not be considered a major point when considering a monitoring system.
For cases where you must push, we offer the Pushgateway.
Short answer: Don't! Use something like the ELK stack instead.
Longer answer: Prometheus is a system to collect and process metrics, not an event logging system. The Raintank blog post Logs and Metrics and Graphs, Oh My! provides more details about the differences between logs and metrics.
If you want to extract Prometheus metrics from application logs, Google's mtail might be helpful.
Prometheus was initially started privately by Matt T. Proud and Julius Volz. The majority of its initial development was sponsored by SoundCloud.
It's now maintained and extended by a wide range of companies and individuals.
Prometheus is released under the Apache 2.0 license.
After extensive research, it has been determined that the correct plural of 'Prometheus' is 'Prometheis'.
Yes, sending SIGHUP to the Prometheus process or an HTTP POST request to the
/-/reload endpoint will reload and apply the configuration file. The
various components attempt to handle failing changes gracefully.
Yes, with the Alertmanager.
Currently, the following external systems are supported:
Yes, we recommend Grafana for production usage. There are also Console templates.
To avoid any kind of timezone confusion, especially when the so-called daylight saving time is involved, we decided to exclusively use Unix time internally and UTC for display purposes in all components of Prometheus. A carefully done timezone selection could be introduced into the UI. Contributions are welcome. See issue #500 for the current state of this effort.
There are a number of client libraries for instrumenting your services with Prometheus metrics. See the client libraries documentation for details.
If you are interested in contributing a client library for a new language, see the exposition formats.
Yes, the Node Exporter exposes an extensive set of machine-level metrics on Linux and other Unix systems such as CPU usage, memory, disk utilization, filesystem fullness, and network bandwidth.
Yes, the SNMP Exporter allows monitoring of devices that support SNMP.
Yes, using the Pushgateway. See also the best practices for monitoring batch jobs.
See the list of exporters and integrations.
Yes, for applications that you cannot instrument directly with the Java client, you can use the JMX Exporter either standalone or as a Java Agent.
Performance across client libraries and languages may vary. For Java, benchmarks indicate that incrementing a counter/gauge with the Java client will take 12-17ns, depending on contention. This is negligible for all but the most latency-critical code.
You are suffering from an unclean shutdown. Prometheus has to shut down cleanly
after a SIGTERM, which might take a while for heavily used servers. If the
server crashes or is killed hard (e.g. OOM kill by the kernel or your runlevel
system got impatient while waiting for Prometheus to shutdown), a crash
recovery has to be performed, which should take less than a minute under normal
circumstances, but can take quite long under certain circumstances. See
crash recovery for details.
See the section about memory usage to configure Prometheus for the amount of memory you have available.
Your storage is under heavy load. Read the section about configuring the local storage to find out how you can tweak settings for better performance.
We restrained ourselves to 64-bit floats to simplify the design. The IEEE 754 double-precision binary floating-point format supports integer precision for values up to 253. Supporting native 64 bit integers would (only) help if you need integer precision above 253 but below 263. In principle, support for different sample value types (including some kind of big integer, supporting even more than 64 bit) could be implemented, but it is not a priority right now. A counter, even if incremented one million times per second, will only run into precision issues after over 285 years.
TLS and basic authentication is gradually being rolled out to the different components. Please follow the different releases and changelogs to know which components have already implemented it.
The components currently supporting TLS and authentication are:
This applies only to inbound connections. Prometheus does support scraping TLS- and auth-enabled targets, and other Prometheus components that create outbound connections have similar support.
This documentation is open-source. Please help improve it by filing issues or pull requests.