Metrics with Prometheus

Overview

Use Metrics with Prometheus to collect and store your metrics as time series data. Prometheus is an open source monitoring toolkit that records your metrics with timestamps and optional, key value pairs called labels, which enables you to create a multi-dimensional data model. Prometheus operates on a pull model over HTTP, where it scrapes metrics from instrumented jobs and supports pushing metrics for jobs. It's particularly well suited for monitoring dynamic service-oriented architectures, offering reliability and independence from external storage systems.

OpenTelemetry can export metrics in a compatible format, allowing Prometheus to scrape them directly from instrumented services. By using OpenTelemetry SDKs and collectors, developers can generate and manage metrics uniformly across distributed systems, then expose them via a Prometheus exporter. This integration simplifies observability pipelines by consolidating telemetry data while maintaining Prometheus's powerful querying and alerting capabilities, making it easier to monitor modern cloud-native applications.

Prometheus Data Model

Prometheus's data model is time series-based. Each time series is uniquely identified by a metric name and a set of labels.

• Metric Name: Specifies the general feature of a system that is measured, such as http_requests_total for the total number of HTTP requests received.
• Labels: Key-value pairs that differentiate characteristics of the metric, such as method="POST", orhandler="/api/tracks".

This multi-dimensional data model allows for powerful and flexible querying, enabling users to filter and aggregate data across various dimensions.

Metric Types

Prometheus supports four primary metric types, each serving distinct purposes:

• Counter: A cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero on restart. Ideal for counting occurrences such as the number of requests served or errors encountered.
• Gauge: Represents a single numerical value that can arbitrarily go up and down. Useful for measured values like temperatures or current memory usage.
• Histogram: Samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. It also provides a sum of all observed values.
• Summary: Similar to a histogram, it samples observations and provides a total count and sum of all observed values, along with configurable quantiles.

Understanding these types is crucial for selecting the appropriate metric type for specific monitoring needs.

Use Cases

To use metrics with Prometheus:

1. Make sure the pool of OpenTelemetry collectors receives OTLP and exports to Prometheus Remote Write.
2. Ensure the Collector joins service discovery with metric resources.
3. If done correctly, the Collector then computes “up” with a staleness marker that is a special signal emitted to indicate that a previously exported metric time series no longer exists..
4. The Collector applies a distinct external label to your metrics.

To use Prometheus directly:

from opentelemetry.exporter.prometheus import PrometheusMetricReader
from prometheus_client import start_http_server

reader = PrometheusMetricReader()
provider = MeterProvider(metric_readers=[reader])
metrics.set_meter_provider(provider)

start_http_server(8000)  # Prometheus will scrape this

Edit the Prometheus config to set up scraping, for example at 15 second intervals:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "my_app"
    static_configs:
      - targets: ["localhost:8000"]

Run Prometheus:

./prometheus --config.file=prometheus.yml

And with a query such as example_counter, you should see metrics with labels.

Limitations

Prometheus through OpenTelemetry has some limitations. Using OpenTelemetry as an intermediary format between two non-compatible formats such as Prometheus and statsd or Prometheus and collectcd. Although not explicitly impossible, these combinations are not supported.