Dashboards and alerts

The SDK repository keeps Grafana dashboards and Prometheus alert rules for the metrics the SDK emits in the observability/ directory. They are versioned next to the code that emits the metrics, and the SDK CI fails when a query names a metric missing from the catalog.

observability/
  dashboards/
    http.json        REST server and REST client
    messaging.json   producers, consumers, and consume lag
    database.json    SQL (sqlDB), Redis (cacheDB), and storage
    runtime.json     Go runtime, GC, and process
  alerts/
    rules.yaml       Prometheus alert rules
    rules_test.yaml  promtool unit tests of the rules

Prerequisites

• metrics enabled and exposed on /metrics, the default behavior; see Observability;
• a Prometheus-compatible backend scraping /metrics, such as Prometheus, Grafana Mimir, VictoriaMetrics, Amazon Managed Prometheus, or Grafana Cloud.

The dashboards and rules query the names /metrics exposes, not the OpenTelemetry names. A backend ingesting the OTLP export instead may name the series differently and is not covered.

Every dashboard filters by the job and instance labels of the scrape. Their drop-downs are filled from target_info, which the SDK exposes on every service.

Import the dashboards

Through the Grafana UI

1. Go to Dashboards → New → Import and upload one of the JSON files of dashboards/.
2. Save it. The dashboard has no datasource baked in: it uses the default Prometheus datasource of the Grafana instance until you pick another one.

Through provisioning

Point a dashboard provider at a copy of dashboards/:

# /etc/grafana/provisioning/dashboards/colibri.yaml
apiVersion: 1
providers:
  - name: colibri-sdk-go
    folder: Colibri
    type: file
    options:
      path: /var/lib/grafana/dashboards/colibri

Choose the datasource

Each dashboard has a Datasource drop-down, a datasource variable restricted to Prometheus, and every panel queries ${datasource}:

• in the UI, pick the datasource and save the dashboard to keep it as the default;
• with provisioning, either make the right datasource the default one (isDefault: true) or edit templating.list[0].current in the JSON, e.g. "current": {"text": "Mimir", "value": "mimir-uid"}.

Load the alert rules

alerts/rules.yaml is a plain Prometheus rule file. Add it to rule_files in prometheus.yml:

rule_files:
  - /etc/prometheus/rules/rules.yaml

For the Prometheus Operator, wrap the groups in a PrometheusRule resource. For Grafana Alerting or Mimir, import it with mimirtool rules load or cortextool rules load.

Alert	Fires when	Severity
ColibriHTTPHighErrorRate	more than 5% of the requests answer 5xx for 5 minutes	critical
ColibriHTTPHighLatency	the p99 request duration is above 1s for 10 minutes	warning
ColibriHTTPPanicRecovered	the REST server recovered a panic	warning
ColibriMessagingPublishFailures	more than 1% of the publishes to a topic fail for 5 minutes	critical
ColibriMessagingConsumerBacklog	the p99 consume lag of a queue is above 60s for 10 minutes	warning
ColibriMessagingConsumerPanics	a consumer panicked processing a message	warning
ColibriSQLPoolSaturated	queries wait for a connection with over 90% of the pool in use	warning
ColibriGoMemoryNearLimit	the Go runtime uses more than 90% of GOMEMLIMIT for 10 minutes	warning
ColibriGoGCPressure	more than 5% of the wall time goes to GC pauses for 10 minutes	warning

tip

The thresholds are starting points. Copy the file and tune them per service rather than editing it in place, so SDK updates do not overwrite your changes.

The backlog alert relies on messaging.consume.lag, the time a message waited in the broker. The queue depth itself is reported by the broker (CloudWatch for SQS, Cloud Monitoring for Pub/Sub, the RabbitMQ Prometheus plugin) and is not covered here.

Try them locally

The SDK development environment runs Prometheus and Grafana with the dashboards and rules of observability/ mounted, so changes show up without an import:

git clone https://github.com/colibriproject-dev/colibri-sdk-go.git
cd colibri-sdk-go/development-environment
make observability

make observability starts only Prometheus and Grafana; make start brings up the emulators too.

Service	Address
Grafana	http://localhost:3030, anonymous admin, dashboards in the Colibri folder
Prometheus	http://localhost:9091, alerts under Alerts

Prometheus scrapes host.docker.internal:8080 and host.docker.internal:8081, where the school-module and finantial-module of the examples repository run. To use your own service, change the targets in development-environment/prometheus/prometheus.yml and reload the configuration:

curl -X POST localhost:9091/-/reload