Metrics

Use metrics to measure volume, latency, and saturation in aggregate. Request and entity identifiers and error messages belong in traces and logs, not in metrics.

Metrics are enabled by default. See Observability to choose between /metrics and the OTLP export.

Create instruments

Create instruments once, when the service starts, and keep them:

package payments

import (
	"github.com/colibriproject-dev/colibri-sdk-go/pkg/base/monitoring"
	monitoringbase "github.com/colibriproject-dev/colibri-sdk-go/pkg/base/monitoring/colibri-monitoring-base"
)

type paymentMetrics struct {
	authorized monitoringbase.Counter
	latency    monitoringbase.HistogramRecorder
}

// newPaymentMetrics runs after colibri.InitializeApp, e.g. from the service constructor.
func newPaymentMetrics() *paymentMetrics {
	return &paymentMetrics{
		authorized: monitoring.Counter(
			"payments.authorized",
			"Number of payment authorizations",
			"{payment}",
		),
		latency: monitoring.Histogram(
			"payments.authorization.duration",
			"Duration of payment authorizations",
			"s",
		),
	}
}

Function	Instrument	Recording
monitoring.Counter	monotonic counter	AddAttrs(ctx, int64, attrs)
monitoring.Histogram	value distribution	RecordAttrs(ctx, float64, attrs)
monitoring.Gauge	current value pushed by the application	RecordAttrs(ctx, float64, attrs)
monitoring.ObservableGauge	current value sampled on every collection	callback

Create instruments after colibri.InitializeApp()

Before monitoring is initialized, the accessors return noop instruments, so a package-level var c = monitoring.Counter(...) records nothing, ever. With metrics disabled the instruments are noop as well, so code never needs to check.

Attributes

Attributes are passed as a monitoringbase.Attrs value. It caches its provider representation, so build it once and reuse it: recording with a reused Attrs does not allocate.

// Built once, outside the hot path: one value per combination.
var (
	cardApproved = monitoringbase.NewAttrs("method", "card", "result", "success")
	cardDeclined = monitoringbase.NewAttrs("method", "card", "result", "error")
)

func (s *PaymentService) authorize(ctx context.Context, order Order) error {
	start := time.Now()
	err := s.gateway.Authorize(ctx, order)

	attrs := cardApproved
	if err != nil {
		attrs = cardDeclined
	}
	s.metrics.authorized.AddAttrs(ctx, 1, attrs)
	s.metrics.latency.RecordAttrs(ctx, time.Since(start).Seconds(), attrs)

	return err
}

NewAttrs takes alternating key/value arguments. A trailing key without a value is ignored, and duplicate keys keep the last value.

When values are only known at runtime but come from a bounded set, build each Attrs once and keep them in a map.

Migrate from Add and Record

Add and Record taking a map[string]string still work but are deprecated: they convert the map on every call.

// Before
requests.Add(ctx, 1, map[string]string{"route": "/api/users"})

// After: build once, reuse
var usersRoute = monitoringbase.NewAttrs("route", "/api/users")
requests.AddAttrs(ctx, 1, usersRoute)

// Or, to migrate mechanically from an existing map
requests.AddAttrs(ctx, 1, monitoringbase.AttrsFromMap(attributes))

Observable gauges

Gauge records a value the application pushes. For a value that is sampled rather than pushed, such as pool size, queue depth, or cache entries, register an observable gauge. Its callback runs on every collection:

highPriority := monitoringbase.NewAttrs("priority", "high")

registration := monitoring.ObservableGauge(
	"orders.pending",
	"Number of orders waiting for payment",
	"{order}",
	func(ctx context.Context) []monitoringbase.Observation {
		return []monitoringbase.Observation{
			{Value: float64(queue.Len()), Attributes: highPriority},
		}
	},
)
defer registration.Unregister()

Register a given name once: each call registers its own callback, and the caller owns its lifetime through Unregister.

Conventions

Naming

• Reuse the OpenTelemetry semantic conventions where they define the metric. The HTTP server metrics are http.server.request.duration and related, not a name of our own.
• Otherwise use domain naming: <domain>.<thing>[.<qualifier>], lowercase, dot-separated, with snake_case inside a segment, such as messaging.process.duration and messaging.in_flight.
• No colibri.* prefix. The origin of a metric is recorded in the catalog, not in the name.
• Leave the unit and the type out of the name. No _seconds, _bytes, _total, or _count: the unit goes in the unit field, and the Prometheus exporter appends the suffixes.
• Name what is measured, not the outcome. Split success and failure with the result attribute, not with two metrics.

Names on /metrics

The Prometheus reader translates the OpenTelemetry name: dots become underscores, the unit is appended as a suffix, and counters get _total. Dashboards and alert rules query the translated name.

OpenTelemetry name	Unit	Prometheus name
messaging.published (counter)	{message}	messaging_published_total
messaging.process.duration (histogram)	s	messaging_process_duration_seconds
storage.transferred (histogram)	By	storage_transferred_bytes
messaging.in_flight (gauge)	{message}	messaging_in_flight

Annotations such as {message} produce no suffix. Histograms are exposed as the _bucket, _sum, and _count series of the translated name. Attributes become labels the same way: http.response.status_code becomes http_response_status_code.

/metrics also serves the Go and process collectors of the Prometheus client (go_gc_duration_seconds, go_memstats_*, process_*). They are not translated and never reach the OTLP exporter.

Units

Units follow UCUM, as the OpenTelemetry conventions require.

Measuring	Unit	Example
Durations	s, as a float64	messaging.process.duration
Sizes	By	storage.transferred
Counts of things	an annotation in braces	{message}, {request}
Dimensionless ratios	1	a cache hit ratio
Percentages	%	go.config.gogc

Record durations in seconds, never milliseconds. The SDK duration histograms use the boundaries in monitoringbase.DurationBucketsSeconds, from 5 ms up to an hour.

Cardinality

Every distinct combination of attribute values is a separate time series, stored and billed on its own. An attribute taking unbounded values turns one metric into millions of series, so every attribute must come from a small, fixed set of values.

These are never metric attributes:

• request identifiers: correlationId, messageId, trace and span ids;
• entity identifiers: userId, tenantId, order, payment, or document ids, e-mails;
• storage object keys and file names;
• raw URL paths and query strings: use the route template (/users/:id), never the path (/users/42);
• error messages, free text, and timestamps.

Also:

• use at most 5 attributes per metric, fewer where possible;
• keep each metric under about a thousand series per instance, multiplying the number of values of each attribute to estimate it;
• remember that values set by the application count too: a message action becomes an attribute of the messaging metrics, so it must be an event name from a fixed set, never an identifier;
• represent outcomes with result, using success and error, and panic where it applies.

Known exceptions

Two metrics from outside the SDK bend this policy. db.sql.client.latency and db.sql.client.calls, from otelsql, carry db.sql.error, the message of a failed call. The http.server.* metrics carry server.address, read from the Host header: behind an ingress or load balancer that only routes known hosts it stays bounded, but a directly exposed service can be sent any value.

Testing metrics

monitoringtest.Install(t) swaps in an in-memory reader for the duration of the test, and installs it as the global meter provider too, so third-party instrumentation records into it. Install it before creating the instruments under test:

package orders_test

import (
	"context"
	"testing"

	"github.com/colibriproject-dev/colibri-sdk-go/pkg/base/monitoring"
	monitoringbase "github.com/colibriproject-dev/colibri-sdk-go/pkg/base/monitoring/colibri-monitoring-base"
	"github.com/colibriproject-dev/colibri-sdk-go/pkg/base/monitoring/monitoringtest"
	"github.com/stretchr/testify/assert"
)

func TestCreateOrder(t *testing.T) {
	recorder := monitoringtest.Install(t)

	// In a real test, create the service that owns the instruments here and exercise it.
	counter := monitoring.Counter("orders.created", "Number of orders created", "{order}")
	counter.AddAttrs(context.Background(), 1, monitoringbase.NewAttrs("channel", "web"))

	created := recorder.Metric(t, "orders.created")
	monitoringtest.AssertShape(t, created, "{order}", "channel")
	assert.Equal(t, int64(1), monitoringtest.CounterValue(t, created, "channel", "web"))
}

Function	Usage
recorder.Collect(t)	Returns every recorded metric, keyed by name.
recorder.Metric(t, name)	Returns one metric and fails the test when it was not recorded.
monitoringtest.AssertShape(t, m, unit, keys...)	Checks the unit and that every data point carries exactly the given keys.
monitoringtest.CounterValue(t, m, kv...)	Value of a counter for an attribute set.
monitoringtest.HistogramCount(t, m, kv...)	Count and sum recorded by a histogram.
monitoringtest.GaugeValue(t, m, kv...)	Current value of a gauge.

Everything is restored when the test ends.

Catalog

Every metric the SDK emits is registered in a catalog, with its name as an exported constant (monitoring.MetricMessagingPublished), type, unit, attributes, and origin. monitoring.Catalog() and monitoring.LookupMetric(name) expose it to tooling, such as checks that dashboards only query metrics that exist.

Emitted by the SDK

Metric	Type	Unit	Attributes	Module
http.server.request.duration	histogram	s	http.request.method, http.response.status_code, http.route, server.address, url.scheme	restserver
http.server.active_requests	up-down counter	{request}	http.request.method, server.address, url.scheme	restserver
http.server.request.body.size	histogram	By	http.request.method, http.response.status_code, http.route, server.address, url.scheme	restserver
http.server.response.body.size	histogram	By	http.request.method, http.response.status_code, http.route, server.address, url.scheme	restserver
http.server.panic.recovered	counter	{panic}	http.request.method, http.route	restserver
messaging.published	counter	{message}	topic, result	messaging
messaging.consumed	counter	{message}	queue, action, result	messaging
messaging.process.duration	histogram	s	queue, action, result	messaging
messaging.rejected	counter	{message}	queue, action, reason	messaging
messaging.in_flight	observable gauge	{message}	queue	messaging
messaging.consume.lag	histogram	s	queue, action	messaging
storage.operation	counter	{operation}	operation, result	storage
storage.operation.duration	histogram	s	operation, result	storage
storage.transferred	histogram	By	operation	storage

Attribute values

• result is success or error; consumed messages also use panic. reason is error or panic.
• action is the event name the application sets on Publish.
• operation is upload, download, or delete.
• http.route is the route template. A panic raised in a middleware, before the route handler sets the template, is recorded with the matched route, or / for a global middleware.
• messaging.consume.lag is measured from the publish time reported by the broker to the moment the consumer picks the message up. A message without a publish time is not recorded, and clock skew that puts the publish time ahead of the consumer is recorded as zero. The queue depth itself is reported by the broker.
• messaging.rejected counts messages nacked without requeue. The SDK leaves them to the broker dead-letter handling, so whether one actually reached a DLQ is reported by the broker, not by the SDK.

Emitted by third-party instrumentation

Prefix	Origin	Module	Content
http.client.*	otelhttp	restclient	HTTP client request duration and body size
db.sql.client.*	otelsql	sqlDB	SQL call latency and counts
db.sql.connections.*	otelsql	sqlDB	open, idle, and active pool connections, waits, and closes
db.client.connections.*	redisotel	cacheDB	Redis pool usage, limits, waits, and timeouts
go.*	runtime	monitoring	memory, allocations, GC, goroutines, and GOMAXPROCS
go_*, process_*	Prometheus client	restserver	Go and process collectors, only on /metrics

The deprecated runtime metrics (runtime.uptime, process.runtime.go.*) are emitted only with OTEL_GO_X_DEPRECATED_RUNTIME_METRICS=true.

The full list, with every third-party metric and its attributes, is in the SDK metrics reference.

To visualize and alert on these metrics, see Dashboards and alerts.