Design notes
Why metrics4s is shaped the way it is. The README says what the API does. This page says what was chosen, what was
rejected and what each choice costs. Several decisions look arbitrary until you know the failure they answer, so the
failures are here too. What was decided and when is in Decisions.
- Why another metrics library
- Principles
- The shape
- Definitions are values, instruments belong to a runtime
- Labels come after the name
- Names follow the conventions or do not compile
- A unit is a type
- Histograms are classic
- Labels are closed
- Series exist from the start
- The hot path does not look labels up
- Effect-agnostic by construction
- What the shared suite found
- One lifecycle shape, one adapter per runtime
- Runtimes live under metrics4s.runtime
- Callbacks are read at scrape time
- Timing an effect
- Measuring an operation
- A gauge forgets what is gone
- The Prometheus backend
- Runtime metrics
- Testing against the real backend
- Dashboards are generated from the definitions
- Integrations
- OpenTelemetry later, behind the same definitions
- The build
- Deliberately not done
- Decisions
Why another metrics library
As of November 2026 no library does what we need: one metrics API over cats-effect, ZIO, Kyo and rapid, backed by the native Prometheus Java client 1.x, with our own metric names.
| Library | What stops us |
|---|---|
| otel4s (Typelevel) | cats-effect only; Prometheus only through OpenTelemetry, which rewrites names (dots, unit suffixes) and brings the OTel SDK |
| prometheus4cats (Permutive) | cats-effect only; 6.0.0 still a release candidate; built for Scala 2.13 first, Scala 3 only on 3.3; client 1.6.1 |
| epimetheus | cats-effect only; client 1.4.3; barely maintained (0.6.1, December 2025) |
| http4s-prometheus-metrics | Still on simpleclient 0.16, incompatible with client 1.x |
| zio-metrics-connectors, kyo-stats-otel | One runtime each; Kyo's goes through the OTel API |
| Kamon | Agent-based model, slow releases (2.8.1, January 2026) |
| metrics-scala | Dropwizard, legacy |
| Micrometer, OTel Java | Java APIs with no effect wrappers |
metrics4s did not start from scratch. Our commons already had the runtime bridge, the runtime metrics of
cats-effect, ZIO and Kyo, the timed and callback-gauge helpers and the testkit. The services that use them
(descriptions, merger) had the naming and dashboard conventions. That code does not belong in commons, so it became
an OSS library that commons can depend on.
Principles
- Scala 3 only.
given-based throughout, with derivation throughMirrorandinlineand no runtime reflection. This is the same approach asmongo4sandlogging4s. - The compiler is the test. Each of these is a compile error rather than a review comment:
- a label value outside its domain;
- a duration recorded as a count;
- a counter named
_total; - a histogram in milliseconds.
- Conventions by construction. The constructors do what
metrics-guide.mdasks people to remember: seconds,_total,classicOnly, pre-created series, closed labels. - No global state. No registry is picked implicitly.
PrometheusRegistry.defaultRegistryis reachable, but only by name, never as a default argument. - The backend is not hidden. The
PrometheusBackendexposes its registry, so anythingmetrics4sdoes not model can still be registered there directly. - One source of truth. A metric definition is one value, used by the service code, the test, the dashboard and the README table. Adding a metric changes one place, not the three the guide lists today.
The shape
flowchart LR
app["your service"] --> core
subgraph core["metrics4s-core (no dependencies)"]
D["Counter · Gauge · Histogram definitions"]
L["LabelValue · Labels"]
E["Effect · Dispatch · Allocated"]
end
RT["runtime<br/>cats-effect · ZIO · Kyo · rapid"] -->|given Effect, Dispatch<br/>runtime metrics| core
PB["metrics4s-prometheus<br/>client 1.x · exporter · JVM"] -->|given Metrics F| core
TK["metrics4s-testkit"] --> PB
GR["metrics4s-grafana<br/>dashboards · README tables"] --> core
INT["integrations<br/>http · http4s · tapir · sttp · mongo"] --> core
| Module | Depends on | What |
|---|---|---|
metrics4s-core |
nothing | Definitions, typed labels, instruments, callbacks, the backend SPI, Effect/Dispatch/Allocated, Metrics[F] |
metrics4s-prometheus |
core, client 1.10 | The backend: registry, exporter, JVM metrics |
metrics4s-cats, -zio, -kyo, -rapid |
core, the runtime | Effect/Dispatch instances, the resource adapter, the runtime's own metrics |
metrics4s-testkit |
prometheus, grafana | A registry per test, typed reads, the raw series map, dashboard coverage |
metrics4s-grafana |
core | Dashboard JSON and README tables from definitions |
metrics4s-http |
core | The HTTP definitions the HTTP integrations share |
metrics4s-http4s, -tapir, -sttp |
http and the library | Server and client instrumentation |
metrics4s-mongo |
core, mongodb-driver-core | Command and connection-pool listeners |
Runtime metrics live in the runtime modules, not in modules of their own. They need nothing the runtime module does not already depend on, because they are written against core's callback API rather than the Prometheus client.
Definitions are values, instruments belong to a runtime
A definition says what a metric is: name, help, kind, unit, labels and buckets. It has no F and touches no
registry, so it can live in an object, and the dashboard generator and the testkit can read it without a runtime.
enum Outcome derives LabelValue:
case Ok, Skipped, Error
object Outcome:
given Labels[Outcome] = Labels.single("outcome")
object Definitions:
val processed = Counter("processor_items", "Processed items by outcome").labelled[Outcome]
val processTime = Histogram.seconds("processor_process_seconds", "Processing of one item", Buckets.Normal)
val cacheSize = Gauge("processor_cache_entries", "Entries in the cache")An instrument is a definition registered on a backend for one F. Registration is the only place that needs a
backend, and it happens once, in the factory of the service's metrics class. That is the shape the services already
use.
trait ProcessorMetrics[F[*]]:
def processed(outcome: Outcome, count: Int): F[Unit]
def processTime(time: FiniteDuration): F[Unit]
def cacheSize(size: Int): F[Unit]
object ProcessorMetrics:
def create[F[*]](using metrics: Metrics[F], F: Effect[F]): F[ProcessorMetrics[F]] =
F.map(metrics.registerAll((Definitions.processed, Definitions.processTime, Definitions.cacheSize)))(Impl(_, _, _))
private final class Impl[F[*]](
processedItems: CounterOf[F, Long, Outcome],
processTimes: HistogramOf[F, FiniteDuration, NoLabels],
cacheEntries: GaugeOf[F, Double, NoLabels],
)(using Effect[F])
extends ProcessorMetrics[F]:
def processed(outcome: Outcome, count: Int): F[Unit] = Effect.whenA(count > 0)(processedItems.add(outcome, count))
def processTime(time: FiniteDuration): F[Unit] = processTimes.observe(time)
def cacheSize(size: Int): F[Unit] = cacheEntries.set(size.toDouble)registerAll takes a tuple of definitions and returns a tuple of instruments, typed by a match type over the
definition types. It is all or nothing: if the third registration fails, the first two are unregistered before the
error is raised. This is the rollback commons' runtime metrics did by hand.
A tuple of eleven instruments is then unpacked into a class by position, where two counters of the same type can swap
places and still compile. registerAs[P] takes a named tuple of definitions instead and fills the case class P. An
inline check compares the tuple's names with the fields, in order, and the instrument types with the field types, so
both mistakes are compile errors with a sentence that says what to change. The registration underneath is
registerAll, rollback included.
A failure is a MetricError, a sealed ADT: name taken, conflicting descriptor, invalid label name, duplicate label
value, domain too large. It is raised in F as a MetricException, never thrown to the caller.
Labels come after the name
The obvious API is Counter[Outcome]("name", "help") next to Counter("name", "help"). Scala 3 refuses the pair. A
call without type arguments matches both alternatives, and the compiler reports an ambiguous overload. Two
workarounds were possible:
- a distinct name for the labelled constructor;
- labels as a step after the name.
The second won. Counter("name", "help").labelled[Outcome] reads in the order a reader asks the questions: what is it
called, then what is it split by. It also keeps one macro entry point per kind of metric.
Names follow the conventions or do not compile
A name is a MetricName, an opaque String. Definitions take a literal, and an inline macro checks it where it is
written, with an error that says why:
Invalid metric name "processor_items_total": it must not end with _total: every counter is exposed with _total addedEvery name must be snake case, with no __ and no trailing _, and a non-empty help. None of the suffixes the
exposition writes itself (_bucket, _count, _sum, _created, _info) may be written by hand, nor _total. The
rest depends on the constructor:
| Constructor | Name must | Records | Exposed as |
|---|---|---|---|
Counter |
end with neither _seconds nor _bytes |
Long |
<name>_total |
Counter.seconds, Counter.bytes |
end with the unit | FiniteDuration, Long |
<name>_total |
Gauge |
end with neither unit | Double |
<name> |
Gauge.seconds, Gauge.bytes |
end with the unit | FiniteDuration, Long |
<name> |
Histogram.seconds, Histogram.bytes |
end with the unit | FiniteDuration, Long |
_bucket, _sum, _count |
Histogram.sizes |
end with neither unit | Long |
_bucket, _sum, _count |
Counters are named without _total. That is what metrics-guide.md asks for and what the client expects: client
1.x strips a _total it is given and adds it back on exposition. The cost is that the name in the code is not the
exact string the dashboard queries. Requiring _total in code would have been just as consistent. The guide's
spelling won because every service and every commons definition already uses it, so nothing has to be renamed on the
way over.
Counter.seconds exists for the runtime counters of idle time (cats_effect_worker_idle_seconds_total), which commons
exported before metrics4s had an opinion on them.
A unit is a type
Histogram.seconds records a FiniteDuration and converts it to seconds itself. There is no observe(Double) on it,
so observe(elapsed.toMillis) does not compile, and descriptions' *_millis histograms, which had to be renamed,
cannot be written in the first place. The unit also goes into the metric's metadata, which is what .unit(SECONDS)
does in the services today.
A definition carries Recorded[V], which turns the recorded type into the Double the backend wants. Which V a
definition takes is fixed by its constructor, not left to an implicit.
Histograms are classic
Every histogram is registered with classicOnly(). Without it client 1.x also writes a native histogram, and our
Prometheus, dashboards and alerts read classic _bucket{le} series.
Buckets is a value, checked at compile time when the bounds are literals and through
Buckets.from: Either[MetricError, Buckets] when they are computed. The bounds must be finite, positive, strictly
increasing and not empty. Buckets.Fast, Buckets.Normal and Buckets.Sizes are the sets the services already use.
Labels are closed
LabelValue[A] is either enumerated, when every value is known and has an index, or bounded, when any value of
the type is acceptable but the values cannot be listed. Labels[L] turns a value of L into a series: label names, a
rendered value per label and, when every label is enumerated, an index into their product.
derives LabelValueworks for an enum or sealed trait whose cases take no parameters. A case with parameters is a compile error. The domain is the cases in order, the index is the ordinal and the rendered value is the case name in snake case:NotFoundbecomesnot_found,HTTPVersionNotSupportedbecomeshttp_version_not_supported. An enum that already carries its string says so withLabelValue.enumerated(Level.values)(_.value). Derivation never looks for avaluefield on its own, so what a derived instance renders depends only on the case names.derives Labelsworks for case classes, and a named tuple gets itsLabelswith no derivation at all. Label names are the field names in snake case. The domain is the product of the fields' domains when they are all enumerated.- A single label needs its name,
Labels.single("outcome"). The name is checked at compile time, soleandquantileare refused. A label name is never guessed from a type name.
Domains known only at startup are enumerated from a value: route templates of a server, the collections of a
database, the operations of a client. LabelValue.enumerated(routes :+ Route.Unmatched)(_.template) makes the domain,
and the integrations normalise every incoming value into it before it reaches the metric. A route nobody declared is
unmatched, never itself.
There are no open labels. An id, a path, an error message or a raw provider name would each be a new series
forever, so LabelValue[String] does not exist. Normalising an outside value into an enum is the job of a small total
function written next to the code that receives it.
Registration checks what the types cannot express:
- label names are valid and not reserved;
- no label is declared twice;
- no two values of an enumerated label render the same string.
Series exist from the start
A labelled series appears only after its first update. So right after a release the panels are empty, and rate()
needs a second sample on top of that. The services worked around it by calling labelValues(v) for each enum value
in create. metrics4s does that at registration: when every label of a metric is enumerated, every combination is
created at zero, and a metric without labels is created at zero too.
The product of enumerated domains can get large: 20 routes × 6 status classes is 120 series of zeros per pod.
Registration refuses a product over MetricsConfig.preCreateLimit (default 512) with MetricError.DomainTooLarge. A
definition that really wants that many says .withoutPreCreation. Refusing, rather than quietly skipping
pre-creation, is deliberate: a cardinality blow-up shows up when the service starts, not as empty panels after a
release.
The MongoDB command histogram is the one built-in definition that opts out. Commands × collections × buckets would be thousands of zeros, so its panels fill as the commands run.
The hot path does not look labels up
On the raw client, counter.labelValues("ok").inc() builds a varargs array and looks the series up in a concurrent
map on every call. For an enumerated domain, metrics4s resolves every data point once, at registration, into an
array indexed by the combination's position in the product (mixed radix over the ordinals). inc(Outcome.Ok) is
then an array read and an inc, with no allocation for a single enum. A bounded label renders its value and does the
client's lookup.
BENCHMARKS.md has the numbers, on an M4 Max:
- one enumerated label: 196 updates per microsecond against the raw client's 152;
- two enumerated labels: 155 against 137;
- one bounded label: 94.
The bounded path first measured 52. Passing the label array to the client without a copy fixed most of that.
Every instrument has unsafe, the same operations returning Unit, for code outside F: a driver listener, an HTTP
client interceptor, a Java callback. The integrations use it.
Effect-agnostic by construction
metrics4s-core depends on nothing. Recording a value only needs delay, so logging4s's single-method Delay would
cover most of the API, but two features need more:
timedhas to read a monotonic clock and see how the effect ended;- a gauge whose value lives in
Fhas to runFon the scrape thread.
So core takes mongo4s's Effect with the same seven abstract members. guaranteeCase is the load-bearing one,
monotonic is derived, and sleep is dropped because nothing here needs it. Without sleep, the cats-effect
instance asks for Sync alone. Beside it sits commons' Dispatch, with its runner given as an Allocated:
trait Dispatch[F[*]]:
def runner: Allocated[F, Dispatch.Runner[F]]
object Dispatch:
trait Runner[F[*]]:
def unsafeRunTimed[A](fa: F[A], timeout: FiniteDuration): AEffect is copied rather than shared with mongo4s; see Deliberately not done. commons'
EffectSuite comes along with it, extended. It is one abstract spec in core's tests that every runtime module
extends, so all four instances are held to the same semantics:
- exit cases;
- finalizer errors, where the original error is kept and the finalizer's is added as suppressed;
- brackets;
Allocatedand the runtime's own resource;- a
Dispatchthat runs an effect from a foreign thread, times out, and cancels what it gave up on.
What the shared suite found
Copying the instances was not enough. On its first run the suite failed on three runtimes, each for a reason worth knowing:
- Kyo 1.0.0-RC7 abandons an interrupted fiber rather than resuming it. The finalizers it owes run with
KyoException("fiber abandoned"), notInterrupted, so aguaranteeCasethat takes the exit case from what Kyo reports sees a cancellation asErrored. The instance records how the body ended instead: the body sets the exit case when it finishes, and a scope that closes with nothing recorded is a cancellation, whatever Kyo reports. Kyo does not wait for those finalizers, so a cancelled fiber's finalizer may finish after the cancellation returns.mongo4s-kyohad the same flaw, and5.1.1fixes it with the same shape. - ZIO's runtime remembers where it was taken.
ZIO.runtimecaptures the current runtime flags, and a runner allocated inside any bracket oracquireReleaseis allocated uninterruptibly. Every fiber it started then ignored the cancellation sent on timeout, so a stuck callback left a fiber behind on every scrape. The runner now runsfa.interruptible. commons'TaskToDispatchInstancehas the same flaw. - rapid cancels by interrupting a thread. An interrupt that reaches running code surfaces as an
InterruptedException, which the rapid instance reports asCanceled. A fiber parked on a callback (aCompletable) is abandoned, and none of its finalizers run. Nothing on this side can change that.RapidCancellationSpecpins the behaviour down, so a rapid release that changes it shows up as a failing test.
One lifecycle shape, one adapter per runtime
The exporter, F-backed callbacks and runtime metrics all hold something that must be released. mongo4s answers
that with a MongoClientResource per runtime, which is fine for one resource but would mean an object per feature
per runtime here. Core instead describes every such thing in one shape:
opaque type Allocated[F[*], A] = F[(A, F[Unit])]It is opaque so that nothing else shaped F[(A, F[Unit])] passes for it. It also composes on its own, with map,
flatMap and use, all built on Effect. flatMap releases in reverse order and releases the first resource when
the second fails to acquire.
Each runtime module turns it into its own idiom once, through metrics4s.runtime.<runtime>.syntax.given:
| Runtime | Adapter | Gives |
|---|---|---|
| cats-effect | allocated.toResource |
Resource[F, A] |
| ZIO | allocated.toScoped |
ZIO[Scope, Throwable, A] |
| Kyo | allocated.toScope |
A < (Scope & Async & Abort[Throwable]) |
| rapid | allocated.use(f) |
rapid has no resource type, so core's use is the idiom |
Runtimes live under metrics4s.runtime
mongo4s puts its runtimes at mongo4s.cats and mongo4s.zio. Here that was the first layout too, and the
integration tests showed what it costs. import metrics4s.* brings metrics4s.cats into scope as cats, and from
then on every import cats.effect... in the same file resolves to the wrong package. The same happens to zio, kyo
and rapid. Since given imports come after the library's imports in our style, this hits every file that imports the
library with a wildcard.
The runtimes therefore live at metrics4s.runtime.cats and its siblings, which also matches the runtime/ directory.
For the same reason the sttp integration is metrics4s.sttpclient: metrics4s.sttp would shadow sttp.client4.
Callbacks are read at scrape time
Some values are cheaper to read when Prometheus asks for them than to keep up to date: the size of a map, the state of
a pool. Every callback gives an Allocated[F, Unit].
metrics.observe(definition)(report)for one metric.reportruns on the exporter's thread on every scrape, so it must be cheap and thread-safe.metrics.observeAll(read)(observations*)for several metrics fed by one read. This is for a source that is expensive or must be read once per scrape. Kyo'sScheduler.status()takes a stack trace of every busy worker. Merger's three callback gauges over the sameMergerState.statscomputed the stats three times per scrape.metrics.observeAllF(read, timeout)(observations*)for a value that is only available inF. It goes throughDispatchwith a timeout, as commons'CallbackGaugedoes.
The backend builds one MultiCollector per callback and gives it full getMetricFamilyDescriptors (type, labels,
metadata). Without them, client 1.10's registry cannot detect a name conflict.
Commons swallowed a failed callback. Here a callback that throws, or an F that times out, still keeps the scrape
alive: its metrics are left out of that scrape and the F is cancelled. The failure is counted in
metrics4s_callback_errors_total{metric}, labelled with the callback's first metric and created at zero when the
callback is registered, so a broken callback is visible instead of being a series that is silently missing. The
counter is read in the same scrape as the callbacks it counts, so a scrape shows the failures up to the one before it.
Timing an effect
histogram.timed(labels)(fa)
histogram.timed(fa)
histogram.timedBy(fa)(result => if result.isRight then Outcome.Ok else Outcome.Error)A success and an error are recorded. A cancellation is not, because its duration is the time until somebody gave up,
not the time the work takes. timedBy picks the labels from the outcome, which lets errors share a histogram with
successes under outcome="error". The effect is taken by name, so nothing in it runs before the clock is read.
Measuring an operation
val fetch = Measured(fetchTime, fetches, FetchOutcome.Failed)((labels, outcome) => (source = labels.source, outcome = outcome))
fetch.measure((source = Source.Kafka))(repository.find(id))(found => if found.isDefined then FetchOutcome.Found else FetchOutcome.Missing)A service that measures its operations writes the same thing for each of them: time it, count it by outcome, where the
outcome comes from the value or is "failed" on an error, and re-raise the error. Written by hand, it is the code most
likely to lie. A wrapper that turns an exception into a value makes the metric wrong in exactly the case it exists
for. Measured writes it once:
- The error is the same instance, re-raised after the recording. If recording fails too, the operation's error still wins, and the recording failure is added to it as suppressed.
- The outcome function runs inside
F, so an exception in it is an error of the effect, not a throw while the effect is built. - A cancellation records nothing, for the reason
timeddoes not. - The histogram keeps the operation's labels, and the counter adds the outcome. A histogram split by outcome would multiply its buckets for a distinction the counter already makes.
A gauge forgets what is gone
A labelled gauge keeps the last value of a series that is no longer updated: a provider that went away stays on the
dashboard with its old number. gauge.set(labels, value) keeps per-series semantics. gauge.replace(values) is the
full recalculation: every series missing from the map is reset to zero when the domain is enumerated (it was
pre-created, so it stays) and removed when the domain is bounded.
The Prometheus backend
metrics4s-prometheus is the only module that knows the Java client.
given Metrics[IO] = PrometheusMetrics[IO](registry)
PrometheusMetrics.jvm[IO](registry)
PrometheusExporter.allocated[IO](9090, registry)
PrometheusScrape.text(registry)- The registry is passed, never assumed.
PrometheusRegistry.defaultRegistryis what other Java libraries register on, so services will often want it, and they write it out. - JVM metrics are the client's own
instrumentation-jvm. Their names are what every dashboard already reads:jvm_memory_used_bytes,jvm_gc_collection_seconds,process_cpu_seconds_total. So they are not reimplemented. - A name registered twice with the same labels is
MetricError.AlreadyRegistered. Any other refusal by the registry isRejected, with the client's message. - Inside the module,
import io.prometheus.metrics.model.snapshots.{Unit as PrometheusUnit, *}is required, because the wildcard import alone shadowsscala.Unit.
Runtime metrics
These are ported from commons with the names, types and labels unchanged, so existing dashboards and alerts keep working. A spec in each runtime module compares the exported families with the list commons exports.
| Runtime | What | Known caveats |
|---|---|---|
| cats-effect | Compute pool (threads, active, searching, blocked), fibers queued and suspended, per-worker counters, timers, CPU starvation count, clock drift histogram sampled by a fiber of its own | 3.7.1: a timer that fires on its own thread is counted twice in _executed, and _canceled lags |
| ZIO | Default and blocking executors (workers, concurrency, queue, enqueued, dequeued), fibers by state and by outcome through a Supervisor |
Only fibers started after the supervisor is added are counted |
| Kyo | One batch callback over Scheduler.status(): workers, load, admission, regulators, per-worker counters |
status() takes a stack trace per busy worker, so it is read once per scrape; regulator averages are NaN until the first measurement and are left out; kyo-kernel declares JMH as a compile dependency, which is excluded |
| rapid | Nothing | rapid 2.9.9 has a TaskMonitor trait but no hook to install one |
The cats-effect clock drift fiber is the one runtime metric that needs scheduling. It is therefore the one that is a
Resource of the runtime rather than a callback.
Testing against the real backend
Tests read real values from a real registry. There are no mocks, and no in-memory backend that could disagree with the Prometheus one. Core's own tests use a fake of the SPI, but only to test core's own logic: pre-creation, rollback, replace.
val testkit = MetricsTestkit()
given Metrics[IO] = testkit.metrics[IO]
...
testkit.value(Definitions.processed, Outcome.Ok) shouldBe Some(3.0)
testkit.value(Definitions.processed, Outcome.Error) shouldBe Some(0.0)- Typed reads take the definition and the labels, so a test cannot misspell a series or forget
_total:valuefor counters and gauges;histogramforcount,sumand cumulativebuckets;labelValuesfor the series that exist. A series that does not exist isNone, which is itself something to test.
- The raw series map,
testkit.series: Map[String, Double], is keyed by the exposition line exactly as commons'MetricsTestkit.snapshotwas. It is for tests that check what a dashboard will read. testkit.missingSeries(dashboard)lists every series a dashboard queries that the registry does not export. That is the "series the dashboard reads" test the guide asks every service to write by hand. The runtime modules use it against the shared rows.
Dashboards are generated from the definitions
Until now, every dashboard was a few thousand lines of hand-written JSON, checked by a Python script and kept in step
with the code by discipline. The definitions already know everything a panel needs (name, type, unit, labels and their
values), so metrics4s-grafana builds the JSON from them.
- The generator applies every rule of the guide, not the author:
- no
__inputsor__requires, and adatasourcevariable; namespace,serviceandpodfromup, and every query filtered by all three;[$__rate_interval], never a fixed window, andsum by (le)in every quantile;- the unit taken from the definition, and
"min": 0; - a fixed
uid, unique ids, a 24-wide grid with nothing overlapping; - the group tag and links.
A custom panel that breaks a rule makes
renderreturn aGrafanaError.
- no
by(_.outcome)is a selector macro, so a label field that does not exist does not compile.byAllandbyLabelcover single labels, whose name the type does not carry.- Filters are typed the same way.
.where(_.outcome =!= Outcome.Ok)is read by a macro that accepts one label field compared with values of its type, and the values are rendered through the label'sLabelValue. A filter typed as a string had to guess thatBadCredentialsrenders asbad_credentials, and matched nothing once the case was renamed. A value outside an enumerated domain failsrender. Stats take the same filters, so a stat for one source or one route is not a custom query either. - One quantile and shares are panels of their own, because services kept writing them as custom queries:
Panel.quantile(h, 0.95)instead of three lines per label value, andPanel.share(counter)(condition)for a hit or error ratio. - Thresholds are colours and increasing numbers.
.withThresholds(Color.Green, 0.01 -> Color.Orange)writes Grafana's steps with the base first. Steps out of order are accepted by Grafana and colour the stat wrongly, sorenderrefuses them. dashboard.missingSeries(definitions)is the drift check without a registry: the series the dashboard's own panels and stats read, minus the ones the definitions serve. The shared rows are left out of it, since the library checks them against the runtime metrics and the integrations' definitions.- The shared rows ship with the library: cats-effect, ZIO, Kyo, the JVM, HTTP server, HTTP client and MongoDB. The
cats-effect and JVM rows query exactly what the hand-written merger and descriptions dashboards query, and a spec
pins this. The legacy
or jvm_memory_bytes_usedfallbacks for pre-commons-4 releases are left out. - The JSON writer is our own. The module only writes JSON. A writer for a tree of objects, arrays, strings and numbers is small enough to own, and owning it keeps the module free of a dependency its users would then have to reconcile with theirs.
- The rendered JSON is a golden file. It is byte-stable, so a test that compares it with the file in the repository catches a metric changed without its dashboard.
MetricsDoc.markdown(definitions*)writes theMetric | Type | Labels | Whattable that eachmetrics/README.mdkept by hand, with the label values listed.
The integration tests import a generated dashboard into Grafana 12. Every query of every panel then has to return data through Grafana's Prometheus datasource, against a Prometheus 3.5 that scrapes a running service.
Integrations
All HTTP integrations share metrics4s-http, so one dashboard row fits whichever library a service uses:
| Metric | Labels |
|---|---|
http_server_requests_total, http_server_request_seconds, http_server_in_flight |
route (a template from a closed set, or unmatched), status (1xx to 5xx) |
http_client_requests_total, http_client_request_seconds, http_client_in_flight |
operation (from a closed set, or other), status (1xx to 5xx, or failed when no response came back) |
- The route includes the method (
GET /events/{id}), so there is no separatemethodlabel. With one, routes × methods × statuses would have to be pre-created. statusis a class, not a code. That keeps the domain small enough to pre-create. A service that needs 404 apart from 400 has the route to tell them apart.- The definitions live on the companions as well.
HttpServerMetrics.requestsis defined with a bounded route, and an instance relabels it with its enumerated routes, so both describe the same series. A dashboard needs the names, not the values, and draws from the companion without the route list.MongoMetricsdoes the same with its collections. - http4s gets server middleware with a classifier from the request to a template, and client middleware with a classifier into operations.
- tapir takes the templates from the endpoints themselves (
TapirMetrics.template). A request no endpoint matched isunmatched. - sttp client 4 goes through its own
ListenerBackend. - MongoDB gets a
CommandListenerand aConnectionPoolListenerfor the driver. They report:- command time and failures by
command(an enum of the common ones, thenother) andcollection(from a closed set, thenother); - pool connections open and checked out, check-out time, and check-out failures by reason.
mongo4stakes them throughMongoClientResource.fromSettings. The integration test runs them against MongoDB 8.2 throughmongo4s.
- command time and failures by
Our rabbit and kafka clients are not OSS, and an OSS library cannot depend on them. Their metrics belong in those
libraries, written against metrics4s-core.
OpenTelemetry later, behind the same definitions
OpenTelemetry would be a second backend, the way logging4s has logback and log4j2. The backend SPI in core speaks in
definitions (name, help, kind, unit, label names, buckets) and cells, with nothing Prometheus-specific in it, so it
does not change when a second backend arrives. What will need care then is naming. An OTel pipeline that ends in
Prometheus adds unit and _total suffixes of its own, so the OTel backend will have to hand over names that come out
unchanged, and a test will have to prove it.
The build
Most of it is copied from mongo4s:
- Scala
3.9.0and sbt 2; requiredJdk = 25, checked on load, because Kyo ships Java 25 class files;-Werrorwith the same warning set, and-WunstableInlineAccessors, because the selector macro and the derivations are inline;- MiMa against
binaryCompatibleWithfrom1.0.0.
The macro objects Checked and Macros are public, in metrics4s.internal. Declared private[metrics4s], every
inline constructor that reaches them made the compiler synthesise an unstable accessor, which is exactly what
-WunstableInlineAccessors exists to refuse.
The it module compiles on the next Scala (3.10.0) against the library built on the LTS. That is how
mongo4s caught an ArrowAssoc deprecation that broke its users. It holds:
- the Docker integration tests (Prometheus, Grafana, MongoDB through Testcontainers);
- an example for every Scala block of the README, which
DocsItSpecmatches against the README, so a block cannot drift from what compiles.
Two things to know when working on it:
- sbt 2 runs only the tests affected by a change with
test; the whole suite istestFull. With IntelliJ open, usesbt --server -batch. - Under
-Werror, warnings from Kyo's inline code cannot be silenced with@nowarn; they are silenced by message, asmongo4sdoes.-source:futurecompiles with only those warnings left.
Deliberately not done
Native histograms. Our Prometheus, dashboards and alerts read classic buckets. When that changes, it is an option
on Histogram, not a change of default.
Summaries. Quantiles computed in the process cannot be aggregated across pods, and every dashboard aggregates across pods.
Open labels. See Labels are closed. A String label would be the single most expensive
mistake the library could make easy.
An implicit default registry. It would remove one given from service setup and bring back the process-wide
coupling that makes metrics leak between tests.
A shared kernel with mongo4s and logging4s. All three need an effect bridge, and a shared Effect would remove
a copy. But every library's release would then depend on the kernel's, and a change one library needs would become a
change for all three. Seven methods are cheaper to copy. A shared EffectSuite in each repository keeps the copies
honest, and here it found two flaws the original copies still have.
simpleclient and Dropwizard compatibility. The point is to be on client 1.x.
A method label on HTTP metrics. See Integrations.
Decisions
Nothing is open.
Decided on 2026-11-07:
- rapid runtime metrics are left out of
0.1.0. rapid 2.9.9 has no hook to install aTaskMonitor, so the hook is to be proposed upstream, and until then a rapid service has the JVM metrics. The item is on the roadmap. - Coordinates: the artifacts are published as
now.sendlineon Maven Central, and the repository is sendline-now/metrics4s. The package staysmetrics4s. The homepage in the POM is oss.sendline.now/metrics4s, the site Sendline's libraries share. It is built in its own repository from this README and the files it links, so this one has no site of its own. - The first public release is
0.2.0.0.1.0was only tested inside Sendline.now. The default branch ismain.
Decided on 2026-11-04:
- Scope of 0.1.0: everything above, the integrations included.
- Counter names are written without
_total. - Enum label values derive as the snake case of the case name; the
valuefield is used only when written out. - A domain product over the pre-creation limit (512) fails registration unless the definition opts out.
- Versioning: the first release is
0.1.0; MiMa and COMPATIBILITY.md checks start at1.0.0.