SENDLINE open source
open source / mongo4s / design notes

Design notes

← back to the README

Why mongo4s is shaped the way it is. The README says what the API does; this says what was chosen, what was rejected, and what it cost. Several of these decisions look arbitrary until you know the failure they were a response to, so the failures are here too.

Principles

  1. Scala 3 only. given-based throughout, derivation via Mirror and inline, no runtime reflection. Not cross-building to 2.13 is what buys macro field selectors, derives, opaque types and typed contextual parameters — the whole reason the API can be type-safe where a cross-built one cannot.
  2. The compiler is the test. A wrong field name, a missing codec, a value of the wrong type for a field: all of these should be compile errors, not runtime exceptions or silently empty result sets.
  3. No global state. No process-wide codec registry, no implicit ambient effect. Everything is resolved at the call site and scoped to the collection it belongs to.
  4. The driver is not hidden. underlying is public on client, database and collection. Anything mongo4s does not model is still reachable, and reaching for it is not a defeat.

The shape: mirror the driver

MongoClient[F, S] → MongoDatabase[F, S] → MongoCollection[F, S, A] follow the official driver's own hierarchy one-to-one. Someone who knows mongodb-driver-reactivestreams can guess where things are; someone who does not can read the driver's documentation and have it apply.

Two type parameters, not one: F[*] is the effect and S[*] is the stream. They are separate because no effect system bundles them — cats-effect pairs with fs2, ZIO ships its own ZStream, and the pairing is a user decision, not ours.

Every method that reaches the server takes (using session: Option[ClientSession] = None) — every one, that is, the driver has a session overload for; estimatedCount and the search-index methods have none to take. Sessions are therefore invisible until you opt in, and joining a transaction requires no change at the call site — the session is given implicitly by withTransaction and picked up by every call inside the block, including repository calls that know nothing about transactions. The alternative — a second overload of every method taking an explicit session, which is what mongo4cats does — doubles the API surface and makes a transactional call textually different from a non-transactional one.

Effect-agnostic by construction

mongo4s-core depends on the Mongo driver and nothing else. Not cats-effect, not fs2. A runtime module is two given instances:

scala
trait Effect[F[*]]:
  def pure[A](a: A): F[A]
  def delay[A](a: => A): F[A]
  def map[A, B](fa: F[A])(f: A => B): F[B]
  def flatMap[A, B](fa: F[A])(f: A => F[B]): F[B]
  def raiseError[A](error: Throwable): F[A]
  def handleErrorWith[A](fa: F[A])(f: Throwable => F[A]): F[A]
  def guaranteeCase[A](fa: F[A])(finalizer: ExitCase => F[Unit]): F[A]
  // suspend, unit, void, attempt, guarantee, onError, bracket, bracketCase, monotonic — derived, all overridable

Seven abstract members. That number is a deliberate ceiling: everything derivable is derived with a default so that supporting a new effect system is an afternoon, not a project, and so that adding capability later does not break implementors.

guaranteeCase is the load-bearing one. It is the only member that could not be dropped without losing something real. It sees ExitCase.Succeeded | Errored | Canceled, and the Canceled case is why an interrupted transaction rolls back immediately instead of lingering on the server until it is reaped, and why a canceled read releases its cursor. An effect abstraction built on Monad + MonadError alone cannot express that, which is the reason Effect exists at all rather than the library taking a cats.effect.Async constraint.

Error handling in the finalizer is defined rather than incidental: a finalizer that itself fails does not replace the error that caused it. It is attached with addSuppressed, so the original failure survives — the rollback error is what you want to see, but the error that triggered the rollback is what you need to debug.

RsBridge[F, S] turns a Reactive-Streams Publisher into F or S:

scala
trait RsBridge[F[*], S[*]]:
  def one[A](publisher: => Publisher[A]): F[A]
  def option[A](publisher: => Publisher[A]): F[Option[A]]
  def list[A](publisher: => Publisher[A]): F[List[A]]
  def unit[A](publisher: => Publisher[A]): F[Unit]
  def stream[A](publisher: => Publisher[A])(using Streamable[S, A]): S[A]
  def liveStream[A](publisher: => Publisher[A])(using Streamable[S, A]): S[A] = stream(publisher)

Every Publisher is taken by name. This is not a style choice. The driver opens a cursor when a publisher is subscribed to, and building an F[A] value is not running it — a by-value parameter would open a cursor for a computation that may never run, or run twice for one that is retried. By-name means the subscription happens when the effect does.

By name is only half of it. Whatever an operation computes before it calls the bridge is computed where the operation is built, and a refusal there is thrown at the caller rather than raised in F, where attempt never sees it. Up to 5.0 that was a publisher built into a val before the call — listIndexes, dropIndex, drop, runCommand, dropCollection, the listing streams — the pipeline aggregate rendered and the filter distinct rendered when they were called (an unclosed polygon ring is refused at render), the options createCollection converted, the page size of findPage and the step of Counters.next; and kyo's stream built its publisher when the stream was defined. Each of them builds inside the effect now. aggregate and distinct keep their stages and filter as the AST and render them when they run, as find always did. DeferredFailureSpec drives them against a client with no server behind it, so every refusal has to come back as a failed IO rather than a thrown exception, and each runtime's bridge spec checks that a publisher is built only when its effect or stream runs.

Streamable[S, A] is a marker typeclass with no methods. It exists so that every streaming call names its element type at the call site, which is what lets the kyo backend supply the per-element Tag its stream type requires. Without it, kyo could not be supported at all without changing every signature.

Codecs are per-collection, never global

The driver's model is a process-wide CodecRegistry: register Codec[A] once, and every collection typed A uses it. mongo4s does not use it for entity codecs at all. A codec is resolved as a given at the call site that creates the collection, and is attached to that collection alone.

The cost is one more implicit parameter. What it buys: medeia, zio-bson, calypso and bson-direct can all be in use in the same process, in the same service, on different collections — and a missing codec is a compile error naming the type, rather than a CodecConfigurationException under load.

The driver's registry still matters twice, and both are load-bearing:

  • It must keep supplying the driver's own BsonDocument codec. A registry that replaces the defaults instead of extending them fails every operation, because mongo4s reads and writes BsonDocument underneath. Pinned by CodecRegistryItSpec.
  • It governs everything past underlying. The escape hatch is the user's, and their registry is what applies there.

On the direct path the derived WireCodec[A] is registered ahead of the client's registry:

scala
CodecRegistries.fromRegistries(CodecRegistries.fromCodecs(derived), underlying.getCodecRegistry)

The other order was tried and was wrong: a Codec[A] in the user's registry silently shadowed the derived codec, so the collection wrote a shape nobody had asked for and no error was raised. Also pinned by a spec.

The codec decides how field names are spelled

WireCodecConfig renames fields on the way out; Filter/Update/Sort render them through a FieldNaming. Those were two separate settings with separate defaults, so a SnakeCase codec on a collection opened with the default identity wrote first_name and then queried firstName — every query matching nothing, silently, which is the exact failure this library says it exists to remove.

The first fix defaulted the collection's naming to the codec's. That was not enough: a caller passing identity explicitly to a snake_case codec brought the silent empty result straight back, and guarding against it meant asking the codec to police a parameter that only existed to contradict it.

So getDirectCollection has no naming parameter. On a direct collection the spelling is not a second decision: Filter, Update, Sort, Index and Projection all render the entity's own field names, and the codec is what wrote them. A derived codec reports the naming it was configured with; a hand-written one declares its own by overriding fieldNaming, which is where a hand-written codec's spelling is decided anyway. There is one source of truth and no way to contradict it.

getCollection keeps its parameter, and that asymmetry is deliberate rather than unfinished: a BsonDocumentCodec comes from medeia, calypso or zio-bson, each with its own naming configuration that mongo4s has no way to read. There the caller is the only one who knows, so the caller is asked.

"One source of truth" only holds if every codec tells it. For a while two did not: a derived sum type passed its naming to the code that built it and then reported identity, and imap/iemap built a fresh codec that forgot the naming of the one underneath. Both wrote first_name and let queries say firstName, the exact failure this section exists to prevent. Both carry it now, and a law spec checks that the naming a codec reports is the spelling it writes, for products, sums and mapped codecs alike.

The one naming is per collection, applied to every derived segment of a path. A nested type derived under a different WireCodecConfig writes its own spelling and is queried through the root's, which is why the README says to derive an entity and what it contains under one config. Following the codec tree segment by segment would fix it at the cost of making every path render depend on a codec, and no one has needed mixed naming inside one document.

Nothing derives without being asked

WireCodec's derivation used to be a given, which meant implicit scope produced one for any type with a Mirror — a nested case class nobody had modelled, an enum, a type inside a List. That reads as convenience and behaves as a hole: a field you forgot to think about gets a codec invented for it, silently, and the promise that a missing codec is a compile error naming the type held for the document path and not for this one.

It is now a plain inline def, so derives WireCodec still works and implicit scope derives nothing. Verified against the ecosystem rather than assumed: circe and zio-json refuse a field whose type has no codec and refuse an opaque type for the same reason, and the same probe run against this codebase's three backend bridges — medeia, calypso, zio-bson — shows none of them re-opens the hole, because none of them derive automatically either.

The one thing a sum must still derive is its own children. Scala gives the cases of an enum nowhere to write derives, so WireSumDerivation summons a child's codec if one exists and derives it otherwise. circe draws the line in exactly the same place, and for the same reason. A field of a product is the opposite case — there is somewhere to write derives, so it is required.

opaque type falls out of this for free and is worth stating: derivation cannot see through one, so an opaque wrapper over String keeps whatever representation its owner gives it rather than borrowing String's. That is the point of declaring one.

What still resolves without derives is the BsonEncoder/BsonDecoder bridge — that converts a codec you already wrote rather than inventing one, and it is how a medeia or zio-bson model reaches the direct path at all. Its cost is the one BsonValue per field it has always cost.

A field can be renamed or left out on its own, and queries follow it. The only rename was FieldNaming.overrides, keyed by Scala name and applied to every class derived under the config, so the generated code wrapped each class in a block with its own given WireCodecConfig to rename from to from_; leaving a field out took a hand-written codec that wrote null. @stored and @notStored are read off the constructor parameters by a macro, the same one that now summons the field codecs — per field, so a @notStored field's type needs none, and a stored field without one is named in the error. A rename only the codec knew would be the silent mismatch this library exists to prevent, so Field.of and selectAs read the same annotations: a @stored segment is a stored segment, and a @notStored field cannot be selected or queried at all. The old make stays for code that inlined the 5.0 derivation; the new path goes through build.

A sum can take the shape a collection already has. The discriminator was _type, always, and nested under value when a case was not a product; a consumer whose documents hold {"StateItem": {…}} modelled the ADT as a case class of four Options and bridged it with iemap. withDiscriminatorField, withWrappedCases and withFallbackCase are options on the config rather than annotations on the sum, since a collection's convention is the same for every sum in it. The fallback has to be a case object: a document of a case the reader has never seen cannot be read as the fields of one it has. A case of a sum derived under a different config is not refused, though it writes a spelling queries will not find: the generated repositories of one consumer derive every case under a naming of its own, and refusing at first use would have failed them in production. @stored is the way out.

Mapping a derived product keeps it a product. imap and iemap used to wrap any codec in a plain WireCodec, which a sum or an Either then nests under value — so the README's own advice, wrap a case class in iemap to validate what it reads, changed the stored shape of every case it was applied to. A mapped FieldCodec is a FieldCodec now, delegating its fields through the two functions. The documents that advice already produced are the cost: they carry the nested shape, and WireCodec.from(codec, codec) — a codec that is deliberately not a FieldCodec — is how they keep reading.

An unread field is marked, not null. A derived product told a field it had not read from one it had by a null in the slot, so a codec that reads null as a value — a Java type, a nullable interop field — was reported missing. The slots are filled with a marker first. Measured, because this loop is the decode path: decoding is unchanged within noise. Reading the codecs' lazy val into a local once per document, which the audit suggested, cost seven percent in readFields and was left out; writeFields takes the local and reads fields by position rather than through productIterator, and is four percent faster for it.

The performance work was measured on what the consumers store, and two of the audit's guesses did not survive it. The codec benchmarks had a seven-field entity; WideEntityBenchmark models the hottest consumer write — optional fields, a sum, an enum-keyed map of twenty-six-field scores — so a change shows where it matters. The audit blamed the Some that indexOf.get returns for part of decode's garbage: the allocation figure does not move when it goes, since the JIT removes it. It also suggested reading the codecs' lazy val into a local, which costs decode seven percent and is applied to encode only. What did show was a copy nobody had named — Tuple.fromArray copying each product's fields once more before the constructor read them — and on the rendering side, snakeCase boxing a Char per letter and the overlap check splitting every path into arrays. Three items were left alone on purpose: aggregateDirect assembling a codec registry per call, which the ten-document row of AggregateBenchmark shows lost in the round trip; the client-level bulk write going through BsonDocument, which would need a registry per client for a method no consumer calls; and running a repository's batches in parallel, which Effect has no combinator for and an ordered write must not do anyway.

A sum's cases are gathered by a macro, not by inline recursion. The audit expected wide products to hit the compiler's 32-deep inline limit; a probe put 120 fields through derives WireCodec without it, because summonAll and constValueTuple no longer recurse. The sum was what failed: its cases' codecs came from a hand-written inline method that called itself once per case, so an enum of 33 cases needed -Xmax-inlines in every consumer's build — which two of them carried at 200 and 256 without knowing why. The macro walks the cases' types and asks, per case, for the case's own WireCodec and otherwise its Mirror, which is what summonFrom asked. WideDerivationSpec round-trips sixty cases and sixty fields under the default limit.

The AST-free path

Every other Scala Mongo library encodes through an intermediate tree: case class → some JSON or BSON AST → org.bson.BsonDocument → wire bytes. bson-direct derives a WireCodec[A] that writes into the driver's BsonWriter directly and reads from its BsonReader, with no intermediate representation at all. It is handed to the driver through a two-method adapter, and the driver calls it with its own BsonBinaryWriter.

That is where the performance comes from — see BENCHMARKS.md — but the reason it is the default codec is dependency footprint: it lives in mongo4s itself, so the default path pulls in no third-party codec library.

Which scalars are native is a correctness question, not only a speed one. A type with no ScalarWireCodec falls back to the BsonEncoder/BsonDecoder bridge, which builds exactly the one BsonValue per field this path exists to avoid — and Instant, ObjectId, UUID and BigDecimal sit in almost every real entity, so the fallback was the common case rather than the exception. Each native instance writes and accepts exactly what the BsonValue path writes and accepts, leniency included. That symmetry is what lets the same collection be opened either way.

It also makes losing one invisible to behavioural tests: the fallback produces byte-identical documents, so dropping a native instance would cost allocations and nothing else — every round-trip and format test would still pass. The spec therefore asserts the structure, that the summoned WireCodec really is a ScalarWireCodec, because nothing observable distinguishes the two.

A ScalarWireCodec answers for its type in queries too. Up to 5.0 a filter, an update, a key, selectAs or distinct needed a BsonEncoder/BsonDecoder, and nothing derived one from the WireCodec a type already had: the consumers carried fifty given BsonEncoder[X] = xCodec.toBsonEncoder lines next to seventy-four scalar codecs, and a type whose line was forgotten simply could not be filtered on. The bridge cannot sit in bson-core, which does not know bson-direct, and an import would be one more thing to forget, so bson-core declares BsonCodecSource — toBsonEncoder, toBsonDecoder — and ScalarWireCodec extends it; the given that turns a source into a BsonEncoder sits in BsonEncoder's companion, where every search for one looks. Where it sits within the companion is the point. It is a parent of its own, unrelated to the default instances, so it is not in BsonInstances.given: imported lexically it would be searched before a type's companion and override a BsonEncoder written there by hand, and an encoder that differs from the wire codec would silently change. Only in implicit scope, it loses to that one and to every default. It is also not offered for document-shaped codecs: WireCodec already falls back to a BsonEncoder/BsonDecoder pair, and a pair that falls back to WireCodec would chase its tail; a product gets toBsonCodec on request. A probe confirmed the ordering before it went in — the bridge, a hand-written encoder in the companion, an imported codec, and the WireCodec fallback, each resolving to the intended instance.

It is strict by construction, and that is a trade, not an oversight. Derivation requires every modelled field to be present unless its decoder supplies a default (Option does). So a projection that drops a modelled field cannot be read back through the entity's own codec. Strictness catches the far more common bug, which is a document that quietly lost a field, so the codec keeps it — and selectAs answers the partial read instead.

selectAs names the shape rather than deriving it from the query. find(...).selectAs[(name: String, age: Int)] builds both the projection and the decoder from one named tuple, so they cannot drift apart. Reading the shape off the selectors instead — select(_.name, _.age), with the type inferred — was tried first and does not work: NamedTuple[N, V] is opaque with V as its lower bound, so a transparent inline result widens to the bare tuple and the labels are gone by the time the caller sees it. The bare tuple still conforms to the named type through that same lower bound, which makes the failure look like success until a field is read by name. Naming the shape in the signature keeps the type where the compiler cannot approximate it away.

Absence is decided by the decoder rather than by the presence of the key. A projection cannot distinguish a field the server omitted from one that was never stored, so selectAs offers the absent value to the field's own BsonDecoder first: Option reads it as None, everything else reports the field as missing. Checking the key before asking the decoder is what made a projected Option disagree with reading the whole entity — the same document, two answers.

aggregate and distinct take their output type from the call site rather than from the collection, so neither could use the collection's own WireCodec. For aggregate that mattered: a pipeline reading a large collection is exactly the hot path bson-direct exists for, and stopping at a BsonDocument there gave the AST-free claim an asterisk. aggregateDirect[B] takes a WireCodec[B] instead and registers it the same way getDirectCollection registers the entity's, so the output is read straight off the wire. It carries the strictness with it — the output type's fields all have to be present, the same rule a stored entity is read under.

The size of that asterisk is now measured rather than assumed: over ten thousand documents aggregateDirect is 1.4× the bridged path against a real server, and over ten it is a wash, because the round trip and the server's work are the whole cost at that scale. Both halves are in BENCHMARKS.md; the second half is the reason aggregate was never deprecated in favour of it.

distinct is left alone deliberately: it reads one BsonValue per result rather than a document, so there is no intermediate tree to skip and nothing to win.

$slice and $meta are shapes the AST was missing

Sort used to be a list of (FieldPath, Boolean), which can say ascending or descending and nothing else. $text had been supported as a filter since 2.0 while the thing you almost always do with it — rank by relevance — had no representation at all, so the feature was half a feature. SortOrder replaces the boolean; TextScore renders {"$meta": "textScore"} and takes an output field name rather than a model field, because that name is invented by the query and must not go through FieldNaming.

$slice is the other shape that did not fit: it is neither an inclusion nor an exclusion, which is exactly why the two-type split could not express it. Rather than add a fourth projection type, all three carry a list of slices, so slice chains onto any of them without committing the projection either way — which matches what the server accepts. The fake simulates it, because taking the first or last N of an array has one right answer; the parity spec checks that answer against a real server for all three forms.

Sorting by text score is the opposite case and the fake refuses it: a score only exists where a text index does. That check runs when the sort is applied rather than inside the comparator, because a comparator is never called for a single-document result and the refusal would have depended on how much data a test happened to insert.

explain returns a document, not a type

The point of a Filter AST is that the query is a value you can build and pass around; the question that follows is whether the server can serve it from an index, and until now nothing in mongo4s could answer it. explain runs the publisher the builder had already configured, so what is explained is the query that would have run, hint and collation included — not a reconstruction of it.

Its result stays a BsonDocument, and that was checked against a real server rather than assumed. Three findings decided it:

  • The top level is a union chosen by the server, not by the caller. aggregate(...).explain() on $match + $limit came back with a top-level queryPlanner and optimizedPipeline: true, because MongoDB collapsed the pipeline into a plain query; the same call on a $group came back with a top-level stages array whose first element wraps an entire queryPlanner under $cursor. One method, two structures, and which one you get depends on an optimisation decision you cannot see from the call site.
  • The plan is a recursive tree over an open set of stages, each with its own fields — COLLSCAN carries direction, IXSCAN carries keyPattern/indexBounds/multiKeyPaths, LIMIT carries limitAmount, PROJECTION_SIMPLE carries transformBy — and a sharded cluster adds a layer per shard.
  • The server versions it. Every response carries explainVersion: "1", which is the server saying it intends to change the format.

A case class mirroring that would compile today and quietly stop matching later. So the raw document is the API.

What is stable is the handful of questions a plan gets asked, and ExplainSummary types those: which indexes were used, which stages appeared, whether a COLLSCAN or a blocking SORT is in there, and the four execution counters. It derives them by walking the document for field names rather than by following a path, which is why one summary type covers all three observed shapes — and why the array case is tested, since that is how both a staged pipeline and a sharded plan nest.

This is deliberately the opposite call from MongoError.DuplicateKey, which refuses to name the index it violated. The difference is the source: there the only thing on offer is a human-readable sentence, and parsing prose is a guess. Here the input is structured, documented BSON with field names stable across every shape observed, and anything unrecognised produces an empty summary rather than a wrong one.

Errors the server raises

mongo4s translates every driver failure into MongoError before any runtime sees it. The reason is the same one behind Filter being an AST rather than driver builders: a library that exists so you never write com.mongodb.* in service code cannot then require you to catch com.mongodb.MongoWriteException and compare getError.getCategory to decide whether a unique index fired.

Where it happens is the interesting part. There is no single point in the bridge: cats and kyo collect through PublisherCollector, zio goes through zio-interop-reactivestreams, and rapid runs everything — including list and unit — through PublisherIterator. Translating in any one of those would have covered some runtimes and some methods. So the translation wraps the Publisher itself, upstream of all of it, which is the only place all four runtimes and all five bridge methods provably meet. RsBridgeBackendSpec asserts it for one, option, list, unit and stream on every runtime, because "wrap the publisher here too" is exactly the kind of thing a fifth runtime would forget.

Classification comes from the driver, not from a table of numbers. ErrorCategory.fromErrorCode already knows which codes mean a duplicate key or an execution timeout, so MongoError asks it instead of hard-coding 11000 and 11001. Only WriteConflict and the two authorisation codes are named here, because the driver has no category for them.

Classification follows the exception's meaning, not where the driver put it in its hierarchy. Three driver classes sit under a parent whose meaning is wrong for them, and each was misfiled until 5.1. MongoOperationTimeoutException — the client's timeoutMS ran out — extends the server-selection MongoTimeoutException, so it read as Unavailable; worse, withTransaction re-commits an unknown commit result unless it was a timeout, and kept re-committing a commit that had already used up its timeoutMS. Refused credentials arrive as MongoSecurityException, a client exception with code -4 and no category, so they read as Failed. A primary stepping down and a node recovering are command exceptions with codes the driver files nowhere, though they mean exactly what Unavailable promises: retry somewhere else, later. They are matched by class now, ahead of their parents.

A write-concern failure is not a failed write, so it is not a case of its own: the document was written, and what failed is the acknowledgement. It is reported on whatever case the exception lands in, as writeConcernErrors, because BulkWriteFailed with an empty failures was otherwise indistinguishable from nothing having gone wrong. A new case would also have broken every exhaustive match over MongoError in a minor release.

A bulk write keeps all of its failures. One bulkWrite or insertMany can fail several documents at once, and collapsing that to a single DuplicateKey would silently drop the rest, so BulkWriteFailed carries every failure with the index of the operation that produced it, and duplicateKeys is a filter over them rather than a different shape.

DuplicateKey does not name the index, on purpose. The server does report keyPattern and keyValue, but the driver's WriteError.getDetails comes back empty for a duplicate key — verified against MongoDB 7 and 8.2 — so the only remaining source is the human-readable message. Parsing it would produce a field that works until a server upgrade rewords the string, then silently returns nothing. The message stays reachable through getMessage.

Translating changed one thing that had nothing to do with errors: withTransaction decides what to retry by asking whether the failure carries TransientTransactionError, and it used to do that by matching MongoException. After translation it no longer sees one. MongoError therefore exposes labels/hasLabel from the wrapped exception, and hasErrorLabel accepts both — the existing retry specs caught this on all four runtimes, which is the argument for having had them.

One numeric rule, two paths

The two decoding paths used to disagree about numbers. BsonDecoder accepted any numeric BSON type for any numeric Scala type, refusing only what would lose information — not whole, or out of range — and that logic had been sharpened over several commits: Decimal128 read exactly rather than through Double, the 2^63 boundary closed, a non-finite value returned as an error instead of a thrown ArithmeticException. The wire codecs had none of it: readInt64 and nothing else. So the same document under the same model read through getCollection and failed through getDirectCollection, with a raw org.bson.BsonInvalidOperationException — a driver type that is neither a BsonError nor a MongoError, escaping both of the error stories this library tells.

The fix is not a second copy of the rule, which is how the paths drifted in the first place. A wire codec now checks the BSON type it is looking at: the declared one is read straight off the reader, and anything else is handed to BsonDecoder — the same instance the AST path uses, reached through its companion. Correctly typed data still allocates nothing; a width mismatch costs one BsonValue, on the read that would previously have thrown. A wrong type now arrives as a BsonError.TypeMismatch rather than a driver exception, which is worth as much as the leniency.

The same shape handles a well-typed value that is still unreadable — a malformed UUID string, a Decimal128 that is NaN. The fast path attempts the conversion and, if it throws, re-runs the value it has already read through BsonDecoder, so the error is the one the AST path would have produced rather than whatever the conversion happened to throw. BsonError.fromThrowable was giving those back re-wrapped as Thrown; it now unwraps a DecodingFailure to the error inside, so a decoding failure reads the same however it reached the surface.

ScalarWireCodecParitySpec runs both paths over every numeric width and asserts they agree — value for value and message for message — which is the guard that keeps them from separating again.

A wire decoding failure carries its path. Up to 5.0 a derived product let a field's failure through bare, so a consumer's forty-field entity failed with Expected string but got null and nothing to say which of eleven string fields was null, or in which nested document. Each field read, array element, map value, sum value and Ior side now wraps a failure in BsonError.Nested as it passes — only on the failure path: the wrapper is an inlined try/catch, so a successful read allocates nothing more and the segment's name is built only when there is something to report. A reader exception (an array where a string was expected) is caught at the same point and reported as a DecodingFailure under the field's name rather than escaping as a driver type. The cost moved to the structure: code that matched a top-level TypeMismatch now sees it inside a Nested.

An operator is declared with the case that renders it

Filter, Update, Stage and Accumulator each carry val key — the MongoDB operator the case renders as — declared on the case itself rather than repeated in toBson:

scala
enum Stage[E](val key: String):
  case MatchStage(filter: Filter[E]) extends Stage[E]("$match")
  case Limit(n: Int)                 extends Stage[E]("$limit")

The rendering then reads BsonDocument(key, ...) and cannot name a different operator than the case it is matching on. Sixty-nine operators live on declarations this way; the dozen strings left in the rendering code are modifiers inside a payload — $each and its friends under $push, $geometry/$minDistance under $near — which belong to no case and are correctly still written where they are used.

The honest limit of this: the exact-JSON specs already caught a mismatched operator, so what changed is locality, not safety. What it buys is that the enum now reads as a table of our AST against MongoDB's vocabulary, and that things previously buried in toBson are visible at a glance — Accumulator.Count renders as $sum, Lookup and LookupPipeline share $lookup. The fake also names a refused stage with stage.key instead of rendering the whole stage just to read its first field.

Eight cases carry an empty key, and that is the shape rather than an oversight. Stage.Raw, Update.Raw, Filter.Raw and Accumulator.Raw carry whatever the caller wrote; Filter.MatchAll is the empty document and Update.Combine is a container rather than an operator; Filter.Eq and Filter.Regex are spelled {path: value} by MongoDB, with no operator at all.

Near was split into Near and NearSphere to make this hold. It used to be one case with a spherical: Boolean, which meant its operator was chosen at render time — the one place a declared key would have been a lie. A boolean discriminator inside a case of an ADT is the modelling smell that an enum exists to remove, so the flag became the case. field.near/field.nearSphere are unchanged.

A transaction reaches as far as the session is in scope

Every operation takes (using session: Option[ClientSession] = None), and withTransaction supplies it through a context function. That gives the good case for free — everything written inside the body joins the transaction with no ceremony — and it hands you one sharp edge: a helper compiled somewhere else took the default, so its writes commit outside the transaction and survive a rollback.

This is not a defect that can be closed by a better default. Implicit propagation is lexical: a method compiled against no session cannot later be told about one, whatever the parameter's type or default. The alternatives are to remove the default and make every non-transactional call pass None, or to thread a session-bound collection through every helper — each buys the edge case at a cost paid on every ordinary call.

So the contract is written down instead, and pinned: SessionScopeItSpec runs both shapes of helper inside a real transaction that then rolls back, and asserts that the one taking (using Option[ClientSession]) loses its write while the one without it keeps it. Documenting a trap without a test that reproduces it is how the documentation drifts away from what the code does.

Cancelling a stream has to wake whoever is waiting on it

rapid reads a publisher through PublisherIterator, whose hasNext blocks on a queue. Cancelling it set the flag and cancelled the subscription — correct as far as the publisher is concerned, and useless to a consumer already parked in queue.take(), which had nothing to wake it. A cancelled or timed-out stream therefore left a thread blocked for the life of the process.

Cancellation now enqueues the end of the stream, under the same terminated compare-and-set that onComplete and onError use, so exactly one terminal reaches the consumer whichever of the three happens first.

The equivalent question for the other three runtimes — does a timeout actually cancel the subscription — was already answered: RsBridgeBackendSpec asserts it for one, option, list and unit on all four, which is what a cross-runtime spec is for.

The query AST

Filter and Update are real enum ADTs that mongo4s interprets itself, not thin wrappers over the driver's opaque Bson builders. That is what makes a second interpreter possible at all (see repositories), and it is what lets the AST be normalized rather than passed through.

Field paths track derived-vs-stored per segment, not per path. Field.of[E, A](_.some.field) is a macro that reads the selector at compile time; each segment it produces is a derived name, spelled through the collection's FieldNaming. Names that are already what the document stores — a map key, an array index, _id, a PrimaryKey's field names, a $lookup's foreign field given as a string — are stored names, used verbatim. Per-segment is the only granularity that works: totals.at("EUR") has to rename totals under snake_case while leaving the map key alone.

A typed path continues through an Option and into a collection's elements, with derived segments. Up to 5.0 Field.of stopped at an Option[Address] — Option has no case fields — and addressField / cityField needed cityField typed against Option[Address], so the only way on was addressField / "city", a stored segment the naming never touches. Under snakeCase that path matched nothing, silently, which is the bug this library exists to remove; a consumer had Option[List[...]] fields it could not reach at all. / now takes the inner type's Field when the outer one holds that type — exactly, in an Option, or as the elements of a collection, the last also through an Option — and appends derived segments. Teaching the macro to read _.address.get.city was the other option, and was turned down: it reads like code that runs, and .get is the one call nobody should be writing. Through an array the path has MongoDB's read semantics, which suits filters, sorts, projections and indexes; updating through it is refused by the server, loudly, and each/matched/where remain the way to name elements in an update.

Adding the overload exposed a resolution rule: two extension methods share their first parameter list — the receiver — and Scala resolves the second list before fixing the receiver's type parameters, so /(inner: Field[A, B]) matched an inner field of any outer type and the pair was ambiguous. The exact form is therefore no longer an extension. It stays in Field as a plain method, binary-identical (nested), for code compiled against 5.0; the one extension / takes a DocumentOf[A, A2], whose exact instance covers the case the old method served.

A join's foreign side is spelled by the foreign collection. lookupWithKeys and graphLookup type their foreign fields against the foreign entity, but up to 5.0 rendered them — and the foreign sub-pipeline — through the naming of the collection running the aggregation, while this record claimed they were stored names. One identity collection joined to one snakeCase collection produced a $lookup on a misspelled field and empty arrays, silently. A foreignNaming parameter defaulting to identity was the obvious fix and the wrong one: it would have broken every project whose collections already share snakeCase, which is most of them. Instead each form has an overload taking the foreign MongoCollection: the handle carries both the name and the naming, and its entity type fixes B. The foreign parts are rendered through that naming when the stage is built — paths become stored segments, the sub-pipeline becomes raw stages — so the Stage cases did not change, and nor did the string forms. Scala refuses default arguments on two overloads of one method, so the handle forms spell let and options as a trailing argument with an overload that leaves them out.

Field[E, A] is opaque over FieldPath, which is opaque over List[Segment]. Both aliases being transparent inside the same file is a live hazard: Field./ calling field.path / segment resolved to its own / and became an infinite loop. Field./ and Field.at must go through the named FieldPath.stored, never the symbolic operator.

Filter.and/or fold all and none away instead of emitting a one-element $and. This is not cosmetic: it is what makes field.in(Nil) produce Filter.none rather than an empty predicate, so deleteMany(Nil) is a safe no-op instead of deleting the collection. An empty list is the most dangerous input a query builder takes, and it gets a defined answer here rather than an emergent one.

What none renders as matters as much as when it is produced. Up to 5.0 it was {$nor: [{}]}: correct, and planned by the server as a collection scan, so an in over an empty list read every document to return none — a consumer carried a guard at the call site and an integration test to keep the query from being sent at all. It is {_id: {$in: []}} now. Every collection indexes _id, so the plan never scans, whatever field the empty in was on; and the same shape stays correct inside $elemMatch and $pull, where an element without an _id simply does not match. Or(Nil) and the negations of all and none are normalised at render time to the same two shapes, since the enum cases are public and get built directly.

Update merges operators of the same name into one sub-document. {$set: {a: 1}} combined with {$set: {b: 2}} is {$set: {a: 1, b: 2}}, not two $set keys the server would reject. Update.Raw merges the same way — but it must clone the caller's document first. Not cloning meant a rendered update wrote into the caller's own BsonDocument: a shared Raw came back permanently carrying fields it never declared, and rendering it twice under different FieldNamings emitted both spellings. An update that would render empty throws rather than sending {}.

Merging cannot keep two changes to one path, so it refuses to try. A sub-document holds one value per key, and for a while the second $inc on a path simply replaced the first — the update that went out was not the one that was written, and nothing said so. The fake, applying the two in turn, answered 3 where the server answered 2, so the same code passed its tests and did something else in production. An update that touches a path twice is now an IllegalArgumentException at render, and so is one whose paths overlap across operators, which the server refuses too — found at the call site rather than as a write error one round trip later. The fake renders an update before applying it, so it refuses the same ones.

A pipeline update is a case of Update, not a separate type. Everything that takes an update — the three update calls, a bulk command, a repository's …ByFilter — takes one without growing an overload for it, which is the point. It renders through toPipeline rather than toBson, since it is a list of stages, and it is refused where the two forms would have to mix: the server takes one or the other, never both.

An operator that only applies to one kind of field says so in its signature. regex asks for TextOf, mod for NumericOf, hasSize and $pop for ElementOf — the same capability pattern the array and numeric operators already used. What the missing bounds allowed was a filter the server accepts and that matches nothing, which is the failure mode the typed AST is there to rule out. A stage option works the same way: $bucketAuto only rounds its boundaries to a granularity over numbers, so bucketAutoRounded asks for NumericValue and the plain bucketAuto takes any field.

A stage that takes several fields takes them one at a time. Scala 3 will not form Field[E, ?] over an opaque alias, so a vararg of fields of unrelated types has no type to be. Stage.unset(a).and(b) builds its list the way Sort and Projection do, and cannot be empty by construction; $densify and $setWindowFields, which came first, take FieldPaths instead.

A window function is not an accumulator. $rank, $shift, $locf and the rest are valid in $setWindowFields and refused in $group, so making them Accumulator cases would have let groupBy("r" -> rank) compile and fail on the server. They are a WindowFunction enum instead, and WindowOutput and WindowFunction share a sealed parent, WindowField, which is what setWindowFields takes. That widened its parameter and the SetWindowFields case's field from WindowOutput to WindowField — a change to the generic signature only, the erased one is Seq/List either way, so MiMa reports nothing and every existing call compiles. The window $derivative and $integral need is a parameter rather than a .over, because the server refuses them without one.

Bitwise operators get their own capability, BitwiseOf. NumericOf admits Double and BigDecimal, which $bit and the $bits filters refuse, so they ask for an integer field — Int, Long, or either in an Option — and a mask of the field's width.

A minor release can add a case, and a trait member with a body, but not an abstract member. Checked rather than assumed before 4.1.0, by adding each to the code and running MiMa against 4.0.0: a new Stage case and a concrete MongoCollection member pass, an abstract MongoCollection member is a ReversedMissingMethodProblem. That is why 4.1.0 grew the ASTs and nothing on the collection or database traits.

Numeric operators go through NumericOf[C, A], mirroring ElementOf[C, A], so an Option[Long] field takes a plain Long. There is deliberately no NumericValue[Option[A]]: None has no numeric encoding, and the obvious stand-in — $inc by zero — is a write that silently does nothing.

set(None) means what the codec means by None. Up to 5.0, field.set(None) encoded the None with BsonEncoder[Option[A]] and wrote null, while a derived WireCodec leaves a None out of the document. One entity had two stored shapes depending on which write touched it last, and $exists, sparse indexes and $type answered differently for each; a consumer wrote nulls for months without knowing, with a codec spec asserting the opposite. The README called it a feature — $set: null and $unset are different intents — and that is true, but the intent behind set(None) on a field the codec omits is almost never "store a null". So set(None) is $unset now, and setNull is the spelling of the other intent, offered only on an Option field. setOnInsert(None) is not $unset — that would remove the field from a document the upsert matched, which $setOnInsert must never touch — it adds nothing, so the inserted document lacks the field the way an encoded None does.

Symbolic aliases (===, =!=, >, >=) exist where they read better, each with a @targetName.

Reads: strict by default, lenient on request

all fails the whole query on the first undecodable document. That is the right default: a decode failure usually means the model and the collection disagree, and finding out immediately beats processing 99 documents and silently dropping one.

But a collection written by more than one version of more than one service will contain documents this version cannot read, and failing the whole page because of one of them is useless. So attempting — on find, aggregate and distinct — reports each document as DecodeResult[A] = Either[BsonError, A]. Transport errors still fail the effect; only decoding is made recoverable, because only decoding is a per-document property.

watchAttempting puts the result one level further in, on the event's documents — ChangeEvent[DecodeResult[A]]. It used to wrap the whole event, and an event whose document failed lost its resume token with it, so a durable consumer could neither checkpoint past the bad document nor avoid reading it again after every restart. The envelope is the server's and always readable; only the document is the model's.

Streaming

Operations expecting at most one document read two, not the whole cursor (SingleResultProbe = 2) — exactly enough to notice a second result under RsBridgeConfig.strictSingleResult, and no more. AggregateQuery.first pushes a $limit into the pipeline for the same reason: bounding work at the server, not in the client.

liveStream exists because fs2 chunks. fs2.interop.reactivestreams.fromPublisher(p, bufferSize) will not emit an element until the chunk is full or the publisher finishes — its own scaladoc says so. With the default bufferSize = 256, a change stream delivered nothing until 256 events had piled up, and a change stream never finishes. find(...).stream only ever worked by accident: its cursor completes, which flushes the partial chunk.

So the bridge has two stream methods. Finite reads keep stream and its buffering, which is what makes them fast. Every watch goes through liveStream, which the cats backend overrides to fromPublisher(p, 1). liveStream has a default implementation delegating to stream, both because of the compatibility promise and because zio (queue-based), rapid (an enqueue-in-onNext iterator) and kyo (EmitStrategy.Eager) have no chunk-fill behaviour to work around.

This was found only because change streams had never actually been exercised — the integration spec was calling cancel on itself and reporting green. Two lessons kept from that: never let an integration spec cancel itself into looking green, and never time a change-stream test with sleep (take the server's operationTime from hello and pass startingAt(now), which is race-free no matter when the stream actually subscribes).

RsBridgeConfig exposes bufferSize, a per-operation timeout and strictSingleResult. The timeout deliberately does not apply to streams: a change stream sitting idle is working, not stuck.

Transactions

scala
client.withTransaction {
  users.insertOne(User("2", "Bob", 41)) // the session is already given here
}

It commits on success and rolls back on failure and on cancellation. The cancellation half is the whole reason Effect.guaranteeCase has the shape it does — and three backends keep it: the rapid one builds guaranteeCase from attempt, which sees success and failure only, so a cancelled rapid transaction is not rolled back. A rollback that itself fails is attached as a suppressed exception rather than replacing the original error.

Retries follow the driver, including which failure means what. A TransientTransactionError says nothing was committed, so the whole transaction is restarted — body and all. An UnknownTransactionCommitResult says the commit may already have landed, so repeating the body could double the write; only the commit is asked again. Treating the two the same in either direction is a data-loss or double-write bug, which is why they are separate paths rather than one retry loop.

Both are bounded by a single deadline taken before the first attempt, so the retries are bounded in total rather than per round — an unbounded chain of individually-quick attempts is the failure mode a per-attempt limit misses. That deadline needs a clock, and Effect had none: monotonic was added with a default reading System.nanoTime, which is what every runtime's own monotonic clock does anyway, so no implementor had to change. cats and zio override it with their runtime's clock, which is what makes the retry window drivable by TestControl/TestClock instead of by waiting.

Retrying is the default because it is the driver's default. TransactionOptions.withoutRetries restores the 2.x behaviour of reporting the first failure.

The retry rules are the specification's, not an approximation of them. Two had drifted. A commit that ran out of maxCommitTime carries the "unknown commit result" label — the driver adds it — so it was asked again until the retry window closed, which turned the limit into a delay; the specification excludes that error from commit retries, and so does this loop now. And a whole-transaction retry pauses first, for a jittered interval that grows by half each attempt from 5 ms to a 500 ms ceiling, so two clients that conflicted do not retry in lockstep. The pause needed a sleep Effect did not have; it has one now, with a default that blocks a thread and an override on every runtime that does not.

A transaction body is built inside the protection, not before it. fa used to be evaluated while the transaction was being wired up, so a body that threw before returning an F — a require, an argument that failed to build — escaped the finalizer and left the transaction open on the session. The default Effect.bracketCase had the same hole, and it was the one kyo and rapid used.

kyo had two more, both about where an interrupt can land. Its bracketCase was the default — acquire, then install the guard — so an interrupt arriving as the acquisition finished left a session open with nothing to close it; a test interrupting two hundred brackets at random points lost about one in a hundred. And its guaranteeCase ran the finalizer of a body that succeeded after the scope had closed, where an interrupt could skip it. kyo 1.0.0-RC7 has no mask that hands back a way to restore interruption, so bracketCase installs the guard first and runs the acquisition in an uninterruptible fiber; the two settle between themselves who goes first, and a guard that comes second waits for the acquisition to finish and releases what it produced. Every finalizer of guaranteeCase now runs inside the scope, which kyo runs whatever happens, and since kyo only logs a finalizer's failure, the failure of the one for a success is kept and raised once the scope has closed.

What the scope tells its finalizers about how it ended is not what ExitCase needs, though. kyo does not resume a fiber interrupted while it waits — on the server, in a sleep — but abandons it, and closes what the fiber owed with KyoException("fiber abandoned") rather than with the interrupt, so 5.1.0, which read the scope's outcome, reported every such cancellation as an error. Nothing caught it: no runtime's spec cancelled a body inside guaranteeCase, and a transaction aborts on Errored as on Canceled. guaranteeCase now records how the body ended as the body ends; a scope that closes with nothing recorded closed on a body that never reached its end, which is a cancellation. The finalizer still runs in the scope, where kyo runs it whatever happens. Recording the ending costs a guaranteeCase 2–4% and 44 bytes; running the body under Async.uninterruptible and the finalizer from a release of its own, the other way to tell, would have forked a fiber for every call, which is most of what a kyo bracket already costs.

The manual path (startTransaction/commitTransaction/abortTransaction with (using Some(session)) at each call site) is still there, and nothing is automatic on it, including the rollback. Both exist because the safe version should be the easy one, not the only one.

Change streams

Every event is a ChangeEvent[A] — a real case class with operationType, documentKey, fullDocument, fullDocumentBeforeChange, updateDescription, resumeToken and clusterTime — not a raw BsonDocument to dig through. watch exists at all three driver scopes: client (whole deployment), database, and collection.

fullDocument defaults to UPDATE_LOOKUP, not the server's default. MongoDB fills the document in only for inserts and replaces, which leaves the most common question — "what does this document look like now?" — unanswered on exactly the events you are usually watching for. The server's default is the surprising one; this is the useful one.

resumeAfter and startAfter clear each other, because the server rejects a stream carrying both.

One known footgun, accepted deliberately: WatchOptions[E].pipeline is typed Seq[Stage[E]], but a change stream pipeline matches against the change event envelope ({operationType, fullDocument, ns, …}), not the document. So a Field.of path renders "age" where the event needs "fullDocument.age". Stage.raw is the right tool there. The alternative was a second, parallel Stage type for events; that was judged worse than one type with a documented sharp edge.

Repositories, and one AST with two interpreters

BaseMongoRepository gives CRUD, batching and paging over a collection. It is open, not generated — extra domain queries are declared directly against collection/Filter/Field, the same escape hatch the driver gives you. A generated repository would have to either predict every query or make you drop out of the abstraction to write one.

Batching is empty-list-safe throughout, which follows from Filter's folding: inFilter(Nil) is Filter.none, so deleteMany(Nil) deletes nothing.

A key can be named by fields rather than by strings. A PrimaryKey was a list of stored names plus a function from key to values, so a consumer wrote every compound key three times — the extractor, the names, and a KeyFields.of("provider_id" -> BsonEncoder[ProviderId].encode(key._2), …) — and single("userId") under snakeCase silently matched nothing. PrimaryKey.of(_.provider, _.providerId) reads the paths off the selectors with the same macro as Field.of and keeps them as derived segments, so the naming applies when the key renders. That needed the key's filters to be built from paths, not names: KeyRef gained keyPaths, concrete and defaulting to the stored names, and eqFilter, inFilter, afterFilter, keyOrder and ensureKeyIndex are built from it. A key that KeyRef users wrote by hand keeps compiling and rendering as before; fieldNames of a key built by of holds the unrenamed names, since there is no collection to rename them against yet.

New repository methods carry bodies on the trait, and some of those bodies throw. One consumer implements Repository by hand, so a new abstract member would break its build and MiMa alike. findOneByFilter and estimatedCount can be built from methods every implementation has, and are. updateByKeys, updateEach and upsertKeeping cannot: the trait has neither a PrimaryKey nor a codec. A default that quietly did something else — upsertKeeping as a plain upsertMany, say — would be the fake that lies, so the trait's body throws, naming BaseMongoRepository as the implementation that has them. upsertKeeping takes the kept fields as FieldPaths, since a wildcard Field[E, ?] over an opaque type does not reduce.

direct answers a Repository, not a BaseMongoRepository. The consumers built every repository over a WireCodec through withoutId, which takes a BsonDocumentCodec: they bridged the codec by hand, passed FieldNaming.snakeCase again although the codec already said it, spelled all four type arguments because nothing fixed E and K, and called .widen because F[BaseMongoRepository[…]] is not an F[Repository[…]]. Their hottest write path ran through BsonDocument twice as a result. BaseMongoRepository's public surface is exactly Repository's, so answering the trait costs nothing, and it lets the type the result is assigned to decide E and K. createDirect stays as it was; changing what an existing factory returns is a binary break, and a default argument cannot be added to it either, so the projection that withoutId exists for is a parameter of the new one.

BulkWriteResult.upsertedIds is keyed by command position, so combine deliberately does not rebase indices; a caller splitting one logical write into batches calls shiftUpsertedIds(offset) per batch first, since only the caller knows how many commands preceded it. Getting this wrong collapsed every batch onto index 0 — a bug that only appears once a write exceeds one batch, which is to say in production and not in tests.

The same rebasing had been done for successes and not for failures: a duplicate in the third batch of 500 came back as index = 0, and its partial result held only the third batch. A caller retrying "the failed command" retried a different one. The repository now reports a batched failure the way the driver reports its own internal batches — one MongoBulkWriteException for the whole write, global indices, and a result counting everything that landed — which keeps BulkWriteFailed a single shape whether the batching was the driver's or the repository's.

Because Filter/Update are an AST rather than driver builders, they can be interpreted twice: once into real Bson for the server, and once against an in-memory buffer. FakeMongoCollection is that second interpreter — filters, every update operator, sorting, paging, projections, distinct and a subset of aggregate are simulated, while watch, explain, $text, $expr, the geospatial operators, Filter.Raw, Stage.Raw and an update carrying arrayFilters throw UnsupportedOperationException naming what was asked for rather than quietly answering wrong. A fake that lies is worse than no fake.

The aggregation subset is drawn along one line: whether MongoDB's answer is unambiguous. $match, $sort, $skip, $limit, $project, $count and $group over $sum/$avg/$min/$max/$first/$last/$push all have exactly one right answer, down to the BSON type — $count and a $sum of Int32s yield an Int32, $avg always yields a Double. $addToSet does not: MongoDB leaves the order of its result undefined, so any array a fake returned would be a guess a test could then depend on. It is refused for that reason rather than for effort, as are $sum/$avg over Decimal128, whose rounding is the server's to define.

The subset is not maintained by reading the manual. FakeAggregateParityItSpec runs the same pipelines through the fake and through a real MongoDB and compares the raw BsonDocuments, so a divergence in value or in numeric width fails the build. That spec is the reason the paragraph above can state the types as fact.

Order is the one thing the fake deliberately does not promise: where MongoDB leaves it undefined, so does the fake, and the two need not agree — $group emits buckets in first-seen order, exactly as find without a Sort returns insertion order. A pipeline whose output order matters has to end in $sort against a real server too.

It ships as its own published module, mongo4s-testkit, so it is usable from a consumer's own tests rather than only inside this build. Alongside it, FakeRepository is a BaseMongoRepository over a FakeMongoCollection — the real repository logic, only the collection faked. Nothing about repository behaviour is reimplemented for tests, so the fake cannot drift from what production does; the only thing standing in for MongoDB is the storage underneath.

What the fake got wrong, and why the first parity spec missed it

The fake shipped a set of answers that differed from MongoDB's: a filter on an array compared the whole array rather than its elements, $regex anchored, $inc truncated to a Long and widened the field, $count reported zero where the server reports nothing, findOneAnd* ignored their sort, $eq null did not match a missing field, and ordering threw on anything that was not a number or a string. Each one makes a test pass that production fails — precisely what "a fake that lies is worse than no fake" was written to prevent.

The parity spec that was supposed to catch this did not, and the reason is worth keeping: it seeded every row with every field and matched non-empty sets. Those are the conditions under which none of these divergences show. A parity spec is only as good as the shapes it seeds — missing fields, empty results, arrays, and the widths a value can take are where two implementations actually part company, so FakeFidelityParityItSpec runs each of them against a container and compares the fake's answer to the server's.

It also fails on the pre-fix fake, all fifteen cases, which is the only evidence that it tests anything.

A second round found the combinations the first did not seed: a duplicate within one insertMany, an unordered bulk past a duplicate, a projection that left a document the codec could not read, an update touching one path twice, a unique secondary index, a stream built before an insert, a no-op update, a replacement without an _id, and numbers of different widths. One of them was pinned by a test that expected the fake's wrong answer, which is a reminder that a green suite describes the implementation it runs against, not the server. Each is now a parity case.

A third round came from the 5.0.1 audit, and its lesson was where a path goes rather than what a value is. The fake read a path one way everywhere — through documents, never by position — and the server reads it three ways: a query walks every element of an array but not into an array inside it, a numeric segment means a position and a field of that name, and an update or a projection reaches into each element by its own rules. Every shape the audit listed failed its new parity case before the fix, the server's answers came out as expected in all of them, and two cases turned up something nobody had listed: an insert stored a generated _id last where the server stores it first, and a replacement counts as unmodified only when it is byte-identical to the stored document as sent — one without an _id counts as modified even when the result is the same. FakePathParityItSpec, FakeUpdatePathParityItSpec and FakeReadParityItSpec hold them.

The fake streams through the RsBridge rather than an emit function the caller supplies. emit took the results as a list, so they were computed when the stream was built and a stream set up before an insert missed it — the one thing a cursor never does. It also meant aggregate, distinct and selectAs could not stream at all, because nobody supplied an emitter for their types. A Publisher that computes its values on subscription, handed to the bridge the runtime already provides, fixes both and removes a parameter every caller had to get right.

The fake's indexes answer as the server's do. listIndexes used to list bare key documents, so a test that read an index's name or unique flag read nothing, and passed or failed for reasons of its own. Re-creating an index recorded it twice, a clash with an existing name or key pattern went through, and dropIndex of a missing index was silent. Adding createIndexes and dropIndexes was the point to fix all of it, each case pinned by a parity spec with the server's own error codes.

The fake takes one lock, not a concurrent map. Its storage was a bare ArrayBuffer, and a test that shared it between fibers saw inserts vanish or throw, two documents with one _id, and a thousand concurrent $incs add up to a few hundred. A TrieMap keyed by _id was the obvious replacement and fixes only the first: every write here is check-then-act — the duplicate check, a unique index scanning the other documents, read-modify-write for $inc, updateMany and a bulk write touching several — so each would need its own CAS loop and some cannot be one at all. It would also lose insertion order, which find without a Sort, updateOne's choice of match and $group's bucket order all rest on, and key 1 and 1L apart where the server treats them as one _id. One ReentrantLock around every operation, and around the moment a stream takes its results, makes each operation atomic and keeps the buffer as it was. It serialises more than MongoDB does, but a serial history is one the server could produce, and the in-memory operations it serialises take microseconds. FakeConcurrencySpec fails against the unlocked buffer — five of its six cases on every run, the duplicate _id one now and then.

The fake refuses what it would get wrong, including what it used to write. A positional path was a path to it, so items.$[].qty became a field literally named $[]. The positional operators are refused by name now, as a tailable cursor, an aggregation expression, Atlas Search and a write for the client-level bulkWrite are — each something the fake cannot answer the way the server would. Where the server refuses outright, the fake refuses with the server's code: a unique index on _id alone (197), a partial filter holding $ne, $nin, $regex, $exists: false, $elemMatch, $size, $mod, $bits or a $nor (67). Each is a parity case against MongoDB 8.2.

The fake opens up, and says what it does not do. All three consumers wrote their own in-memory repository rather than use FakeRepository: it was final, so it could not take a service's own repository trait, and it wanted a BsonDocumentCodec where they had a WireCodec. It is open now and builds from a WireCodec with that codec's naming. What it still does not do is transactions: every method takes a session and ignores it, so a withTransaction body over the fake commits as it goes. Simulating abort would mean a second, transactional copy of the store for a case that needs a replica set to mean anything, so the README says so instead.

Comparison was the fake's widest gap, and it was a refusal pretending to be a rule. The fake compared two values only when they had the same type and threw cannot compare otherwise — so a range filter, a sort, $min/$max or a group extreme over a field that was null in one document failed the test, and every codec except the direct one writes None as null. The server has a total order instead: types first, by BSON's bracket order, then values within a bracket, and per operation a rule on top — a range filter matches only its operand's bracket and goes element by element through an array; a sort reads a missing field as null, an array by its least element ascending and its greatest descending, an empty array below null; a group's $min/$max skip null. The parity case seeds one field with every type side by side, and the fake matched the server's answers on the first run — which the spec was then made to print once, to be sure it had compared something.

Aggregation expressions are a phantom-typed tree

Expr[-E, +A] reads a document of E and computes an A, but the tree underneath is untyped: every node is an Expr[Any, Nothing], which conforms to every Expr[E, A], and the types live only on the factories and operators that build them. That keeps the renderer one match over ten node shapes instead of a case per operator, and it is why Expr.raw costs nothing to offer.

The variance does real work. A literal reads no document, so it is an Expr[Any, A], and contravariance in E lets it stand wherever an Expr[Order, A] is expected. The binary operators take [E2 <: E] so that lit + field and field + lit both come out as an Expr[Order, …] — inference instantiates E2 to its upper bound because it appears contravariantly in the result. Covariance in A is what made a first draft wrong: qty.expr < note.expr compiled, because A widened to Int | String. Comparisons now take a second type B and ask for a Comparison[A, B], which exists for one type with itself and for two numbers, since the server compares an Int with a Long correctly.

Arithmetic asks for Widened[A, B, R], which encodes the server's own widening — Int with Long is Long, anything with Double is Double, anything with BigDecimal is BigDecimal — so price * qty needs no cast. An Option[A] field reads as A (Unwrapped), because inside an expression a missing value is a null the operators propagate, and orElse is $ifNull for the cases where that is not wanted.

Two names were chosen by the compiler rather than by taste. Navigating into an element is get, not /: with / overloaded for division and navigation, inference inside "name" := ... picked the wrong alternative. And the comparisons have named forms — equalTo, lt — beside the symbolic ones, because ScalaTest's Matchers bring a === by implicit conversion that wins over an extension method inside a spec, which is exactly where users write expressions to test them.

The fake does not evaluate expressions. A faithful evaluator means the server's null handling, numeric widening and date arithmetic for every operator, and one that is subtly wrong is worse than one that refuses; every stage, projection and filter that holds an expression is refused by name, and integration tests cover them against a real server instead.

Search: a typed core, a raw edge

ROADMAP.md used to defer $search because nothing here could run it. Testcontainers 2 ships MongoDBAtlasLocalContainer, over the mongodb-atlas-local image, and that removed the reason: search indexes, $search, $searchMeta and $vectorSearch all run in the integration suite now. What is typed is the core most code reaches for — the text operators, equality, ranges, exists, compound, vector search with its pre-filter, and dynamic and vector index definitions — with SearchOperator.raw and SearchIndex.of for the long tail, which is large and moves fast. The text operators compile on text fields and the ranges on fields the server orders, the same capability rule the filters follow.

A write the client sends keeps its collection's codec

The client-level bulkWrite addresses writes by namespace, and this library is built on typed collections. The two meet at collection.write(command): the collection that knows how to encode its entity and spell its fields turns its own WriteCommand into an addressed write, and the client only sends the batch. Nothing about the write is untyped, and the driver's client-level codec registry — which knows nothing of mongo4s codecs — never sees an entity, because the document is already a BsonDocument when it arrives.

Refusing what would be dropped

Some options mean nothing in some places, and the driver's answer is to ignore them. A sort on updateMany is sent nowhere; a comment on one command of a client-level bulk has no place in the command. Both are refused here, with a message saying where the option belongs, because a write that quietly ignores half of what it was told is the kind of bug that survives every test. The rule is the same one the fake follows: answer correctly or say no.

A tailable cursor is a live stream

A tailable cursor never completes, and fs2's fromPublisher emits only when its chunk fills or the publisher finishes — so a tailing find bridged like any other delivered nothing at all. The cursor type decides the bridge: a tailable find streams through RsBridge.liveStream, the chunk-of-one bridge change streams already use, and every other find keeps its batching. all on a tailable cursor still never returns, which is the cursor's nature rather than the bridge's.

Insert-if-absent writes only on insert

There were two ways to insert an entity unless its key is taken: insertOne and catch DuplicateKey, or an upsert whose every field sits under $setOnInsert. The second is the one insertIfAbsent uses. A duplicate-key error inside a transaction ends the transaction, so catching it leaves the caller holding a transaction that can only abort, while an upsert that matches simply writes nothing and reports wasUpserted = false. The entity's fields go under $setOnInsert by their stored names, key fields included — the server accepts setting on insert the value the equality already pins, which ConditionalWritesItSpec checks for an _id key, a field key and a compound one.

The answer is only as strong as the key's uniqueness. On _id the server guarantees it; on any other key it takes the index ensureKeyIndex builds, and without that index two racing calls can both insert.

A counter is a document

MongoDB has no sequence, and the usual stand-in has one shape: a document per counter, moved by findOneAndUpdate with $inc, upsert and the updated document returned. One round trip, atomic on the server, and the first call creates the counter — so there is no separate "create" step to forget, and no read-then-write window for two callers to share a number in. MongoCounters is that shape behind a name, keyed by _id so the only index it needs is the one every collection already has. A step larger than one reserves a block, and the value returned is the block's last. A step below one is refused: a counter that moves back hands out numbers twice.

It is a separate trait and a separate class rather than a repository method, because a counter has no entity type and no key of the caller's — it is its own collection with its own fixed codec — and a test wants it as a whole: FakeCounters is the real implementation over a fake collection, which is why the fake had to learn update-based upserts first.

A default stands in for the database only when asked

WireCodec fails on a missing field unless its type has a missing value of its own — Option's None. A case-class default is not consulted, and that stays the default after withDecodeMissingAsDefault arrived. The default belongs to the constructor; using it on a read means a document that never held the field reads back as if it did. For a field added to a model that is exactly what is wanted, and the flag says so. Otherwise it is a quiet substitution: an Option[Status] = Some(Active) reads a missing field as Some(Active) rather than None, and a reader cannot tell a stored value from an invented one.

Defaults are read by a macro, because Mirror carries labels and types but not defaults. It reaches the companion's default getters and wraps each in a thunk, so a default is evaluated per decode as the constructor would evaluate it, and a generic case class's getter is applied to the decoded type's arguments. The thunks are built whatever the flag says — the configuration is a value, known only at run time — and are used only when it is on.

The fake builds an upserted document the way the server does

An update-based upsert that matches nothing inserts a document the server assembles from the filter and then updates. The rules are not in one place in the manual, so FakeUpsertParityItSpec asked the server shape by shape and the fake follows its answers: every equality in a conjunction is seeded, dotted paths as nested documents; an $in or an $or with a single value or branch counts as an equality, and with several they seed nothing; an $all is one equality per value, so two values on one path are refused, with code 54, the same way two $eq on one path are; $elemMatch and the range operators seed nothing. The update then applies on top, $setOnInsert included, and an _id the filter did not pin is generated and placed first. A replacement keeps only the filter's _id.

The build fails on a warning

The warnings the build turns on are only a test if something fails on them, so -Werror is on — except for the -source:future check, which exists to see what a later compiler will say and would otherwise never get to say it. One message is silenced by text: kyo's Frame macro expands to @nowarn annotations that suppress nothing, in code this project does not own.

The integration tests compile on the next Scala

The library is built on an LTS and used from whatever compiler its users are on, and inline code and macros expand in the user's module, against the TASTy the LTS wrote. 5.0.0 shipped an a -> b in the defaults macro and in PrimaryKey.compound. The 3.9-built TASTy names Predef.ArrowAssoc, which 3.10 deprecates, so every caller on 3.10 with -Werror failed to compile. Nothing here could see it: the build is on 3.9, and even a 3.10 build of the library's own sources resolves -> to the new extension method. Only a module compiled by the newer compiler against the older build reproduces what a user gets.

it is that module. It depends on the library's projects, which compile on Versions.scala3, and compiles itself on Versions.scala3Next; it already derives codecs with defaults and builds compound keys, and with the old code it fails exactly as a user did. The unit tests stay on the LTS, so both sides of the jar are compiled on every build. Inline and quoted code also writes a tuple rather than -> from now on.

A new organization is a major version

Moving from org.mongo4s to now.sendline changes no class, and MiMa agrees, so by the letter of semver it could have been 5.1.2. It is 6.0.0 because a build tool sees a new organization as a new library: it will not evict org.mongo4s 5.1.1 in favour of now.sendline 5.1.2, and with both in a graph the same classes come from two jars, in classpath order, with no warning. That is a change the user has to make by hand, in build.sbt, and a version bump that looks routine would hide it. A major release signals that the build needs editing; the migration guide then says the edit is one line.

The package stays mongo4s. Renaming it to now.sendline.mongo4s would turn a one-line build change into an edit of every import, and buy nothing a short package name does not already have.

The old coordinates are not left to rot. org.mongo4s 6.0.0 is a relocation: a pom per module with <relocation> naming now.sendline, which Maven and Coursier follow to the module of the same name and version, so a version bump, by hand or from Scala Steward, lands on the new organization. That was checked against sbt 2.0.11 by resolving org.mongo4s:mongo4s-core_3:6.0.0 from a scratch repository and getting the now.sendline jars. A relocation cannot reach back, though: a build that pins 5.1.1 under org.mongo4s stays there until someone edits it, which is what the migration guide covers.

Evolving the API

Three commitments, and the mechanics that make each one keepable:

New Effect/RsBridge members carry default implementations. Implementing either typeclass yourself keeps compiling across minor releases. liveStream and Effect.monotonic were both added this way.

Binary compatibility is checked, not asserted. MiMa runs in CI against the previous release. Every deliberate break is a MiMa exclusion in build.sbt and an entry in COMPATIBILITY.md — waivers are visible, not silent. A major release such as 5.0.0 needs no filters at all: it is allowed to break, so the list there is a migration guide rather than a set of waivers.

MiMa has a blind spot, and the compiler covers it. An inline method that reaches a non-public member makes the compiler synthesize a public accessor for it. That accessor's name is not part of the stable ABI: recompiling can rename it, and code that inlined the old one fails at link time. MiMa never sees this, because the accessor is synthesized rather than declared. -WunstableInlineAccessors reports each one and @publicInBinary pins it. Three existed the first time the flag was switched on — both derivations' make, and FieldMacro reached from Field.of. For a library this inline-heavy the check is not optional, so it is on in the build rather than left to whoever remembers.

Configuration types are builders, not case classes. This one was learned the expensive way. Adding a field with a default to a case class is source-compatible but never binary-compatible: default arguments are resolved at the call site, so the JVM only ever sees the old arity, and apply, copy and the constructor all change. Since derivation options will keep arriving, WireCodecConfig became a final class with a private constructor and withX methods — one break taken once, and every option after it is additive:

scala
given WireCodecConfig = WireCodecConfig.SnakeCase.withOmitNoneFields(false)

WatchOptions was the counter-example here for a while — a case class with withX on top, carrying the same hazard, with a note that it should get the same treatment the next time it changed. Adding showExpandedEvents was that time, so it is now a final class with a private constructor too, and the note has been paid off rather than restated.

Deliberately not done

A discriminator-free Either/Ior codec. Dropping _type and trying branch A then branch B via BsonReader.getMark()/.reset() is technically feasible — the driver supports mark/reset. It was rejected because it is silently unsafe whenever the two branches are structurally similar enough that both decode "successfully": the wrong branch wins with no error. That is precisely the bug class the codec audit existed to eliminate.

Building the key index without being asked. A repository whose key is not _id is not unique until something builds its index, and the repository could do it on creation. It does not: indexes are as often owned by migration scripts as by the application, a build on start-up competes with every replica starting at once, and an index the library builds behind the user's back is one their migrations then find already there. ensureKeyIndex is one call for those who want it.

A global CodecProvider convenience. It would remove one implicit parameter and reintroduce process-wide coupling between unrelated collections. Not worth it.

Streaming bulk inputs. insertMany and bulkWrite take a Seq, not an S, so the whole input is in memory before the first batch goes out. Consuming an S would need RsBridge to fold a stream into F, which none of the four runtimes shares a shape for, and the loop it would save is three lines in each of them — chunk the stream, insertMany each chunk. Until one runtime-neutral way to do it is worth a new abstract member, that loop is the answer.

Simulating pipeline updates in the fake. A pipeline update computes from the document with aggregation expressions, and the fake does not evaluate expressions. It refuses them by name.

Simulating $bucketAuto, $geoNear and renameCollection in the fake. $bucketAuto's boundaries come out of the server's own balancing, which a fake would have to copy rule for rule to agree with; $geoNear needs the spherical distances the fake does not compute for $near either; and a rename needs a second namespace, which a single-collection fake does not have. Each is refused by name.

Checking where $geoNear sits in a pipeline. The server refuses it anywhere but first, with a message naming it. A check in aggregate would restate that rule without adding anything to it.

Simulating $median, $percentile, $setWindowFields and $redact in the fake. The first two are computed by the server's approximate method, which a fake could only imitate; windows and $redact evaluate aggregation expressions, which the fake does not. Each is refused by name. $fill, which only fills values in sort order, is simulated — including the explicit null the server writes where there is nothing to fill from.

Batching rapid's demand. rapid's stream asks the cursor for one more document each time it takes one, where fs2 asks by the chunk. Measured before changing it: against a real MongoDB, rapid's findStream runs at 3,179 ops/s against 3,174 for cats-effect and 3,044 for zio, in the same sitting (BENCHMARKS.md), and at 3,262 against 3,109 and 3,029 when the 5.1.0 build was measured. A per-element request is lost in the decode and the round trip, so the simpler code stays.

Copying every BsonDocument the API accepts. Filter.Raw, Stage.Raw, hints and resume tokens are taken as given, not cloned: mongo4s never writes into them, so aliasing only matters if the caller mutates a document after handing it over. Update.Raw is the exception, cloned because rendering used to write into it.

Bounding a live stream with rapid's Stream.take(n). Not a mongo4s decision, but worth knowing: rapid decides Step.Stop inside transform, so it needs element n+1 to arrive before it stops. On an infinite change stream carrying exactly n events it blocks forever. Not fixable from here.

A circe bridge. There is no mongo4s-bson-circe module, and there is not going to be one. circe is a JSON codec: routing an entity through it means case class → circe → some BSON tree → org.bson, which is the same case class ↔ circe ↔ mongo4cats.Bson ↔ org.Bson trip BENCHMARKS.md measures at 1.9–3.5× slower on decode. That cost is the reason mongo4s exists; shipping a first-class path back to it would be arguing against the library's own premise. A model already on circe should get a BsonDocumentCodec[A] written against BSON directly — bson-direct derives one with no third-party dependency at all.

built from DESIGN.md at v6.0.0 edit this page on GitHub