Four bugs that silently broke RxDB sync, and how they were fixed

Personal OS keeps its data in the browser with RxDB and syncs it to a Spring Boot and PostgreSQL backend through RxDB's custom replication. The bugs that cost the most time there had one thing in common: nothing reported an error. Edits disappeared, a collection stopped syncing, or every write failed after a reload, and the logs looked normal. This post walks through four of them, what caused each, and the fix.

How changes reach other devices in real time, with Server-Sent Events, is covered in SSE with Spring Boot and Redis Pub/Sub.

The protocol in one paragraph

Every successful write on the server gets a version from a per-user sequence that only goes up. That number is both the replication checkpoint (a device pulls "everything after version N") and the way to detect conflicts: a push says which version it assumed, and the server rejects it if the row has moved on. When two devices really did edit the same row, the conflict handler keeps the copy with the higher server version, which in practice is the one already on the server; the client's updated_at only breaks ties between copies at the same version. An edit made offline still goes through on reconnect, as long as no other device changed that row in the meantime. The cost is that when one did, the offline edit is dropped whole, not merged. For one person's data on a few devices that's rare, and it's the trade-off this design accepts.

Bug 1: rows created on this device lost every later edit

The symptom: create a block, then rename it. The rename looked saved, then reverted after the next sync. Deleting it did nothing either.

What happened, step by step:

1. A new row is created locally with version = 0 and pushed. The push succeeds, but the server's response lists only conflicts; it doesn't send back the version it assigned.
2. The next pull brings the row with its real version, say 1. RxDB first asks the conflict handler's isEqual(pulled, local) whether anything changed.
3. isEqual had a shortcut: if either side had version === 0, return false. RxDB treated the documents as different, but in that step it didn't write the pulled one locally either.
4. The local row stayed at version = 0 forever. Every later edit was pushed with "assumed version 0", the server was at 1, so every edit was a conflict, and the conflict handler discarded the user's change.

The fix was to delete the shortcut and compare the three fields that matter:

function isEqual(a, b) {
  return (a.version ?? 0) === (b.version ?? 0)
    && a._deleted === b._deleted
    && a.updated_at === b.updated_at
}

A pulled row at version 1 now differs from the local row at version 0, so RxDB writes the real version down. A local edit not yet pushed has the same version as the server but a newer updated_at, so it still isn't overwritten by a pull.

RxDB calls isEqual very often and requires it to be synchronous, so comparing three scalar fields is also the cheap option; a deep comparison of whole documents isn't.

The bug was found and pinned with an integration test that runs a full sync loop with real RxDB against a mock server that follows the backend's contract, plus backend tests for the server side of the same scenario. Unit tests of the conflict handler alone had passed throughout.

Bug 2: a column that only exists on the client

The large tree post describes parent_key: a non-null copy of parent_id that exists only so the browser can index it. The server has no such column. That broke sync in two directions.

Push. The client sent parent_key along with every block. The server validates column names and rejects unknown ones with HTTP 400 rather than ignoring them. Every push of that collection failed, the change stayed in RxDB's queue, and was retried forever. The fix strips local-only fields at the last moment before a document leaves the device:

export function stripLocalOnly(doc) { /* removes parent_key, category_key */ }

Pull. Locally, parent_key is filled by preInsert and preSave hooks. But replication writes pulled documents through RxDB's storage layer directly, not through the collection, so those hooks never run on the pull path. Pulled blocks arrived without parent_key; in development that failed schema validation, and in production the blocks were saved but invisible to every query on parent_key, so freshly synced blocks were missing from the library tree. The fix adds the derived fields in the pull handler's modifier.

That fix had its own trap. RxDB always sets additionalProperties: false at the top level of a schema, so adding a field the collection doesn't declare makes every pull of that collection fail. snapshot_nodes has category_id but no category_key, so a hard-coded "add both keys" would have broken it. The derivation now reads which derived columns each collection's schema declares, instead of naming collections in code.

The general rule: a local-only derived column needs handling at three places, the write hooks, the push modifier and the pull modifier, and only the first one is obvious.

Bug 3: a migration that finished but never said so

After a schema change, a real device ended up with every write to one collection failing with RxDB error COL25, thrown because the collection was flagged as still migrating. The data showed up fine; reordering, editing or deleting anything in that collection failed. Reloading didn't help.

The state on disk, captured from a real device, explained it:

rx-migration-status|library_blocks-v-4  → DONE, 1704/1704, 100%
collection|library_blocks-3             → still alive (_deleted = false)

After moving the data, RxDB's migration has two more steps: delete the metadata document of the old collection version, then clear its "migration in progress" flag. If the process dies between those two steps (a closed tab, a power cut, iOS killing a background tab), the old metadata survives. On the next start, RxDB sees it, decides a migration is needed and sets the flag again, but the migration has nothing left to do, so the flag is never cleared. Awaiting migratePromise() doesn't help either: it waits for work that is already done and never reports it.

The fix performs the step RxDB missed, marking the old metadata document as deleted, but only when both conditions hold: a status record says the migration is DONE, and the old metadata is still alive. A migration that is genuinely RUNNING is left alone, because deleting its metadata would lose the data not yet moved. The repair touches only RxDB's bookkeeping documents, never user data, and runs before migratePromise().

Bug 4: a safety net that tore itself

A device that comes back after a long time can't trust its checkpoint: the server purges deletion records (tombstones) after a while, so an old checkpoint might miss deletes. The safety net: if the last successful sync was more than 60 days ago, drop the checkpoint and pull the whole account again.

Three details made it worse than no safety net:

• "Never synced" was treated as "too old". A full resync clears the last-sync timestamp, so the next start found no timestamp, resynced again, and so on: every app start downloaded the whole account. A missing timestamp now means "not stale"; the checkpoint is 0 anyway, so the first pull runs normally.
• The timestamp was only written on some code paths. It is now written after every successful pull.
• A clock moved backwards produces a timestamp in the future; the age can't be known, so it is reset to now instead of forcing a resync.

There is also an invariant between client and server: the client's 60 days must stay shorter than the server's tombstone retention, 90 days. If it were longer, a device returning after 75 days would skip the full resync while the server had already purged the deletes it missed, and deleted rows would come back to life on that device. The client reads the server's retention from a config endpoint at startup and logs a loud error if the invariant is broken. It only warns: a deployment mistake shouldn't stop sync entirely.

Not a sync bug, but just as quiet: a white screen from chunk splitting

To improve caching, RxDB and RxJS were moved into their own vendor chunk with a manual chunks rule. Moving RxJS pulled in modules that React's chunk also needed, which created a circular dependency between two chunks. Module initialization order broke, the RxDB chunk called an export of the React chunk before it existed (Cm is not a function in the minified build), and production showed a white screen. The development server, which doesn't bundle, was fine.

The split was reverted and the production build checked in preview. The caching gain wasn't worth an app that might not open, and the lesson is to test chunking changes on a production build, never only on the dev server.

Less load while nobody is looking

One more change that didn't fix a bug but cut server load: only the leader tab opens the Server-Sent Events stream, a tab hidden for more than five minutes closes it and reopens on return, and each reconnection is followed by one check against the server's latest version. While the stream is healthy, the periodic safety pull runs every 30 minutes instead of more often.

Summary

The common lesson: when sync goes wrong, it rarely throws. Each of these was found by looking at the state on a real device or by a test that ran the whole loop, not by reading error logs.

If you've tracked down sync bugs in RxDB or another local-first stack that never threw an error, I'd love to compare notes. My email is on the home page.