FeaturesSecurityPricingCompareDevelopersBlogStart Free
BlogAugust 4, 2026

How we test our own data room

Four instruments, one committed baseline that fails the build, and a list of the defects that every one of them missed. The last part is the part worth reading.

Four instruments, four questions

Every published number on this site comes out of one of four places, and each answers a question the others cannot.

What each instrument measures, and where it runs.
CriterionQuestion it answersRuns where
Fidelity baselineDoes the viewer still draw what the file says?Local and on demand
Pixel assertionsIs the watermark still in the served bytes?Playwright suite
Deployment invariantsDoes the code that is deployed still carry the control?CI, against the live database
Black-box replayDoes the original exploit still work?CI, public key only
What each instrument measures, and where it runs.

The first is a scored comparison against LibreOffice with a committed floor per dimension. Any dimension that scores below its floor prints the regression and exits 1, so a fidelity loss fails the run rather than reaching a reader. Floors are updated only by an explicit flag, which makes lowering the bar a deliberate act that shows up in a diff.

The second asserts on real pixels produced by the real burner rather than on the code that calls it. It also asserts source ordering, that the burn happens before the image is encoded, because that ordering is the entire control and a refactor could silently reverse it.

The third is the one most suites skip. It reads the deployed function's source text out of the database catalogue and fails if either permission predicate has gone missing, or if a client-reachable role has regained execute permission. A test of the source tree cannot see a migration that ran differently in production.

The fourth replays closed exploits from outside with only the public key that ships in the browser bundle. It counts a permission-denied error as a pass and treats an ambiguous error as a failure, so a probe that stops working cannot quietly start reporting success.

  1. Claim
    Stated precisely enough to be wrong
  2. Instrument
    Measured by something that can fail
  3. Floor
    Committed, lowered only on purpose
  4. Gate
    Regression exits non-zero in CI
  5. Replay
    Original attack re-run from outside
The loop a claim goes through before it appears on this site.

Reproducing it

node tests/fidelity/run.mjs --rebuild
npx playwright test -c tests/rasterization/playwright.config.ts watermark-burn
node --experimental-strip-types tests/security/org-isolation.test.ts
node scripts/security/check-function-privileges.mjs --require-db
13 / 13
Isolation probes passing
Public key only, 4 Aug 2026
4
Producer dialects gated on every fidelity run
tests/fidelity/baseline.json
64
Crop windows swept per watermark run
tests/rasterization

What none of it caught

Several of the worst defects we have shipped were invisible to every suite we had, and appeared only when the product ran on real infrastructure against a real browser. They share a shape: the code was correct and the thing around the code was not.

  • The model endpoint was unreachable from any browserbypassable

    The parser passed 158 tests. The reverse proxy had no route for it, so every request fell through to a 403, and the handler sent no cross-origin header, so a browser would have discarded the response anyway. Found by walking the deploy path by hand.

  • A route prefix quietly protected the wrong pagesbypassable

    The middleware guarded a path prefix, and a second, unrelated route began with the same characters. Every unit test of the guard passed, because the guard was right about the prefix it was given.

  • A background job could never queue workbypassable

    The reindex command was correct in every part a test exercised and could not enqueue a single job in practice.

  • The permission predicate was right, its caller was notbypassable

    The filter inside the retrieval query was correct and tested. The function could still be called directly with a substituted participant identifier, and it returned 30 chunks across 13 documents to someone with no relationship to the room.

Real defects, and the layer each one lived in. None was reachable by a test of the code.

One more, in the opposite direction. We recorded that legacy .xls frozen panes were unrecoverable because the converter dropped them, and wrote that down as a permanent limit. It was wrong, and a later attempt recovered them. A limitation logged as permanent is a claim like any other, and it deserves the same re-testing as a feature.

The pattern is not that testing failed. It is that a test of the code answers a narrower question than most people hear when they are told the code is tested. Configuration, routing, grants, deployment and reachability all sit outside it, and every one of them has produced a defect here. That is why two of the four instruments probe the deployed system rather than the source tree, and why the deploy path now gets walked by hand before a release rather than inferred from a green suite.

The individual measurements have their own pages: the fidelity scores, the crop sweep, and the permission fence, each with the method that produced it.

Related: The spreadsheet fidelity test · Do watermarks survive screenshots · What a citation has to point at · Pricing

Pricing is public, with no per-page fees and a 14-day trial that does not ask for a card. See the tiers.

FAQ

Questions about how we verify our own claims.

Ask which layer each control sits in, ask what the test measures, and ask what it does not measure. A vendor who can name the false negatives their suite has is telling you they have looked. A vendor whose answer is a certification logo is telling you someone else looked, once, at a scope you have not read.
Re-running the original attack against the fixed system, from outside, with only the credentials an attacker would have. It answers a different question from the unit test written alongside the fix. The unit test asks whether the new code behaves as intended; the replay asks whether the exploit still works.
No. It can prove specific properties hold on the code paths it exercises. Several of the worst defects we have shipped were invisible to every suite we had, because they lived in configuration, routing and deployment rather than in code. That is a limit of the method, not a scandal, and it is why the deployed system is probed rather than only the source.
No. We hold no SOC 2 certification and no other third-party security certification. Everything described here is our own testing of our own system, published so it can be checked rather than taken on trust.
Start Free