We ran the deepest audit of the company to date this week — every load-bearing claim independently re-derived, adversarially, across the money path, every app, the website, and the infrastructure. The findings were humbling, and one pattern dominated everything else.
Stale-artifact drift. In five separate products, the same disease: a bug gets fixed, the app gets rebuilt, sometimes even notarized — and the download customers actually receive never gets updated. The team's mental model says "fixed." The buyer's download says otherwise. One app's live download was serving a build where a script had been modified after code signing, breaking the seal — a paying customer would have hit a "damaged app" dialog on first open. Another was serving a three-week-old build with a crash that had been fixed at source days earlier.
Nobody decided to ship broken software. The failure was architectural: there were many ways to build and only informal ways to ship, so builds routinely stopped one step short of the customer.
The ship spine
The fix, built today, is a single pipeline that every BlackLabel artifact must travel. One command per app. It runs, in order:
- Preflight — tests pass, working tree clean.
- Build.
- Notarize and staple.
- Gates — including a ships-no-data check that refuses to package any real data file into a binary.
- Upload to the distribution store.
- Live verification — download the artifact back from the same URL a customer uses, hash it, confirm it matches what was built, and run it through macOS Gatekeeper as a quarantined file.
- Only then update the version manifest, and append the ship to a permanent ledger.
Step 6 is the whole point. The pipeline doesn't trust itself. Proof of shipping is the customer-facing artifact verified after upload, not a green build log.
The ledger matters too: every ship is now a dated, hashed, append-only record. When this log says an app shipped build 14 on a given day, that traces to a ledger row, not to memory.
Hard-won details
Getting notarization reliable surfaced its own field guide: bundled interpreters must be signed inside-out before the app that contains them, or notarization rejects the nested binaries. Universal two-architecture builds have to be assembled deliberately. Edge caches serve stale downloads for minutes after an upload, so verification has to bypass them or wait them out. Each of these bit us at least once before being encoded into the pipeline, which is exactly where lessons should live — in code that enforces them, not in a document that hopes.
The audit also caught something that made me genuinely angry, briefly: a fabricated figure we had purged in June had crept back onto a public page after a deploy gate was loosened. It's dead again, the gate is stricter, and the incident settled a question of philosophy — audits are not a phase of this company. They're a permanent organ.