vibecheck inspects vibecheck: the dogfooding write-up.

2026-05-09 · vibecheck team · 5 min read · Operations

Quick answer Every vibecheck deploy is followed by a smoke test that runs vibecheck against vibecheck's own deployed URL. The current self-scan grade is D: one high-severity finding, csp_unsafe_inline_scripts, accepted publicly because the alternative (a fully hash-pinned CSP) exceeds Cloudflare Pages' 16 KB per-header limit. Every other detector family — Supabase, Firebase, JWT, secrets, CORS, paths, framework, etc. — comes back clean. The point of dogfooding isn't that the tool's own deploy is perfect; it's that the tool runs against itself, finds real things, and publishes the findings.

If you build a security scanner and don't run it against your own deploy, you're not really building a security scanner. You're building marketing for a security scanner. We don't ship anything to vibecheck.themeridianlab.com that hasn't passed a smoke run that includes a self-scan.

This post documents what the self-scan actually returns, why one finding is currently accepted, and what would change if it were any other deploy.

What the smoke does

Every deploy ends with scripts/smoke.sh running 28 checks against the freshly-deployed URL. Three of those are /api/scan calls:

  1. Test fixture scan. Hits /test-fixture/leaky-app.html — a deliberately-broken page seeded with a fake Supabase service_role JWT, a Stripe live key, and an open Realtime Database URL. Must return grade F with severity critical. If grade is anything else, the scanner stopped detecting things.
  2. JWT fixture scan. Hits /test-fixture/leaky-jwt.html — five hand-crafted JWTs that each trigger one of the five JWT rules. Smoke asserts all five rule names appear in the output.
  3. Self-scan. Hits vibecheck.themeridianlab.com/ itself. Asserts grade A through D, with the explicit exception that the only acceptable critical/high finding is csp_unsafe_inline_scripts. Any other critical/high finding fails the check.

The first two protect against detector regressions. The third protects against everything else: a leaked secret committed by mistake, a CORS misconfig accidentally introduced, a CSP weakness that wasn't there yesterday.

The CSP finding we accepted

When the CSP detector shipped, it immediately flagged vibecheck's own deploy. csp_unsafe_inline_scripts — high severity. Self-scan grade went from B to D in one deploy.

The finding is real. Our script-src directive includes 'unsafe-inline' because we embed inline <script type="application/ld+json"> blocks for SEO on every page (BlogPosting, BreadcrumbList, FAQPage schema). Across ~30 pages, that's about 320 unique inline blocks.

The fix would be a fully hash-pinned script-src — compute the SHA-256 of every inline block, list them all in the CSP. We wrote the build script (scripts/build-csp.ts) and ran it. The resulting CSP was 18 KB. Cloudflare Pages caps individual response headers at 16 KB; oversized headers are silently dropped. The "fix" left the deploy with no CSP at all — strictly worse than what we started with.

We considered the alternatives:

We chose the last option. The trade-off lives in public/_headers as a comment, in the CSP article as a section, and in the smoke threshold as an explicit exception (only csp_unsafe_inline_scripts is accepted; any other critical/high finding fails the build).

The shape of this trade-off matters more than the trade-off itself. Lots of security tools reach grade A by ignoring the parts of their own deploy that don't pass. We'd rather show the D, explain it, and have the smoke gate prevent it from quietly getting worse.

What else the self-scan checks

Everything else comes back clean. The 14 other detector families currently emit nothing against vibecheck.themeridianlab.com:

What we'd fix on a real customer's deploy

For a customer's app — anything with user-generated content, forms, comments, or markdown rendering — the CSP finding would be unacceptable. The trade-off only works when there's no XSS sink.

The fix order we'd recommend:

  1. Switch from 'unsafe-inline' to nonces. Generate a fresh random nonce per request server-side, inject into the CSP header AND every legitimate inline script tag. Attacker-injected scripts won't have the nonce. Detailed walkthrough in /fix/csp_unsafe_inline_scripts.
  2. If nonces require a refactor, hashes work for static inline content. Compute SHA-256 of each block at build time. Pin the hashes. Limited to projects whose total hash list fits under the host's header size cap (16 KB on Cloudflare Pages, varies elsewhere).
  3. Audit for other XSS sinks first. The CSP is the second line of defence. The first line is "don't echo user input into HTML without escaping." Found those sinks first means CSP is genuinely defence-in-depth rather than the only thing standing between an attacker and code execution.

Why this is the public version of the post

Most security tools have an internal checklist that includes "scan our own deploy" and a public surface that says "we are secure." Those two things don't necessarily contradict, but they're often answering different questions. This post is the public version of the checklist. The grade. The single accepted finding. The reasoning. The smoke threshold.

You can run a self-scan yourself: curl -X POST https://vibecheck.themeridianlab.com/api/scan -H "Content-Type: application/json" -d '{"url": "https://vibecheck.themeridianlab.com/", "consent": true}'. The result you get should match what we're claiming here. If it doesn't, file an issue.

Inspect your own deploy

Related