AI copywriting copilot with confidentiality gates

Two-sided confidentiality gate. The old gate (detectConfidentialLeak) ran on the draft on the way out. The new gate – shipped as PR #345 – runs the same class of check on the grounding facts on the way in. The prompt never sees the confidential material in the first place, so the model can't reveal it even if the leak-out check has a blind spot.

Metric classification via regex. The CONFIDENTIAL_METRIC_RE regex catches metrics whose labels or values contain pipeline vocabulary (signals, leads, prospects, deals, tracked, briefing, targets, accounts, customers) and holds them out of the grounding block. Outcome metrics ("5+ hours saved") flow through unchanged. The regex is broad enough to catch the failure that happened and narrow enough not to censor legitimate outcome data, which was measured on the two real work items that had already been drafted.

Protected-client-name heuristic. Vision-pass reads of client dashboards used to lift UI vocabulary as proper nouns – "Results", "All", "Download PDF" – which the leak-out check then rejected as unfamiliar company names, blocking clean drafts. protectedClientNames() filters generic UI vocabulary, calendar tokens ("Aug", weekday "Monday" but not "Monday.com"), and sector labels, while preserving real entity names. On the two real work items, false positives dropped to zero without weakening the block on real client names.

Model stratification by use case. Opus 4.6 for the interactive chat (user-facing quality), Sonnet 4.6 for batch drafting (throughput), Haiku 4.5 for cheap non-writing judgments – scene inference, image verification, same-person confirmation. Haiku is used deliberately for checks where the alternative is no check at all, with the reasoning captured in the file's header comment.

Alternatives considered and rejected:

• Rebuild the leak-out check with a stronger model. Would have improved recall on the same class of failure but not addressed the root cause: material the model shouldn't have was flowing into the prompt.
• Skip vision-pass entirely. The vision-pass is the source of most of the quality lift on real work items. Removing it removes the product's edge.
• Client-supplied allowlists. Puts the burden on clients to enumerate every confidential noun in their own business – this fails the moment a client onboards a new prospect.

• Migrated per-user cron from Vercel to Trigger.dev v4. Weekly schedules.task per user with an externalId of userId:postingContext, three-retry budget per draft, exact-time firing. Vercel cron's global scheduler couldn't express any of that.
• LinkedIn scope parsing tolerant of both delimiters. /[\s,]+/ splits on either whitespace or comma so grantIncludesOrgScopes works regardless of which LinkedIn representation the day's OAuth response ships with.
• Langfuse instrumentation on every Claude call. Traces are tagged with the resolved user ID (resolveTraceUserId); the Langfuse instance is explicitly flushed and nulled on function teardown to avoid losing telemetry when Vercel terminates the serverless function mid-request.
• Cost tracking at the API layer. Every Claude call is tagged with its model in a shared resolver so spend can be totalled by route and context, with Opus/Sonnet input/output cost ratios hardcoded as constants.

• 208 test files, 3,485+ assertions. Confidentiality tests use the actual leaking-draft example and verify both directions: real prospect names in ingested metrics are caught before the prompt, and the same names never appear in generated drafts.
• Three evaluation suites. Brand adherence, draft quality, framing A/B, and avatar likeness. Avatar likeness runs against real Gemini and Replicate vision, gated on an explicit env var (AVATAR_LIKENESS_EVAL=1) – the eval costs $4 and takes 20–30 minutes per run, so it never fires accidentally.
• 80% minimum unit + integration + E2E coverage. House rule, enforced.

1. AI safety patterns in production – two-sided confidentiality gate, incident-driven design, false-positive measurement against real work items.
2. Multi-model orchestration under cost discipline – Opus/Sonnet/Haiku split by use case, tagged per-call cost tracking, model choice justified in code.
3. OAuth integration debugging – LinkedIn's dual-format scope response, dual-app choreography for personal-vs-org posting.
4. Trigger.dev v4 for per-user scheduling – per-user externalId, retry budgets, exact-time firing, migration path from Vercel cron.
5. Evaluation discipline – three eval suites, gated expensive evals, avatar-likeness runs against real vision APIs when explicitly requested.

1. Gate on the way in, not just on the way out. If a leak-out check has a blind spot and the material was in the prompt, the failure is inevitable. Don't put material the model shouldn't have into the model.
2. Model stratification is a cost tool as much as a quality tool. Opus everywhere reads well and burns money; Haiku everywhere is unusable. The right rule is per-use-case, written down where the model is chosen.
3. The check that catches the incident should catch every incident of the same shape. A regex the client-facing team can read beats a system-prompt tweak that only the author understands.