The four signals every AI content program should log from day one

When a draft is bad, the operator needs to know inside thirty seconds whether the brief was thin, whether the voice gate caught it, whether the model regressed, or whether the reviewer skipped a step. Generic LLM logging answers none of those questions. Instrumenting AI content is the discipline of logging the four signals that actually identify which decision broke. This article is for senior operators running a weekly content program who want the framework before they invest in tooling, not after.

Most AI content programs start with logging API calls. Token counts, latency, model version, prompt and completion. The data is real; it just answers the wrong question. When a draft underperforms, the operator does not need to know the model returned a 200 in 2.4 seconds. The operator needs to know which brand-side decision broke. Instrumenting AI content shifts the logging stack from API observability to decision observability, and four signals do all the work.

The four signals for instrumenting AI content

The framework is four signals, logged at the decision points in the pipeline, structured so the operator can answer the question "which decision broke?" without rerunning the draft. Brief completeness is signal one. Voice gate findings is signal two. Quality reviewer score is signal three. Reviewer behavior is signal four. Together these four tell you which decision broke when a draft underperforms. Without them you are swapping models and praying.

Signal 1: Brief completeness

Score every brief against the schema before the draft runs. A 7 out of 10 brief produces a 7 out of 10 draft. The score is mechanical: count the fields filled, the specificity of the key points (do they have numbers, names, mechanisms, or are they vague), the sharpness of the angle, and the presence of references. A brief that fails completeness should be fixed before the draft runs, not after.

Logging brief completeness alongside draft outcomes turns the dataset into something diagnostic. Eight weeks in, the operator can look at the worst-performing drafts of the quarter and read the brief completeness score next to them. If the bad drafts cluster on low brief scores, the fix is upstream of the model entirely. If they cluster on high brief scores, the model or the voice gate is the actual problem and replacing the brief layer will not move the metric.

Signal 2: Voice gate findings

Every output is scanned for forbidden vocabulary, brand-direction violations, and AI-pattern leaks. The voice gate logs every finding even when the draft passes, because the findings themselves are the signal. A draft that passed the gate by one finding tells the operator something different from a draft that passed by twenty. The pattern of findings over time also tells the operator whether the model has regressed against the voice contract.

Voice gate findings are the canary for model drift. When the same brief schema starts producing drafts with more findings per draft over a four-week window, the model has shifted, the prompt context has changed, or the brand-direction document needs an update. Instrumenting AI content with this signal catches drift before the operator notices it in the published feed.

Signal 3: Quality reviewer score

A human reviewer scores every draft on a fixed rubric. Substance, voice, hook strength, ship-readiness. The score is logged. The reviewer rubric is fixed because variable rubrics produce un-comparable scores; the same draft scored against a drifting rubric tells the operator nothing about whether quality is going up or down across weeks.

Reviewer score paired with brief completeness and voice gate findings becomes a diagnostic triple. A draft with a strong brief, clean voice gate findings, and a low reviewer score points the operator at the model or the prompt. A draft with a weak brief and a low reviewer score points the operator at the strategist. Without all three signals, the operator cannot tell which.

Signal 4: Reviewer behavior

How long the reviewer spent on the draft. Whether they edited the body or only the headline. Whether they kicked it back to the drafter for a second pass or pushed it through. Reviewer behavior is the signal everybody forgets to log because it feels meta, but it is the one that catches the slowest regression: the reviewer who has stopped doing the work.

A reviewer who used to spend eight minutes per draft and now spends ninety seconds is not a more experienced reviewer. They are a less engaged one, and the quality of the program is degrading regardless of what the other three signals say. Instrumenting AI content with reviewer behavior is what catches the human side of the regression, and the human side is the dominant cause of program decay over twelve months.

Worked example: diagnosing a bad week

Illustrative, not a real incident. The Friday review pulls eight drafts from the week. Three are flagged as below ship-quality. Without the four signals, the operator and the strategist argue: was it the brief, was it the model, was it the reviewer? With the four signals, the answer takes thirty seconds. All three flagged drafts had brief completeness scores in the low band. Voice gate findings were normal. Reviewer time was normal. The signal points clean: the strategist had a thin week on briefs. Fix is upstream, model and reviewer are not the problem, no time wasted swapping out either.

Now invert the example. Brief completeness scores are normal across all eight drafts. Voice gate findings are double the four-week trailing average across all eight. Reviewer time is normal. The signal points to a model or prompt regression. The fix is to roll the prompt context back to last week and rerun the comparison. Without the voice gate signal logged at the draft level, the operator would have spent the weekend rewriting briefs that were already fine.

Runbook for instrumenting AI content

1. Define the brief schema and the completeness rubric. Score every brief before the draft runs. Log the score alongside the draft. 2. Build the voice gate as a deterministic scanner: forbidden vocabulary list, AI-pattern detectors, brand-direction violation checks. Log every finding, not just the pass or fail flag. 3. Define the reviewer rubric on substance, voice, hook strength, and ship-readiness. Fix the rubric and resist iterating it; a fixed rubric is what makes scores comparable across weeks. 4. Capture reviewer behavior automatically: time on draft, edit depth, kick-back count. The reviewer should not have to log it manually; the tooling captures it. 5. Pair all four signals on every draft record. The dataset is the unit of analysis, not any single draft. 6. Set a weekly review where the operator scans the four signals across the week of drafts. Look for clusters: low briefs paired with low reviewer scores point at the strategist, normal briefs paired with high voice gate findings point at the model, normal everything paired with shrinking reviewer time points at the reviewer. 7. Change one variable at a time. If three things changed and the metric moved, you do not know which change moved it. Instrumenting AI content is only worth the cost if the operator runs the program with experimental discipline.

When instrumenting AI content is overkill

The four-signal framework is calibrated for a weekly content program with at least one specialist channel running on cadence. For a brand shipping one post a week or experimenting with the format, the instrumentation cost outruns the diagnostic value. Ship for two months on a lighter setup, then layer the four signals in once cadence is real and the diagnostic question becomes worth answering.

It is also overkill if the brand has not yet decided on a voice contract. The voice gate signal is meaningless without forbidden vocabulary and brand-direction inputs, and instrumenting against an undefined contract just produces noise. The right sequence is voice contract first, brief schema second, instrumentation third. Skipping ahead produces a logging stack that captures decisions nobody made.

What success looks like

On a program that has been instrumented for a quarter, the visible signals are: time-to-diagnose a bad draft drops from a half-day argument to thirty seconds, model swaps stop happening on hunch and start happening on signal, and reviewer engagement gets caught and corrected before it shows up in published quality. On the AI Lab side, the published outcome bands are 20 percent retention lift and 60 percent operator time saved on the workflows the four signals make legible.

The qualitative signal is the operator who, six months in, can read a single draft record and tell you exactly which decision in the pipeline produced it. That literacy is the actual asset. The signals are the means; the diagnostic literacy is the end.

FAQ

What are the four signals for instrumenting AI content? Brief completeness, voice gate findings, quality reviewer score, and reviewer behavior. Together they tell the operator which decision broke when a draft underperforms.

Why is brief completeness logged separately from the draft? Because a 7 out of 10 brief produces a 7 out of 10 draft, and the operator needs to know whether the regression sits upstream of the model or downstream of it. Logging completeness next to the draft answers the question without rerunning anything.

How is reviewer behavior captured without adding manual work? The tooling captures time on draft, edit depth, and kick-back count automatically as the reviewer interacts with the draft. The reviewer never has to log anything manually; the captured behavior is the signal.

When should a brand start instrumenting AI content? After the voice contract is written and after the program has been shipping cadence-rate for at least four to six weeks. Earlier than that, instrumentation captures decisions nobody has actually made yet.

What is the alternative to the four signals? Generic LLM logging tells you the API call returned what. It does not tell you which brand-side decision broke. The alternative is swapping models and praying.

Read more

- https://www.arthea.ai/ai-lab - https://www.arthea.ai/article/free-the-weekly-brief-template - https://www.arthea.ai/article/why-klaviyo-flows-quietly-stop-converting

If you want a 30-minute review on instrumenting AI content inside your own program, the calendar is at arthea.ai/book.