Your agent made a call. Can you show me why?

Thursday night one of our members asked the group what to make of Jev, the new model from TypeSafe that came out of stealth this week. Quick recap for anyone who missed it: Jev does not write. You hand it a blob of state (a listing, a support ticket, a deal) and a question with fixed answers, and in under half a second it hands back a decision and a probability. Input tokens cost four cents per million. Output is free. The vendor's own line is "unstructured state in, typed probabilistic decisions out," and the pitch is that it cannot hallucinate because it never generates a string.

Someone in the resale world had already posted that it was the perfect model for scoring arbitrage deals at sub-200ms speeds. I quote-posted him with a prediction: Jev is going to be a very loud conversation, and maybe five people on my timeline will understand its proper use, because they have not solved for observability.

That was a gut reaction. So I did the thing I always do when I want to check one. I asked my agent to make the case, and it said this:

Jev gives no rationale. In an arbitrage loop that fires several times a second, you get a stream of opaque probabilities. Without logging the exact state sent, the question wording, the returned distribution, the threshold you used, and the eventual P&L, you cannot tell calibration drift from a bad listing format from a prompt the model reads too literally. Most people will drop Jev into a loop and treat 0.91 as a fact. The few who instrument state to decision to outcome will actually know whether it is making money.

I posted that to the group, typed "build your observability layer!", and then, about four minutes later, typed "I need to write an article on observability." This is that article.

I will tell you up front where I stand. I have built observability and governance architectures for very large clients. I gave one of them to the group that same night, a four-layer diagram a law firm has to have in place and documented before it can legally point AI at client data. And in my own businesses I have not done nearly enough of it. That gap is the honest reason to write this.

What observability means

Strip the vendor language away and observability is one question, asked about a system you did not watch run:

Can I reconstruct what happened, and why, from the evidence it left behind?

Not "did it run." Not "is the process alive." I taught those in July with the heartbeat playbook, and in August I found out the hard way that alive and producing are different claims. This is the next question up the ladder, and it is the one that matters once your agent is making decisions instead of just moving files.

For an agent, the evidence breaks into three pieces:

1. What it saw. The exact inputs at the moment of the decision. The listing as it looked at 2:14am, not the listing as it looks when you check at breakfast.
2. What it decided, and how sure it was. The answer, the score or probability if it gave one, the threshold you set, and the version of the instructions it was running.
3. What happened next. The outcome, joined back to the decision. Did the buy make money. Did the customer reply. Did the case get opened.

If you can pull those three for any decision your agent made last week, you have observability. If you can only pull the last message it sent you, you have a chat log.

A model like Jev makes this stark because it gives you nothing else. There is no paragraph explaining itself. A number comes back. One of the best threads I read this week made the point that this is arguably better, because an LLM's explanation is written after the answer, as plausible text, not as a record of how it got there. A probability logged at decision time is a more honest audit record than a paragraph the model composes afterward. But only if you logged it.

Why it matters

Three reasons, in the order they will cost you.

Silent failure is the default failure mode for agents. Traditional software crashes and pages someone. Agents return the wrong answer with a success code. One developer's post-mortem this week, which I keep rereading: "Six hours of agent compute. Wrong output. No error." Another burned seventy percent of a monthly usage quota on a sub-agent stuck in a loop fixing its own bug, and only found out from the plan meter. I have my own version. A proxy I run in front of one model counted 1,251 authentication failures over a stretch of days, logged every single one, and escalated none of them. The chat channel it fed looked fine. The intelligence had quietly drained out of a system that reported itself healthy.

You cannot tune what you cannot see. This is the Jev point specifically, and it generalizes to every scoring, routing, and classifying job you hand an agent. Two people ran independent calibration tests on Jev this week and got different answers. One found the confidence honestly fell with accuracy (0.72 down to 0.08 as it got things wrong), which is exactly what you want. Another found a ten-line text baseline beat it on calibration for his data. Both could be right for their workloads. You will never know which is true for yours without a log of decisions joined to outcomes. Without it, "0.91" is a vibe.

The liability lands on you. Same week as the Jev thread, I told the group about a distributor we bought a hundred units from. Their site says invoices available on request. Their customer service inbox is an AI bot. Four rounds of asking for a document invoice, four rounds of it emailing us our order details, no way to reach a human. That company now has an AI making commitments in its name with no record anyone is reading. In 2024 a Canadian tribunal ordered Air Canada to honor a refund its chatbot invented, and the line from the ruling is the whole lesson: it makes no difference whether the information comes from a static page or a chatbot. This June a federal court in Mississippi barred two lawyers for two years and cancelled a trial because AI-drafted filings cited cases that did not exist and nobody could produce a record of checking them. Colorado just re-enacted its AI law with a three-year record-keeping requirement for anyone using AI in consequential decisions, effective January. Europe's version requires deployers to keep automatic logs at least six months. None of that is aimed at a reseller with two agents. All of it is where the wind is blowing, and the gurus selling "AI receptionist in a weekend" have never read a page of it.

What it looks like in the enterprise

Here is the four-layer picture I showed the group. It was designed for a law firm's finance data platform, where an ethical wall between lawyers is a professional obligation, not an IT setting, and a client's outside counsel can audit the firm's controls on demand. I am simplifying, but the shape is what matters.

Plane 1, identity. Who is allowed to see what, decided in one firm-owned place, with an owner, an approver, a valid period, and a revocation path. Fail closed: if the system cannot resolve who you are, you get nothing.

Plane 2, enforcement. The rules from Plane 1 applied on every path the data can be reached, and tested on every path. The sentence I make every audience hear once: a wall that holds in your document system but leaks through a finance dashboard is no wall at all.

Plane 3, governance. Catalogs, labels, classifications, data owners, exception rules. All of it advisory. A label is never the lock. This plane feeds enforcement; it is never assumed to enforce anything by itself.

Plane 4, evidence. This is the observability plane, and it is the one to steal. Eight separate log sources (activity events, storage diagnostics, query logs, sign-in logs, source-system logs, and critically the history of the authorization decisions themselves) feed one governed evidence store that the platform team cannot silently rewrite. Retention set by policy. A copy kept off-platform. And out of that store come two answers a client auditor will ask for: who could see this matter on this date, and who did open this report. Two different questions, from two different sources, correlated.

There is one more piece of that diagram, and it is the part most vendors leave off. An amber box that states the design boundary out loud: no single native log proves which rows came back on every engine. Where the client needs that level of proof, we constrain the system to the paths we can instrument. We say what the logs cannot prove, before an auditor says it for us.

That architecture costs real money. The firm needs it because a leaked wall is a disqualification motion. What it buys them is not the logs. It is the ability to answer a question under pressure with evidence instead of memory.

And here is the scar from the enterprise side that I want you to carry into the next section. I worked with a professional services firm that had excellent telemetry. Months of it. Coverage climbing, thousands of events captured, a six hundred percent spike in one metric that should have set off alarms. Zero percent of alerts had ever been triaged. Nobody owned the reading. An unread dashboard is indistinguishable from no dashboard, and may be worse, because it makes you feel covered.

Translating it to a business of one

You do not need four planes. You need the two ideas underneath Plane 4, and a habit.

Idea one: the evidence lives somewhere the agent cannot rewrite. In the enterprise that is a governed store with restricted write paths. For you it is an append-only file or table the agent writes to and never edits. If your agent can go back and tidy its own history, you have a diary, not a log.

Idea two: every decision carries the three pieces and an ID. What it saw, what it decided and how sure, what happened. The ID is what lets you join the third piece back to the first two a week later. Without the join, you have a pile of decisions and a separate pile of outcomes and no way to learn from either.

The habit: somebody reads it. Fifteen minutes a week. This is the whole difference between the firm with zero percent triage and a firm with a working control. You are the triage team. Put it on the calendar or it will not happen, and I am saying that as the person who let a review queue go seventeen days stale in August.

Two things you already have that most people are throwing away:

The harness you run keeps a transcript. Hermes writes every session to a local database with the full message history, tool calls, tool results, and token counts. Claude Code writes a JSONL file per session. Codex writes a history file. That is your raw evidence, already collected. The catch: Hermes auto-prunes ended sessions after ninety days by default. If you want an audit trail, that setting has to be turned off or the retention raised. A log that deletes itself on a schedule you did not choose is not an audit trail.

If you route models through OpenRouter, there are two toggles in the dashboard, no code, that store every prompt and completion and fan each request out to a free logging tool or your own storage. For a non-engineer that is the lowest-friction path to real observability I found this week, with one gap: Claude Code and Codex on subscription login do not pass through it.

Then the practice that costs nothing and catches the most. I told the group Thursday that I audit my data by asking my agent CEO questions. "How did you come up with that number?" "Show me the data points you used." A member replied that she had just done a SKU and cash ledger audit and her agent found inbound shipping double-counted from the day it was first built, quietly inflating expenses. Another member closes every month by handing his agent the marketplace reports and asking which numbers in the books do not match, then fixing the field so next month it will. Those are observability practices. The only difference between them and the enterprise is that nobody wrote them down and nobody scheduled them.

One more, borrowed from the best Jev writeup I read: shadow mode. Before an agent decision is allowed to act, run it for a week next to the current behavior, logging its answer and confidence alongside what you would have done. Then compare. If the log is empty at the end of the week, you have learned something important about your setup before it cost you anything.

Your move

Pick one decision your agent makes without you. A deal score, a listing flag, a reply it sends, a price it sets. Then answer three questions from evidence, not memory:

1. Can I see what it saw at the moment it decided?
2. Can I see what it decided and how sure it was?
3. Can I tie that decision to what happened next?

Score each yes or no. Three yeses and you have observability for that decision. Anything less, the playbook below is the fix. My own score, for the reselling automation I described to the group this week: one out of three. I can see the decision. I cannot reliably see the state it decided on, and I have not joined outcomes. That number is why this issue exists.

The playbook: the operator's decision log

Operators: this is the full build, written so you can hand it to your agent and say "build this." Same test as the heartbeat playbook. If your agent gets stuck implementing it from this document, reply and tell me where.

What you are building, in one paragraph

An append-only decision log, one row per agent decision, holding a snapshot of the input state, the decision with its confidence and the rules version that produced it, and a slot for the outcome that gets filled in later using a shared ID. Plus a weekly fifteen-minute review that the agent prepares and you read. That is it. The enterprise version has eight sources and a governed store. Yours has one file and a calendar entry.

Design rules, in priority order

1. Append only. The agent may add rows. It may never modify or delete one. If it needs to correct something, it adds a correction row that points at the original. Enforce this with structure, not instruction: a file the agent's write tool can append to but the edit tool cannot touch, or a database table with insert permission and nothing else. A rule in a prompt is a promise. A missing permission is a fact.
2. Snapshot the input, do not reference it. Store the listing as it was, not a link to the listing. The link will show you today's price. The decision was made on Tuesday's.
3. Every row gets an ID at decision time. Not at review time. The ID is what the outcome row will carry back.
4. Log before you act. The row is written before the buy, the send, the flag. If the action fails, you still have the record that it was attempted.
5. Version the rules. Any decision made under a prompt, threshold, or scoring rubric carries a version tag. When you change the rubric, bump the version. Otherwise a week of decisions under two different rubrics looks like drift when it is really an edit you made and forgot.
6. The agent that decides is not the agent that reviews. Same principle as the heartbeat watcher. A separate scheduled job reads the log and writes the weekly brief. It should be too simple to be interesting.

The row

Nine fields. Add more only when a review has shown you needed one.

Outcome rows are the same log, different shape: decision_id, ts, outcome (sold / no reply / case opened / refund / etc.), value (P&L, hours saved, or whatever the job's unit of success is), and outcome_source (who or what recorded it). Outcome rows arrive days or weeks after decision rows. That is fine. The ID does the work.

Where it lives

Pick the one your agent can already write to.

• A JSONL file (one JSON object per line) in a folder the agent can append to. Simplest. Works with every harness. Your agent can read the whole thing with one command.
• A SQLite table if you already have a local database for the business, which several of you do. Grant insert only.
• A spreadsheet tab if that is where your business already lives. Weaker on append-only enforcement, but a log that exists in Sheets beats a perfect log that never got built.

Keep a copy off the machine that runs the agent. Nightly sync of the file to cloud storage is enough. The enterprise calls this the off-platform copy; you call it "the log survives my laptop."

Shadow mode before live mode

For any new decision the agent will make on its own:

1. Week one: the agent logs its decision and confidence on every case, with action_taken: none. You keep doing what you were doing.
2. End of week one: the review job lines up the agent's calls against yours. Where did it agree, where did it disagree, what was its confidence on the disagreements.
3. If the disagreements are ones you would have been fine with, promote it. If not, fix the rubric, bump the version, run another week.

For scoring models specifically, including Jev and anything else that hands you a probability, this week tells you what the number means for your data. If everything above 0.8 turned out right and everything below 0.6 turned out wrong, you have a threshold. If the confidence and the outcomes are unrelated, you have a coin flip with decimals, and you found out in shadow mode instead of with your money.

The weekly brief

A scheduled job, separate from every decision-making agent, runs once a week and writes you a short brief from the log. Five sections, in order:

1. Volume and cost. Decisions this week by job, and what they cost. The first thing that goes wrong is usually visible here as a number that doubled.
2. Close calls. Every decision where confidence landed within a band of the threshold (say, plus or minus 0.1). These are the rows to read. Near-misses are where a rubric is wrong.
3. Outcome joins. Decisions from prior weeks that now have outcomes. Hit rate by job and by confidence band. This is the calibration curve, and after a month it will tell you whether the model's confidence means anything.
4. Orphans. Decisions older than the expected outcome window with no outcome row. This is the part that fails silently: the outcome collection stopped and nobody noticed. It is the heartbeat principle applied to the log itself.
5. Version changes. Any rules_version change this week, so you can see a shift in results next to the edit that caused it.

Read it. Fifteen minutes. Silence from the brief means the job did not run, not that all is well; if the brief does not arrive on the scheduled day, that is your alert.

What the log cannot prove

Say this out loud the way the law firm architecture does. Your log proves what the agent was given and what it returned. It does not prove the agent's explanation is the real reason; a generated rationale is plausible text, not a trace. It does not prove the input snapshot was complete if the agent gathered it. And if the agent both decides and records outcomes, it can be wrong twice in a way that looks consistent. Where an outcome matters, have it recorded by something other than the agent that made the decision: the marketplace report, the bank feed, you.

The ten-minute start

You do not need the whole thing today.

1. Turn off auto-prune (or raise retention) in whichever harness you run, so the transcripts you already have stop deleting themselves.
2. Pick one decision. Add the nine-field row to whatever the agent already does before it acts. Shadow mode, action_taken: none.
3. Put a fifteen-minute block on next Friday to read what it wrote.

That is the enterprise's Plane 4, scaled to a business of one. The firm needs it to survive an audit. You need it to know whether the machine is making you money, and to be able to answer for it when it is not.

Steal this: the review job's first version is one prompt. "Read the decision log. List every decision from the last seven days where confidence was within 0.1 of the threshold. For each, show the input snapshot and the decision. Then list every decision older than fourteen days with no matching outcome row." Run it Friday. That is your observability layer, day one.

Hit reply and tell me what's on your bench this week. I read every one.