Using Jev with Hermes Agent to improve results

Most of what we ask a language model to do is not writing. It's deciding.

Is this email a bill or a newsletter? Did the page load correctly, or am I looking at a login wall? Should this action be allowed to fire? We hand those questions to a trillion-parameter model that answers by generating a paragraph of text, which we then parse with a regex hoping to find "yes" somewhere in it.

That's a strange way to make a decision. And it's expensive, slow, and (because the answer arrives wrapped in prose) only loosely checkable.

Jev is an argument that those questions deserve a different kind of model entirely. I've spent the last few days wiring it into my Hermes agent to see whether the argument holds up. This is what I found, including the part where I deliberately did not let it do very much.

What Jev actually is

Jev is a decision model from TypeSafe. It's the first production model I've used that belongs in a different category than "chat model": it does not generate text. It takes a state (some text or JSON, up to a large budget) and a set of typed questions, and returns probabilities.

There are three primitives:

Noul: a yes/no question, answered with a probability. Does this email convey urgency? → 0.95
Choice: pick one of N options, answered with a distribution over them. Which team handles this? → { billing: 0.92, technical: 0.08 }
Score: place something on an ordered scale, answered with a position and a distribution.

That's the whole interface. No prose. No chain-of-thought to read. No markdown to strip. The output is already structured, because structure is the only thing it can produce. "Typed output" isn't a formatting option you request, it's the model's entire output space.

The other thing worth knowing: output tokens are free. You pay for input only. A typical decision call (a few hundred tokens of state plus a question schema) costs about $0.00002. Not "cheap relative to the frontier." Cheap relative to thinking about the cost.

Access is easy if you already use OpenRouter: POST /api/alpha/decisions with typesafe/jev-1.13. Same API key I use for everything else, no new vendor account, no new billing relationship. That mattered to me. I wasn't going to stand up another account to test a hypothesis.

Why a decision model pairs well with an LLM

The intuition is that LLMs are generalists and you pay for that generality on every call, even when the task is narrow. Jev is the opposite: useless for writing, extremely good at closing a question.

Concretely, four things improve when you split the work:

1. Cost collapses. My email triage was running a full agent model over every message to answer "what is this?" That's thousands of tokens per email to produce a category label. Jev answers the same question for ~560 input tokens at $0.042/M. Measured against my own agent-model calls, that's 11x cheaper than a short classification prompt and 36x cheaper than the same job handed to my current default model. The ratio matters less than the consequence: at this price you stop rationing. I went from "should I classify every email?" to "of course I classify every email."

2. Latency collapses. Median round trip on a decision is about 0.3 seconds. A decision is not a thought process; it doesn't need to be.

3. The output can't be malformed. This is the underrated one. When an LLM classifies, you get "This appears to be a newsletter.". Now you own a regex, an edge case where it says "not a newsletter" and your matcher finds the word and flips the sign, and a silent failure mode when the model decides to be helpful and explain itself instead. With Jev there is no parsing step, so there is no parsing bug. The branch in your code is a float comparison.

4. You get a number, and the number is a gate. A probability lets you write policy that an LLM's "yes" never could: auto-accept above 0.85, escalate between 0.6 and 0.85, refuse or ask below that. Calibration becomes an engineering decision rather than a vibe. Confident calls go fast and cheap; uncertain calls get the expensive model. That's the hybrid pattern the papers on decision-model gating keep landing on, and it's obvious why: you only pay for reasoning when reasoning is the missing ingredient.

What it does to Hermes skills

Hermes is the agent that runs my automation: cron-scheduled briefs, email triage, a daily check on a rewards site, GPS map generation. Its skills are markdown instruction files backed by Python scripts. The interesting question wasn't "can Jev answer questions," it was "where in that stack does a cheap, typed decision beat both a rule and an LLM?"

Four answers, in descending order of how confident I am:

Gates in front of expensive work. Every cron job has steps that either go or don't: is the session still logged in, did the page render or am I looking at a captcha, did the scrape return results. Today those are DOM selectors and string matches, brittle in exactly the way that produces a silent no-op at 7:30 AM. A Noul question over the page text turns "did this work" into a labeled answer, with an explicit low-confidence path that notifies me instead of guessing.

Classification where rules have already failed. This is where Jev earned its place in my stack, and I'll show the receipts below.

A second opinion before irreversible actions. A Delete or a Send is a place where a cheap independent check pays for itself the first time it fires. Not a replacement for guardrails, but an additional voice with a confidence threshold.

Skill routing. Theoretically appealing: one model picks which of my skills to load instead of a large prompt reading all of them. But the payoff is currently small, because skills load on demand anyway. I'd rather not add a network hop to every message for a marginal token saving. Filed under "revisit."

The experiment: four days, 167 decisions

I wired Jev into my email pipeline as a pure observer. Every new email went to Jev with the same information my rules see (sender and subject) and was asked to pick a category. Both verdicts were logged side by side. Jev changed nothing; it only voted.

Four days and 167 calls later (total cost: $0.004), the raw agreement number was 56%. That sounds like a failure, and it isn't. It's a lesson in reading your own taxonomy:

Google Alerts: 18/18 perfect
Windsurf: 3/3
Jobs: 7/9 (78%) · Newsletters: 18/25 (72%)
Marketing: 3/5 · Bills: 3/6 · Ads: 1/2
"Other": 40/99 (40%)

The 56% is dominated by the fact that 99 of 167 emails landed in other, my catch-all bucket for "the rules didn't recognize you." Jev has no catch-all instinct. It looks at those emails and says what they actually are. So the disagreement concentrated precisely where my rules had already given up.

And in that pile were three emails I would have been annoyed to lose:

"Your credit card payment is scheduled" → Jev said bills, p=1.00. My rules said other.
"Thanks for your payment" → Jev said bills, p=0.92. My rules said other.
"Annual Report due in 10 weeks, DESIGNING WEBS" → Jev said bills, p=0.86. My rules said other.

Other is on the auto-delete list. A credit card payment notice, a payment confirmation, and a state-required annual filing deadline for my own business would all have been deleted as junk. I'd already been burned by this class of bug once before (a renewal notice misclassified and swept away), and here was a cheap model catching a statutory deadline from a sender line and a subject.

That's the whole case for decision models in one example: not that Jev writes better than a frontier model, but that a $0.00002 call correctly labeled something my hand-written rules got wrong, and the failure mode was deletion of mail I wanted.

The part where I didn't trust it

Here's what I did not do: let it delete anything.

Even with that evidence, authorizing deletions on four days of data and 167 samples would be reckless. Bills still sits at 3/6, a coin flip on the category that matters most. And there's a trap worth stating plainly: agreement with your existing process is not correctness. The literature on decision gating is explicit that models can agree with a stronger model more often than either agrees with ground truth, because they share the same blind spots. My rules aren't ground truth. Neither is the LLM. A high agree-rate tells you two systems coincide, not that the answer is right.

So I wired the guard so it can only ever move in the safe direction:

If the rules are about to file an email as other (which auto-deletes) and Jev says bills with p >= 0.85, the email is moved to Bills and kept, and the summary labels it (kept · jev).

That's it. It can convert a deletion into a kept email. It cannot delete, cannot override a keep rule, cannot touch a category where I haven't measured it. Worst case is an email sitting in my inbox with a label telling me a decision model made the call, which is a bug I can live with.

Deletion-side authority stays in shadow mode until I have a few hundred samples and at least thirty per category. If that never justifies itself, the guard stays one-directional forever and I've lost nothing but a fraction of a cent.

Practical notes if you want to try this

A few things I'd have wanted to know going in:

Design for Jev being absent. Every call in my pipeline is wrapped so a failure returns None and the code falls through to exactly the behavior it had before Jev existed. A decision model is an optimization, and optimizations should not be able to break the pipeline.

One call, two uses. The verdict feeds both the guard and the log. I compute it once per email and cache it by message id. Naively I'd have called Jev twice, once to act and once to observe, doubling latency and cost for no reason.

Ask one thing at a time. My schema mixes three Noul questions with a Choice. That's fine, but every question you add is input tokens on every call. Start with the question you'll actually branch on.

Keep the probability. The first instinct on wiring a classifier is to grab choice and throw away probabilities. The probabilities are the whole value: they're what let you decide not to decide.

Shadow first, and keep the log. My log is JSONL, one line per decision, with both verdicts and the confidence. It cost nothing to write and it's the only reason I can tell you the category breakdown above instead of a story about how it "felt pretty good."

Where this goes

The honest summary is that Jev is a small, sharp tool with a narrow shape. It can't write your docs, it can't explain itself, and if you ask it something requiring synthesis you'll get a confident number that means nothing. What it can do is close a question cheaply, deterministically, and transparently, and then tell you how sure it is, which is more than most classifiers offer and considerably more than a paragraph of prose.

For an agent, that's a genuinely useful new primitive. Not a replacement for the model doing the reasoning, but the thing standing in front of it saying: this one's obvious, don't spend a dollar on it.

Four days of data bought me a mail-loss guard, a caught statutory deadline, and a much better picture of where my own heuristics are weak. That's a decent return on four-tenths of a cent.