On Jev and System One Models

On September 15, 2026, Diogo Almeida, co-author of InstructGPT^{1 1}A series of OpenAI language models trained with RLHF to follow instructions. ¹A series of OpenAI language models trained with RLHF to follow instructions. and contributor to OpenAI’s work on RLHF,^{2 2}Reinforcement Learning with Human Feedback, a training technique used in many modern LLM training pipelines. ²Reinforcement Learning with Human Feedback, a training technique used in many modern LLM training pipelines. announced his new company and product: Jev, by TypeSafe AI.

This model broke the internet (or X, mostly) and became immediately very popular, advertised as a GPT-5.6 Terra-level model for a fraction of the cost and orders of magnitude faster; but what even is Jev?

Why is it special?

Jev is described as a System One model, so, what is that? You may have heard of “Thinking, Fast and Slow”, a popular book by psychologist Daniel Kahneman; I personally heard of this book for the first time in the ML space, around the time OpenAI’s o1^{3 3}OpenAI's first o-series reasoning model, trained to spend more computation reasoning before answering. ³OpenAI's first o-series reasoning model, trained to spend more computation reasoning before answering. was released. Sorry for the digression. The point is that this book popularized the theory of the differentiation of modes of thought: “System 1” is the intuitive, fast thinking mode, while System 2 is the slow, deliberate, and logical mode.

You can think of the usual reasoning-enabled LLM as a System 2 model: the thinking process is deliberate thinking, while the text after that (the actual answer) is either actions (tool/function calling) or speech.

Jev instead, being marketed as a System 1 model, is not an LLM (at least according to TypeSafe’s description, more on that later); it can’t generate text, and its output is calibrated probabilities: given many questions in which Jev answers with a probability of 80% for its chosen answer, about 80% of those answers should be correct. That’s the calibration claim, at least.

So, not an LLM?

Maybe not. Or maybe…

Many hypotheses on Jev’s architecture are being shared on the web, with about as many reproductions, using different architectures and training techniques. Some people are even accusing TypeSafe AI of getting the spotlight for a project they already released months prior, with some examples being Laya and GLiNER.

But are these really the same thing? While both these models and Jev return probabilities over a schema as an output (which yes, was not a “scientific breakthrough” ideated by TypeSafe AI), Jev seems to be a much more knowledgeable model than either Laya or GLiNER. So, did TypeSafe just train a bigger Laya? No, probably not.^{4 4}This is my hypothesis, not a disclosed architecture. TypeSafe describes a new architecture and Reinforcement Learning for Calibrated Decisions (RLCD), but hasn't published enough detail to establish Jev's backbone or parameter count. ⁴This is my hypothesis, not a disclosed architecture. TypeSafe describes a new architecture and Reinforcement Learning for Calibrated Decisions (RLCD), but hasn't published enough detail to establish Jev's backbone or parameter count.

Laya is a BERT-inspired model, based on ModernBERT-large, a roughly 400M-parameter language model with bidirectional attention, usually ideal to be tuned as a classifier. GLiNER is encoder-based too.

The problem is the cost of scaling this from scratch: to get a more knowledgeable model, you need a bigger model trained on a lot more data. Masked language models also get their direct prediction targets from only a subset of the input positions, unlike causal LLMs, which predict the next token at nearly every position.^{5 5}The original BERT recipe selected 15% of positions for prediction; other recipes use different masking rates. The other tokens still provide context and receive gradients, so this is not by itself proof that encoders are less compute-efficient at every scale but it would a costly experiment given the lack of pre-existing research on the topic. ⁵The original BERT recipe selected 15% of positions for prediction; other recipes use different masking rates. The other tokens still provide context and receive gradients, so this is not by itself proof that encoders are less compute-efficient at every scale but it would a costly experiment given the lack of pre-existing research on the topic.

Do you know what already has that large-scale training behind it and is available in different sizes, licenses, and flavors online? Yes, LLMs!

But, LLMs generate text, not probabilities

Wrong! And this is, again, nothing new.

LLMs have been adapted to tasks different from text generation for years already: they have been used as sentence embedders, as scorers/judges, as classifiers.^{6 6}Scaling Sentence Embeddings with Large Language Models; LLM-as-a-Verifier: A General-Purpose Verification Framework; and Language Models are Few-Shot Learners, which includes classification by comparing the probabilities of possible completions. ⁶Scaling Sentence Embeddings with Large Language Models; LLM-as-a-Verifier: A General-Purpose Verification Framework; and Language Models are Few-Shot Learners, which includes classification by comparing the probabilities of possible completions.

These methods share the same base idea: a single forward pass on a well-trained LLM is able to leverage its immense knowledge and (partly) capabilities for tasks different from text generation, without the latency of generating a whole response.

For the uninitiated, a single forward pass will give you the distribution for the next token over the full vocabulary of the model (e.g. “rate this conversation, only answer with a number from 1 to 10”, and you can then take the scores for the answer tokens, as in LLM-as-a-Verifier), plus the hidden states for each token position (generally used when you want to use an LLM as an embedding model).

One detail: those answers need to be single tokens for this readout to work in one pass. The LLM-as-a-Verifier paper actually suggests using letters, A, B, C and so on, even for score levels, then renormalize their logits with a softmax. This will be useful later.

LLMs are usually overconfident in text, isn’t that a problem here?

Yes, that is a big problem of LLMs, they can tell you something wrong is certainly right, and that’s certainly bad for a Jev-like model in which confidence should be meaningful; and yet, this topic was already encountered.

In 2022 Anthropic released a paper titled “Language Models (Mostly) Know What They Know”, which tries to answer the question “can a language model tell when it knows something, and when it probably doesn’t?”, and the answer, surprisingly, is mostly yes, especially with bigger models.

A preceding paper titled “How Can We Know When Language Models Know?” finds that LLM probabilities are often miscalibrated (the top token score is not automatically a reliable confidence estimate), but also finds improvements with scale and studies techniques such as temperature scaling to improve calibration on a fitted domain.

Temperature scaling is simply dividing the logits by a positive number before the softmax: above one, the distribution becomes flatter; below one, sharper. You fit that number on development examples. It doesn’t change which answer wins, only how confident the model is about it.

So, what we got till now is that:

• bigger models can have more reliable scores as a confidence metric, though scale alone doesn’t guarantee calibration;
• you probably don’t want to train a 30B BERT model from scratch;
• temperature scaling can improve calibration on fitted domains.

Can we replicate Jev?

I think so, yeah, and from the evidence we got I would do that using an LLM. I’m not the first to think of this, and from a new benchmark called JevBench you can actually see some of those LLM-based reproductions.

The surprising thing is that you don’t even need to tune an LLM to act like Jev, and instead taking the token distribution for the first token after our Jev-like input can already give satisfactory results.

How do I know this?

My own reproduction of Jev

I started by benchmarking some models against Jev on different datasets: from multiple-choice knowledge tasks (ARC-Easy, SciQ, OpenBookQA, CommonsenseQA), to soft-label tasks (ChaosNLI), to similarity-score prediction (STS-B), and a lot more.

Unfortunately I am not able to share Jev’s results because of the publication restrictions in the Jev’s license I had to sign, though what I think I can say is that on the knowledge and NLI tasks, Jev scored very similarly to Qwen3.8-27B with temperature scaling enabled. The comparison varies by task; it’s not the same ranking everywhere.

For Qwen I used the Q4_K_M quantization through llama.cpp, with reasoning disabled: one pass, reading the label probabilities, NOT asking it to generate a confidence score. In the tables + T means a temperature fitted on that environment’s dev split.

So, with that in mind, I wanted to create something, to tune some models, to make something I myself would use daily. I ended up choosing Liquid’s LFM2.5-VL family of models, in sizes 450M, 1.6B and 3B parameters.

YES, these are vision-enabled models, and they are also built to be pretty fast (LFM2’s architecture is crazy good for inference speed).

Aaaaand, after benchmarking them, they were not great, not the usual download-and-use models at least.

These are test NLL scores (cross-entropy for the soft-label tasks), lower is better. “Untuned” means the weights haven’t been fine-tuned; the models still get a dev-fitted temperature.

The brackets are 95% confidence intervals, from a bootstrap over groups of related examples.^{8 8}Groups are questions for MCQ, shared premises for ChaosNLI, articles for CivilComments, connected components of shared sentences for STS-B, households for Folktables, source images for CIFAR-10/C, and slides for Camelyon17. These intervals are conditional on each checkpoint, not intervals over training seeds. The evaluation uses the repository's splits, including custom grouped splits. ⁸Groups are questions for MCQ, shared premises for ChaosNLI, articles for CivilComments, connected components of shared sentences for STS-B, households for Folktables, source images for CIFAR-10/C, and slides for Camelyon17. These intervals are conditional on each checkpoint, not intervals over training seeds. The evaluation uses the repository's splits, including custom grouped splits.

So it was time to tune some models… I present you… alpha-sys-1.

The models are on Hugging Face: alpha-sys-1-450M, alpha-sys-1-1.6B, and alpha-sys-1-3B. Code and instructions are on GitHub.

And, since these are vision models, here are the image tasks too. This table shows the change in loss compared to the untuned model + T, with a paired confidence interval: negative means the tuned model is better.

About those metrics

These scores emphasize getting the probabilities right. For a question with one correct answer, NLL is the negative log of the probability assigned to that answer: confidently wrong answers are punished much more than uncertain ones.

And as we remember from before: among answers assigned a probability of 80%, about 80% should be correct. That does not mean “be more confident”; it means be confident when you have reason to be, and uncertain when you don’t.

Some datasets don’t have just one correct label, though. ChaosNLI has distributions from 100 annotators; CivilComments has annotator shares for toxicity. There, cross-entropy measures how well the model matches that distribution, rather than pretending everyone agreed with the majority.

I also measured Brier score, which uses squared probability error. Both Brier and log loss are proper scoring rules: in expectation, the best thing to report is the true distribution. They reward useful predictions as well as calibration, so the loss can also improve because the model learns more about the task.

Why not just use ECE, the expected calibration error? Because a model that always answers with the correct population base rate can have perfect ECE without telling you which examples are different. I report it too, just not alone.

One naming detail: the API’s confidence field summarizes how concentrated the distribution is, using normalized entropy. The confidence used in the evaluation is the top answer’s probability, which is what the 80% example above refers to.

How did I train them?

The idea is pretty simple: fine-tune the model to predict the answer distribution, instead of training it to write an answer.

I used cross-entropy against a target called y_soft: one-hot when the dataset has one answer, and a soft distribution when we have several annotations. No RLCD, no teacher model for these checkpoints, just supervised fine-tuning. The loss is computed over the full vocabulary, with target mass on the answer-label tokens, so the model also learns to put its probability mass on the labels we actually read.

Every released model is trained on the same mixture of seven environments:

• MCQ: ARC-Easy, SciQ, OpenBookQA and CommonsenseQA.
• ChaosNLI: natural-language inference with human disagreement.
• CivilComments-WILDS: toxicity, using annotator shares.
• STS-B: sentence similarity. I spread each annotator mean over the two neighboring score levels; this is a constructed target, not a histogram of the original votes.
• Folktables: income prediction from tabular records serialized as text, training on California 2014 and testing across states and years.
• CIFAR-10: clean images for training, clean and corrupted images for evaluation.
• Camelyon17-WILDS: histopathology patches, with held-out hospitals in evaluation.

All three sizes use LoRA rank 32, learning rate 1e-4, and an effective batch size of 32. The vision tower stays frozen; the adapters train the language model and projector, and are then merged into the released weights. Each run fits on a 24 GB GPU, though the 3B needed smaller micro-batches.

I shuffle the choice options on every draw, so the model can’t simply learn to prefer the first answer. The environments are sampled in proportion to the square root of their training-set sizes. Why not equally? Well, I tried that, and STS-B kept seeing the same small dataset while the image tasks were still learning. It overfit and pulled the stopping point too early for the other tasks.

Another thing that didn’t work: adding ordinary single-label NLI examples to get more training volume. They taught certainty, which is exactly what the disagreement labels in ChaosNLI punish. Keeping only its 100-annotator training rows worked much better in the single-domain experiments.

I trained three seeds per size, with a budget of 10,500 steps per mixture and dev evaluation every 300 steps. The saved checkpoint is the one with the lowest mean dev cross-entropy across environments, not the highest accuracy. The released seed is also chosen on dev only; it happened to be seed 1 for all three sizes.

The tables show those released checkpoints, without temperature scaling on top. The full aggregate results include all three seeds and their paired comparisons. Every released model improves over its untuned + T baseline on all seven environments; one of the other 3B seeds has a CIFAR interval that includes zero.

Why that name?

I’m answering this right away: alpha stands for the first revision of an AI system I am working on, sys-1 means System 1, so alpha-sys-1 will be the System 1 model used in that system, as simple as that.

It does not perform like Qwen at all

Yeah, even the 3B model does not have enough knowledge to solve some tasks that instead are easy for Qwen (and Jev), but on most of the environments I tuned it on, it is still way better than Laya:

Not everywhere: Laya with temperature scaling is competitive on CivilComments, and it wins on BoolQ, which I hadn’t trained on. These are results for the English checkpoint through Laya 0.1.6.

And on tasks it hasn’t seen?

I also tested on BoolQ and Yelp review stars, neither of which is in my fine-tuning mixture:

Here the improvements aren’t consistent. The models mostly stay around their untuned bases, sometimes better, sometimes worse. “Unseen” means unseen in my fine-tuning, not necessarily unseen in pretraining or in Laya’s training. The untuned LFM references in this table all use the first assistant token, plus a dev-fitted temperature.

I also tried leaving an entire environment out of training at 1.6B, without using its dev split either. Only the held-out MCQ fold passed; ChaosNLI and CIFAR didn’t. So training on a few tasks doesn’t automatically teach the model to be calibrated on a new one. The seven-environment leave-one-out experiment is still unfinished.

Now can you imagine what would happen if we added external engram memory to this model? Yeah… stay tuned for that.

@misc{desantis2026onjevandsystemonemodels,
  author = {Marco De Santis},
  title = {On Jev and System One Models},
  year = {2026},
  month = sep,
  url = {https://marcodsn.me/blog/on-jev-and-system-one-models}
}