forced wordcelification (of a math model)

10 Nov, 2026

This is a work-in-progress.

{Written : 27 October 2026 | Repo : [WIP!] }

Paradigma is a new auto-research startup based in Rome that has recently released the delightful model Limite 1B - Violetto. Self-evidently, Limite is tiny little thing, {{but it punches above its weight in mathematics.}} <- revise

(From hereon in, I'll refer to the final model as Violetto and the base model as Limite, with the whole 'family' being Limite/Violetto)

To be completely honest with you, I don't care much about how good Violetto is at math. Okay, that's a lie. I wouldn't be so interested in the model if it were an utter lobotomite, but my fascination with restricted-domain pretraining is what begets that interest in the first place. And Limite/Violetto is the rare case of a modern model with significantly restricted pretraining that is not disabled by design (*cough* training on random bullshit like children's stories *cough*).

I mean, this is usually what post-training looks like! While I'm quite the proponent of pre-training diversity and thoughtfulness in data curriculum construction, there are so many neat properties that come out of regime like this, especially combined with even more brutally math-centric post-training.

Namely, despite the base model also being trained on a mathmaxxed corpus, Violetto's post-training kind of cooked its other abilities.

Paradigma uses the above as an example of an "instruction following failure," but it feels less like an issue of instruction following and more like one of parametric knowledge access; ... [ do more writing lazy ; it feels like it's drawn towards mitosis ]

[TODO: run the above prompt on the side-by-side!]

The first example I thought of was dangling a concept more associated with a certain problem in mathematics than its actual origin in the model's face-- Konigsberg, of Seven Bridges Fame. [Okay, I know it is of a lot of historical fame as well. I fucking love Prussia. The fact that it goes by Kaliningrad now probably does not help Violetto at all, but Limite still gets the answer, so it's not too obscure of a fact as to not be recallable by the base model.]

With these two motivating examples, I decided I was interested in eliciting OOD behaviour from this model, where OOD vaguely means "not possible without steering or interventions," because the math-heavy curation of its pretraining corpus and its poor non-mathematical instruction following make it difficult to fully gauge what Violetto actually knows.

Intuitively, a model trained mostly on mathematics will find elucidating mathematical ideas quite easy, while the internal pathways that allow generalist models to properly identify and tap their knowledge base may not have been reinforced during training, or have even formed in the first place! Thus, I hypothesize that we may be able to properly assess how much "moribund" knowledge Violetto really learned through interventions on its residual stream.

If this is the case, it's likely that we can improve its mathematical abilities through making its crystallized knowledge more accessible. Also, because we have access to both the base and post-trained model, we can observe the pernicious effects (if any) of RL & SFT post-training on a narrow domain, contrasting that with just pre-training on that more narrow domain. Also, because this is cool.

On my depressing unoriginality

Evaluations

To get an idea of what I'm working with, I decided to run evals myself. These should all be repeatable via the setup in my repo.

Math

With a fixed budget of 16,384 tokens, we get the results below. Obviously, this is not what Violetto and co. were evaluated on; the model seemed awfully verbose for how small it is, so I wanted to try both this...

...and a longer math eval using Paradigma's own suite, of course. However, due to compute constraints and the fact that I'm not particularly distrustful of Paradigma, I decided to only bother recreating the AIME 2026 Violetto run; it came out at 93.9% vs the tech report's 94.01%, so that checks out. (It used a median of 17k tokens per problem, but went up to 120k on the difficult ones!)

General Knowledge (loglik)

(Done via raw-text loglik, where random MMLU chance is 25)

Generative Short Answer

(Basically what you'd expect from a regular eval, judged by gpt-6-luna with low effort on 24 October 2026)

These results are a little... uh... abysmal, but not unexpected.

Post-Training Damage?

Transplanting (lol)

Something interesting about the architecture of this model is that it uses a lot of NanoGPT Speedrun tricks. One that I was particularly eyeing was its use of gated value embeddings every third layer.

For all intents and purposes, this is tantamount to having a second embedding table with a row for every token in our input vocabulary. Instead of entering the residual stream directly, each token's row is added straight to attention every third layer.

So, in a value-embedding-affected layer, for each token position $s$ in the sequence, let $x_s$ be that position's residual stream vector entering the layer, and $t_s$ the token id sitting there. The values fed to attention are

where $E[t_s]$ is the value for that token from the second embedding table. This is somewhat reminiscent of DeepSeek's Engram, which they demonstrated in their own paper the recall effects of.

A corollary of this is that the mid-layer patching that [transfers recall in normal models does not work on Limite/Violetto. (Actually, Violetto doesn't know enough facts to be fairly tested, so I couldn't include it below).

To demonstrate the difference between Limite and Violetto here, I decided to run a small test on the value-embedding table. (And because I am me, I went straight to geography. Sue me, it's easy!)

For each of the 51 countries below, I

1. Looked it up in the value-embedding table (if its name contained multiple tokens, I averaged them. This only affected 4/51 countries.)
2. Passed that row through a single head in layer 46 (the last layer that reads the value-embedding table; I ran some experiments before this that showed patching value-embeddings has basically no effect on recall for every value-embedding layer except this one [why is that the case? late layer -> affects readout heavily, maybe], where I got 0.70 recall fidelity from the patch). I took the best heads for both Limite and Violetto (3 and 8, respectively). Selecting a single best did better than picking multiple heads, probably due to specialization, so I feel it's acceptable in a readout-only experiment.
3. Read the vector produced by step (2) with the LM head unembedding
4. Took the score of the first token of the capital and ranked them by the scores for each country (this, of course, only measures how likely it is in comparison to the 50 other capitals!)

(Many of these capitals are not represented as one token; I've provided the full capital names for reading convenience, but something like Hanoi really starts with H, so that's what I checked the probs for. This could be a minor reason why tokens like Hav and Kab with more "country-capital-ness" to them attract so many of the countries in Limite, but it's much more of a head-specific thing, as you can see by comparing to another head's outputs. This effect is also minorly visible in Limite's chart!)

While this gives us a nice figure, it doesn't feel exhaustive enough to conclude anything about what knowledge Violetto actually has-- we only see what it can output. It could be true that Violetto's access to its knowledge has been trained out of it, while the knowledge still remains moribund in its weights. (Essentially, that readout is extremely fucked-up.)

To attempt and address this, I decided to Frankenstein Limite and Violetto with components of each other. (Soup is a limite-base variant made by averaging different checkpoints; I won't get too much into it, as there are more details in their tech report, but I use as a sort of control here.)

I originally planned to only use CounterFact, but limite-base only got 71 out of 1020 total facts right there. Because I didn't want to encounter any weird floor-effect issues, I added 203 random, well-known facts, bringing the total pool to 1223. Recall is based on 198 facts limite-base got right. Stated facts are 300 random facts from a pool of 1223 total facts, whether the model knows them or not, with the true fact written into the prompt, meant to test readout of facts. Made-up people are 400 random stories generated by a script, which I figured may be easier than the stated facts section because there is almost no no real-world knowledge tied to them at all.

(You can check out all three categories' data here!)

todo write up these

(damage exp)