# What nobody knows yet

> The open problems: reading a model's insides, sudden abilities, and what understanding means here.

Lesson 9 of 9 in [How LLMs actually work](https://learnfold.app/learn/llms/), a free course from Learnfold.

You now know the recipe. Tokens, embeddings, attention, a mountain of text, and a round of tuning. Here's the uncomfortable part. Knowing the recipe isn't the same as knowing what the finished model is doing, and the people who build these systems say so openly.

### Built, but not fully readable

Every step of the computation is visible. You can print any of the billions of numbers at any moment. What's missing is an explanation in human terms. Which numbers mean "this sentence is sarcastic", and how does the model get from a question to an answer?

That gap is the subject of interpretability research, and it has made real progress. Researchers have pulled millions of features out of large models, directions that light up for specific concepts. One fires for a particular bridge, another for bugs in code, another for flattery. Turning one of those features up or down changes what the model writes in the expected way. They've traced small circuits too, like how a model seems to pick a rhyming word before writing the line that leads up to it. Still, these maps cover a fraction of what's going on. Nobody can yet read a large model the way you'd read a program.

### Does written reasoning show the real reasoning?

When a model writes out its working before answering, it's tempting to treat that as a transcript of its thinking. Studies suggest it often isn't fully faithful. A model can reach an answer for one reason and write down a different, tidier one, or leave out a hint that actually swayed it. The working is useful, and it often helps the model get the answer right. It is still text the model produced, and it can leave things out.

### Do abilities appear suddenly?

As models grew, some abilities seemed to switch on abruptly. A task would sit near zero at one size and then jump to good at the next. People called these emergent abilities, and they fed both excitement and worry, since you can't prepare for what you can't predict.

Other researchers argued that much of the suddenness comes from how tests are scored. If a long sum gets zero credit unless every digit is right, steady improvement underneath looks like a sudden leap on top. Score it more finely and many of those jumps smooth out. The argument isn't settled. Some abilities still seem hard to forecast, even when the underlying error falls smoothly.



Move the accuracy slider slowly. Partial credit improves steadily while the all-or-nothing score rises more sharply near the top.

### Does it understand?

One camp says these models are elaborate pattern matchers, stitching text together with no picture of the world behind it. Another points to evidence of internal structure. In one well-known experiment, a small model trained only on lists of moves from the board game Othello turned out to contain a readable picture of the board, though it had never been shown one. Both camps can point to real behavior. Models solve problems they've never seen, and they also fail at small twists on easy puzzles in ways no person would.

Part of the disagreement is about the word. "Understand" was made for people, and a language model is a different kind of thing. A more useful question is narrower. On this kind of task, how reliable is it, and how would I check?

That's a good place to end the course and start asking your own questions. Any sentence here can become a page of its own.

### Rabbit holes

> How do researchers find a feature for one concept among billions of numbers?

> What did the Othello experiment actually show, and what are its limits?

> If a model's written reasoning isn't faithful, how could anyone check what it really did?

Read online: https://learnfold.app/learn/llms/open-questions/