LLM architecture mental models
, 4 min read
Model selection begins with the workload. Benchmarks and parameter counts help only after the team defines the input, required behavior, latency budget, deployment constraints, and failures it can tolerate.
These mental models help application engineers read a model card and turn architecture details into production questions.
Dense and mixture-of-experts models
A dense model uses the same parameter set for each token. A mixture-of-experts model routes tokens through a subset of specialized feed-forward blocks.
The distinction changes how to read size. Total parameters describe stored capacity. Activated parameters give a better signal for compute per token. A large MoE model can activate far fewer parameters than its headline number suggests, though it may still require substantial memory and more complex serving.
Ask:
- How many parameters are active for each token?
- Can the serving stack place and route the experts efficiently?
- Does routing behavior remain stable on the target workload?
- Is the latency distribution acceptable, not only the average?
Dense models often offer simpler deployment behavior. MoE models can offer a useful quality-to-compute tradeoff when the serving system supports them well.
Context is capacity, not memory
A context window states how many tokens a model can accept. It does not promise equal attention to every part of that input, reliable retrieval, or durable memory between calls.
Long context affects cost and latency through attention computation and the key-value cache. It also changes evaluation. A model can accept a long document and still miss a fact in the middle, combine unrelated passages, or follow an instruction from the wrong source.
Ask:
- How much context does the real task need?
- Which information must remain exact?
- Can retrieval reduce the input before generation?
- Does the model find and use evidence at the positions your application creates?
Treat persistent memory as a separate system. It needs rules for what to store, how to retrieve it, when to expire it, and how a user can correct it.
Attention design shapes long-context cost
Architectures use techniques such as grouped-query attention, multi-query attention, sliding windows, and optimized attention kernels to reduce the cost of long inputs and decoding.
The model specification alone does not determine application performance. Hardware, serving framework, batch behavior, prompt length, output length, and cache reuse all affect the result.
Measure the full request path with representative traffic. A theoretical context limit has little value if the target latency or concurrency collapses long before that limit.
Tokenization is part of the bill
Models do not read characters or words directly. Their tokenizer can represent the same source text with different token counts.
This matters for code, identifiers, multilingual content, structured records, and specialized terminology. More tokens increase input cost and consume context capacity. Token boundaries can also affect extraction and constrained output.
Run the tokenizer against real inputs before estimating cost. Averages from general English text will not describe every product.
Quantization trades precision for operating cost
Quantization stores or computes model values with fewer bits. It can reduce memory use and increase throughput, but the effect on quality depends on the model, method, hardware, and task.
Small regressions can matter in tool arguments, structured output, numerical work, or rare edge cases even when general text still reads well.
Evaluate the quantized build against the same acceptance set as the original model. Compare failure categories and latency rather than relying on a single aggregate score.
Post-training defines behavior
Two models with similar base architecture can behave differently after instruction tuning, preference optimization, safety training, and tool-use training.
Application engineers experience this as behavior: whether the model follows a schema, calls the right tool, asks for missing information, cites supplied evidence, and refuses outside the intended boundary.
Prompt work cannot repair every post-training mismatch. A smaller model trained for the required interaction may outperform a larger general model in a bounded workflow.
Failure behavior beats the leaderboard
Average benchmark performance hides the cases that determine production trust.
Build an evaluation set around the system's decisions. Include ambiguous requests, missing data, conflicting context, long inputs, invalid tool results, and requests that should reach a person. Record how the model fails and whether the surrounding system can detect the failure.
The best model for an application is often the one whose errors the team can recognize and contain.
Keep a model decision record
For each candidate, record:
- The workload and representative inputs
- Required output behavior
- Context and tokenizer measurements
- Latency, throughput, and cost under expected traffic
- Failure categories and escalation policy
- Security, privacy, and deployment constraints
That record turns model selection into an engineering decision. It also makes the next evaluation faster when models, prices, and product requirements change.