Mental model: no free rent in your mind
This one is for the people who keep replaying what someone said or did: the person who insulted you, betrayed you, or let you down, and somehow still takes up your evenings.
This one is for the people who keep replaying what someone said or did: the person who insulted you, betrayed you, or let you down, and somehow still takes up your evenings.
This is one I come back to whenever a comment, a review, or a passing remark gets under my skin. It's for the moments when someone's words make you angry, leave you disappointed, or make you want to quit.
Machine learning is one of the most expensive ways to express a rule. It introduces uncertain output, evaluation work, data dependencies, and a system that changes when its inputs change. Those costs are justified when the problem contains patterns that ordinary software cannot express well.
I use three questions before a team commits to ML. The order matters because each question rules out a cheaper and more controllable approach.
Leadership became more concrete for me as my responsibility moved beyond the code. A technical decision changes a system. A leadership decision changes who can make the next hundred decisions, and how well they make them.
These are the principles I return to while building products and teams.
Model selection begins with the workload. Benchmarks and parameter counts help only after the team defines the input, required behavior, latency budget, deployment constraints, and failures it can tolerate.
These mental models help application engineers read a model card and turn architecture details into production questions.
As an application AI engineer, you don't need to train LLMs—but you must understand how they're built to make correct architectural decisions. Think of it like understanding database internals without writing a storage engine.
This guide consolidates the critical parameters you need to evaluate when selecting, deploying, or debugging LLMs in production.

| Aspect | Details |
|---|---|
| What it is | Next-token prediction (left-to-right) generator |
| Key questions | Is it decoder-only? Any special heads for classification/tooling? |
| Why it matters | Determines best-fit tasks (generation vs embedding) and streaming behavior |
| Common options | Decoder-only (GPT/LLaMA/Qwen), Encoder-decoder (T5), Encoder-only (BERT) |
| What you control | Prompting pattern, stop sequences, structured output constraints |
| Tradeoffs | Decoder-only can hallucinate; not ideal alone for retrieval/classification |
| Smell test | Code assistant or refactor agent → decoder-only is the right choice |
| Aspect | Details |
|---|---|
| What it is | Stored capacity vs per-token compute actually used |
| Key questions | Is it MoE? What's the activated compute per token? |
| Why it matters | Cost/latency can be close to a small model with large-model quality |
| Common options | Dense, MoE (top-k), Hybrid |
| What you control | Batch sizing, expert parallelism, routing config (if exposed) |
| Tradeoffs | MoE can have load imbalance; routing instability; harder to serve |
| Smell test | "30B total, ~3B active" → cheap inference, big VRAM footprint |
| Parameter | What it is | Why it matters | Tradeoffs |
|---|---|---|---|
| Layers (Depth) | Transformer block count (24, 32, 48, 80+) | Correlates with multi-step reasoning and abstraction | Deeper = higher latency; harder to quantize aggressively |
| Hidden size (Width) | Representation capacity per layer (2k–10k+) | Affects richness of representations + code reasoning | Wider increases VRAM + compute; may not help if data mix is weak |
| Aspect | Details |
|---|---|
| What it is | Attention parallelism mechanism; GQA/MQA reduces KV cache size |
| Key questions | Is it MHA, GQA, or MQA? What's the Q:KV head ratio? |
| Why it matters | Big driver of long-context feasibility and throughput |
| Common options | MHA (Multi-Head), GQA (Grouped Query), MQA (Single KV) |
| What you control | KV cache precision, batching, context window policy |
| Tradeoffs | Too aggressive KV sharing can slightly reduce quality |
| Smell test | GQA (32Q/4KV) → much smaller KV cache for long context |
| Aspect | Details |
|---|---|
| What it is | Maximum tokens handled as trained |
| Key questions | Native or RoPE-scaled? What's quality past 32k/64k? |
| Why it matters | Long-context correctness is non-linear; "supports" ≠ "understands" |
| Common options | 8k / 16k / 32k / 128k / 256k+ |
| What you control | Chunking, sliding window, retrieval strategy, summarization |
| Tradeoffs | KV cache explosion; "lost in the middle" phenomenon; higher latency |
| Smell test | If it forgets key constraints in long prompts → long-context weakness |
| Aspect | Details |
|---|---|
| What it is | How the model represents token positions |
| Key questions | RoPE? ALiBi? Any long-context scaling (NTK/YaRN)? |
| Why it matters | Governs long-range generalization |
| Common options | RoPE, ALiBi, Absolute, RoPE scaling variants |
| What you control | Prompt structure (headings), chunking order, retrieval placement |
| Tradeoffs | Scaling hacks can break beyond the trained window |
| Smell test | Native 256k trained usually beats RoPE-hacked 256k |
Only applicable to MoE models (e.g., Mixtral, Qwen-MoE, DeepSeek-V2)
| Parameter | What it is | Key questions | Tradeoffs |
|---|---|---|---|
| # Experts | Count of expert FFNs (8–128+) | How many? Are they specialized (code/math)? | Bigger memory footprint; more complex routing |
| Activated experts (top-k) | How many experts run per token | k=2/4/8? Does it vary? | Higher k = better quality, more compute; tail latency spikes |
| Router / load balancing | Mechanism selecting experts | Token-level routing? Load balance loss? | "Hot" experts cause congestion; quality variance |
| Parameter | What it is | Key questions | Engineering impact |
|---|---|---|---|
| Tokenizer + vocab | How text/code becomes tokens | BPE/SentencePiece? Code-optimized? Multilingual? | Bad tokenizer → 20–40% extra tokens; worse code formatting |
| Activation function | Non-linearity in FFN layers | SwiGLU? GeGLU? ReLU? | Some activations quantize better than others |
| Aspect | Details |
|---|---|
| What it is | Pretrain gives capability; post-train adds instruction-following/alignment |
| Key questions | How much instruction data? RLHF? Tool-use training? Safety tuning? |
| Why it matters | "Usable in prod" depends more on post-train than size |
| Common options | SFT only, RLHF/DPO/IPO, Tool-use finetune, Safety filters |
| What you control | System prompt, policy layer, tool schema, eval harness |
| Tradeoffs | Over-alignment/refusals; shallow SFT can be prompt-fragile |
| Smell test | If it ignores instructions or breaks JSON easily → weak post-train |
| Aspect | Details |
|---|---|
| What it is | Ratio of code/text/math + freshness |
| Key questions | How much code? Which languages? License-clean? |
| Why it matters | Strong predictor of code quality and hallucination rate |
| Common options | Code-heavy, Balanced, Math-heavy |
| Tradeoffs | Data contamination/licensing risk; stale frameworks |
| Smell test | If it's bad at modern React/TypeScript patterns → stale data mix |
| Parameter | What it is | Key questions | Tradeoffs |
|---|---|---|---|
| Precision | Numerical format (FP32/BF16/FP16/FP8) | FP16 vs BF16 vs FP8? Mixed precision? | FP16 can overflow; FP8 needs calibration |
| Quantization | Lower-bit weights (INT8/INT4) | GPTQ/AWQ? KV quantization? | Biggest lever for $/token but can break math/JSON/tool calls |
| Parameter | What it is | Key questions | Why it matters |
|---|---|---|---|
| Attention kernel | GPU attention algorithm | FlashAttention2/3? Paged attention? | Can mean 2–4× throughput difference |
| KV cache strategy | How past tokens' K/V are stored | KV precision? Paged KV? Prefix caching? | Dominant factor for long-context serving economics |
| Parameter | Details |
|---|---|
| Batching strategy | Static vs continuous batching; drives throughput and cost |
| Parallelism (TP/PP/EP) | Tensor/Pipeline/Expert parallelism; determines scaling + infra cost |
| Aspect | Details |
|---|---|
| What it is | Sampling parameters for generation |
| Key questions | Temperature/top-p/top-k? Repetition penalty? |
| Common profiles | Code mode: temp 0–0.2, low top-p; Creative: temp 0.7–1.0 |
| Tradeoffs | Too creative → wrong code; too greedy → brittle phrasing |
| Aspect | Details |
|---|---|
| What it is | Whether model learned tool usage |
| Key questions | Native tool-calling? Or prompt-hacked? |
| Why it matters | Determines agent reliability |
| Common options | Trained function calling, XML tags, Prompt-only |
| What you control | Tool schema design, retries, self-heal loops |
| Smell test | If it keeps "thinking" instead of calling a tool → weak tool training |
| Aspect | Details |
|---|---|
| What it is | Ability to emit valid JSON/DSL |
| Key questions | Does it support JSON schema / constrained decoding? |
| Why it matters | Critical for agents and tool calls |
| Common options | Free-form, JSON mode, CFG/constrained decoding |
| What you control | Use constrained decoding, schema validation + retries |
| Smell test | If tool calls break 5% of the time → production pain |
| Metric | What to measure | Why it matters |
|---|---|---|
| Throughput | Tokens/sec, req/sec at target context length | Real production cost = throughput × latency |
| Latency | Time-to-first-token (TTFT) + per-token decode speed | UX + agent planning loops |
| Determinism | Same input → same output? | Needed for CI, debugging, compliance |
| Aspect | Details |
|---|---|
| Safety/refusal | Guardrails (inline or post-hoc classifier); can impact valid request completion |
| Context management | How you place info in prompt; RAG placement; "lost in middle" mitigation |
| Failure modes | Known weak spots: Where does it hallucinate? Over-refuse? Loop? |
| Parameter | What it is | Key questions | Tradeoffs |
|---|---|---|---|
| Fine-tuning compatibility | Whether you can adapt it | LoRA/QLoRA? Full finetune? SFT pipeline? | Drift, regressions, catastrophic forgetting |
| Instruction tuning depth | Breadth/quality of instruction-following | Multi-turn? Code-specific SFT? | Overfit to chatty style; refusals |
Before deploying any model, run these smell tests:
Remember: Predictability beats raw intelligence. A model that's slightly less capable but reliable is often better for production than a brilliant but brittle one.
AI customization is a spectrum. At one end is zero-cost prompt engineering; at the other is pre-training a large language model (LLM) from scratch. Each stage brings different levels of complexity, data requirements and organizational investment. Understanding the stages helps you pick the right approach for your business or research needs.
The main strategies for tuning or customizing an AI model are:
Prompt engineering
At the simplest end of the spectrum lies prompt engineering. Instead of altering the model's parameters, you craft carefully worded prompts to guide its outputs. It needs no additional data or lengthy training. It's quick, cost-effective and gives immediate improvements, but offers less fine-grained control.
Retrieval-augmented generation (RAG)
RAG brings external knowledge sources, such as a vector database or an up-to-date knowledge base, into the model's responses. By retrieving relevant information at query time, the model can give more accurate and contextually rich answers without retraining. It adds complexity and inference cost, but remains a powerful way to update the model's "knowledge" on the fly.
Fine-tuning
For organizations that need more specialized control, fine-tuning is the next level. Training a pre-existing LLM on your domain-specific data refines its behavior and outputs. It requires labeled data, compute and a longer training period than prompt engineering or RAG, but it gives granular control over the model's performance and can significantly improve domain-specific accuracy.
Pre-training
The most resource-intensive end of the spectrum is pre-training an LLM from scratch on massive datasets. It provides maximum control and a tailor-made model architecture aligned with your needs. It also demands significant time (often weeks or months), computational power and large-scale curated data. It's typically pursued by large organizations or research teams aiming for pioneering innovations.

The chart above plots these stages against time/complexity and response quality/organizational maturity. As you move from prompt engineering to RAG, then to fine-tuning and finally pre-training, both the resource investment and the level of achievable specificity increase.

Source: Databricks blog.
The table above compares the methods across these attributes:
| Method | Data requirements | Training time | Advantages | Considerations |
|---|---|---|---|---|
| Prompt engineering | None | None | Fast and cost-effective | Limited control |
| RAG | External knowledge base | Moderate (e.g., computing embeddings) | Dynamically updated context | Longer prompts and increased inference costs |
| Fine-tuning | Thousands of domain-specific samples | Moderate to long (depending on dataset size) | Granular domain specialization | Requires labeled data and more compute |
| Pre-training | Billions to trillions of tokens | Long (days to many weeks) | Maximum control and tailor-fit | Extremely resource-intensive |
The right stage depends on your organization's goals, resources and maturity:
| If you need… | Start with |
|---|---|
| Immediate results, no additional data | Prompt engineering |
| Dynamic knowledge integration | Retrieval augmentation (RAG) |
| Specific domain expertise | Fine-tuning |
| Full control and customization | Pre-training |
By understanding each stage's trade-offs, you can invest in the approach that best fits your constraints and aspirations.
Interactive tools and references for seeing how transformer models work inside.
An interactive tool for visualizing and understanding the inner workings of transformer models. It shows how these models process and generate text across layers and attention heads.
A visualization platform with detailed, intuitive explorations of large language models (LLMs). It helps you build a deeper understanding of their structure, parameters and behavior.
A comprehensive resource on the circuit-level functioning of transformer models. It focuses on frameworks and simplifications that demystify their decision-making processes and internal representations.
Let's break this into two key parts: the attention mechanism and the transformer architecture. The architecture builds on attention, so mastering attention first is critical to understanding the full transformer model.
The attention mechanism is the foundation of Transformers. It allows the model to focus on relevant parts of the input sequence when generating outputs, addressing limitations of RNNs (like long-range dependencies).
Attention is built around the Query (Q), Key (K), and Value (V) paradigm.
Query (Q):
Key (K):
Value (V):
Attention computes relevance scores for each token pair using dot products:
Input sentence: "The cat sat on the mat."
If the focus is on "cat":
Q) = "cat".K) and Value (V).A single attention mechanism may miss nuanced relationships (e.g., syntax vs. semantics). Multi-Head Attention solves this by:
Transformers process sequences in parallel, so they lack inherent order information (unlike RNNs). Positional Encoding adds order back:
The Transformer builds on attention by combining Self-Attention, Feed-Forward Networks (FFN), and Residual Connections in a modular, stackable architecture.
Encodes the input into a rich contextual representation.
One encoder block:
Multi-head self-attention:
Feed-forward network (FFN):
Residual connections and layer normalization:
Structure:
Generates the output sequence step-by-step.
One decoder block:
Masked multi-head attention:
Cross-attention:
Feed-forward network and residual connections:
Structure:
A busy calendar and a busy mind will destroy your ability to create anything great.
"Gather together the things that change for the same reasons. Separate those things that change for different reasons."
A reading list of August 2024 arXiv papers, grouped by theme. Each entry links to the paper PDF with an excerpt from its abstract; my own comments are marked as notes. Source: code_your_own_AI.
GraphRAG leverages structural information across entities to enable more precise and comprehensive retrieval, capturing relational knowledge and facilitating more accurate, context-aware responses. Given the novelty and potential of GraphRAG, a systematic review of current technologies is imperative. This paper provides the first comprehensive overview of GraphRAG methodologies. We formalize the GraphRAG workflow, encompassing Graph-Based Indexing, Graph-Guided Retrieval, and Graph-Enhanced Generation. We then outline the core technologies and training methods at each stage.
Verilog is a popular hardware description language for designing and modeling digital systems; thus, Verilog generation is one of the emerging areas of research to facilitate the design process. In this work, we propose VerilogCoder, a system of multiple Artificial Intelligence (AI) agents for Verilog code generation, to autonomously write Verilog code and fix syntax and functional errors using collaborative Verilog tools (i.e., syntax checker, simulator, and waveform tracer).
Advancements in multi-agent LLM frameworks as well as emergent capabilities such as in-context learning have enabled LLMs to employ more sophisticated reasoning strategies, such as 'chain-of-thought' reasoning (CoT). We propose LLM-ACTR, which shows improved decision-making capabilities over LLMs by integrating intermediate representations extracted from a well-establish neuro-symbolic system: the ACT-R cognitive architecture.
This is an exception: I'm including a paper on military defence because it explains the civilian possibilities offered by neuro-symbolic AI. Otherwise, I try to keep military context out of my content.
Contributing to the growing body of research, this study represents a comprehensive exploration of the extensive possibilities offered by Neuro-Symbolic AI.
Microsoft hides its automatic prompt optimization technique for enhancing weaker models on complex tasks here.
Hand-crafting high quality prompts to optimize the performance of language models is a complicated and labor-intensive process. We propose Concept Distillation (CD), an automatic prompt optimization technique for enhancing weaker models on complex tasks.
Reinforcement Learning (RL) is a potent tool for sequential decision-making and has achieved performance surpassing human capabilities across many challenging real-world tasks. As the extension of RL in the multi-agent system domain, multi-agent RL (MARL) not only need to learn the control policy but also requires consideration regarding interactions with all other agents in the environment, mutual influences among different system components, and the distribution of computational resources.
Brain-inspired spiking neural networks (SNNs) have received widespread attention. Unlike traditional artificial neural networks (ANNs), which transfer information using intensive floating-point values, SNNs transfer discrete 0-1 spikes between neurons, providing a more efficient neuromorphic computing paradigm.
Spiking neural networks (SNNs) are becoming a promising alternative to conventional artificial neural networks (ANNs) due to their rich neural dynamics and the implementation of energy-efficient neuromorphic chips.
A new update on the transformer's attention mechanism.
The quadratic computational and memory complexities of self attention have limited its scalability for modeling users' long range behaviour sequences. To address this problem, we propose ELASTIC, an Efficient Linear Attention for SequenTial Interest Compression, requiring only linear time complexity and decoupling model capacity from computational cost.
The attention module in a transformer is a smoothed cubic spline. Viewed in this manner, this mysterious but critical component of a transformer becomes a natural development of an old notion deeply entrenched in classical approximation theory.
We propose a Siamese Multiple Attention Temporal Convolutional Network (Siamese MA-TCN) to capitalize on the strengths of both TCN architecture and multi-head self-attention, enabling the proficient extraction of both local and long-term dependencies.
Mental models are powerful tools for understanding the world, making better decisions, and solving problems. Think of them as mental shortcuts or frameworks that help you navigate complexity.
I write these up as short posts on the Blog, one idea at a time:
See the full series under Mental Models on the Blog.
Remember: The best way to use mental models is to practice applying them to real-life situations. Start with a few that resonate with you and gradually expand your toolkit.
Using retrieval-augmented generation (RAG) in trading means integrating live data, such as news and economic indicators, with AI agents and advanced AI models to make informed trading decisions.
This note covers the components you need, the end-to-end workflow, an example implementation in Python, and the considerations to keep in mind.
Live Data Feeds:
Trading Platform APIs:
RAG Model:
Trading Algorithms:
Data Retrieval:
Query Generation:
Retrieval Process:
Generation of Insights:
Trading Algorithm Integration:
Execution of Trades:
A minimal Python sketch of the loop: fetch and index news, generate insights, then act on them.
import requests
from elasticsearch import Elasticsearch
import openai
# Configure APIs and Elasticsearch
news_api_url = "https://newsapi.org/v2/everything"
stock_api_url = "https://www.alphavantage.co/query"
broker_api_url = "https://broker-api.example.com/trade"
es = Elasticsearch("http://localhost:9200")
# Fetch live news and stock prices
def fetch_news(api_key, query):
response = requests.get(news_api_url, params={"q": query, "apiKey": api_key})
return response.json()["articles"]
def fetch_stock_price(api_key, symbol):
response = requests.get(stock_api_url, params={"function": "TIME_SERIES_INTRADAY", "symbol": symbol, "interval": "1min", "apikey": api_key})
return response.json()["Time Series (1min)"]
# Index news articles in Elasticsearch
def index_news_articles(articles):
for article in articles:
es.index(index="news", document=article)
# Retrieve and generate insights using OpenAI
def generate_insights(query):
results = es.search(index="news", query={"match": {"content": query}})
context = " ".join([hit["_source"]["content"] for hit in results["hits"]["hits"]])
response = openai.Completion.create(engine="text-davinci-003", prompt=f"Analyze the following news and provide trading insights: {context}", max_tokens=150)
return response.choices[0].text.strip()
# Execute trade
def execute_trade(api_key, symbol, action, quantity):
response = requests.post(broker_api_url, json={"symbol": symbol, "action": action, "quantity": quantity}, headers={"Authorization": f"Bearer {api_key}"})
return response.json()
# Main trading loop
def trading_loop():
news_api_key = "YOUR_NEWS_API_KEY"
stock_api_key = "YOUR_STOCK_API_KEY"
broker_api_key = "YOUR_BROKER_API_KEY"
query = "impact of current events on tech stocks"
# Fetch and index news articles
articles = fetch_news(news_api_key, query)
index_news_articles(articles)
# Generate insights
insights = generate_insights(query)
print(f"Generated Insights: {insights}")
# Make trading decision
if "positive sentiment" in insights:
execute_trade(broker_api_key, "AAPL", "buy", 10)
elif "negative sentiment" in insights:
execute_trade(broker_api_key, "AAPL", "sell", 10)
# Run the trading loop
trading_loop()
Using RAG in trading can provide a significant edge by leveraging real-time data and advanced AI models to make informed decisions.
The AI Answer Evaluator is a streamlined tool designed to assess the quality of AI-generated answers by comparing them against expected answers. It uses LLM to identify factual discrepancies, missing information, and errors, and provides a detailed evaluation with a score out of 10.s
The app also includes a customizable evaluation prompt template and offers insights into token usage, including cost estimation. Perfect for developers, researchers, and educators, this app helps analyze and refine AI outputs effectively while ensuring clarity and precision in results.
This is an app about comparing the return on investment (ROI) of buying a flat and investing in the market. It discusses buying a flat, living in it for ten years, and then selling it.
Ever wondered how some AI systems seem to know exactly what you need? They might be using a technique called retrieval-augmented generation (RAG). It's like giving the AI a little research assistant:
In its most basic form, a RAG application runs three steps:
User request ──► Retrieval ──► Augmentation (request + supporting data) ──► Generation ──► Response

Yeah, brains, experience, and connections are great. But Altman says the real secret sauce is something else: never giving up.
He's seen it time and time again. The founders who win aren't always the smartest ones in the room. They're the ones who keep going when things get tough. The ones who say, "I'll figure this out" even when it seems impossible.
The best part? You don't have to be born with this grit. Altman believes you can learn it, just like any other skill.
Altman breaks it down into three things you need to nail:
Altman has another gem: If you can't explain your idea in 25 words or less, you're in trouble. The best founders can pitch their vision clearly and concisely.
This might sound crazy, but Altman says it's easier to start a hard startup than an easy one. Why? Because a bold vision attracts passionate people. No one wants to work on something boring.
Being a founder is hard work, no doubt about it. But if you have the grit, the focus, and a clear vision, you've got a fighting chance.
So, go out there and build something amazing!
Triples are a fundamental concept in knowledge graphs, especially those based on the Resource Description Framework (RDF).
A triple is a data structure with three components:
Together, these three parts form a statement about the data, often likened to a simple sentence with a subject, verb and object. For example:
| Subject | Predicate | Object |
|---|---|---|
| "Alice" | "knows" | "Bob" |
This triple can be represented as:
<Alice> <knows> <Bob>
In RDF, triples make assertions about resources as subject–predicate–object expressions. They can be stored in an RDF graph, where each subject and object is a node and each predicate is an edge connecting two nodes.
<http://example.org/Alice> <http://example.org/knows> <http://example.org/Bob> .
<http://example.org/Alice> <http://example.org/age> "30" .
<http://example.org/Bob> <http://example.org/age> "25" .
In this example:
Triples are the building blocks of RDF-based knowledge graphs. They represent complex information in a structured form that machines can easily parse and query. This enables powerful querying and the ability to infer new information from existing triples.
Knowledge graphs often use query languages such as SPARQL to query these triples and extract meaningful information from the interconnected data.
Triples are essential for representing data in RDF and knowledge graphs. They allow a structured, flexible and semantically rich representation of information.