Skip to main content
Dheeraj Inampudi
Engineer, company builder, and founder of EigenH AI
View all authors

When to use machine learning

· 4 min read

Machine learning is one of the most expensive ways to express a rule. It introduces uncertain output, evaluation work, data dependencies, and a system that changes when its inputs change. Those costs are justified when the problem contains patterns that ordinary software cannot express well.

I use three questions before a team commits to ML. The order matters because each question rules out a cheaper and more controllable approach.

Leadership principles I practice

· 3 min read

Leadership became more concrete for me as my responsibility moved beyond the code. A technical decision changes a system. A leadership decision changes who can make the next hundred decisions, and how well they make them.

These are the principles I return to while building products and teams.

LLM architecture mental models

· 4 min read

Model selection begins with the workload. Benchmarks and parameter counts help only after the team defines the input, required behavior, latency budget, deployment constraints, and failures it can tolerate.

These mental models help application engineers read a model card and turn architecture details into production questions.

LLM evaluation guide for AI engineers

· 7 min read

As an application AI engineer, you don't need to train LLMs—but you must understand how they're built to make correct architectural decisions. Think of it like understanding database internals without writing a storage engine.

This guide consolidates the critical parameters you need to evaluate when selecting, deploying, or debugging LLMs in production.

LLM evaluation framework infographic: eight areas around a core evaluation framework — model architecture, context and position, MoE experts, tokenization, training and alignment, inference and serving, agent and tools, and production metrics


Model architecture​

Model type​

AspectDetails
What it isNext-token prediction (left-to-right) generator
Key questionsIs it decoder-only? Any special heads for classification/tooling?
Why it mattersDetermines best-fit tasks (generation vs embedding) and streaming behavior
Common optionsDecoder-only (GPT/LLaMA/Qwen), Encoder-decoder (T5), Encoder-only (BERT)
What you controlPrompting pattern, stop sequences, structured output constraints
TradeoffsDecoder-only can hallucinate; not ideal alone for retrieval/classification
Smell testCode assistant or refactor agent → decoder-only is the right choice

Parameters: total vs activated (MoE)​

AspectDetails
What it isStored capacity vs per-token compute actually used
Key questionsIs it MoE? What's the activated compute per token?
Why it mattersCost/latency can be close to a small model with large-model quality
Common optionsDense, MoE (top-k), Hybrid
What you controlBatch sizing, expert parallelism, routing config (if exposed)
TradeoffsMoE can have load imbalance; routing instability; harder to serve
Smell test"30B total, ~3B active" → cheap inference, big VRAM footprint

Depth and width​

ParameterWhat it isWhy it mattersTradeoffs
Layers (Depth)Transformer block count (24, 32, 48, 80+)Correlates with multi-step reasoning and abstractionDeeper = higher latency; harder to quantize aggressively
Hidden size (Width)Representation capacity per layer (2k–10k+)Affects richness of representations + code reasoningWider increases VRAM + compute; may not help if data mix is weak

Attention architecture​

AspectDetails
What it isAttention parallelism mechanism; GQA/MQA reduces KV cache size
Key questionsIs it MHA, GQA, or MQA? What's the Q:KV head ratio?
Why it mattersBig driver of long-context feasibility and throughput
Common optionsMHA (Multi-Head), GQA (Grouped Query), MQA (Single KV)
What you controlKV cache precision, batching, context window policy
TradeoffsToo aggressive KV sharing can slightly reduce quality
Smell testGQA (32Q/4KV) → much smaller KV cache for long context

Context and positioning​

Context length​

AspectDetails
What it isMaximum tokens handled as trained
Key questionsNative or RoPE-scaled? What's quality past 32k/64k?
Why it mattersLong-context correctness is non-linear; "supports" ≠ "understands"
Common options8k / 16k / 32k / 128k / 256k+
What you controlChunking, sliding window, retrieval strategy, summarization
TradeoffsKV cache explosion; "lost in the middle" phenomenon; higher latency
Smell testIf it forgets key constraints in long prompts → long-context weakness

Positional encoding​

AspectDetails
What it isHow the model represents token positions
Key questionsRoPE? ALiBi? Any long-context scaling (NTK/YaRN)?
Why it mattersGoverns long-range generalization
Common optionsRoPE, ALiBi, Absolute, RoPE scaling variants
What you controlPrompt structure (headings), chunking order, retrieval placement
TradeoffsScaling hacks can break beyond the trained window
Smell testNative 256k trained usually beats RoPE-hacked 256k

Mixture of experts (MoE)​

Only applicable to MoE models (e.g., Mixtral, Qwen-MoE, DeepSeek-V2)

ParameterWhat it isKey questionsTradeoffs
# ExpertsCount of expert FFNs (8–128+)How many? Are they specialized (code/math)?Bigger memory footprint; more complex routing
Activated experts (top-k)How many experts run per tokenk=2/4/8? Does it vary?Higher k = better quality, more compute; tail latency spikes
Router / load balancingMechanism selecting expertsToken-level routing? Load balance loss?"Hot" experts cause congestion; quality variance

Tokenization and internals​

ParameterWhat it isKey questionsEngineering impact
Tokenizer + vocabHow text/code becomes tokensBPE/SentencePiece? Code-optimized? Multilingual?Bad tokenizer → 20–40% extra tokens; worse code formatting
Activation functionNon-linearity in FFN layersSwiGLU? GeGLU? ReLU?Some activations quantize better than others

Training and alignment​

Training stage​

AspectDetails
What it isPretrain gives capability; post-train adds instruction-following/alignment
Key questionsHow much instruction data? RLHF? Tool-use training? Safety tuning?
Why it matters"Usable in prod" depends more on post-train than size
Common optionsSFT only, RLHF/DPO/IPO, Tool-use finetune, Safety filters
What you controlSystem prompt, policy layer, tool schema, eval harness
TradeoffsOver-alignment/refusals; shallow SFT can be prompt-fragile
Smell testIf it ignores instructions or breaks JSON easily → weak post-train

Training data mix​

AspectDetails
What it isRatio of code/text/math + freshness
Key questionsHow much code? Which languages? License-clean?
Why it mattersStrong predictor of code quality and hallucination rate
Common optionsCode-heavy, Balanced, Math-heavy
TradeoffsData contamination/licensing risk; stale frameworks
Smell testIf it's bad at modern React/TypeScript patterns → stale data mix

Inference and serving​

Precision and quantization​

ParameterWhat it isKey questionsTradeoffs
PrecisionNumerical format (FP32/BF16/FP16/FP8)FP16 vs BF16 vs FP8? Mixed precision?FP16 can overflow; FP8 needs calibration
QuantizationLower-bit weights (INT8/INT4)GPTQ/AWQ? KV quantization?Biggest lever for $/token but can break math/JSON/tool calls

Attention and memory​

ParameterWhat it isKey questionsWhy it matters
Attention kernelGPU attention algorithmFlashAttention2/3? Paged attention?Can mean 2–4× throughput difference
KV cache strategyHow past tokens' K/V are storedKV precision? Paged KV? Prefix caching?Dominant factor for long-context serving economics

Batching and parallelism​

ParameterDetails
Batching strategyStatic vs continuous batching; drives throughput and cost
Parallelism (TP/PP/EP)Tensor/Pipeline/Expert parallelism; determines scaling + infra cost

Decoding parameters​

AspectDetails
What it isSampling parameters for generation
Key questionsTemperature/top-p/top-k? Repetition penalty?
Common profilesCode mode: temp 0–0.2, low top-p; Creative: temp 0.7–1.0
TradeoffsToo creative → wrong code; too greedy → brittle phrasing

Agent and tool use​

Tool and function calling​

AspectDetails
What it isWhether model learned tool usage
Key questionsNative tool-calling? Or prompt-hacked?
Why it mattersDetermines agent reliability
Common optionsTrained function calling, XML tags, Prompt-only
What you controlTool schema design, retries, self-heal loops
Smell testIf it keeps "thinking" instead of calling a tool → weak tool training

Structured output​

AspectDetails
What it isAbility to emit valid JSON/DSL
Key questionsDoes it support JSON schema / constrained decoding?
Why it mattersCritical for agents and tool calls
Common optionsFree-form, JSON mode, CFG/constrained decoding
What you controlUse constrained decoding, schema validation + retries
Smell testIf tool calls break 5% of the time → production pain

Production metrics​

Performance​

MetricWhat to measureWhy it matters
ThroughputTokens/sec, req/sec at target context lengthReal production cost = throughput × latency
LatencyTime-to-first-token (TTFT) + per-token decode speedUX + agent planning loops
DeterminismSame input → same output?Needed for CI, debugging, compliance

Reliability​

AspectDetails
Safety/refusalGuardrails (inline or post-hoc classifier); can impact valid request completion
Context managementHow you place info in prompt; RAG placement; "lost in middle" mitigation
Failure modesKnown weak spots: Where does it hallucinate? Over-refuse? Loop?

Customization​

ParameterWhat it isKey questionsTradeoffs
Fine-tuning compatibilityWhether you can adapt itLoRA/QLoRA? Full finetune? SFT pipeline?Drift, regressions, catastrophic forgetting
Instruction tuning depthBreadth/quality of instruction-followingMulti-turn? Code-specific SFT?Overfit to chatty style; refusals

Quick evaluation checklist​

Before deploying any model, run these smell tests:

  • JSON/Structured output — Can it reliably emit valid JSON for 100 requests?
  • Tool calling — Does it call tools correctly, or does it "think" endlessly?
  • Long context — Does it remember constraints from 50k tokens ago?
  • Code patterns — Does it know modern frameworks (React 18, TypeScript 5)?
  • Determinism — Do tests flake? (1/20 runs fail = determinism gap)
  • Refusals — Does it refuse benign requests too often?

Remember: Predictability beats raw intelligence. A model that's slightly less capable but reliable is often better for production than a brilliant but brittle one.

Stages of AI tuning

· 3 min read

AI customization is a spectrum. At one end is zero-cost prompt engineering; at the other is pre-training a large language model (LLM) from scratch. Each stage brings different levels of complexity, data requirements and organizational investment. Understanding the stages helps you pick the right approach for your business or research needs.

The four stages​

The main strategies for tuning or customizing an AI model are:

  1. Prompt engineering

    At the simplest end of the spectrum lies prompt engineering. Instead of altering the model's parameters, you craft carefully worded prompts to guide its outputs. It needs no additional data or lengthy training. It's quick, cost-effective and gives immediate improvements, but offers less fine-grained control.

  2. Retrieval-augmented generation (RAG)

    RAG brings external knowledge sources, such as a vector database or an up-to-date knowledge base, into the model's responses. By retrieving relevant information at query time, the model can give more accurate and contextually rich answers without retraining. It adds complexity and inference cost, but remains a powerful way to update the model's "knowledge" on the fly.

  3. Fine-tuning

    For organizations that need more specialized control, fine-tuning is the next level. Training a pre-existing LLM on your domain-specific data refines its behavior and outputs. It requires labeled data, compute and a longer training period than prompt engineering or RAG, but it gives granular control over the model's performance and can significantly improve domain-specific accuracy.

  4. Pre-training

    The most resource-intensive end of the spectrum is pre-training an LLM from scratch on massive datasets. It provides maximum control and a tailor-made model architecture aligned with your needs. It also demands significant time (often weeks or months), computational power and large-scale curated data. It's typically pursued by large organizations or research teams aiming for pioneering innovations.

Visualizing the spectrum​

Chart plotting prompt engineering, RAG, fine-tuning and pre-training with time and complexity on the vertical axis and response quality and organizational maturity on the horizontal axis

The chart above plots these stages against time/complexity and response quality/organizational maturity. As you move from prompt engineering to RAG, then to fine-tuning and finally pre-training, both the resource investment and the level of achievable specificity increase.

Detailed comparison​

Comparison table of prompt engineering, RAG, fine-tuning and pre-training with their definitions, primary use cases, data requirements, advantages and considerations

Source: Databricks blog.

The table above compares the methods across these attributes:

MethodData requirementsTraining timeAdvantagesConsiderations
Prompt engineeringNoneNoneFast and cost-effectiveLimited control
RAGExternal knowledge baseModerate (e.g., computing embeddings)Dynamically updated contextLonger prompts and increased inference costs
Fine-tuningThousands of domain-specific samplesModerate to long (depending on dataset size)Granular domain specializationRequires labeled data and more compute
Pre-trainingBillions to trillions of tokensLong (days to many weeks)Maximum control and tailor-fitExtremely resource-intensive

Choosing the right approach​

The right stage depends on your organization's goals, resources and maturity:

If you need…Start with
Immediate results, no additional dataPrompt engineering
Dynamic knowledge integrationRetrieval augmentation (RAG)
Specific domain expertiseFine-tuning
Full control and customizationPre-training

By understanding each stage's trade-offs, you can invest in the approach that best fits your constraints and aspirations.

Transformer links

· One min read

Interactive tools and references for seeing how transformer models work inside.

Transformer Explainer​

An interactive tool for visualizing and understanding the inner workings of transformer models. It shows how these models process and generate text across layers and attention heads.

Open Transformer Explainer

LLM Visualization​

A visualization platform with detailed, intuitive explorations of large language models (LLMs). It helps you build a deeper understanding of their structure, parameters and behavior.

Open LLM Visualization

Transformer Circuits​

A comprehensive resource on the circuit-level functioning of transformer models. It focuses on frameworks and simplifications that demystify their decision-making processes and internal representations.

Open A Mathematical Framework for Transformer Circuits

Understanding attention and transformer architectures

· 5 min read

Three mental models​

  • RNNs (Storyteller): Reads a story word by word, remembering only what came before to predict what comes next.
  • CNNs (Photographer): Focuses on details in patches of an image and combines them to form a complete picture.
  • Transformers (Collaborator): Reads the entire story at once and learns relationships between all parts simultaneously, identifying which details to focus on globally.

Let's break this into two key parts: the attention mechanism and the transformer architecture. The architecture builds on attention, so mastering attention first is critical to understanding the full transformer model.

1. Attention mechanism​

The attention mechanism is the foundation of Transformers. It allows the model to focus on relevant parts of the input sequence when generating outputs, addressing limitations of RNNs (like long-range dependencies).

Core idea​

  • Instead of processing sequences step by step (like RNNs), attention compares all tokens in the input to each other to determine relevance.
  • It assigns weights (importance scores) to each token based on how much they contribute to understanding a given token or query.

Key components of attention​

Attention is built around the Query (Q), Key (K), and Value (V) paradigm.

  1. Query (Q):

    • Represents what you're looking for.
    • Example: In the sentence "The cat sat on the mat," if we're focusing on "cat," the query is related to "cat."
  2. Key (K):

    • Represents the characteristics of each token.
    • Example: Every word in the sequence has a key, encoding its meaning.
  3. Value (V):

    • Represents the actual content of the tokens.
    • Example: "The", "cat", "sat", etc., are passed as values.

Self-attention formula​

Attention computes relevance scores for each token pair using dot products:

Attention(Q, K, V)=softmax(QKTdk)V\text{Attention(Q, K, V)} = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V
  • QKᵀ: Measures similarity between query and key.
  • dₖ: Scaling factor to avoid large dot products in high dimensions.
  • softmax: Converts scores into probabilities (attention weights).
  • V: Final weighted sum of values, representing the focused context.

Example of self-attention​

Input sentence: "The cat sat on the mat."

If the focus is on "cat":

  • Query (Q) = "cat".
  • Each word in the sentence has a Key (K) and Value (V).
  • The attention mechanism calculates how much each word contributes to "cat" using QKᵀ:
    • "The" → Low weight (irrelevant).
    • "sat" → High weight (verb related to "cat").
    • "mat" → Low weight (less connected).

2. Multi-head attention​

A single attention mechanism may miss nuanced relationships (e.g., syntax vs. semantics). Multi-Head Attention solves this by:

  • Running multiple attention mechanisms (heads) in parallel.
  • Each head captures different relationships (e.g., word order, context, synonyms).
  • Outputs are concatenated and projected into a final representation.
MultiHead(Q, K, V)=Concat(head1,head2,…,headh)WO\text{MultiHead(Q, K, V)} = \text{Concat}(\text{head}_1, \text{head}_2, \dots, \text{head}_h)W^O

3. Positional encoding​

Transformers process sequences in parallel, so they lack inherent order information (unlike RNNs). Positional Encoding adds order back:

  • Each token embedding is enriched with a positional vector.
  • Common approach: Use sine and cosine functions for each position.
PosEnc(pos, 2i)=sin⁡(pos100002i/d),PosEnc(pos, 2i+1)=cos⁡(pos100002i/d)\text{PosEnc(pos, 2i)} = \sin(\frac{\text{pos}}{10000^{2i/d}}), \quad \text{PosEnc(pos, 2i+1)} = \cos(\frac{\text{pos}}{10000^{2i/d}})

4. Transformer architecture​

The Transformer builds on attention by combining Self-Attention, Feed-Forward Networks (FFN), and Residual Connections in a modular, stackable architecture.

Transformer encoder​

Encodes the input into a rich contextual representation.

One encoder block:

  1. Multi-head self-attention:

    • Computes relationships between all tokens in the input.
    • Outputs context-aware embeddings for each token.
  2. Feed-forward network (FFN):

    • Applies two dense layers with non-linear activation.
    • Refines the token representations.
  3. Residual connections and layer normalization:

    • Residual connections stabilize training by adding the input back to the output of attention/FFN layers.
    • Layer normalization improves gradient flow.

Structure:

Input Embeddings→Self-Attention→FFN→NormalizationOutput Embeddings\text{Input Embeddings} \xrightarrow{\text{Self-Attention}} \xrightarrow{\text{FFN}} \xrightarrow{\text{Normalization}} \text{Output Embeddings}

Transformer decoder​

Generates the output sequence step-by-step.

One decoder block:

  1. Masked multi-head attention:

    • Ensures the decoder only attends to already-generated tokens (causal attention).
  2. Cross-attention:

    • Allows the decoder to attend to the encoder's outputs for context (important for tasks like translation).
  3. Feed-forward network and residual connections:

    • Similar to the encoder.

Structure:

Generated Embeddings→Masked Self-Attention→Cross-Attention→FFNNext Token Embedding\text{Generated Embeddings} \xrightarrow{\text{Masked Self-Attention}} \xrightarrow{\text{Cross-Attention}} \xrightarrow{\text{FFN}} \text{Next Token Embedding}

End-to-end transformer workflow​

  1. Input processing:
    • Tokenize input (e.g., "The cat sat").
    • Add positional encodings.
  2. Encoder:
    • Processes input embeddings into contextual representations.
  3. Decoder:
    • Predicts the next token based on encoder outputs and already-generated tokens.
  4. Output:
    • Decodes token predictions into readable text.

Key strengths of transformers​

  1. Handles long-range dependencies: Self-attention captures relationships between distant tokens.
  2. Parallelizable: Processes sequences in parallel, unlike RNNs.
  3. Scalable: Stacking multiple encoder/decoder blocks improves performance on large datasets.
  4. Adaptable: Works across NLP, vision, and multimodal tasks.

Next steps​

  1. Experiment with attention matrices (visualizing weights).
  2. Implement a small Transformer from scratch to solidify understanding.
  3. Explore pre-trained models (e.g., BERT, GPT) to see Transformers in action.

Quotes and phrases

· 2 min read

Rules for Life (Genius Rules)​

  1. Be Ready to Start Over
  2. When Inspired, Act Immediately
  3. Choose to Be Yourself
  4. Do What Feels Like Play to You, But Looks Like Work to Others
  5. Choose Your Desires Carefully
  6. Fall in Love with Reading
  7. Master the Basics
  8. Be an Owner
  9. Use the New Leverage
  10. Create Empty Space
  11. Overcome the Need for External Validation

A busy calendar and a busy mind will destroy your ability to create anything great.


General Quotes​

  • "You must expect great things of yourself before you can do them." — Michael Jordan
  • "Any person capable of angering you becomes your master." — Epictetus
  • "The greatest sin is to think that you are weak." — Swami Vivekananda
  • "A question well put is half solved." — Charles Kettering
  • "Would you build a beautiful house and let someone you dislike live in it for free? Then why let them occupy space in your mind rent-free?" — Sadhguru

Personal Reflections​

Own​

  • "We must accept finite disappointment, but never lose infinite hope."
  • "When your own feelings override the facts, you will stay in the dark."
  • "A fool will get you to their level and win the argument with experience. Don’t fall for it."

Leadership Lessons​

  • "Having access to people smarter than you is a blessing, not a threat."
  • "When you focus on you, you grow. When you focus on shit, shit grows."
  • "Remember, you don't need to be aggressive. You need to be clear and concise."
  • "Jo dikhta hai, wahi bikta hai." (What is visible, sells.)

Tech Principles​

  • Quote from "Clean Architecture":

    "Gather together the things that change for the same reasons. Separate those things that change for different reasons."

  • "Don’t let the perfect get in the way of good enough."
  • "The smarter I get, the more I realize that I can’t be sure of much."
  • "As developers, we don't do things because they are easy; we do them because we thought they were going to be easy."

Business​

  • "Businesses today are data rich and insights poor."

Starting Over: 3 Things to Remember​

  1. People who did it before you aren’t super smarter than you.
  2. You’re not bad at it; you’re just starting out.
  3. There’s nothing you need to know that you can’t find out or figure out.

Research paper picks

· 5 min read

A reading list of August 2024 arXiv papers, grouped by theme. Each entry links to the paper PDF with an excerpt from its abstract; my own comments are marked as notes. Source: code_your_own_AI.

Retrieval and knowledge graphs​

Graph Retrieval-Augmented Generation: A Survey​

arxiv.org/pdf/2408.08921

GraphRAG leverages structural information across entities to enable more precise and comprehensive retrieval, capturing relational knowledge and facilitating more accurate, context-aware responses. Given the novelty and potential of GraphRAG, a systematic review of current technologies is imperative. This paper provides the first comprehensive overview of GraphRAG methodologies. We formalize the GraphRAG workflow, encompassing Graph-Based Indexing, Graph-Guided Retrieval, and Graph-Enhanced Generation. We then outline the core technologies and training methods at each stage.

Agents, reasoning and prompting​

VerilogCoder: Autonomous Verilog Coding Agents with Graph-based Planning and Abstract Syntax Tree (AST)-based Waveform Tracing Tool​

arxiv.org/pdf/2408.08927

Verilog is a popular hardware description language for designing and modeling digital systems; thus, Verilog generation is one of the emerging areas of research to facilitate the design process. In this work, we propose VerilogCoder, a system of multiple Artificial Intelligence (AI) agents for Verilog code generation, to autonomously write Verilog code and fix syntax and functional errors using collaborative Verilog tools (i.e., syntax checker, simulator, and waveform tracer).

Cognitive LLMs: Towards Integrating Cognitive Architectures and Large Language Models for Manufacturing Decision-making​

arxiv.org/pdf/2408.09176

Advancements in multi-agent LLM frameworks as well as emergent capabilities such as in-context learning have enabled LLMs to employ more sophisticated reasoning strategies, such as 'chain-of-thought' reasoning (CoT). We propose LLM-ACTR, which shows improved decision-making capabilities over LLMs by integrating intermediate representations extracted from a well-establish neuro-symbolic system: the ACT-R cognitive architecture.

Neuro-Symbolic AI for Military Applications​

arxiv.org/pdf/2408.09224

note

This is an exception: I'm including a paper on military defence because it explains the civilian possibilities offered by neuro-symbolic AI. Otherwise, I try to keep military context out of my content.

Contributing to the growing body of research, this study represents a comprehensive exploration of the extensive possibilities offered by Neuro-Symbolic AI.

Concept Distillation from Strong to Weak Models via Hypotheses-to-Theories Prompting​

arxiv.org/pdf/2408.09365

note

Microsoft hides its automatic prompt optimization technique for enhancing weaker models on complex tasks here.

Hand-crafting high quality prompts to optimize the performance of language models is a complicated and labor-intensive process. We propose Concept Distillation (CD), an automatic prompt optimization technique for enhancing weaker models on complex tasks.

Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey​

arxiv.org/pdf/2408.09675

Reinforcement Learning (RL) is a potent tool for sequential decision-making and has achieved performance surpassing human capabilities across many challenging real-world tasks. As the extension of RL in the multi-agent system domain, multi-agent RL (MARL) not only need to learn the control policy but also requires consideration regarding interactions with all other agents in the environment, mutual influences among different system components, and the distribution of computational resources.

Spiking neural networks​

Temporal Reversed Training for Spiking Neural Networks with Generalized Spatio-Temporal Representation​

arxiv.org/pdf/2408.09108

Brain-inspired spiking neural networks (SNNs) have received widespread attention. Unlike traditional artificial neural networks (ANNs), which transfer information using intensive floating-point values, SNNs transfer discrete 0-1 spikes between neurons, providing a more efficient neuromorphic computing paradigm.

Obtaining Optimal Spiking Neural Network in Sequence Learning via CRNN-SNN Conversion​

arxiv.org/pdf/2408.09403

Spiking neural networks (SNNs) are becoming a promising alternative to conventional artificial neural networks (ANNs) due to their rich neural dynamics and the implementation of energy-efficient neuromorphic chips.

Attention and sequence modeling​

ELASTIC: Efficient Linear Attention for Sequential Interest Compression​

arxiv.org/pdf/2408.09380

note

A new update on the transformer's attention mechanism.

The quadratic computational and memory complexities of self attention have limited its scalability for modeling users' long range behaviour sequences. To address this problem, we propose ELASTIC, an Efficient Linear Attention for SequenTial Interest Compression, requiring only linear time complexity and decoupling model capacity from computational cost.

Attention is a smoothed cubic spline​

arxiv.org/pdf/2408.09624

The attention module in a transformer is a smoothed cubic spline. Viewed in this manner, this mysterious but critical component of a transformer becomes a natural development of an old notion deeply entrenched in classical approximation theory.

Siamese Multiple Attention Temporal Convolution Networks for Human Mobility Signature Identification​

arxiv.org/pdf/2408.09230

We propose a Siamese Multiple Attention Temporal Convolutional Network (Siamese MA-TCN) to capitalize on the strengths of both TCN architecture and multi-head self-attention, enabling the proficient extraction of both local and long-term dependencies.

Mental models

· 3 min read

Mental models are powerful tools for understanding the world, making better decisions, and solving problems. Think of them as mental shortcuts or frameworks that help you navigate complexity.

My own mental models​

I write these up as short posts on the Blog, one idea at a time:

See the full series under Mental Models on the Blog.

Must-Know Mental Models for Everyday Life​

1. How to Make Smart Decisions Without Getting Lucky​

2. The Feynman Learning Technique​

  • Key Idea: Learning is not just about memorization. To truly understand something, explain it simply, identify gaps in your knowledge, and refine your understanding.
  • Learn More: The Feynman Learning Technique

3. Understanding Your Circle of Competence​

Mental Models for Making Decisions​

4. Mental Models: The Best Way to Make Intelligent Decisions (~100 Models Explained)​

Mental Models for Startup Founders​

5. Mental models for startup founders: a free book​

  • Key Idea: This free book provides a curated collection of mental models specifically tailored for startup founders. Discover how to apply these models to navigate the challenges of entrepreneurship.
  • Learn More: Mental models for startup founders: a free book

More Resources​

  • Nav.al: A treasure trove of insights from Naval Ravikant, a successful entrepreneur and investor. Explore his ideas on wealth, happiness, and decision-making.
  • Farnam Street Blog: This blog delves into mental models, decision-making, and critical thinking skills.

Remember: The best way to use mental models is to practice applying them to real-life situations. Start with a few that resonate with you and gradually expand your toolkit.

RAG approach in trading

· 4 min read

Using retrieval-augmented generation (RAG) in trading means integrating live data, such as news and economic indicators, with AI agents and advanced AI models to make informed trading decisions.

This note covers the components you need, the end-to-end workflow, an example implementation in Python, and the considerations to keep in mind.

Components required​

  1. Live Data Feeds:

    • News APIs: Use APIs like NewsAPI, Alpha Vantage News, or any specific financial news provider to get real-time news updates.
    • Stock Market APIs: Use APIs like Alpha Vantage, IEX Cloud, or Yahoo Finance to get live stock prices and other financial metrics.
  2. Trading Platform APIs:

    • Brokerage APIs: Integrate with brokerage APIs like Alpaca, Interactive Brokers, or Robinhood to execute buy/sell orders programmatically.
  3. RAG Model:

    • Retrieval Component: Use Elasticsearch or another retrieval system to index and retrieve relevant news articles and financial data based on queries.
    • Generation Component: Use a transformer-based model like GPT-3 or a custom fine-tuned model to analyze the retrieved information and generate insights or trading signals.
  4. Trading Algorithms:

    • Develop or use existing trading algorithms that can process the insights generated by the RAG model to make buy/sell decisions.

Workflow​

  1. Data Retrieval:

    • Set up a pipeline to continuously fetch live news articles and stock prices using the respective APIs.
    • Index the news articles and other relevant data using a retrieval system like Elasticsearch.
  2. Query Generation:

    • Create a query generator that formulates relevant queries based on the current stock market context. For example, the query could be "impact of current events on tech stocks."
  3. Retrieval Process:

    • Use the retrieval component to fetch relevant news articles and data based on the generated queries.
  4. Generation of Insights:

    • Use the generation component of the RAG model to analyze the retrieved information and generate trading insights. For example, the model could summarize how recent news might affect certain stocks.
  5. Trading Algorithm Integration:

    • Integrate the insights with your trading algorithms. The trading algorithm should be designed to interpret the insights and make buy/sell decisions.
    • For example, if the RAG model indicates positive sentiment around a stock, the trading algorithm might decide to buy that stock.
  6. Execution of Trades:

    • Use the brokerage API to execute the trades based on the decisions made by the trading algorithm.

Example implementation​

A minimal Python sketch of the loop: fetch and index news, generate insights, then act on them.

import requests
from elasticsearch import Elasticsearch
import openai

# Configure APIs and Elasticsearch
news_api_url = "https://newsapi.org/v2/everything"
stock_api_url = "https://www.alphavantage.co/query"
broker_api_url = "https://broker-api.example.com/trade"
es = Elasticsearch("http://localhost:9200")

# Fetch live news and stock prices
def fetch_news(api_key, query):
response = requests.get(news_api_url, params={"q": query, "apiKey": api_key})
return response.json()["articles"]

def fetch_stock_price(api_key, symbol):
response = requests.get(stock_api_url, params={"function": "TIME_SERIES_INTRADAY", "symbol": symbol, "interval": "1min", "apikey": api_key})
return response.json()["Time Series (1min)"]

# Index news articles in Elasticsearch
def index_news_articles(articles):
for article in articles:
es.index(index="news", document=article)

# Retrieve and generate insights using OpenAI
def generate_insights(query):
results = es.search(index="news", query={"match": {"content": query}})
context = " ".join([hit["_source"]["content"] for hit in results["hits"]["hits"]])
response = openai.Completion.create(engine="text-davinci-003", prompt=f"Analyze the following news and provide trading insights: {context}", max_tokens=150)
return response.choices[0].text.strip()

# Execute trade
def execute_trade(api_key, symbol, action, quantity):
response = requests.post(broker_api_url, json={"symbol": symbol, "action": action, "quantity": quantity}, headers={"Authorization": f"Bearer {api_key}"})
return response.json()

# Main trading loop
def trading_loop():
news_api_key = "YOUR_NEWS_API_KEY"
stock_api_key = "YOUR_STOCK_API_KEY"
broker_api_key = "YOUR_BROKER_API_KEY"
query = "impact of current events on tech stocks"

# Fetch and index news articles
articles = fetch_news(news_api_key, query)
index_news_articles(articles)

# Generate insights
insights = generate_insights(query)
print(f"Generated Insights: {insights}")

# Make trading decision
if "positive sentiment" in insights:
execute_trade(broker_api_key, "AAPL", "buy", 10)
elif "negative sentiment" in insights:
execute_trade(broker_api_key, "AAPL", "sell", 10)

# Run the trading loop
trading_loop()

Considerations​

  • Data Latency: Ensure that the data retrieval and processing pipeline is optimized for minimal latency to make timely trading decisions.
  • Model Accuracy: Fine-tune your RAG model on financial data to improve the accuracy of generated insights.
  • Risk Management: Incorporate risk management strategies in your trading algorithms to mitigate potential losses.
  • Regulatory Compliance: Ensure that your trading activities comply with relevant financial regulations and laws.

Takeaway​

Using RAG in trading can provide a significant edge by leveraging real-time data and advanced AI models to make informed decisions.

Streamlit apps

· One min read

AI Answer Evaluator​

AI Answer Evaluator

The AI Answer Evaluator is a streamlined tool designed to assess the quality of AI-generated answers by comparing them against expected answers. It uses LLM to identify factual discrepancies, missing information, and errors, and provides a detailed evaluation with a score out of 10.s

The app also includes a customizable evaluation prompt template and offers insights into token usage, including cost estimation. Perfect for developers, researchers, and educators, this app helps analyze and refine AI outputs effectively while ensuring clarity and precision in results.

House vs Market ROI App​

House vs Market ROI App

This is an app about comparing the return on investment (ROI) of buying a flat and investing in the market. It discusses buying a flat, living in it for ten years, and then selling it.

RAG intro

· 2 min read

Ever wondered how some AI systems seem to know exactly what you need? They might be using a technique called retrieval-augmented generation (RAG). It's like giving the AI a little research assistant:

  1. When you ask a question, the AI first looks for relevant information in its own knowledge base or in external sources.
  2. It combines what it found with your question to create a detailed prompt.
  3. It uses a powerful language model to generate a response that's both informed and tailored to your needs.

RAG definition​

In its most basic form, a RAG application runs three steps:

  • Retrieval: The user's request is used to query an outside source of information. This might mean querying a vector store, running a keyword search over some text, or querying a SQL database. The goal of the retrieval step is to obtain supporting data that helps the LLM provide a useful response.
  • Augmentation: The supporting data from the retrieval step is combined with the user's request, often using a template with additional formatting and instructions for the LLM, to create a prompt.
  • Generation: The resulting prompt is passed to the LLM, which generates a response to the user's request.
User request ──► Retrieval ──► Augmentation (request + supporting data) ──► Generation ──► Response

Sam Altman's talk for founders: it's all about grit

· 2 min read

alt text

Forget the Fancy Stuff, It's All About Grit​

Yeah, brains, experience, and connections are great. But Altman says the real secret sauce is something else: never giving up.

He's seen it time and time again. The founders who win aren't always the smartest ones in the room. They're the ones who keep going when things get tough. The ones who say, "I'll figure this out" even when it seems impossible.

The best part? You don't have to be born with this grit. Altman believes you can learn it, just like any other skill.

Three Things to Focus On​

Altman breaks it down into three things you need to nail:

  1. Laser Focus: Don't get distracted by shiny objects. Pick your goal and go after it with everything you've got.
  2. Believe in Yourself: This isn't just about positive thinking. You need to really believe your idea can change the world. That confidence is contagious.
  3. Build Real Relationships: Founding a company is a team sport. You need to connect with people, whether it's your team, your customers, or your investors.

Keep It Simple, Stupid​

Altman has another gem: If you can't explain your idea in 25 words or less, you're in trouble. The best founders can pitch their vision clearly and concisely.

Aim High, It's Actually Easier​

This might sound crazy, but Altman says it's easier to start a hard startup than an easy one. Why? Because a bold vision attracts passionate people. No one wants to work on something boring.

Bottom Line​

Being a founder is hard work, no doubt about it. But if you have the grit, the focus, and a clear vision, you've got a fighting chance.

So, go out there and build something amazing!

Graph triples

· 2 min read

Triples are a fundamental concept in knowledge graphs, especially those based on the Resource Description Framework (RDF).

What a triple is​

A triple is a data structure with three components:

  1. Subject: the entity or resource being described.
  2. Predicate: the attribute or relationship of the subject.
  3. Object: the value, or another entity, that the subject is related to.

Together, these three parts form a statement about the data, often likened to a simple sentence with a subject, verb and object. For example:

SubjectPredicateObject
"Alice""knows""Bob"

This triple can be represented as:

<Alice> <knows> <Bob>

Triples in RDF​

In RDF, triples make assertions about resources as subject–predicate–object expressions. They can be stored in an RDF graph, where each subject and object is a node and each predicate is an edge connecting two nodes.

Example triples in RDF format​

<http://example.org/Alice> <http://example.org/knows> <http://example.org/Bob> .
<http://example.org/Alice> <http://example.org/age> "30" .
<http://example.org/Bob> <http://example.org/age> "25" .

In this example:

  • The first triple states that Alice knows Bob.
  • The second triple states that Alice's age is 30.
  • The third triple states that Bob's age is 25.

Usage in knowledge graphs​

Triples are the building blocks of RDF-based knowledge graphs. They represent complex information in a structured form that machines can easily parse and query. This enables powerful querying and the ability to infer new information from existing triples.

Knowledge graphs often use query languages such as SPARQL to query these triples and extract meaningful information from the interconnected data.

Summary​

Triples are essential for representing data in RDF and knowledge graphs. They allow a structured, flexible and semantically rich representation of information.