Last updated on 7/7/2026.

Contents:

1. AI Product Management (7)

2. Generative AI: Basic Terms (13)

3. AI Models (11)

4. Prompting & Context Management (20)

5. Retrieval-Augmented Generation (17)

6. AI Agents (20)

7. Agent Loops & Orchestration (19)

8. Data & Feature Management (6)

9. AI Evals (AI Evaluation Systems) (20)

10. Deployment & Operations (20)

11. Model Performance & Cost (5)

12. Safety, Ethics & Alignment (8)

13. Neural Networks & Transformers (9)

1. AI Product Management

Agentic Engineering (aka. Vibe Engineering)

Building software with AI agents using real engineering discipline: clear specs, version control, tests, evals, artifact review, security checks, and production standards. It keeps the speed of AI-assisted building while adding the constraints needed to make the result scalable, secure, and maintainable.

The term was popularized by Andrej Karpathy in 2026. The same shift from vibe coding to engineering discipline was explored in Sep 2025 in The Product Compass newsletter:

AI Literacy & AI Intuition

The ability to understand foundational AI concepts (e.g., LLMs, AI Agents, MCP), perform error analysis, and iteratively refine prompts or models based on hands-on experimentation. I recommend developing it by working with AI, not studying theory.

AI Product Manager

A PM who works on AI-powered products and features. Unlike traditional PMs, AI PMs must understand the technology and develop an AI intuition.

AI-Powered Product Manager (aka AI-Native PM)

A PM who uses AI as part of their day-to-day work: research, synthesis, discovery, prototyping, writing, analysis, and delivery. The distinction is about how the PM works, not necessarily what product they manage.

Paweł’s note: soon, this may stop being a separate category. AI will likely become part of the default PM toolkit.

Product Manager

The team member especially responsible for the two risk areas:

• Customer value: Do customers want this product or feature? Can we actually deliver on the value promised?

• Business viability: Will it work for different parts of the business?

The most important activity for a PM is participating in Product Discovery. Its goal is to address key risks before implementation, to avoid wasting time and resources.

Spec-Driven Development

Writing a clear specification of what should be built and using it as the source of truth that an agent (and its loops and tests) work toward.

Opinion: It never worked for humans. Waterfall lost to agile because you discover what you actually need while building, not in a spec written up front. Thariq's field guide is the clearest proof it's the same for agents: the unknowns surface during the work and after it.

The fix isn't hunting down every unknown unknown either. It's communicating the context beyond the goal, the why and the strategy and the constraints, so the agent navigates the unknowns you never listed.

I've been calling this intent engineering since January 2026.

Vibe Coding

Building software by describing what you want to an AI and accepting what it produces, with little attention to how the code works. It is fast and great for prototypes, but it skips the engineering needed to make something you can scale, secure, and maintain.

Also see ‘Agentic Engineering.’

2. Generative AI: Basic Terms

Deep Learning

A subset of machine learning that uses neural networks with many layers (deep neural networks). It can be applied to supervised, unsupervised, and reinforcement learning.

Direct Preference Optimization (DPO)

A preference-tuning method that aligns a model directly on preferred vs rejected outputs without a separate reward model.

Fine-Tuning

The process of adapting a pre-trained model with additional training so it behaves better for a specific task, domain, style, or preference. Common approaches include supervised fine-tuning (SFT) on examples of desired outputs, and preference/post-training methods such as DPO, ORPO, and KTO.

Generative AI (Gen AI)

AI systems, typically based on large pre-trained models, that generate new content (such as text, images, audio, or code) by learning patterns from large datasets.

Kahneman-Tversky Optimization (KTO)

A model-alignment method that learns from desirable vs undesirable outputs rather than pairwise preference comparisons.

Odds Ratio Preference Optimization (ORPO)

A reference-model-free preference optimization method that uses odds ratios to favor preferred responses during fine-tuning.

Reinforcement Learning

A training method where a model learns by interacting with an environment. It receives rewards for desired actions and penalties for undesired ones (e.g., AlphaGo).

Reinforcement Learning from Human Feedback (RLHF)

A technique to align LLMs with human preferences. It involves training a reward model based on human ratings of model outputs and then using that reward model to guide reinforcement learning that updates the LLM.

Supervised Fine-Tuning (SFT)

Fine-tuning a model on examples of desired inputs and outputs so it imitates the target behavior, format, or domain.

Supervised Learning

A training method where a model learns from a labeled dataset, meaning each piece of input data is paired with a correct output. This is the basis for fine-tuning.

Transformers

A neural network architecture designed for sequential data, like text. Its key innovation is the self-attention mechanism, which weighs the importance of different parts of the input.

Unsupervised Learning

A training method where a model learns from unlabeled data, discovering patterns and structures on its own.

Subscribe now

3. AI Models

Agentic AI (Agentic AI)

The full product system around a model: prompts, context, retrieval, tools, memory, evals, UI, permissions, and operations.

Language Action Models (LAM, advanced)

Models designed not just to understand language but to take action based on it, such as interacting with software or APIs.

Large Language Models (LLMs)

Models trained on large datasets to generate and understand human-like text. They combine supervised, unsupervised, and deep learning, making them excellent for natural language understanding and generation (e.g., ChatGPT).

Latent Concept Models (LCM, advanced)

Powerful models that excel at capturing nuanced or hidden concepts within data.

Masked Language Models (MLM, advanced)

Models trained to predict masked or missing words in a sentence. This approach helps them master context and understand language structure deeply.

Mixture of Experts (MoE)

An architecture that combines outputs from multiple specialized "expert" models. It intelligently routes queries to the most relevant expert, improving performance and efficiency.

Reasoning Model

A model trained to work through a problem in explicit steps before giving an answer, rather than responding in one shot. It tends to do better on math, code, and multi-step logic, and usually costs more and runs slower in exchange.

Segment Anything Models (SAM, advanced)

Models designed for precise image segmentation, capable of identifying and outlining objects in an image with high detail and accuracy.

Small Language Models (SLMs)

More compact models that are ideal for efficiency and speed, especially in resource-constrained environments or for specialized tasks.

Test-Time Compute (Inference-Time Scaling)

Spending more computation while answering a question (more reasoning steps, more attempts, more checking) to get a better result, instead of relying only on a bigger trained model. It is a way to buy quality at answer time rather than at training time.

Vision-Language Models (VLMs)

Models that handle both text and images, bridging the gap between visuals and language to understand and generate content based on both modalities.

4. Prompting & Context Management

Autonomy Boundaries

Explicit limits on what an agent may decide or do without human approval.

Chain-of-Thought (CoT) Prompting

A prompting technique that encourages a model to solve a problem step by step instead of jumping straight to an answer. In products, you usually should not ask the model to reveal hidden/private reasoning. Ask for a plan, concise rationale, intermediate checks, or a brief explanation of the answer when that helps the user.

Constraints

Rules that narrow acceptable behavior, outputs, tools, data, or actions for an AI system.

Context Engineering

The practice of deciding what information goes into the model’s context window for a given task, and in what form. As tasks move from single prompts to agents, choosing and structuring the right context (instructions, retrieved data, memory, tool results) matters more than wording any single prompt.

Context Management

Since the model has a memory limit, you need to be smart about what info you include. Context management means deciding what matters most, trimming what doesn’t, and structuring it in a way the model can understand and use. Specific techniques include prompt engineering (yes, it’s not dead, despite what some say) and RAG. Some make the mistake of including fine-tuning, but it’s not a context management technique - it modifies the model itself.

Context Window (Token Limit)

The amount of information a model can consider in a single request, measured in tokens. It may include the user message, system instructions, prior conversation, retrieved documents, tool results, and other context. If the request exceeds the limit, the system may truncate, summarize, reject, or lose earlier context depending on the implementation.

Desired Outcomes (Evidence)

Concrete measurable success criteria that let an agent, evaluator, or PM know whether the work succeeded.

Extended Thinking (Thinking Budget)

A mode where the model is given room to reason step by step before answering, with a budget that caps how much of that reasoning it can do. More thinking budget can improve hard-problem accuracy at the cost of higher latency and spend.

Few-Shot Learning

Instead of zero examples, you give the model a few. Just enough to show what kind of response you want. Great for formatting tasks, tone matching, or guiding the model when clarity matters without fine-tuning.

Health Metrics

Metrics that show whether an AI system or agent is behaving reliably over time. Used to prevent system degradation while pursuing the current objective.

Intent Engineering

Describing what you want and why, with success criteria and boundaries, rather than dictating each step. It shifts your job from writing instructions to specifying the outcome and letting the agent work out the how.

Objectives

The desired end state an agent should pursue, distinct from the step-by-step instructions. It should explain why the objective matters. Also see, “Desired Outcomes.”

Prompt Engineering

Crafting instructions that steer a model’s response in a single interaction. It covers wording, examples, formatting, constraints, and output structure. Prompt engineering is still useful, but for agents it is only one layer: Context Engineering decides what information the agent gets, and Intent Engineering defines the goal, outcomes, and boundaries it should work within.

Prompt Caching

Reusing the model’s processing of a repeated chunk of context (such as a long system prompt or document) across calls instead of paying to process it again. It lowers cost and latency when many requests share the same large, stable prefix.

Prompt Injection

An attack where malicious instructions are hidden in prompts to trick an LLM into bypassing its safety protocols, disclosing private data, or deviating from its intended behavior. For example, telling the model to ignore earlier instructions. A real security concern in AI systems.

ReAct

A prompting/agent pattern where the model alternates between reasoning about what to do and taking actions with tools.

Strategic Context

Business, user, market, or product context that helps the model choose better tradeoffs.

System Context

Product, technical, policy, data, and operational context the AI needs to act correctly inside a system.

Temperature

A setting that controls how creative or predictable the model’s response is. Low values make it focused and safe. High values let it get more varied or helpful for brainstorming, less so for factual tasks.

Zero-Shot Learning

Instructing an LLM to perform a task without providing any examples in the prompt, relying entirely on its pre-trained knowledge.

5. Retrieval-Augmented Generation

Agentic RAG

A version of RAG where the AI decides on its own when to retrieve information, what to retrieve, and how to use it.

Adaptive RAG

Enhanced RAG that makes retrieval and generation process more dynamic and self-correcting, adapting to the complexity of a query and the quality of retrieved information.

CAG (Cache-Augmented Generation)

An alternative to RAG. Instead of retrieving data during each interaction, CAG loads relevant information into the model’s context beforehand. This speeds things up and works well when the data is stable and fits in the model’s memory. It is also called Context-Augmented Generation.

Chunking

Splitting documents into retrievable pieces so the system can fetch the most relevant context.

Document Search

Searching over documents or files to retrieve useful context for an AI answer or agent action.

Embedding

A way to convert data, like text or images, into numerical vectors that capture meaning. These vectors help the model understand relationships between different pieces of data and allow for similarity searches.

Graph Database

A database optimized for relationships between entities, useful when connections matter more than isolated documents.

Hybrid RAG

A retrieval approach that combines multiple retrieval methods to improve the quality of context provided to the model. Typically, it merges dense retrieval (using vector similarity) with sparse retrieval (like keyword-based search or BM25). This way, it captures both semantic meaning and exact keyword matches, making the retrieval more robust.

HyDE (Hypothetical Document Embeddings, advanced)

A method that boosts retrieval by having the model generate a fictional answer to the query. That generated answer is turned into an embedding, which is then used to find real documents that are similar. This helps connect short queries to more complete responses.

Knowledge Graph

A structured map of entities and relationships that helps AI systems reason over connected information.

Knowledge System

The full setup that determines what an agent can know: files, memory, RAG, vector stores, graphs, tool results, and instructions.

Markdown

A lightweight text format often used as durable, readable context for agents.

OKF (Open Knowledge Format)

An open specification introduced by Google Cloud in 2026 for representing knowledge as a directory of Markdown files with YAML frontmatter. OKF formalizes the LLM-wiki pattern: curated context, metadata, and links that humans can edit and agents can read without a proprietary tool or API. It is useful for portable agent knowledge bases, especially when knowledge should live near docs, code, or a team wiki.

Previously, we discussed similar structures in What I Learned Building a Self-Improving Agentic System with Claude.

Reranking

A technique used in RAG that improves retrieval results by reordering the initial set of documents or chunks. A model evaluates which items are most relevant and puts the best ones at the top.

Retrieval-Augmented Generation (RAG)

A method where an AI system retrieves relevant information from external sources such as documents, databases, vector stores, or file systems, then uses that information as context to produce a better answer. Retrieval can happen before generation or dynamically during an agent run when the agent decides it needs more information.

Second Brain

A personal or team knowledge base that agents can read from and update across work. The term was popularized by Andrej Karpathy in 2026. Previously, we covered it in What I Learned Building a Self-Improving Agentic System with Claude.

Vector Databases (Vector Stores)

Specialized databases that store and search embeddings efficiently. They are built to handle high-dimensional vectors and find similar items quickly, even at large scale.

You might also benefit from out Interactive RAG simulator:

Popular tools:

6. AI Agents

Agent Instruction File (CLAUDE.md / AGENTS.md)

A repo or workspace file that gives an agent standing instructions, project context, and operating rules.

Agent Payments Protocol / AP2 / A2P

A protocol layer for authorizing and securing payments initiated by agents.

Agent SDK

A developer toolkit by Anthropic for building agents with tools, instructions, memory, handoffs, and eval hooks.

AI Agent

A system where a model works inside a harness that gives it tools, context, permissions, memory, guardrails, and feedback. The agent can decide what steps to take, which tools to use, and how to adapt as it works toward a goal. The model matters, but the surrounding system is what makes the agent useful and reliable.

AI Chain (advanced)

A setup broader than prompt chaining, where different components (including prompts, code, external tools, or even other models) are connected in a sequence. Each component passes its output to the next, forming a pipeline. AI Chains are often used to build full workflows, not just chains of prompts. For example, we extract info using an LLM, validate it with code, enrich it with an API call, then ask the LLM to write a final summary. Think n8n.

Agent-to-Agent (A2A)

A protocol that allows AI agents to talk directly to each other. It supports things like sharing results, passing off tasks, or negotiating responsibilities between agents.

Agent-to-User Interface (A2UI)

A protocol for agents to return structured, interactive UI instead of only text.

Agent Memory

The information an agent carries beyond a single conversation, such as past results, user preferences, or learned rules, so it does not start from scratch each run. Storing every interaction is not the same as learning. Memory only compounds when experience gets curated into rules the next run follows.

Agentic Workflows

Automated systems where an LLM and supporting tools are connected through predefined logic. The steps are set in code, but the LLM adds intelligence by handling inputs, making decisions, or generating content along the way.

Computer Use

A capability where an agent operates a computer the way a person would, by reading the screen and controlling the mouse and keyboard. It lets an agent work with software that has no API, at the cost of being slower and more error-prone than direct tool calls.

Handoff

Passing work from one agent, system, or human to another with enough context to continue safely.

LLM Workflow

A structured process where LLM calls, tools, rules, and deterministic steps are arranged to complete a task.

Managed Agent

An agent hosted and operated by a platform, where infrastructure, scaling, or some orchestration is abstracted away. For example Amazon Bedrock AgentCore, Claude Managed Agents.

Model Context Protocol (MCP)

An open standard introduced by Anthropic for connecting AI models and agents to external tools, APIs, and data sources in a consistent way. MCP is mainly about agent-to-tool and agent-to-data connections. Agent-to-Agent (A2A) is the separate pattern/protocol family for agents communicating with other agents.

Multi-Agent System

An architecture where several AI agents work together to solve tasks. Each agent may specialize in something different. They communicate, share information, or divide responsibilities to handle complex problems.

Product Agent

An agent embedded inside a product or business process that runs for users or teams with designed tools, guardrails, and handoffs. For example n8n, LangChain, LangGraph, Agent SDK.

Prompt Chaining

A method where you split a task into smaller steps and run them through the LLM one at a time. Each step is handled with its own prompt, and the output from one step is passed into the next prompt. This helps guide the model through complex tasks or multi-stage reasoning. For example, we ask the model to summarize a text, then ask it to generate a title based on that summary.

Subagent

A separate agent that a main agent spawns to handle a focused piece of work, then reports back. Splitting work across subagents keeps each one's context clean and lets parts of a task run in parallel.

Tool Use (Function Calling)

When a model is given a set of tools or functions it can call (search, code execution, an API, a database) and decides when to call them and with what inputs. This is what lets an agent act on the world instead of only producing text.

Workspace Agent

An agent that runs on your own work: research, discovery, writing, prototyping, coding, analysis, and delivery tasks. For example, Claude Code, Codex App, Cowork.

7. Agent Loops & Orchestration

Agentic Back-Pressure

Signals that push back on an agent and constrain what it can do next, such as a failing test, a type error, or a rejected build. Back-pressure keeps a loop from drifting by forcing each pass to satisfy a real constraint before it continues.

Agent Harness (aka. “Orchestration”)

The supporting system around the model that makes an agent useful: retrieval, tools, memory, verification, and guardrails. The harness, not the raw model, is where most of the reliability and leverage of an agent actually comes from.

The term was adopted by the AI community in February 2026, we’ve been discussing it in The Product Compass newsletter since July 2025:

Agent Loop

An agent that repeats one cycle (act, check, adjust) and keeps going until a stop condition is met. The check inside the cycle is what makes it a loop and not a fixed routine, e.g. cluster a feedback backlog into themes, re-cluster, and stop when the themes stop changing.

Compounding Slop

The failure mode where a loop runs unattended with a weak stop condition and produces more and more low-quality output that nobody reviews. A loop running unattended is also a loop failing unattended, so the same technique either compounds value or compounds garbage depending on the check that ends it.

Dynamic Workflow

A process that adapts within a single run, branching on what it finds, then finishing. One pass, no repetition, e.g. read each incoming ticket, decide if it is a bug or a feature request, and route each to the right queue.

Hook

A trigger or extension point that runs code, checks, or automation before or after an agent action.

Loop Engineering

Designing the system that runs the agent instead of writing each prompt by hand. You define the goal, the iteration, the independent check for “done,” and the cadence, then let the loop do the task. The phrase was popularized in mid-2026, but the underlying control loop is old.

Maker-Checker (Sub-Agent Split)

A loop design where one agent does the work and a separate, independent agent (or test, or model) decides whether it is correct. Keeping the maker and the checker apart is what keeps a loop honest, since an agent grading its own output tends to approve it.

No-Code Agent Builder

A visual tool for building agent workflows and automations without writing most of the code, e.g., n8n.

Open Loop vs Closed Loop

A closed loop feeds its result back in and adjusts on the next pass, so it self-corrects toward a goal. An open loop runs once and stops with no feedback. Most agent loops worth building are closed loops, because the feedback is where the value comes from.

Orchestrator (Fleet)

The controlling agent or system that starts other agents, hands them tasks, watches their results, and decides what runs next. Running many agents under one orchestrator is sometimes called managing a fleet, and the orchestrator is where you observe failures across runs and stop the bad ones.

Orchestration-First Design

Designing the steps, tools, constraints, checks, and handoffs around a model instead of relying on open-ended autonomy.

Ralph Wiggum Loop

The simplest possible loop: feed the same prompt to a coding agent over and over with a fresh context each time, relying on tests and the build to push the work toward correct. Named by Geoffrey Huntley (early 2025), it is the ancestor of today’s loop techniques and runs one task at a time.

Routine

A fixed sequence of steps an agent runs the same way every time, with no judgment and no stop condition. It just ends when the steps are done, e.g. pull the open tickets, sort by date, post the list to Slack.

Self-Prompting

When the agent writes its own next prompt instead of reusing one you wrote. This is the real shift behind calling loops the successor to prompt engineering: you design the prompt generator and the goal, not each individual prompt.

Scheduled Task (Cron Trigger)

A clock that starts a routine, workflow, or loop on a cadence such as hourly or daily. The schedule is a separate question from whether the thing it starts is actually a loop, e.g. run the competitor scan every morning at 8.

Stop Condition

The rule that decides when a loop is finished and should exit. It is the hardest and most valuable part of a loop, because a loop with a loose or missing stop condition runs forever and piles up low-quality output instead of stopping at a good result.

Skill (SKILL.md)

A reusable, named capability you write down in a file so an agent can load it on demand and apply it the same way every time. Skills that encode your specific decisions and procedures compound in value, while skills that just restate what the model already knows add little.

Tool Result

The output returned by a tool call, which becomes context for the agent’s next decision.

8. Data & Feature Management

Concept Drift

A specific kind of Data Drift where the underlying relationship between inputs and the target variable changes over time. For example, if customer behavior evolves, a model trained on past preferences might no longer make accurate predictions.

Data Annotation / Labeling

The process of tagging or categorizing raw data to create a labeled dataset for supervised learning. This step is essential for training models to recognize patterns and usually requires a lot of manual effort. Another context in which we use these terms is error analysis (part of AI evals).

Data Augmentation

A method used to expand and diversify a training dataset by making slight changes to existing data. Examples include rotating images, adding noise, or rephrasing sentences to help the model generalize better.

Data Drift

When the data a model sees in production starts to differ from the data it was trained on. This shift can reduce accuracy because the model is no longer operating in the same conditions it learned from.

Feature Store (advanced)

A feature store is like a shared database where machine learning teams keep useful data pieces (called features) that help models make predictions. These features are cleaned and ready to use, so teams don’t have to rebuild them every time. It also makes sure the model uses the same information when it’s being trained and when it’s used in the real world.

Model Drift

This is a broader, more casual term than Data Drift and Concept Drift. It usually means that a model's performance is getting worse over time but doesn’t specify why. It might be caused by data drift, concept drift, or something else like bugs.

9. AI Evals (AI Evaluation Systems)

Action-Based Evals

Evals that judge whether the agent took the right action, not just whether the final text sounded good.

AI Evaluation System

A system for measuring whether an AI product works in practice. It can include code-based evals, LLM-as-judge checks, human review, action-based evals, observability, traces, production monitoring, and error analysis. Good eval systems measure not only final outputs, but also whether the agent took the right actions and behaved reliably over time.

AI Observability

Monitoring the behavior of AI systems through traces, tool calls, costs, latency, errors, eval scores, and user outcomes.

Axial Coding

Grouping labeled errors into higher-level failure modes during qualitative error analysis.

Bottom-Up vs. Top-Down Analysis

Two approaches to defining AI metrics. Bottom-up analysis identifies application-specific failure modes directly from the data. Top-down analysis applies generic metrics (like hallucination or toxicity) that may miss domain-specific nuances.

Code-Based Evals

Automated evaluators implemented as deterministic scripts (e.g., using Python with XML, SQL, or regex checks) that objectively flag specific, rule-based failure modes.

Error Analysis

The process of reviewing model outputs (LLM traces), labeling errors, and classifying them into distinct failure modes. This is repeated iteratively until no new failure modes emerge and is a high-ROI activity in AI product development.

F1-Score

The harmonic mean of precision and recall, providing a single score that balances both metrics.

Failure Modes

Coherent and non-overlapping categories of errors that emerge from analyzing LLM traces (e.g., hallucination, incorrect format, or missed instruction). Each binary failure type is easy to recognize and forms the basis for targeted metrics.

Human Eval

Human review of outputs, traces, or actions to judge quality, safety, usefulness, or correctness.

Model Eval

Evaluation of a model’s behavior on a defined task set, separate from evaluating the whole product system.

LLM Traces

A record of an AI system’s run, usually including the user query, prompts or messages, retrieved context, tool calls, tool results, metadata, costs, latency, and final output. Traces are the raw material for debugging, observability, evals, and error analysis. They do not always include hidden model reasoning.

LLM-as-Judge Evals

Evaluation modules where a secondary LLM acts as the "judge," assessing outputs against a single failure mode in a binary pass/fail format. This is useful for complex or subjective checks but requires monitoring for alignment with human judgment.

Open Coding

The early error-analysis pass where reviewers label traces freely before consolidating categories.

Precision

A metric that measures the proportion of true positives out of all positive predictions (True Positives / (True Positives + False Positives)).

Recall (True Positive Rate, TPR)

A metric that measures how often a system correctly identifies the cases you defined as positive: True Positives / (True Positives + False Negatives). Define the positive class first. If “positive” means a passing output, recall asks how often the judge says Pass when a human said Pass. If “positive” means a failure mode, recall asks how often the eval catches that failure when it is truly present.

Synthetic Traces

Artificially generated LLM traces created by sampling combinations of hypothesized failure dimensions. These are converted into natural-language queries and run through the model pipeline when production data is scarce.

Theoretical Saturation

The point in an error analysis cycle when reviewing additional LLM traces yields no new failure modes, indicating that the failure taxonomy is complete.

True Negative Rate (TNR)

A metric that measures how often a model correctly identifies a negative case. For judge alignment, this answers: "How often does the judge say 'Fail' when a human said 'Fail'?"

Unit Tests

Deterministic checks for specific expected behavior in code, prompts, tools, or AI workflows.

10. Deployment & Operations

AEO (Answer Engine Optimization)

Optimizing content so AI answer engines can understand, cite, and present it as a direct answer.

Branching

Separating development work into isolated code branches before merging and shipping.

CI/CD (Continuous Integration/Continuous Deployment)

Automated testing, building, and deployment pipelines that help teams ship safely.

Dev/Test/Prod Environments

Separate environments for building, validating, and running production software.

Edge Protection / WAF

Web application firewall protection that blocks malicious traffic before it reaches the app. E.g., Cloudflare.

GEO (Generative Engine Optimization)

Optimizing content and entities to appear accurately in AI-generated answers and summaries.

Hosting

Running an app or agent on infrastructure users can access reliably.

JWT

A signed token format commonly used for authentication and authorization, known, e.g., from Clerk.

Latency

The time it takes for an AI system to respond after getting a request. Lower latency means faster responses, which is especially important in real-time applications like chatbots or fraud detection.

MLOps / ModelOps

A set of tools and practices that help teams manage AI models throughout their lifecycle. This includes training, testing, deploying to production, monitoring performance, and updating models as needed. The goal is to make working with AI models as smooth and reliable as software engineering.

Model-as-a-Service (MaaS)

A way to use pre-trained AI models through a cloud API, without needing to build or host the models yourself. Examples include the OpenAI API or Gemini API. This lets teams quickly add AI features to their apps with less effort and cost.

Penetration Testing

Security testing that actively probes an app for exploitable weaknesses.

Performance Testing

Testing how an app behaves under load, latency, or resource constraints.

RLS (Row-Level Security)

Database rules that restrict which rows a user or service can access.

Secrets

Private credentials such as API keys, tokens, and passwords that must be protected.

SEO

Optimizing pages so search engines can discover, understand, and rank them.

SQL Migration

A controlled database schema change, such as adding tables, columns, indexes, or constraints.

Usage Limits

Caps on requests, tokens, spend, actions, or tool calls to control cost and risk.

Throughput

The number of requests or tasks an AI system can process in a given time, such as per second or per minute. High throughput is important for scaling and handling many users at once.

XSS (Cross-Site Scripting)

A web security flaw where attackers inject scripts into pages viewed by users.

Popular tools:

11. Model Performance & Cost

Inference

The process of using a trained model to make predictions on new data. In the case of large language models, this means generating a response based on the input prompt. It’s what happens when you “ask” the model something.

LLM Limits

Practical constraints of language models, including context size, latency, cost, reliability, hallucination risk, and tool-use errors.

Time to First Token (TTFT)

The time it takes from sending a request to when the first token of the response appears. Even if the full answer takes longer, a fast TTFT makes the system feel more responsive to the user.

Tokens Per Second

A measure of how fast a language model generates output. It tracks how many tokens (chunks of text) the model produces each second. Faster speeds mean quicker responses and better user experience.

Quantization

A method to shrink the size of a model and make it run faster. It works by converting the model’s internal numbers from high precision (like 32-bit floats) to smaller ones (like 8-bit integers). This helps reduce memory usage and speed up inference with only a small drop in accuracy.

12. Safety, Ethics & Alignment

AI Slop

Low-quality, generic AI-generated content produced at high volume with little human judgment. It usually looks plausible on the surface but adds no real value, and it is the typical output of an unattended loop with a weak stop condition.

Bias & Fairness

AI models can pick up biased patterns from the data they were trained on. This can lead to unfair or harmful outcomes, especially when decisions affect people. Fairness means detecting, reducing, and preventing those biases from impacting results.

Eval-Driven Development

Building an AI feature by first defining how you will measure whether it works, then improving the system against that evaluation set. The evals become the spec: you fix the underlying cause of each failure and re-run the full set until outputs clear the bar.

Paweł: This approach directly contradicts the spirit of AI Evals

Explainability / Interpretability

The ability to understand why a model made a certain prediction. This helps people trust the system, find errors, and meet legal or ethical standards. Techniques might show which inputs mattered most or how decisions were made step by step.

Guardrails

Controls that keep an AI system inside allowed behavior. Guardrails can include prompt rules, permissions, tool allowlists, validators, policy checks, human approvals, runtime blockers, rate limits, and output filters. For agents, guardrails should be architectural, not just polite instructions in the prompt.

Hallucination

When a language model makes something up and presents it as if it were true. The response might sound convincing, but the facts are incorrect or completely invented. This is a major issue for tasks that require accuracy or trust, especially in AI Agents and Multi-Agent Systems.

Human-in-the-Loop (HITL)

A setup where humans stay involved in the AI’s workflow. People might review outputs, make the final decision, or correct the model when it’s unsure. This helps improve safety, quality, and accountability, especially in sensitive or high-risk situations.

Red Teaming

Actively trying to break, misuse, or stress-test an AI system before users or attackers do.

Subscribe now

13. Neural Networks & Transformers

Activation Function (advanced)

A mathematical rule inside a neuron that decides how it reacts to input. It helps the network capture complex patterns by adding non-linear behavior to the model.

Backpropagation (advanced)

The learning process used in neural networks. It calculates how much each weight contributed to the error and then updates those weights to reduce the loss in future predictions.

Feedforward Network (advanced)

A part of the transformer that processes the results from the self-attention step. It’s a small neural network that helps the model refine its understanding before passing data to the next layer.

Loss Function

A way to measure how far off the model’s predictions are from the correct answers. It gives the model feedback during training so it knows what to improve.

Masked Attention (advanced)

Used in transformer decoders to make sure the model doesn’t look ahead when generating text. This is important during training so the model learns to predict the next word without cheating.

Neural Networks

A type of machine learning model inspired by how the human brain works. It’s made up of layers of connected units called neurons. Each neuron processes input data and passes signals forward to help the model learn patterns.

Positional Encoding (advanced)

Since transformers don’t process data in order by default, this adds information about the position of each token in the sequence. It helps the model understand the order of words.

Self-Attention (advanced)

A key mechanism in transformers that lets each token in a sequence look at the others. This helps the model understand relationships between words, even if they’re far apart.

Tokenization

The step where input text is broken down into smaller parts, called tokens. These can be words, characters, or subwords. The model processes these tokens instead of raw text.

Weights of a Model

The internal settings of a neural network that get adjusted during training. They control how strongly each input affects the model’s prediction and are what the model "learns" over time.

Keep Learning

Thanks for reading The Product Compass!

You might also explore our practical step-by-step guides available in the AI Product Management section and office hours recordings.

For example, users who signed up in a specific month or completed a particular onboarding step. By tracking each cohort, you can identify where users drop off and take action to improve retention, reduce churn, and drive growth.

1. What Is Cohort Analysis?

Cohort analysis groups users by a shared characteristic, like their sign-up date or a key action. You then track how each cohort’s behavior changes over time.

For example, if you want to see why new users stop using your app after two weeks, you can analyze the “September 2025 Sign-Ups” cohort and see how their activity evolves day by day or week by week.

Why It Matters: It answers not just “What’s happening?” but “Who’s affected?” and “When did it start?” That’s far more actionable than a broad average that lumps everyone together.

2. Two Types of Cohorts a PM Should Know

There are two types of cohorts you can use in customer cohort analysis:

1. Acquisition Cohorts

   • Group users by the date they joined (e.g., all September sign-ups).

   • Helps you see if changes to onboarding or marketing in that period influenced retention or churn.

2. Behavioral Cohorts

   • Group users by in-app actions (e.g., completed onboarding, used a key feature).

   • Reveals which behaviors correlate with higher retention. For example, users who completed a setup checklist may churn less than those who skip it.

3. Netflix Case Study: The “Hooked Episode”

Background

In 2015, Netflix publicly shared that they analyzed when new viewers got “hooked” on a series, meaning the exact episode after which they would finish the entire season.

They discovered, for instance, that viewers who reached Episode 2 of Breaking Bad or Episode 3 of House of Cards were far more likely to keep watching. And stay subscribed.

How It Relates to Cohort Analysis

Let’s break it down:

• Cohort Definition: One way to view this is to treat “people who started a new show” in a given month (or week) as a cohort.

• Behavior Tracked: Netflix monitored how many episodes each user watched, grouping them by the specific show and episode at which they continued or dropped off.

• Outcome: By comparing cohorts who made it to the “hooked episode” vs. those who didn’t, Netflix found that hitting that milestone correlated strongly with higher retention.

• Action Taken: This insight influenced Netflix’s strategy:

  • Auto-Play: They refined the auto-play feature so users quickly move from one episode to the next, making it easier to reach that “hooked” threshold.

  • Personalized Recommendations: They promoted the next episode or similar shows more prominently to encourage continuous viewing.

  • Content Investment: They doubled down on producing or acquiring shows with strong “hooking” potential.

Results

• Thanks to retention cohort analysis, Netflix ensured that users who reached the hooking episode had a significantly lower churn rate.

• Netflix continued using these data-driven insights to refine how they present shows to new viewers, ultimately boosting overall retention and customer lifetime value (CLV).

Share

4. How to Run a Cohort Analysis in Product Management

Cohort Analysis is relatively straightforward. Here are the basic steps:

1. Define Your Question: Are you investigating churn, feature adoption, or onboarding success?

2. Pick a Cohort Type: Time-based (acquisition) or action-based (behavioral).

3. Track Over Time: Use analytics tools (e.g., Mixpanel, Amplitude) or set up custom dashboards. Monitor how many users remain active or complete the next step each day, week, or month.

4. Identify Patterns: Look for significant drop-offs or sudden changes. Investigate if a new feature or marketing campaign correlates with those changes.

5. Take Action: Form a hypothesis, run experiments, and see if your cohorts improve. Repeat until you see sustained gains.

5. Cohort Analysis for The Product Compass Newsletter (real data)

Here, you can see a real example of a cohort analysis for this newsletter:

• Rows: Your subscription start date

• Columns: How many readers turned into paid subscribers after x months (%)

I just realized we’ve never had such a massive conversion.

Subscribe now

How to perform a similar cohort analysis with Excel and ChatGPT

Let’s take Substack as an example. To perform a similar cohort analysis, you can:

1. Export subscribers’ data to Excel, but only include what we need: Subscription type, Subscription date, and First payment at.

2. Remove the Email and Name columns if they are present. We don’t want to share this data.

3. Upload it to ChatGPT-4o. Prompt:

Think step-by-step to generate a cumulative cohort analysis table based on the attached Excel file.

Rows:

• Represent the month when the subscription started (Subscription date), formatted as yyyy-mm.

• Display only cohorts from 2024-01 to 2024-12, starting with 2024-01.

Columns:

• Represent the number of months after the subscription started (First payment at - Subscription date in full months).

• Include only months that have elapsed for each cohort.

• Example: If today is 2024-06, then:

  • The 2024-01 cohort can have data for months 0 to 5,

  • The 2024-03 cohort can have data for months 0 to 3,

  • The 2024-06 cohort can only have data for month 0.

• Any future months per cohort should be left empty dynamically.

• Display only columns from 0 to 12.

Cells:

• Contain the cumulative percentage of subscriptions that upgraded to a paid plan (Monthly Subscriber or Yearly Subscriber in Subscription type) in this month or earlier. This means that values in each row can only grow when moving to the right.

• The percentage should be calculated relative to all subscriptions started in the given cohort month.

• Leave cells empty (and not 0.0) if the cell is in the future. Today is 2/16/2025.

• Round values to 2 decimal places.

• Display a number corresponding to each cell, inside the cell, if not empty.

Visualization:

• Apply a smooth gradient color from #FFFFFF (0) to #2e75a9 (max visible value in the table).

• Ensure that each cohort only displays the months that have actually elapsed, forming a diagonal shape in the table.

6. Four Best Practices

When performing cohort analysis:

• Prioritize Leading Indicators: If you see user complaints spike early, that might predict future churn.

• Combine Qualitative Insights: Talk to users in the drop-off cohort. Surveys and interviews can tell you Why they’re leaving.

• Avoid Vanity Metrics: A big user count doesn’t help if they don’t stick around. Cohort analysis emphasizes retention, which is more actionable.

• Iterate Constantly: Cohorts can reveal new problems once you fix the old ones. Keep measuring and improving.

FAQ

Q: What’s the difference between cohort analysis and funnel analysis?

A: Funnel analysis shows how users move through specific steps (e.g., sign-up to purchase). Cohort analysis groups users by a shared trait over time.

In practice, they often work together. Use funnel analysis to find drop-offs in a process, then cohort analysis to see how different user groups behave.

Q: Should I start with acquisition or behavioral cohorts?

A: Acquisition cohorts are easier to start with (e.g., “all users who signed up in October”). Behavioral cohorts provide deeper insight into why users churn.

Many product teams use both.

Q: How do I pick the right time intervals?

A: If your product sees daily engagement (like a messaging app), track cohorts by day or week. For B2B SaaS with monthly plans, monthly intervals often make more sense.

Q: Can cohort analysis help with upsells?

A: That’s one of the core use-cases! By grouping users who upgraded or bought add-ons, you can see if certain behaviors (e.g., frequent feature usage) predict upsells. Then, optimize the journey for that behavior.

Q: What tools do I need?

A: Popular analytics platforms like Mixpanel, Amplitude, or Google Analytics can generate cohort reports. Data Analysts can build custom reports with Microsoft Business Intelligence, Google BigQuery (SQL-like), Tableau, or spreadsheets, though it may be more time-consuming.

Enjoy this?

You might also like:

By mapping out these steps, you can identify where users drop off, understand why it happens, and then experiment to improve conversions.

Whether running a B2B SaaS platform or a B2C streaming service, funnel analysis helps you focus on the right metrics rather than getting lost in vanity numbers.

Subscribe now

1. What Is Funnel Analysis?

A funnel is a set of steps users follow from the moment they land on your product (top of the funnel) to the final action you want them to take (bottom of the funnel).

Typical stages might include:

• Awareness: Users discover your product via ads, organic search, or social media.

• Consideration: They explore features, read reviews, or compare plans.

• Conversion: They sign up, make a purchase, or subscribe.

• Retention: They continue using your product over time.

Your funnel stages may differ depending on your business model, but the principle is the same: track each step to find bottlenecks.

2. Key Metrics to Measure

Some of the key metrics include:

• Bounce Rate: The percentage of users who leave without taking any next step. A high bounce rate often indicates unclear messaging or poor user experience.

• Conversion Rate: The percentage of users who complete a desired action (e.g., sign up, purchase). This is your primary metric for funnel success.

• Drop-off Points: Where users abandon the process (e.g., after adding to the cart but before checkout). Identify these steps to guide improvements.

• Time to Value (TTV): How quickly new users experience the product’s core benefit. Faster TTV often increases the chance of moving users down the funnel.

As highlighted in Are You Tracking the Right Metrics, focus on metrics that are easy to underatand and compare and change your behavior. Data should inform decisions, not just accumulate in reports.

3. Netflix Funnel Analysis Example

Netflix is famous for using funnel analysis to streamline user onboarding and boost retention. One of their funnels might look like this:

1. Landing Page: Users arrive and see the “Try Netflix Free” or subscription offer.

2. Plan Selection: They pick a plan (Basic, Standard, Premium).

3. Sign-up & Payment: They enter their details and billing info.

4. First Watch: They begin streaming a show or movie.

5. Recurring Subscription: They stay on as paying members.

Imagine that Netflix noticed many users dropped off at the Plan Selection stage, often due to confusion about the differences between plans.

By testing clearer plan descriptions and simpler plan choices, Netflix reduced friction and increased conversion rates. They also measured time to first watch as a key metric.

The faster a new subscriber starts streaming (TTV), the more likely they will remain long-term.

4. Five Tips for Effective Funnel Analysis in Product Management

Funnel analysis typically goes like this:

1. Map Your Stages: Define the steps you want users to take. Keep it simple.

2. Track Each Step: Use analytics tools (e.g., Mixpanel, Google Analytics, Amplitude) to record how many users advance or drop off.

3. Dig Deeper: When you see a significant drop-off, investigate. Product analytics help you understand the What. If the reason isn’t obvious, use qualitative methods like user interviews to learn Why.

4. Experiment: Ideate, formulate a hypothesis, and change one variable at a time (e.g., messaging, page layout) to see if conversions improve.

5. Monitor Leading vs. Lagging Indicators: Lagging indicators (like monthly churn) tell you what happened. Leading indicators (like daily user complaints) can predict future churn.

Share

FAQ

Q: What is the difference between a funnel analysis and the AARRR framework?

A: Funnel analysis focuses on user flow for a specific goal (like sign-ups). AARRR covers five lifecycle stages (Acquisition, Activation, Retention, Revenue, Referral) to track a broader customer journey. Both approaches are complementary.

Q: How many funnel stages should I track?

A: It depends on your product. Start with a simple top, middle, and bottom. Expand if you need more granularity, like sign-up vs. payment vs. activation steps.

Q: What if my funnel has multiple user paths?

A: Segment your funnel by user cohort, traffic source, or plan type. This helps you see which user groups convert best and where friction is highest.

Q: How do I handle drop-offs?

A: Identify the stage with the highest drop-off, form a hypothesis, and run an experiment. For example, if you lose users at checkout, simplify the payment process or offer guest checkout.

Q: How can I measure funnel success over time?

A: Compare conversion rates week over week or month over month. Look for consistent improvement or significant changes after product updates.

Q: How do the stages of a marketing funnel differ from those of a sales funnel?

A: Some identify marketing funnel analysis vs. sales funnel analysis. But that's the same approach applied to different steps the user or customer takes.

Marketing funnel typically includes stages like awareness, interest, consideration, and intent, focusing on attracting and nurturing leads.

A sales funnel (depending on B2B/B2C) emphasizes stages such as prospecting, lead qualification, proposal, negotiation, and closing. It focuses on converting leads into customers.

Q: How does SEO impact the top of the marketing funnel?

A: SEO plays a crucial role in the awareness stage by attracting potential customers to your website through targeted keywords and high-quality, sharable content. This post is applying that tactic in practice.

Liked this?

You might also enjoy:

Google’s HEART framework helps with this.

Created by Google’s UX research team, HEART stands for Happiness, Engagement, Adoption, Retention, and Task Success. It’s a simple way to track your product's user experience (UX).

In this post, we break down each HEART element with practical tips, examples, and a Google Maps Case Study.

Before we go further, remember that data is only helpful if it leads to better decisions. Instead of tracking everything, focus on a few key metrics that align with your strategy and objectives.

Let’s dive into the details.

Subscribe now

1. Happiness

Definition: The user’s attitude or satisfaction. This is subjective and often measured with surveys or in-app feedback.

Example: For a SaaS platform, you might send a quick survey after users complete a key milestone. High satisfaction scores usually predict strong word-of-mouth.

My advice: Track changes in Happiness alongside other metrics. A short-term dip in sentiment might not be critical, but a long-term decline signals deeper problems.

2. Engagement

Definition: How often and how deeply users interact with your product.

Example: On a streaming platform, total watch time or daily viewing sessions can measure Engagement. In a collaboration tool, it might be the number of documents created per week.

My advice: Focus on meaningful Engagement. Increasing time spent on a page isn’t always good if it means users are confused or stuck. Tie Engagement to real value for both users and the business.

3. Adoption

Definition: The number of new users (or customers) over time. This can also mean how quickly users embrace new features.

Example: If you offer a free trial, track how many sign-ups convert to paying customers. For product-led growth, measure Time to Value (TTV) - the time before users experience your product’s core benefit.

My advice: Reduce friction in onboarding. Make it easier for users to reach value quickly. If new users see the benefit fast (the “Aha moment”), they’ll be more likely to stay.

4. Retention

Definition: How well your product keeps users over time. A “sticky” product means users return regularly.

Example: If 200 people sign up in January, how many still use your product by March? If 120 remain active, your retention is 60%.

My advice: Retention often follows Happiness. If satisfaction drops, churn might rise soon after. Keep users engaged by delivering real value, not just short-term hacks. (For example, LinkedIn once introduced games to “prep your mind for the workday.”)

5. Task Success

Definition: Measures how effectively users complete specific tasks. This includes efficiency (time to complete), effectiveness (tasks completed), and error rate.

Example: If you’re testing a new feature, assign a task, like creating a project, and see how many users finish without errors. Compare results to a benchmark or competitor’s product.

My advice: You can run remote usability tests at scale, e.g., using Maze. Pair Task Success metrics with qualitative insights, such as user interviews, to understand Why users fail.

Case Study: Google’s Heart Framework for Google Maps

An example of Google Heart Framework (my analysis):

How to Implement the HEART Framework (3 Steps)

Here’s how to implement the Google HEART Framework in practice:

Step 1: Goals

Identify your product or feature goals. What are the desired outcomes? Are you improving onboarding, reducing churn, or boosting user satisfaction?

If your team runs OKRs, a goal in this table maps to your Objective, and the metrics you pick in Step 3 become Key Results or KPIs. The difference: OKR vs KPI: What’s the Difference?.

Step 2: Signals

Determine which user actions indicate success or failure for each goal. Failures, like incomplete tasks, are often easier to identify than successes.

Step 3: Metrics

Map these signals to specific HEART metrics. For example, if your goal is faster onboarding, track setup completion time (Task Success) or first-time user satisfaction (Happiness).

When Google’s teams use HEART, they get user-centric, data-informed insights. You can do the same. Just adjust these metrics to your product and audience.

FAQ

Q: Who created the Google HEART framework?

A: Google’s UX research team (led by Kerry Rodden, Hilary Hutchinson, and Xin Fu) created HEART to measure user satisfaction and product success.

Q: Is the HEART framework only for UX teams?

A: No. Product managers, designers, and engineers can all use HEART metrics to drive user-centric decisions.

Q: Do I need all five HEART metrics at once?

A: Not necessarily. Focus on the metrics that match your immediate goals. You can add more over time.

Q: How do I collect Happiness data?

A: Surveys (Customer Satisfaction - CSAT, Customer Effort Score - CES), often as in-app polls, can all measure user satisfaction. Combine them with other metrics for context.

Q: Can I benchmark Task Success against competitors?

A: Absolutely! Running usability tests on both products can show where you excel and where you might be behind.

Liked this?

You might also enjoy:

Below, we break down each stage with examples of key SaaS metrics.

Subscribe now

Before we go further, I can't emphasize this enough: you can't track everything. Focus on a few key indicators that matter most for your product.

Now, let’s dive into the details.

1. Acquisition

Acquisition is how you attract new users or customers.

• Key Metrics: Cost Per Acquisition (CPA), Customer Acquisition Cost (CAC), Bounce Rate, Click-Through Rate (CTR).

• Example: If you run a language-learning app, track sign-ups from Facebook Ads. Check how many convert after seeing a paid ad. Compare that to sign-ups from your SEO blog posts. If ads cost more than the revenue from each new user, your CAC is too high.

My advice: focus on channels that attract the right audience, not just big numbers.

For example, I could cover popular and controversial topics on X, like politics, and link to this newsletter. But that wouldn’t help in selecting the right audience.

2. Activation

Activation happens when a user first realizes and engages with the core value of your product.

• Key Metrics:

  • Time to Value (TTV),

  • Onboarding Completion Rate,

  • Trial-to-Paid Conversion,

  • User Activation Rate,

  • Feature Adoption Rate.

• Example: For a B2B SaaS product, measure how quickly a new user reaches a key activation milestone, such as successfully integrating their CRM and seeing actionable insights. If 60% complete this step in the first week while 40% drop off, you may need to simplify the process or provide better guidance.

My advice: A smoother activation increases the likelihood of higher engagement and retention later.

3. Retention

Retention measures how many users continue engaging with your product's core value over time.

Key Metrics:

• Churn Rate,

• Retention Rate (1-day, 7-day, 30-day, etc.),

• Cohort Retention (how different user groups retain over time),

• User Renewal Rate (for subscriptions),

• Feature Retention (do they keep using key features?).

Example: If 100 people sign up for your subscription-based fitness app in January, how many are still completing workouts by March? If 50 remain active, your two-month retention rate is 50%.

My advice: High retention usually means users consistently get value from your product. Instead of just short-term funnel hacking, focus on:

• Delivering repeatable value tied to your North Star Metric (e.g., “workouts completed”).

• Reducing friction to re-engage (e.g., quick-start workouts).

• Features that help users track progress and stay motivated (e.g., progress dashboards, personalized reminders, or social sharing).

4. Revenue

Revenue measures how much money your product generates and how it scales over time.

• Key Metrics:

  • Monthly Recurring Revenue (MRR),

  • Annual Recurring Revenue (ARR),

  • Customer Lifetime Value (LTV),

  • Expansion Revenue, MRR / ARR (recurring revenue growth),

  • LTV:CAC Ratio (sustainability of revenue),

  • Expansion Revenue (upsells, cross-sells),

  • Revenue Churn (lost revenue from cancellations).

• Example: If you run a SaaS with tiered plans, track how many users upgrade, downgrade, or cancel each month. If your MRR is $10,000 in one month and $12,000 the next, you've added $2,000 in net MRR. But, if high-value accounts churn, your net revenue could shrink despite new upgrades.

My advice: Balance top-line revenue with costs so you don’t overspend on acquisition. While we often simplify metrics, the ultimate goal is not just revenue but sustainable profitability.

5. Referral

Referral happens when existing users bring in new users, often through word-of-mouth or incentive programs.

• Key Metrics:

  • Virality Coefficient (how many new users each existing user brings in),

  • Referral Conversion Rate (% of referred users who sign up),

  • Customer Referral Rate (% of existing users who refer others),

  • Some use NPS as a supporting metric, but it’s controversial. I prefer measuring actual customer behavior.

• Example: Launch a referral program that rewards both the referrer and the new user, such as free credits. Track how many sign-ups come from shared links and whether those users become active. If your virality coefficient is above 1, your user base can grow exponentially without additional marketing costs.

My advice: Referrals can be a powerful, low-cost growth channel, especially for products with built-in network effects. But they are challenging. You need to offer clear, must-have value that makes users want to share.

A great referral strategy makes the product itself more useful when shared (e.g., Dropbox's extra storage for referrals) or provides strong incentives to spread the word.

Share

Keep It Simple

You can’t track every metric. Focus on just a few that align with your North Star Metric, strategy, or objectives.

Once you know which metric to focus on, set the target with an OKR: OKR vs KPI: What’s the Difference?.

FAQ

Q: What does AARRR stand for?

A: It stands for Acquisition, Activation, Retention, Revenue, and Referral. These five stages cover your entire growth funnel.

Q: Why is it called Pirate Metrics?

A: Because saying the letters A-A-R-R-R sounds like a pirate’s “Arrr.”

Q: Who created the AARRR framework?

A: Dave McClure, an investor and entrepreneur, popularized it to help startups focus on the entire user journey.

Q: How do I start measuring AARRR metrics?

A: You can’t track everything. Start with just a few of the most impactful metrics. For instance, if churn is high, focus on retention strategies before scaling acquisition.

I also recommend you read Are You Tracking the Right Metrics. For example, track changes over time and look at ratios (e.g., conversion rate) rather than raw numbers.

Q: How do I choose which metric to focus on first?

A: Identify your biggest problem or opportunity. For example, if you’re losing users right after sign-up, focus on Activation. If you’re not generating enough revenue, focus on Revenue.

Q: Should I track every sub-metric in each stage?

A: Pick no more than just 2-3 key metrics in each stage (if any). You can analyze more metrics “ad hoc” as you grow.

Q: What if I don’t have enough data?

A: Start small. Track basic metrics and pair them with qualitative insights (user interviews, surveys). As your user base grows, expand your analytics.

Liked this?

You might also enjoy:

Every week, I share actionable tips and resources for PMs.

Here are a few posts you might have recently missed:

• Introduction to AI PM: Neural Networks, Transformers, and LLMs

• Crossing the Chasm: The Ultimate Guide For PMs

• 12 Proven Sources of Insights to Fuel Your Product Discovery

• 5 Steps to Negotiate Your Product Manager Job Offer in a Tough Market

• Assumption Prioritization Canvas: How to Identify And Test The Right Assumptions

Consider subscribing and upgrading your account for the full experience.

Subscribe now

I created a dedicated section of the newsletter, “Glossary and Examples,” to discuss common PM terms and frameworks with examples. Most of the posts from that area will be published on workdays, in addition to our weekly schedule.

Previously, you were not notified about the first post I’ve been testing since July:

In today’s post, we discuss examples of product strategy:

1. Product Strategy: Editable Template

2. Product Strategy: Examples

   1. Google Maps,

   2. Netflix,

   3. OpenAI.

3. Product Strategy Canvas: Alternatives

4. Premium: An Editable Notion Template

Product Strategy: Editable Template

The Product Strategy Canvas combines strategic elements to help you define and communicate your product strategy.

Read more and download a free template

You can learn more and get a free editable template by reading this free post.

Product Strategy Examples

Example 1: Product Strategy for Google Maps

1. Vision

Organize the world's geographical information and make it universally accessible and useful for everyone.

2. Market Segments

• Individuals who need reliable navigation and local information.

• Businesses that need mapping services for logistics and customer engagement.

• Developers who need geospatial APIs to enhance their products.

3. Relative Costs

Optimizing for unique value rather than competing on cost.

4. Value Proposition

For individuals needing reliable navigation and local information:

• Before: People relied on paper maps or basic GPS devices, leading to inaccurate directions and difficulty finding local services.

• How: Google Maps provides real-time navigation, traffic updates, public transit information, and detailed local business data accessible on multiple devices.

• After: Users enjoy accurate, efficient navigation and can easily discover and engage with local businesses and services.

• Alternatives: Differentiated by global coverage, real-time data, Street View, and integration with other Google services like Google Search and Google Assistant.

5. Trade-offs

• No Dedicated Hardware: Focus solely on software solutions.

• Data Collection for Services: Collect user data to improve services, raising privacy considerations.

• No User Fees: Monetize via advertising and enterprise offerings rather than direct user fees.

6. Key Metrics

• Daily Active Users (DAU): The number of unique daily users.

• Session Duration: The average time spent per session to assess user engagement.

• Data Accuracy: The accuracy of mapping data by monitoring error reports and resolution times.

• User-Generated Content: The volume and quality of user contributions like reviews, photos, and map edits.

7. Growth

• User-Generated Content: Encourage users to add reviews, photos, and local insights through programs like Local Guides.

• Expanding into Local Markets: Localize content and features to meet regional needs and preferences.

• Partnerships with Businesses: Collaborate with local businesses to enhance listings and provide detailed information.

• Integration with the Google Ecosystem: Leverage other Google services (e.g., Search, Google Assistant) to drive user engagement.

8. Capabilities

• Advanced Data Collection: Utilize data from 1,000+ authoritative sources for comprehensive mapping.

• Machine Learning and AI: Leverage AI to improve navigation, predict traffic, and improve the user experience.

• Global Partnerships: Partner with governments, transit agencies, and businesses worldwide to improve data quality.

9. Can't / Won't

Competitors can't easily replicate Google Maps' extensive mapping data, real-time updates, user-generated contributions, and integration within the Google ecosystem without significant investment and technological development.

Example 2: Product Strategy for Netflix

1. Vision

Entertain the world by providing the best streaming entertainment service.

2. Market Segments

• Individuals who need on-demand, ad-free entertainment across various genres.

• Regional viewers who desire localized and diverse content tailored to their preferences.

• Families looking for shared entertainment options suitable for all ages.

3. Relative Costs

Optimizing for unique value through exclusive and original content rather than competing on the lowest subscription price.

4. Value Proposition

For individuals with ad-free plans:

• Before: Audiences were limited to scheduled TV programming with advertisements and had restricted access to diverse content.

• How: A vast library of movies, TV shows, and original productions available anytime and without ads on multiple devices.

• After: Users enjoy a personalized, uninterrupted viewing experience adjusted to their preferences.

• Alternatives: Differentiated by original productions, user-friendly interface, and recommendation algorithms.

5. Trade-offs

• No Live TV Broadcasts: Does not offer live sports or news.

• No Free Tier: Avoids an ad-supported free option to maintain a premium user experience.

• No User-Generated Content: Avoids user-generated content to ensure quality control.

6. Key Metrics

• Subscriber Growth Rate: The monthly or quarterly percentage increase in global paid subscribers (e.g., 10% quarterly growth).

• Churn Rate: The percentage of subscribers canceling their subscription each month, aiming to keep it below a specific threshold (e.g., under 2% monthly).

• Average Viewing Hours per User: Calculate the average number of hours streamed per user weekly to assess engagement (e.g., aiming for 10 hours/week).

7. Growth

• Original Content Production: Invest in diverse, high-quality original productions to attract new subscribers.

• Market Expansion: Expand into new international markets with localized content.

• Enhanced Personalization: Use machine learning to personalize user experiences.

8. Capabilities

• Content Production and Acquisition: Creating and acquiring compelling original content.

• Streaming Infrastructure: Technology for reliable, high-quality streaming.

• Machine Learning: Use machine learning to provide personalized content recommendations.

9. Can't / Won't

Competitors cannot easily match Netflix's scale of original content production and advanced recommendation system. Many productions are available only on Netflix, making them impossible to copy.

Example 3: Product Strategy for OpenAI

1. Vision

Ensure that artificial general intelligence (AGI) benefits all of humanity by advancing AI technology responsibly.

2. Market Segments

• Individuals who need AI-powered assistance for learning, productivity, and entertainment.

• Developers who need advanced AI tools, APIs, and models.

• Businesses that need AI solutions to improve operations and services.

3. Relative Costs

Compete on performance-to-cost ratio with smaller models while maintaining unique value (the best benchmarks) for top-tier performance.

4. Value Proposition

For individuals who need AI-powered assistance:

• Before: Limited access to advanced AI due to high costs and technical barriers.

• How: Accessible applications (e.g., ChatGPT) with affordable subscriptions.

• After: Enhanced productivity, learning, and entertainment through intuitive, easy-to-use AI.

• Alternatives: Differentiated by superior AI capabilities, partnerships, constant innovation, and ethical commitment.

5. Trade-offs

• No Open-Source: Advanced models remain proprietary.

• No Custom Solutions: Focus on versatile, general-purpose AI rather than niche customization.

• No Hardware Development: Rely on partners for infrastructure needs instead of developing custom hardware.

6. Key Metrics

• Monthly Revenue: Monthly revenue from subscriptions and usage-based fees.

• Monthly Active Users (MAU): User engagement across different platforms.

• AI Benchmarks: Maintain top-tier performance in industry-standard datasets, ensuring a competitive performance-to-cost ratio.

• Policy Compliance Rate: Ensure compliance with safety guidelines.

7. Growth

• Innovation: Continuous R&D to improve AI models and cut costs.

• Strategic Partnerships: Leverage collaborations (e.g., with Microsoft) for resources and integration into widely-used platforms.

• Funding Initiatives: Seek additional investments to support growth and transition to a sustainable for-profit model.

8. Capabilities

• AI R&D: World-class research and development capabilities.

• Strategic Partnerships: Strong partnerships for technological and market reach.

• Infrastructure: Scalable systems for training and deploying large-scale AI models.

• Ethics & Safety: Dedicated teams to ensure ethical and safe AI.

9. Can't / Won't

Competitors can't easily replicate OpenAI's combination of advanced AI technology, strategic partnerships — especially with Microsoft's enterprise reach — and commitment to ethical AI without significant investment and expertise.

Share

Product Strategy Canvas: Alternatives

Product Strategy Canvas is not the only available tool. Depending on what you are looking for, you might also consider:

• For a new product:

  • Startup Canvas (Product Strategy and a Business Model for a new product)

  • Lean Canvas (File > Download)

  • Business Model Canvas with Examples

• For an existing product, a better choice would be Product Strategy Canvas.

Premium: An Editable Notion Template

My premium subscribers can access a Notion template that they can easily duplicate to their Notion account.

Our premium PM resources: https://www.productcompass.pm/p/your-premium-pm-resources

Thanks for reading The Product Compass!

It's great to learn and grow together with you.

Have a fantastic weekend and a productive week ahead,

Paweł

1. Business Model Canvas: Editable Template

2. Business Model Canvas: Examples

   1. Google Maps

   2. Airbnb

   3. Uber

3. Business Model Canvas: Cons

4. Business Model Canvas: Alternatives

5. Premium: An Editable Notion Template

Subscribe now

Business Model Canvas: Editable Template

The Business Model Canvas developed by Strategyzer is not just a business model. It combines elements of strategy and business model. As the authors state, it’s a “strategic management and entrepreneurial tool.”

Get from Google Drive (File > Download)

Business Model Canvas: Examples

Example 1: Business Model Canvas for Google Maps

Google Maps, one of the most widely used navigation apps globally, offers a comprehensive mapping service that integrates real-time traffic updates, street views, and detailed geographical information.

Its business model is built around providing reliable and user-friendly navigation solutions while monetizing through various revenue streams.

1. Key Partners:

• Map Data Providers: Companies and organizations that supply geographical data, satellite imagery, and traffic information.

• Local Businesses: Partnerships with local businesses for advertising and featured listings.

• Technology Partners: Companies that provide cloud services, APIs, and other technological support.

• Government Agencies: Collaboration for access to public infrastructure data and regulations.

• Automotive Companies: Partnerships for integrating Google Maps into vehicle navigation systems.

2. Key Activities:

• Map Data Collection and Updating: Gathering and updating geographical data, traffic information, and user-generated content.

• Product Development: Enhancing features such as navigation, real-time traffic updates, and street view.

• Advertising Management: Managing and optimizing ad placements and sponsored listings.

• Customer Support: Providing support for users and businesses using Google Maps.

• Platform Maintenance: Ensuring the reliability and scalability of the Google Maps platform.

3. Key Resources:

• Technical Infrastructure: Data centers, servers, and cloud computing resources.

• Talent: Engineers, designers, product managers, and data scientists.

• User Data: Anonymized user data to improve service accuracy and personalized experiences.

• Brand and Reputation: Google’s strong brand and trust among users.

4. Value Propositions:

• Accurate Navigation: Reliable directions and real-time traffic updates.

• Comprehensive Mapping: Detailed maps with a wide range of geographical information.

• User Convenience: Easy-to-use interface with features like voice navigation and offline maps.

• Local Business Visibility: Platform for businesses to advertise and increase visibility.

• Integration: Seamless integration with other Google services and third-party applications.

5. Customer Relationships:

• Self-Service: Users can access and use Google Maps without direct interaction with Google.

• Community Engagement: Users contribute to the platform through reviews, photos, and map edits.

• Personalized Experience: Tailored recommendations and routes based on user preferences and history.

• Support Channels: Online help centers, forums, and customer support for troubleshooting and feedback.

6. Channels:

• Mobile Apps: Google Maps app available on iOS and Android.

• Web Interface: Accessible through any web browser.

• APIs: Google Maps API for integration with third-party apps and services.

• Automotive Systems: Embedded in car navigation systems.

• Smart Devices: Integration with smart devices like Google Home.

7. Customer Segments:

• General Public: Individuals using Google Maps for personal navigation and location searches.

• Businesses: Local businesses and enterprises leveraging Google Maps for advertising and visibility.

• Developers: Companies and developers using Google Maps API to enhance their own applications.

• Tourists: Travelers seeking directions, local attractions, and services.

• Commuters: Daily commuters relying on real-time traffic updates and efficient route planning.

8. Cost Structure:

• Infrastructure Costs: Maintaining servers, data centers, and cloud services.

• R&D Expenses: Continuous improvement and development of new features.

• Salaries: Compensation for employees across various departments.

• Marketing and Sales: Promoting the service and acquiring new users and business partners.

• Licensing Fees: Payments to data providers and third-party services.

9. Revenue Streams:

• Advertising: Revenue from local business ads and sponsored listings.

• API Usage Fees: Fees charged to developers and businesses using the Google Maps API.

• Data Licensing: Selling anonymized data insights to businesses and researchers.

• Premium Features: Subscription fees for advanced features and enterprise solutions.

• Partnership Deals: Revenue from strategic partnerships and collaborations.

Example 2: Business Model Canvas for Airbnb

Airbnb revolutionized the hospitality industry by connecting travelers with hosts offering unique accommodations worldwide.

Its platform enables property owners to monetize their spaces while providing travelers with diverse lodging options beyond traditional hotels.

1. Key Partners:

• Hosts: Property owners and renters who list their spaces on Airbnb.

• Local Governments: Collaborations to ensure compliance with local regulations and laws.

• Payment Processors: Companies like Stripe and PayPal for secure payment transactions.

• Travel Insurance Providers: Partners offering insurance options for hosts and guests.

• Service Providers: Cleaning services, photographers for property listings, and maintenance services.

• Tech Infrastructure Providers: Cloud service providers and other tech partners.

2. Key Activities:

• Platform Development: Continual development and enhancement of the Airbnb platform.

• Marketing and Sales: Attracting hosts and guests through various marketing channels.

• Customer Support: Providing support for hosts and guests for a seamless experience.

• Community Engagement: Building and maintaining a community of trust among users.

• Compliance and Safety: Ensuring properties and transactions comply with local laws and safety standards.

3. Key Resources:

• Platform Technology: The Airbnb website and mobile app.

• User Data: Data on user preferences, behaviors, and transactions.

• Brand Reputation: Strong global brand recognition and trust.

• Human Resources: Talented employees in tech, marketing, customer support, and compliance.

• Financial Resources: Funds for operational costs, marketing, and growth initiatives.

4. Value Propositions:

• Unique Accommodations: A wide range of lodging options from single rooms to entire homes, offering unique experiences.

• Affordability: Competitive pricing compared to traditional hotels.

• Local Experience: Authentic local experiences through stays with local hosts.

• Flexibility: Flexible booking options for both short and long-term stays.

• Trust and Safety: Secure payment system, verified profiles, and reviews for trust.

5. Customer Relationships:

• Self-Service: Easy-to-use platform for booking and listing properties.

• Personalized Service: Personalized recommendations and tailored experiences.

• Community Support: Host and guest support via help centers, forums, and customer service.

• Trust Building: Reviews, ratings, and verified profiles to build trust among users.

6. Channels:

• Website: The primary platform for booking and listing properties.

• Mobile App: Accessible on iOS and Android for on-the-go use.

• Social Media: Platforms like Facebook, Instagram, and Twitter for marketing and community engagement.

• Email Marketing: Regular newsletters and promotional emails to users.

• Partnerships: Collaborations with travel agencies, tourism boards, and other travel-related businesses.

7. Customer Segments:

• Travelers: Individuals and groups seeking unique lodging options.

• Business Travelers: Professionals needing short-term accommodations for work trips.

• Families: Families looking for spacious and family-friendly accommodations.

• Long-Term Renters: Individuals seeking longer stays, such as digital nomads or temporary relocations.

• Hosts: Property owners and renters looking to monetize their extra space.

8. Cost Structure:

• Platform Maintenance: Costs associated with developing and maintaining the Airbnb platform.

• Marketing and Sales: Expenses for attracting new hosts and guests.

• Customer Support: Costs of providing support to users.

• Legal and Compliance: Ensuring compliance with local laws and regulations.

• Operational Costs: Day-to-day operations, including salaries and administrative expenses.

9. Revenue Streams:

• Service Fees: Fees charged to guests and hosts for each booking.

• Host Service Fees: A percentage of the booking fee charged to hosts.

• Guest Service Fees: A percentage of the booking fee charged to guests.

• Experiences: Revenue from Airbnb Experiences, where hosts offer activities and tours.

• Premium Listings: Fees for premium placement or featured listings on the platform.

Example 3: Business Model Canvas for Uber

Uber has transformed urban transportation by offering a convenient and efficient ride-sharing service accessible through a mobile app.

Its business model focuses on connecting drivers with riders, ensuring competitive pricing, and maintaining high service quality through advanced technology and extensive partnerships.

1. Key Partners:

• Drivers: Independent contractors who use their own vehicles to provide rides.

• Fleet Operators: Companies that provide vehicles and drivers for Uber's various services.

• Payment Processors: Financial partners like PayPal and credit card companies for handling transactions.

• Regulatory Bodies: Government agencies ensuring compliance with local transportation laws.

• Technology Providers: Partners providing mapping, GPS, and other technological infrastructure.

• Vehicle Manufacturers: Partnerships for vehicle leasing and purchase deals for drivers.

2. Key Activities:

• Platform Development: Continuous development and maintenance of the Uber app and backend systems.

• Marketing and Sales: Attracting drivers and riders through advertising and promotional activities.

• Customer Support: Providing assistance to both riders and drivers.

• Regulatory Compliance: Ensuring compliance with local, regional, and national regulations.

• Driver Recruitment and Retention: Strategies to attract and keep drivers on the platform.

3. Key Resources:

• Technology Platform: The Uber app and backend systems.

• User Data: Data on rider and driver behavior and preferences.

• Brand Reputation: Strong global brand presence and recognition.

• Human Resources: Skilled employees in technology, marketing, and operations.

• Financial Resources: Capital for operations, marketing, and expansion.

4. Value Propositions:

• Convenience: Easy booking of rides via the app.

• Availability: Wide availability of rides at any time.

• Affordability: Competitive pricing compared to traditional taxi services.

• Safety: Background checks for drivers, GPS tracking, and in-app safety features.

• Variety: Different ride options (e.g., UberX, UberPOOL, UberBLACK) to suit various needs and budgets.

5. Customer Relationships:

• Self-Service: App-based booking and payments without the need for direct interaction.

• Customer Support: 24/7 support through the app for both riders and drivers.

• Ratings and Reviews: Two-way rating system to maintain service quality and trust.

• Personalization: Personalized ride suggestions and promotions based on user behavior.

6. Channels:

• Mobile App: Primary platform for booking rides, available on iOS and Android.

• Website: Information about services and company.

• Social Media: Platforms like Facebook, Twitter, and Instagram for engagement and promotion.

• Email Marketing: Newsletters and promotional emails to users.

• Partnerships: Collaborations with other businesses and services to expand reach.

7. Customer Segments:

• General Public: Individuals needing rides for various purposes.

• Business Travelers: Professionals needing reliable transportation for work-related travel.

• Commuters: Daily commuters looking for convenient and cost-effective transport options.

• Tourists: Visitors needing transportation in unfamiliar cities.

• Drivers: Individuals looking to earn income through ride-sharing.

8. Cost Structure:

• Technology Costs: Development and maintenance of the Uber app and infrastructure.

• Driver Payments: Earnings paid out to drivers.

• Marketing and Sales: Costs for advertising, promotions, and customer acquisition.

• Customer Support: Expenses related to providing 24/7 support.

• Regulatory Costs: Expenses related to compliance with transportation laws.

• Operational Costs: General administrative and operational expenses.

9. Revenue Streams:

• Ride Fees: Commission from each ride booked through the platform.

• Surge Pricing: Additional revenue from higher fares during peak demand times.

• Subscription Services: Fees from services like Uber Pass or Uber for Business.

• Advertising: Revenue from in-app advertising and promotions.

• Partnerships: Revenue from strategic partnerships and integrations with other services.

Share

Business Model Canvas: Cons

While the Business Model Canvas is the most popular canvas for a new product, what could be improved:

• It doesn’t mention a vision. Why are we doing this? Why should your employees wake up every day and go to work? It's especially critical for startups, founders included.

• It doesn’t mention that you should ensure that your competitors can’t/won’t copy your strategy without sacrificing their existing businesses. That's essential. The moment you succeed, someone will try to do the same.

• It doesn’t mention trade-offs, which are the things you don’t want to do. They are particularly important for startups, where it’s often tempting to capture every possible opportunity. Trade-offs create focus.

• It doesn’t mention key metrics. How will you test if your strategy and business model are working?

• It doesn’t mention relative costs, which are an essential strategic choice. Consider Apple vs. Xiaomi.

• It doesn’t explain what a Value Proposition really is. In addition, Value Proposition Canvas also doesn’t make that clear. Read more.

• In my opinion, “Customer Relationships” and “Key Activities” are unnecessary. Virtually no one uses the former, and the latter could be included in the Can’t/Won’t section. If it existed.

Business Model Canvas: Alternatives

Business Model Canvas is not the only available tool. Depending on what you are looking for, you might also consider:

• For a new product:

  • Startup Canvas (Product Strategy and a Business Model for a new product)

  • Lean Canvas (File > Download)

• For an existing product, a better choice will be:

  • Product Strategy Canvas and Product Strategy Canvas Examples

Premium: An Editable Notion Template

My premium subscribers can access a Notion template that they can easily duplicate to their Notion account.

Our premium PM resources: https://www.productcompass.pm/p/your-premium-pm-resources