What I learned from Andrew Ng's Agentic AI course

One prompt against a workflow

Ng starts with how most of us use a language model: one prompt in, one answer out. He compares it to asking a person to write an essay from the first word to the last in a single pass, without ever pressing backspace. People don't do their best work that way, and models don't either.

An agentic workflow splits the work into steps. For an essay, the model writes an outline, decides whether it needs web research, searches and downloads pages, writes a first draft, reads the draft to see what needs changing, and revises it. It takes longer and produces a much better result.

Open at full size

Source: Drawn from module 1 of the course

Ng's definition is simple. An agentic AI workflow is a process where an LLM-based app executes multiple steps to complete a task.

Degrees of autonomy

Ng explained why he uses "agentic" as an adjective. A few years ago people argued about whether a system was a "true agent" or not. Calling systems agentic to different degrees let everyone stop arguing about definitions and get on with building.

He described a spectrum:

• Less autonomous: the developer fixes every step in advance, and any tools are hard-coded. The model's freedom is mostly in the text it writes.
• Semi-autonomous: the model makes some decisions and chooses between tools, but the tools are predefined.
• Highly autonomous: the model decides the sequence of steps itself, and some systems can even write new tools.

His practical point is that the less autonomous end holds a lot of valuable business applications being built today. The highly autonomous end is less controllable, less predictable, and still an area of active research.

Why bother with a workflow

Ng gave 3 benefits.

Performance. His team's data on HumanEval, a coding benchmark, showed GPT-4 answering zero-shot at about 67%, a large jump over GPT-3.5. GPT-3.5 placed inside an agentic workflow, where it writes code, reflects on it and improves it, did better than GPT-4 on its own. GPT-4 inside a workflow improved further again.

Parallelism. In his black hole essay example, 3 model calls generate search ideas in parallel. Each search finds 3 pages, so 9 pages are downloaded at the same time. A person would read them one at a time.

Modularity. You can swap any single step. Ng swaps search engines often (Google, Bing, DuckDuckGo, Tavily, You.com and others) and tries different models from different providers for different steps.

Real examples, easiest to hardest

1. Invoice processing. Convert the PDF to text, check it's an invoice, pull out the biller, address, amount due and due date, and write them to a database. It's easy because there's a clear process to follow.
2. Customer order enquiries. Extract the order details from an email, look up the order in a database, draft a reply and send it for human review.
3. General customer service. Here the steps aren't known in advance. "Do you have black jeans or blue jeans?" needs several stock checks. "I'd like to return the beach towel I bought" needs a check that the customer bought it, a check of the return policy (within 30 days and unused, in the example), and then a return slip.
4. Computer use. An agent drives a web browser. Ng showed one checking seats on United flights from San Francisco to Washington DCA. It got stuck on the United site, moved to Google Flights, and found the answer. He was clear that computer use still fails often and isn't reliable enough for mission-critical work.

Tasks are easier when there's a clear process or an existing operating procedure, and when the input is text. They're harder when the steps aren't known in advance or the input is images or audio.

Task decomposition

Ng calls this one of the most important skills in the course. You look at how a person would do the job, split it into steps, and ask of each step: can a language model or a tool do this? If not, you split that step further.

His research agent example went from 1 step, to 3 steps (outline, search, write), and then he split "write" into draft, critique and revise because the essays felt inconsistent from beginning to end. He said you rarely get the decomposition right the first time.

The building blocks he lists are language models (and multimodal models), specialised AI models such as PDF-to-text or text-to-speech, APIs for things like web search, email and calendars, information retrieval including databases and RAG, and code execution.

A first look at evals

Ng said the single biggest predictor of whether a team builds agents well is whether it runs a disciplined evaluation process. His example: after building a customer service agent, the team noticed it kept mentioning competitors ("Unlike RivalCo, we make returns easy"). Nobody could have predicted that before building it. The fix was a piece of code that counts competitor names in the output.

For subjective quality you can use another model as a judge, but Ng warned that models are poor at giving 1 to 5 scores. He also introduced the difference between end-to-end evals (the whole system) and component-level evals (one step), and the habit of reading traces, the intermediate outputs of each step.

The 4 design patterns

Module 1 ends with the 4 patterns the rest of the course covers:

1. Reflection: the model critiques and improves its own output.
2. Tool use: the model can request functions to search, calculate, email and more.
3. Planning: the model decides the sequence of actions. Ng's example came from the HuggingGPT paper, where a model chained pose detection, pose-to-image, image-to-text and text-to-speech.²
4. Multi-agent collaboration: several agents with different roles work together. He mentioned ChatDev, a virtual software company of agents playing CEO, programmer, tester and designer.³ In the multiagent debate study (Du et al. 2023), multi-agent setups beat single agents on biographies (73.8% against 66.0%), MMLU (71.1% against 63.9%) and chess moves (45.2% against 29.3%).⁴

Open at full size

Source: Drawn from module 1 of the course

The basic idea

Ng compared reflection to rereading an email before you send it. You spot the vague date, the typo and the missing signature, and you fix them. With a model, you prompt it for a first draft, then prompt it again to review the draft and write a better version.

For code, the second step can use a different model. Ng said reasoning models are often good at finding bugs.

He repeated one caveat several times: reflection is not magic. It often gives a modest boost, and sometimes very little.

The evidence

The Self-Refine study (Madaan et al. 2023) compared direct generation with reflection across several models and tasks.⁵ Reflection improved results on tasks like dialogue responses, sentiment reversal, code readability and constrained generation. Maths reasoning barely moved, with gains of 0.0 to 0.2 points. Ng's advice was to measure it on your own application.

Writing a reflection prompt

His rules: tell the model clearly to review the draft, give it specific criteria to check, and tell it what to output. One example came from his team at AI Fund, which used reflection in production to brainstorm startup domain names. The prompt checked whether each name was easy to pronounce and whether it meant something negative in other languages, then asked for a shortlist.

He also said he learned to write better prompts by reading other people's prompts, including the ones inside open source projects.

Reflection on images

In the chart lab, a model wrote code to plot coffee sales for Q1 2024 against Q1 2025. The first chart was a stacked bar chart that was hard to read. A multimodal model was given both the code and the image, asked to critique the chart for readability, clarity and completeness, and asked to rewrite the code. The second version was a clearer grouped bar chart.

Measuring whether reflection helps

For objective tasks, build a small set of questions with known answers. In Ng's SQL example, the workflow got 87% right without reflection and 95% with it.¹ He said to re-run the eval every time you change the reflection prompt.

For subjective tasks, he showed why asking a model "which of these 2 charts is better?" doesn't work well. The answers are sensitive to wording, often don't match human experts, and suffer from position bias, where many models prefer whichever option comes first. The better method is a rubric of yes or no checks (has a clear title, has axis labels, uses an appropriate chart type) added up into a score.

External feedback

This was the main lesson of module 2. Reflection is much stronger when the model gets new information from outside itself. Ng drew 3 curves of performance against time spent on prompt engineering. Tuning a plain prompt flattens out. Adding reflection lifts it higher, and adding external feedback lifts it highest. His advice: when prompt tuning stops paying off, add reflection and look for external feedback.

Open at full size

Source: Drawn from module 2 of the course

His examples of external feedback:

• Code: run it and pass the error messages back.
• Competitor names: use pattern matching to find them, then ask for a rewrite.
• Facts: use web search. His example was an essay saying the Taj Mahal was built in 1648, when it was commissioned in 1631 and finished in 1648.
• Length: models still don't follow word limits well, so count the words in code and pass the exact count back.

The SQL lab showed this clearly. The model's query returned total sales of -190,571.46 because the database stored sales as negative numbers. When the model reviewed only the query text, it said the query was correct. When it was also shown the result, it saw the negative number and fixed the query.

What tool use is

Ng's opening example was "What time is it?" A model trained months ago can't know. Give it a get_current_time function and it can answer.

The key detail is that the model never runs the function. It requests the call, your code runs it, and the result goes back to the model in the conversation. The model also decides whether to use a tool at all. Asked how much caffeine is in green tea, it answers directly without checking the time.

With several tools, it picks which one and in what order. In his calendar example, the model checked the calendar, chose a free slot on Thursday and then booked the meeting with Alice.

Open at full size

Source: Drawn from module 3 of the course

How it works underneath

Ng showed the manual version first. You tell the model in the system prompt to output a specific pattern, such as FUNCTION: get_current_time(). Your code looks for that pattern, runs the function and sends the result back.

Then he showed aisuite, the open source library he built with friends to call many model providers with the same syntax.⁶ You pass Python functions directly as tools. aisuite reads the function's name, docstring and parameters and builds the JSON schema the model sees, so the docstring is the description the model uses to decide when to call the tool. The max_turns setting lets aisuite run the request, run, return loop for you, with a ceiling so it can't loop forever.

Code execution

Ng's argument for code execution: if you build a calculator, you'd need a separate tool for addition, square roots, exponents and every other button. Instead, you ask the model to write Python between <execute_python> tags, extract the code, run it and pass the result back. He said he's been surprised several times by how clever the code was.

Code execution pairs naturally with reflection, because the error message is external feedback.

He was direct about the risk. Someone on his team had a coding agent run rm *.py in a project directory, which deleted the Python files. They were backed up on GitHub. His advice is to run model-written code in a sandbox, with Docker or E2B as options, while admitting many developers don't.

MCP

The Model Context Protocol, proposed by Anthropic, solves a repeated-work problem.⁷ Without it, every app writes its own connector for every service such as Slack, Google Drive, GitHub or Postgres, which means m × n pieces of work. With MCP, each service is wrapped once as a server and each app connects once as a client, so it becomes m + n. MCP covers both resources (data to read) and tools (actions to take). Ng demonstrated Claude Desktop using a GitHub MCP server to summarise a README and list recent pull requests.

Ng called this "maybe the most important module of this entire course".

Build first, then measure

His advice is to build a quick and dirty version, look at its outputs, and let the failures tell you what to measure.

Three examples:

• Invoices: the agent kept mixing up the invoice date and the due date. The eval: hand-label the due dates of 10 to 20 invoices, ask the model to output dates as YYYY/MM/DD, extract them with a regular expression and count how many match.
• Marketing captions: the limit was 10 words, and the outputs often went over. The eval counts words, with the same rule for every example.
• Research reports: the agent sometimes missed important points. The eval: an expert writes 3 to 5 talking points per topic, and a model judge counts how many appear in the essay.

That gives his 2 × 2 of evals. You evaluate with code or with a model judge, and with ground truth for each example or one rule for all. He said starting quick and dirty is fine, that you should improve the eval when it disagrees with your own judgement, and that places where the agent is worse than a human expert are good places to work on next.

Error analysis

When the research agent produced weak essays, any step could be to blame: the search terms, the search results, the choice of sources or the writing. Ng's method is to read the traces of the failed runs and record which step went wrong in a spreadsheet.

In his black hole example, the search terms looked fine, but the results were children's news sites and blogs, with no scientific papers among them. Across the examples, search results caused 45% of the problems, choosing sources 10% and search terms 5%.¹ So the search engine was the place to work.

Open at full size

Source: Ng's error analysis of the research agent, module 4 of the course

He added a fairness rule. Judge each step on what it was given. The source picker chose weak sources because every option it was handed was weak.

Two more examples:

• Invoices: the model choosing the wrong date caused 87% of errors and PDF-to-text 15%. The percentages add up to more than 100 because one invoice can fail at both steps.
• Customer emails: the model's database query caused 75% of errors, the drafted email 30% and the database itself 4%.

Component-level evals

Once you know which step to fix, you can test that step on its own. For web search, an expert lists the sources a good search should find, and code scores the results against that list, for example with an F1 score. You can then try different search engines, numbers of results and date ranges quickly, and finish with an end-to-end eval to confirm the whole system improved.

Improving a step

For steps that aren't language models (search, RAG, code execution, other ML models), tune their settings or replace them. For language model steps, Ng's options are:

1. Improve the prompt, with clearer instructions or examples.
2. Try a different model.
3. Split the step into smaller steps.
4. Fine-tune, which he leaves until last because it costs much more developer time.

He showed why model choice matters with a PII redaction test. A small model, Llama 3.1 8B, missed a person's name, left part of the address exposed and ignored the requested output format. A larger frontier model got it right. He builds his intuition by trying new models, keeping a personal set of test questions and reading other people's prompts.

Latency and cost

Ng gets quality right first, then looks at speed, then cost. He said his team has several times had so many users that cost became a problem, which he called a good problem to have.

His method is to time and price every step. For the research agent, the times were 7 seconds for search terms, 5 for web search, 3 for picking sources, 11 for fetching pages and 18 for writing the draft. Fetching pages in parallel or using a faster model or provider are the obvious places to save time.

The costs per run on his slide were $0.0004 for the search terms, $0.016 for web search, $0.0004 for picking sources, $0.002 for fetching, $0.03 for PDF to text and $0.009 for the draft.¹ Added up, that's about 5.8 cents per run, and the language model steps are a small part of it.

The process

Ng sums up development as a loop between building and analysing: examining outputs and traces, building evals, doing error analysis and running component-level evals. His warning is that less experienced teams spend too much time building and too little analysing.

Planning

The example is a sunglasses shop. The customer asks, "Do you have any round sunglasses in stock that are under $100?" The model is given tools such as get_item_descriptions, check_inventory, get_item_price and process_item_return, and asked to write a step-by-step plan. It plans to find round frames, check stock, then check prices, and each step runs with the previous step's output.

With the same tools, a return request produces a different plan. Nobody hard-coded either sequence. An email assistant example worked the same way: find Bob's dinner email, reply to it, then archive it.

Ng said planning works really well in agentic coding tools, which write a checklist and work through it. Elsewhere it's still more experimental and harder to control, because you don't know in advance what plan the model will write.

Plans as JSON

For code to execute a plan reliably, the plan needs structure. Ng asks for JSON with a step number, description, tool name and arguments for each step. In his ranking JSON is best, XML is also good, Markdown is less reliable and plain text least.

Plans as code

The coffee machine example showed the limit of tools. With tools like get_column_max, get_column_mean and filter_rows, "Which month had the highest sales of hot chocolate?" needs a long chain of steps repeated for every month. "How many unique transactions last week?" needs a new tool. Ng described this approach as brittle and inefficient, with teams continuously dealing with edge cases.

The alternative is to ask the model to write Python, for example with pandas, between <execute_python> tags. The code's comments are the plan and the code carries it out. The model already knows hundreds or thousands of library functions. In the study Ng showed (Wang et al. 2024, "Executable Code Actions Elicit Better LLM Agents"), writing code as the action beat writing JSON or plain text across the models tested.⁸ The sandbox advice from module 3 still applies, and Ng noted that some applications still need custom tools.

Open at full size

Source: Drawn from module 5 of the course

Multi-agent systems

Ng's analogy is that even on one computer, programmers split work into processes and threads because it's easier to build. In the same way, you can ask which team you'd hire for a job. One person is the other option.

For a marketing brochure about sunglasses, that might be a researcher with web search, a graphic designer with image generation or code execution for charts, and a writer who needs no tools. Each agent is a model prompted to play its role. They can work in a line, with the researcher passing to the designer and both passing to the writer. Or a marketing manager agent can plan the work and delegate to them. Agents built this way can also be reused in other projects.

Communication patterns

Ng described 4:

1. Linear: each agent passes its work to the next. One of the 2 most common.
2. Hierarchical: a manager coordinates the other agents, and it's usually simpler to have each report back to the manager. The other of the 2 most common.
3. Deeper hierarchy: some agents have their own sub-agents, such as a web researcher and fact checker under the researcher. Used less because it's more complex.
4. All-to-all: any agent can message any other. Ng said the results are hard to predict, so it suits applications that can tolerate some chaos.

He said many software frameworks now make these patterns easier to build.

The graded labs were:

• A reflective essay workflow that drafts, reflects and revises.
• A research agent that searches arXiv and the web with a hand-written tool-calling loop, reflects on and rewrites its report as JSON, and converts it into an HTML page.
• A multi-agent research system where a planning agent writes the plan, and an executor sends each step to a research agent (arXiv, Tavily and Wikipedia), a writer agent or an editor agent, passing the earlier results along as context.

The ungraded labs covered SQL generation with reflection, chart generation with multimodal reflection, turning Python functions into tools, an email assistant, component-level evals for web search, a customer service agent for a sunglasses shop that writes its own database code as its plan, and a marketing team of agents that researches trends, generates a campaign image, writes a quote and packages it all into a report.

In the final video, Ng said that when his team and others interview people for jobs, the interviews often test the skills this course teaches. He ended by asking us to use these skills responsibly and to go build cool stuff.

If you want to start, the course's own order is the one to follow. Pick a task with a clear process and text as its input. Split it into steps and ask of each one whether a model or a tool can do it. Build the quick and dirty version, read its outputs, and write an eval with 10 to 20 examples. Add reflection where prompt tuning stops paying off, and give the model outside feedback before you give it more freedom.

I'm using these notes as the blueprint for the agents I build for my own work.