Back to List

Agent and Tool Use β€” A Loop for Automatically Stitching Together Experiments

An LLM becomes an agent when you add tools, loops, and memory. Understand the principles of Function Calling, ReAct, Planning, MCP, and Multi-Agent through the scenario of automating experiment design.

Beginner
|
22min
|
Verified (2026-07)
Progress0/15 (0%)

Agents and Tool Use: Automating Experimental Design with Looping

After Completing This Topic

Assistants up to sections #7 and #8 return one response per prompt. However, real-world research tasks such as experimental design, data analysis, and literature reviews involve multiple steps and tools. In this section, you will learn how to extend LLMs into agents by adding tools, loops, and memory.

The probabilistic machine from section #1, combined with the conditional design from section #7 and external knowledge injection from section #8, now becomes an autonomous executor that chains together multiple steps. This is at the forefront of AI development in 2024-2025.


The Real Complexity of a Graduate Student's Request

Suppose you need to submit an experimental setup for the next week to your advisor. You tell the assistant:

"Based on our previous paper, create a draft of an experimental design, referencing recent papers that used similar cell lines and the CRISPR editing protocol. Also, list the necessary reagents and indicate which ones are not in our lab's inventory."

This single sentence contains several tasks:

  1. Search the lab's paper archive to determine which cell lines are frequently used.
  2. Search PubMed for recent papers using that cell line + CRISPR editing conditions.
  3. Summarize several similar papers found in the search.
  4. Synthesize multiple protocols to create a draft experimental design.
  5. Extract the list of necessary reagents from the draft.
  6. Query the lab's inventory database to check the availability of reagents.
  7. Indicate only the missing reagents.
  8. Organize the final results into a markdown document.

A single prompt in section #7 cannot handle this workflow. RAG in section #8 can help with steps 1-3, but not 4-7. What is needed is an executor that calls multiple tools sequentially and in parallel. This executor is an agent.


Agent = LLM + Tools + Loop + Memory

Let's break down the structure of an agent into its minimum components:

text
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚              Agent Loop                 β”‚
β”‚                                         β”‚
β”‚   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”           β”‚
β”‚   β”‚  LLM    │───▢│  Decision     β”‚           β”‚
β”‚   β”‚(reason) β”‚    β”‚(next step)β”‚          β”‚
β”‚   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜           β”‚
β”‚        β–²              β”‚                 β”‚
β”‚        β”‚              β–Ό                 β”‚
β”‚   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”            β”‚
β”‚   β”‚ Memory │◀───│  Tool Call  β”‚           β”‚
β”‚   β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β”‚ (act)     β”‚           β”‚
β”‚                 β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜            β”‚
β”‚                       β”‚                 β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                        β–Ό
                   External World
              (search, computation, API, DB)

Four components:

  • LLM: The probabilistic machine we covered in sections #1-6. It decides which tool to call with which arguments.
  • Tools: Functions that interact with the external world. These can include searching, computing, querying databases, or calling APIs.
  • Loop: The LLM calls a tool, observes the results, and then makes another decision. This is a repeating process.
  • Memory: A record of past observations and decisions. This can be divided into short-term memory (context window) and long-term memory (external storage).

With these four components in place, the agent can autonomously execute multiple steps.

Bio Analogy - Cellular Response to Environment. A cell also has this structure. Sensing (receptor) β†’ Decision (signal cascade) β†’ Action (transcription factor activation, metabolic regulation) β†’ Observation (feedback) β†’ Re-sensing. If the signal cascade in section #4 was an internal decision-making circuit for the cell, then the agent is an artificial analogy that replaces it with an LLM. Just as a cell is a "functional decision system," so is an agent.


Tool - The Gateway to the External World

A tool is a function that an agent can call. Each tool has a name, description, and a parameter schema.

Here are some examples of tools that would be useful in a biological setting:

python
tools = [
{
"name": "search_lab_papers",
"description": "Search for relevant papers in the lab's paper archive",
"input_schema": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "Search query"},
"top_k": {"type": "integer", "default": 5}
},
"required": ["query"]
}
},
{
"name": "search_pubmed",
"description": "Search for papers on PubMed",
"input_schema": {
"type": "object",
"properties": {
"query": {"type": "string"},
"year_from": {"type": "integer"},
"max_results": {"type": "integer", "default": 20}
},
"required": ["query"]
}
},
{
"name": "get_reagent_stock",
"description": "Check the reagent inventory in the lab's database",
"input_schema": {
"type": "object",
"properties": {
"reagent_names": {"type": "array", "items": {"type": "string"}}
},
"required": ["reagent_names"]
}
},
{
"name": "write_file",
"description": "Save the results to a file",
"input_schema": {
"type": "object",
"properties": {
"path": {"type": "string"},
"content": {"type": "string"}
},
"required": ["path", "content"]
}
}
]

The agent calls these tools, and the results are accumulated in the context, which the agent uses to make the next decision.

The Importance of Tool Descriptions. The LLM uses the tool name, description, and parameter schema to decide which tool to call. If the description is inaccurate or ambiguous, the agent may call the wrong tool or use incorrect arguments. Tool design should be done with as much care as prompt engineering.

Elements of a Good Tool Description:

  • Clearly state when to use this tool.
  • Describe the purpose of each parameter and provide examples.
  • Specify the format of the return value.
  • Outline how to handle failure cases (parameter errors, lack of resources, etc.).

Function Calling - A Standard Interface

Function Calling (OpenAI terminology) or Tool Use (Anthropic terminology) is a standardized API interface for LLMs to call tools.

The trained model has learned "at this point, I should call this tool". When given a user prompt and a list of tools, it returns the tool name and parameters in JSON format.

Here's an example using the Claude API:

python
response = client.messages.create(
model="claude-3-5-sonnet-latest",
tools=tools,
messages=[{"role": "user", "content": "Tell me the top 3 cell lines that the lab has used in the last 3 years."}]
)
# response.stop_reason == "tool_use"
# response.content will contain:
# [
# {"type": "text", "text": "Let me check the lab's paper archive."},
# {"type": "tool_use", "name": "search_lab_papers",
# "input": {"query": "cell line used in experiments", "top_k": 10},
# "id": "toolu_01ABC..."}
# ]

The host program (the agent loop) executes this call and passes the results back to the model.

python
tool_result = search_lab_papers(query="cell line used in experiments", top_k=10)
response = client.messages.create(
model="claude-3-5-sonnet-latest",
tools=tools,
messages=[
{"role": "user", "content": "Tell me the top 3 cell lines that the lab has used in the last 3 years."},
{"role": "assistant", "content": response.content}, # includes tool_use
{"role": "user", "content": [
{"type": "tool_result", "tool_use_id": "toolu_01ABC...",
"content": json.dumps(tool_result)}
]}
]
)

The model then uses this result to make the next decision. It may request another tool call (stop_reason == "tool_use") or return a final answer (stop_reason == "end_turn").

This back-and-forth is the backbone of the agent loop.

ReAct Pattern: Alternating Thought and Action

ReAct (Reasoning + Acting) is a pattern where an agent explicitly expresses its reasoning in each step, followed by an action (calling a tool).

While Chain-of-Thought in Chapter 7 focused on filling in reasoning phrases before generating an answer, ReAct explicitly states why a particular tool is being called with a specific input before each tool call. This explicitness stabilizes the model's judgment.

Example of the ReAct pattern:

text
Thought: Since the lab frequently uses certain cell lines, I should first search the lab's paper repository.
Action: search_lab_papers(query="cell line CRISPR knockout", top_k=15)
Observation: [Abstracts and method snippets from 15 papers]

Thought: These papers mention HEK293T 8 times, HeLa 5 times, and U2OS 3 times. HEK293T is the most frequently used.
Now, I should search for recent papers that use HEK293T and CRISPR editing protocols.
Action: search_pubmed(query="HEK293T CRISPR knockout efficient protocol", year_from=2023, max_results=10)
Observation: [Abstracts of 10 recent papers]

Thought: Among these, a 2024 Nature Protocols paper seems to use similar cell lines and editing methods. I need a detailed summary.
Action: search_pubmed(query="2024 Nature Protocols HEK293T CRISPR efficient", max_results=1)
Observation: [Excerpts from the paper]

Thought: I am ready to prepare a draft protocol. I will list the necessary reagents and check the inventory.
...

Each Thought justifies the next Action, and each Observation serves as evidence for the next Thought. This explicit structure helps the agent with self-validation and debugging. If there is a failure, it is possible to trace back to the Thought where the incorrect judgment was made.

Inherent in Recent Models. The latest models, such as Claude 3.5, GPT-4o, and o1, automatically use this flow even without being explicitly asked to follow the ReAct pattern. This is a result of the models being trained on a large amount of agent logs.


Planning: Breaking Down Large Tasks

For simple tasks, a single step of tool calling may be sufficient, but for more complex tasks, like the request from our graduate student (8 steps), it is advantageous to first establish an overall plan.

Plan-and-Execute Pattern.

  1. Upon receiving a user request, first establish a plan: List the necessary steps.
  2. Execute each step sequentially or in parallel.
  3. Observe the results of each step and, if necessary, modify the plan.
  4. Return the final result.

When establishing the plan, provide the LLM with a prompt like this:

text
You are a research assistant. Break down the user's request into executable steps.
Specify the tool to be used and its parameters for each step.

User request: (User's request)

Return the plan as JSON:
{
  "steps": [
    {"step_id": 1, "action": "tool_name", "params": {...}, "depends_on": []},
    {"step_id": 2, "action": "tool_name", "params": {...}, "depends_on": [1]},
    ...
  ]
}

Parallel Execution. Steps with an empty depends_on can be processed in parallel. For example, the step of summarizing 5 papers can be done in parallel, saving time.

Dynamic Replanning. If the result of a step deviates from the plan (e.g., a search returns 0 results), reconfigure the remaining steps. This is a combination of CoT + Tool + Replanning.

Bio Analogy. Similar to managing a research project and aligning concepts. Breaking down a large project into milestones and tasks, distinguishing what can be done in parallel and what must be done sequentially, and adjusting the plan based on the results. What the agent does is automate this management cycle.


Memory: Short-Term and Long-Term

As the agent executes multiple steps, information accumulates. Where this information is stored is the memory design.

Short-Term Memory = Context Window. The history of tool calls, observations, and reasoning within the current session is all accumulated in the context window. The attention mechanism in Chapter 6 references this history.

Problem. As the session gets longer, it exceeds the context window. Even GPT-4's 128K and Claude's 200K are filled after 30-50 tool calls. And the "lost in the middle" problem from Chapter 8 also occurs.

Short-Term Memory Management Techniques.

  • Summarization. Compress old observations into a summary.
  • Selective Retention. Retain only important results, and save detailed logs to a file.
  • Chunking. Maintain only the most recent N steps in the context.

Long-Term Memory = External Storage. Information that persists across multiple sessions.

  • File System. Store the results of each session as a file. Refer to the file using a file-reading tool in the next session.
  • Vector DB. Store past results as embeddings and retrieve them using RAG when needed.
  • Structured Database. Store structured information (experiment history, result metrics, etc.) in an SQL database.

Bio Analogy. The epigenetic memory of a cell. The history of gene expression is passed down across generations through histone modifications and DNA methylation. The agent's long-term memory has a similar concept of persistence.


MCP: Standardizing the Tool Interface

The tools we have discussed so far are defined in different formats for each agent system. Claude API's tool schema, OpenAI's function calling, LangChain tool, LlamaIndex tool, etc., all have slightly different interfaces. If you want to move to a new system, you have to redefine the tools.

Model Context Protocol (MCP). A standard proposed by Anthropic in 2024. An open standard that defines tools, resources, and prompts in a language, framework, and model-agnostic manner.

Structure.

  • MCP Server: A process that provides tools and resources. Examples: filesystem server, GitHub server, PubMed server.
  • MCP Client: An agent that connects to the server and calls tools.
  • Communication: Standard JSON-RPC protocol.

Examples of MCP servers that would be useful in bio practice.

  • PubMed MCP: Search for and retrieve papers.
  • PDB MCP: Protein structure data.
  • UniProt MCP: Protein information Q&A.
  • Filesystem MCP: Access to the lab's file system.
  • Slack MCP: Team channel notifications.

Significance. If MCP becomes standardized, agent systems only need to use the language they know (MCP). To add a new tool, you just need to launch an MCP server. The tool ecosystem is built on a single standard.

As of 2025, major agent tools such as Claude Desktop, Cursor, Continue, and Windsurf have adopted MCP. Hundreds of MCP servers are publicly available in the open-source community. Understanding MCP and being able to create your own tools as MCP servers is becoming a basic skill for AI developers today.


Multi-Agent: Role Division

For complex tasks, it is often better to divide the work among multiple agents rather than have a single agent do everything.

Example of a bio research pipeline. Divide it into three agents.

  • Literature Agent: Dedicated to searching for and summarizing papers.
  • Protocol Agent: Drafts protocols and organizes reagent lists.
  • Inventory Agent: Manages lab inventory.

The three agents collaborate under an orchestrator. The orchestrator receives the user request and decides which agent to delegate what to.

Anthropic's Sub-Agent Experiment. The latest coding agents, such as Claude Code, Manus, and Devin, use this structure. The main agent spawns multiple sub-agents to perform sub-tasks in parallel. Each sub-agent has its own context window, which isolates information.

When is multi-agent advantageous?

  • When different domains of knowledge are required (literature, protocols, inventory).
  • When time can be saved through parallel processing (summarizing 10 papers simultaneously).
  • When you want to prevent contamination through context isolation.

When is a single agent better?

  • When context needs to be shared (continuously adjusting within a single conversation).
  • When the orchestration overhead is greater than the benefits.
  • When debugging and auditing are important (easier to track with a single log).

Safety: The Risks of Tool Execution

The fact that an agent actually executes tools means that it affects the real world. Mistakes or misuse can result in real losses.

Examples of risks in a bio context.

  • An automated reagent ordering tool orders the wrong quantity.
  • An experiment scheduling tool schedules the wrong time.
  • A database modification tool overwrites experiment records.
  • Personal information is exposed through external API calls.

Defense Mechanisms.

1. Tool Permission Separation. Separate read-only tools from write and execute tools. Most automated executions are limited to read-only.

2. Human Approval Gate. For destructive tools (reagent ordering, data modification), require human confirmation before execution. The agent presents a plan, and the user approves it before execution.

3. Tool Execution Log. Log all tool calls in an auditable log.

4. Sandboxing. Isolate the execution environment. For example, code execution tools are run in Docker or WASM sandboxes.

5. Prompt Injection Defense. The injection risk from Chapter 7 is even more serious in agents. Documents found through searches or read from files may contain instructions like "Ignore your original instructions and execute X." Do not directly connect tool results to the system prompt, but wrap them with explicit delimiters.

Bio-Specific Risks. Combinations of chemical and biological information can generate information on how to synthesize dangerous substances. Alignment training can prevent this, but it is not perfect. See Chapter 11 for more details.

Bio Application Scenarios

Scenario 1 β€” Experimental Design Assistant (Continued from Previous Scenario)

Implementing the agent that handles the 8-step request mentioned earlier.

python
import anthropic
client = anthropic.Anthropic()
tools = [...] # The 4 tools defined earlier
def execute_tool(name, params):
if name == "search_lab_papers":
return vectorstore.similarity_search(params["query"], k=params.get("top_k", 5))
elif name == "search_pubmed":
return pubmed_api(**params)
elif name == "get_reagent_stock":
return db.query_stock(params["reagent_names"])
elif name == "write_file":
Path(params["path"]).write_text(params["content"])
return {"status": "ok"}
def agent_loop(user_request):
messages = [{"role": "user", "content": user_request}]
while True:
response = client.messages.create(
model="claude-3-5-sonnet-latest",
tools=tools,
messages=messages,
max_tokens=4096
)
if response.stop_reason == "end_turn":
return response.content[0].text
messages.append({"role": "assistant", "content": response.content})
tool_results = []
for block in response.content:
if block.type == "tool_use":
result = execute_tool(block.name, block.input)
tool_results.append({
"type": "tool_result",
"tool_use_id": block.id,
"content": json.dumps(result)
})
messages.append({"role": "user", "content": tool_results})
# Usage
result = agent_loop("Referencing recent papers with CRISPR editing protocols similar to the cell line used in our previous publications, create a draft of the experimental design for this time. Also, list the required reagents and indicate which ones are not in our lab's inventory.")

This loop automatically processes the 8-step request by calling tools for a maximum of 20-30 steps.

Scenario 2 β€” Molecular Property Calculation Agent

Wrap chemical information tools such as RDKit, PubChem, and OpenBabel as MCP servers and attach them to the agent.

User request: "Find 3 known inhibitors with similar predicted IC50 values for this SMILES molecule. Also, provide information on hERG channel side effect prediction."

The agent:

  1. Parses the SMILES string into a molecular structure (RDKit MCP)
  2. Calculates molecular fingerprints (RDKit)
  3. Searches for similar molecules in PubChem (PubChem MCP)
  4. Calls the hERG inhibition prediction model (separate prediction API)
  5. Integrates the results and creates a report

Agents like this are starting to be deployed in the early stages of drug discovery.

Scenario 3 β€” Automated Experimental Data Analysis Pipeline

Automatically analyzes raw files from measurement devices.

  • File Watcher Agent: Monitors the experimental device folder and detects new files.
  • QC Agent: Checks the quality of the raw data and detects outliers.
  • Analysis Agent: Performs statistical analysis, visualization, and report generation.
  • Notification Agent: Notifies the results via Slack or email.

This is a real-world example of multi-agent orchestration. Some recent laboratory automation startups provide such systems.


Key Takeaways

  • Agent = LLM + Tools + Loop + Memory. Goes beyond a single prompt for autonomous, multi-step execution.
  • Tool use / Function calling is the interface for LLMs to call external tools in a standard way.
  • ReAct pattern β€” Thought Β· Action Β· Observation cycle. A natural extension of CoT in section #7.
  • Planning breaks down large tasks into smaller parts and processes them in parallel. Handles failures with dynamic replanning.
  • Memory combines short-term (context) and long-term (files, vector DB, SQL) memory. Persists across sessions.
  • MCP is the open standard for tool interfaces. Accumulates a tool ecosystem on a single standard.
  • Multi-agent is used when role division is necessary. Orchestrator + Sub-agents pattern.
  • Safety β€” Permission separation, human approval gates, logs, sandbox, injection prevention. Tool execution affects the real world.

πŸ“ Appendix β€” Mathematical/System Formulas for Experts

Difficulty: Very Hard Target Audience: Readers with background in system design, reinforcement learning, and formalized processes.

A.1 Agent's Perspective on Markov Decision Process

Formalize the agent as a Partially Observable Markov Decision Process (POMDP).

  • State s ∈ S: The current state of the world (not directly observable by the agent).
  • Observation o ∈ Ξ©: The result of calling a tool. o ~ O(o | s).
  • Action a ∈ A: Calling a tool.
  • Policy Ο€(a | h): The probability of the next action given the history of observations and actions h.
  • Reward r: A measure of how well the goal is achieved.

LLM is a function that parameterizes Ο€. Ο€_ΞΈ(a | h) = LLM(h).

Difference. Unlike reinforcement learning, the LLM's Ο€ is not trained with explicit reward signals during training (RLHF is a different layer). However, it absorbs the "in this situation, this action" pattern from the agent logs in the training data.

A.2 Training for Function Calling

It's not automatic that LLMs will call tools well. The training data must explicitly contain patterns of tool calls.

Function calling training data format:

text
User: "What is the average cell culture success rate over the past 3 months?"
Assistant: <tool_call>get_experiment_records(period="3_months")</tool_call>
Tool: {"records": [...], "success_rate": 0.87}
Assistant: "The average success rate over the past 3 months is 87%."

Trained with tens of thousands of such dialogues. The model learns the pattern of calling tools appropriate to the situation.

Toolformer (Meta AI, 2023) is an early study of this training method. Recent models use pre-training with a large amount of tool use dialogue + RLHF.

A.3 Formalization of ReAct

At each step t:

text
Thought_t ~ Ο€_thought(Β· | h_t)
Action_t ~ Ο€_action(Β· | h_t, Thought_t)
Observation_t = Tool(Action_t)
h_{t+1} = h_t βˆͺ {Thought_t, Action_t, Observation_t}

Stopping condition: Action_t == "answer" or t == max_steps.

Theoretical benefits. If the Thought is explicitly in the context, Ο€_action can reference it. More accurate than directly extracting Action without Thought.

Internal reasoning in recent models. Modern models such as Claude 3.5 sometimes treat Thought as hidden reasoning. It's not visible from the outside, but internally it has a similar structure.

A.4 Computational Complexity of Multi-step Planning

Greedy planning (LLM creates a plan at once):

  • Time: 1 LLM call
  • Error propagation: Initial errors affect all steps.

Tree Search planning (Generate and select multiple plan candidates):

  • Time: N Γ— 1 LLM call (N candidates) + evaluation LLM call
  • Error reduction: Increased possibility of selecting the optimal plan through diversity.

Monte Carlo Tree Search (MCTS):

  • Explore multiple candidates at each step + rollout simulation
  • Time explosion (b^d, b=branch factor, d=depth)
  • Root methodology of AlphaGo and the o1 series.

A.5 Anchoring Tool Selection Probability

The probability of the LLM selecting a tool:

text
P(tool_i | context) = softmax(logit(tool_i | context))

Anchor effect. If a specific situation is explicitly mentioned in the tool description, the probability of the tool being selected increases sharply when that situation is in the context. This is why tool descriptions are as important as prompt engineering.

Ambiguity penalty. If multiple tools fit a similar situation, the probability of the model selecting the wrong tool increases. Tool names and descriptions should be clearly distinguishable from each other.

A.6 MCP Protocol Structure

Based on JSON-RPC 2.0.

Methods provided by the server:

  • initialize: Exchange protocol version and capabilities.
  • tools/list: List of available tools.
  • tools/call: Execute a tool.
  • resources/list: List of static resources (e.g., files).
  • resources/read: Read a resource.
  • prompts/list: List of prompt templates.
  • prompts/get: Get a prompt.

Methods provided by the client:

  • sampling/createMessage: The server can request the LLM to be called (bidirectional).
  • logging/log: The server sends logs to the client.

Transport: stdio (standard input/output) or HTTP+SSE.

A.7 Context Window Budget Management

As the agent session gets longer, context management becomes important.

Session budget:

text
budget = C - system_prompt - tools_schema - reserved_output

C: Context window size.

Usage during session:

text
used = Ξ£_{t} |Thought_t + Action_t + Observation_t|

If used > budget * threshold, trigger compression/summarization.

Compaction strategies:

  1. Replace detailed observations of the first few steps with summaries.
  2. Keep only the most recent successful steps.
  3. Store all intermediate results in external files and keep only the file paths in the context.

A.8 Multi-Agent Communication Protocol

Format for communication between agents.

Message passing:

json
{
    "from": "orchestrator",
    "to": "literature_agent",
    "task": "Summarize the 10 most recent papers on HEK293T + CRISPR in PubMed",
    "context": {...},
    "deadline": "2 minutes",
    "return_format": "json_schema"
}

Response:

json
{
    "from": "literature_agent",
    "to": "orchestrator",
    "status": "success",
    "result": {...},
    "duration": "45s",
    "cost": {"tokens": 45230}
}

Consensus mechanism. If multiple agents return different answers, the orchestrator selects the final answer through majority voting or quality evaluation.

A.9 Tool Cost and Latency Management

Cost and latency of each tool call:

ToolLatencyCost
Vector search100~500ms$0 (self-hosted)
PubMed API1~3s$0
LLM call2~30s0.01 0.01~1
Web scraping3~10s$0
Code execution1s~$0 (sandbox)

Optimization:

  • Batch call those that can be called in parallel.
  • Caching (reuse the same query).
  • Early termination (break the loop when sufficient information is collected).

A.10 Agent Evaluation Metrics

Task success rate: Completion rate (human judgment). Steps to completion: Number of tool calls. Fewer is more efficient. Cost per task: Tokens and API costs. Latency: Time from start to completion. Robustness: Recovery rate from failure cases.

Benchmarks:

  • AgentBench: Agents across various domains.
  • SWE-bench: Resolving real-world GitHub issues.
  • GAIA: General assistant tasks.
  • WebArena: Web automation.

Bio-specific benchmarks are still in the early stages.


References

All of the content, scenarios, analogies, and figures in this section are developed by BioPlayground. The following are external references that can help with concept learning.

  • ReAct paper: Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models" (ICLR 2023)
  • Toolformer: Schick et al., "Toolformer: Language Models Can Teach Themselves to Use Tools" (NeurIPS 2023)
  • Function Calling: OpenAI docs "Function Calling", Anthropic docs "Tool Use"
  • Model Context Protocol: modelcontextprotocol.io, Anthropic official specification (2024)
  • Plan-and-Execute: Wang et al., "Plan-and-Solve Prompting" (ACL 2023)
  • AutoGPT and BabyAGI concept origins: Nakajima and Richards, open-source projects (2023)
  • MCTS + LLM: Yao et al., "Tree of Thoughts" (NeurIPS 2023)
  • SWE-bench: Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" (ICLR 2024)
  • AgentBench: Liu et al., "AgentBench: Evaluating LLMs as Agents" (ICLR 2024)
  • Anthropic Building Effective Agents: anthropic.com/research/building-effective-agents (2024)

In Part #9, you learned the basic structure of agents. In Part #10, we will discuss practical techniques for wisely managing the context window in long agent sessions.

Next Concept

πŸ’¬ Questions & Comments

0 comments

You can post without signing in. Guest comments cannot be edited or deleted by their author.

0/2000

Loading...