Test a CrewAI Project

How to test crews and flows deterministically, without API keys, and how to debug real runs with events and traces.

Verified against crewai 1.15.22 and 1.15.23 on 2026-10-01. Live-tested with real LLMs on 2026-10-01. Run crewai version first; if the major/minor differs, re-verify the version-sensitive rows with the ask-docs skill.

1. Why tests need a stub LLM

A real LLM gives different text on every run, costs money, and needs a key. Every Agent without an explicit llm resolves to gpt-4.1-mini and fails at kickoff with ValueError: OPENAI_API_KEY is required when no key is set, so a CI run with no secrets cannot even start. You cannot assert on output you cannot predict.

A stub LLM fixes the responses, so assertions are exact. It keeps every real crewai code path: prompt building, {placeholder} interpolation, task context, tool execution, guardrail retries, pydantic conversion and flow routing. It also records every call, so you can assert on the prompts crewai built. That is where most bugs show up: a missing input, a dropped context, a tool that was never listed.

Test in tiers:

2. The custom BaseLLM contract in 1.15

from crewai import BaseLLM. The only abstract method is call. crewai passes from_task, from_agent and (for structured tasks) response_model as keyword arguments, so call must accept them.

references/stub_llm.py implements all of this. Copy it; do not write your own unless you need to.

3. Set up the test harness

1. Copy references/stub_llm.py to tests/stub_llm.py in the project.
2. uv add --dev pytest, then run tests with uv run pytest.
3. Do not run uv run test to mean pytest. In a scaffolded crew project test is a project script that runs crewai test (section 8), which calls a real model.
4. Add tests/.crewai-test-storage/ to .gitignore.

# tests/conftest.py
import os
import socket
from pathlib import Path

# Optional, test-only switches. The AGENTS.md that `crewai create` writes leaves
# turning observability off to the user, so ask before uncommenting them.
# Without them the socket guard below does not cover everything:
# - crewai telemetry is still sent to telemetry.crewai.com when the test process
#   exits, after the guard has been undone;
# - if tracing is on (CREWAI_TRACING_ENABLED=true or tracing=True) and a CrewAI
#   credential exists (CREWAI_USER_PAT, or just a saved `crewai login` on the
#   machine), kickoff requests a trace grant, the guard refuses it, and the test
#   fails with TraceGrantError.
# If uncommented, they must run before crewai is imported; never put them in the project's .env.
# os.environ["CREWAI_DISABLE_TELEMETRY"] = "true"
# os.environ["CREWAI_TRACING_ENABLED"] = "false"
# os.environ["OTEL_SDK_DISABLED"] = "true"
# Absolute path: crewai's SQLite files and memory store go here instead of your user data dir.
os.environ["CREWAI_STORAGE_DIR"] = str(Path(__file__).parent / ".crewai-test-storage")

import pytest


@pytest.fixture(autouse=True)
def offline(monkeypatch, tmp_path):
    def refuse(*args, **kwargs):
        raise RuntimeError("test tried to use the network")

    monkeypatch.setattr(socket.socket, "connect", refuse)
    monkeypatch.chdir(tmp_path)  # output_file= and flow file writes land in a temp dir

Why each line:

• import crewai alone creates <platform user data dir>/<cwd name>/. On macOS that is ~/Library/Application Support/<cwd name>/. Kickoffs write SQLite files there. An absolute CREWAI_STORAGE_DIR set before the import redirects all of it.
• The socket guard turns any accidental real-LLM call into an immediate failure instead of a hang or a bill.
• crewai's own internals emit DeprecationWarnings on every kickoff. To quiet them:

# pyproject.toml
[tool.pytest.ini_options]
filterwarnings = ["ignore::DeprecationWarning:crewai.*"]

4. Driving the stub

StubLLM picks each response from three sources, in this order:

Tool calls go through the ReAct loop. tool_call(name, args) builds the Thought/Action/Action Input text. The real tool _run executes, and its result comes back to the next LLM call as Observation: ....

# tests/test_structured_and_tools.py
from pydantic import BaseModel

from crewai import Agent, Crew, Task
from crewai.tools import tool
from stub_llm import StubLLM, prompt_text, tool_call


class Brief(BaseModel):
    title: str
    bullets: list[str]


tool_inputs = []


@tool("Word Counter")
def word_counter(text: str) -> int:
    """Count the words in a piece of text."""
    tool_inputs.append(text)
    return len(text.split())


def test_tool_call_then_structured_answer():
    llm = StubLLM(responses=[
        tool_call("Word Counter", {"text": "bees make honey"}),  # researcher: use the tool
        "Final Answer: 3 words",                                 # researcher: answer
        {"title": "Bees", "bullets": ["honey"]},                 # writer: JSON for Brief
    ])
    researcher = Agent(role="Researcher", goal="Count words", backstory="Careful.",
                       tools=[word_counter], llm=llm)
    writer = Agent(role="Writer", goal="Write briefs", backstory="Concise.", llm=llm)
    count = Task(description="Count the words", expected_output="A count", agent=researcher)
    brief = Task(description="Write a brief", expected_output="Title and bullets",
                 agent=writer, output_pydantic=Brief)

    out = Crew(agents=[researcher, writer], tasks=[count, brief]).kickoff()

    assert tool_inputs == ["bees make honey"]               # the real tool ran
    assert "Observation: 3" in prompt_text(llm.calls[1])    # its result went back to the LLM
    assert llm.calls[0]["response_model"] is None           # agents WITH tools get no response_model
    assert llm.calls[2]["response_model"] is Brief
    assert out.pydantic == Brief(title="Bees", bullets=["honey"])

Rules that keep stub tests stable:

5. Testing a crew and asserting on prompts

Make the LLM injectable. For a factory function, take llm=None and pass it to every Agent (None means crewai's default model in production). For a scaffolded @CrewBase class, swap the LLM after .crew():

# tests/test_crew.py
from pathlib import Path

from acme_research.crew import AcmeResearch
from stub_llm import StubLLM, prompt_text


def crew_with(llm):
    crew = AcmeResearch().crew()
    for agent in crew.agents:
        agent.llm = llm
    return crew


def test_research_crew_output_and_prompts():
    llm = StubLLM(responses=["- bees pollinate crops", "# Bees report"])
    out = crew_with(llm).kickoff(inputs={"topic": "bees", "current_year": "2026"})

    assert out.raw == "# Bees report"
    assert [t.raw for t in out.tasks_output] == ["- bees pollinate crops", "# Bees report"]
    assert len(llm.calls) == 2

    first = prompt_text(llm.calls[0])
    assert "bees Senior Data Researcher" in first                   # {topic} filled in agents.yaml
    assert "the current year is 2026" in first                      # {current_year} in tasks.yaml
    assert "- bees pollinate crops" in prompt_text(llm.calls[1])    # task 1 output is task 2 context
    assert Path("report.md").read_text() == "# Bees report"         # output_file, in tmp cwd

llm.calls[i] holds messages (a snapshot of the system+user messages crewai sent), response_model, tools and response. prompt_text(call) joins the message contents. Useful things to assert on:

6. Testing guardrails

A function guardrail returns (True, value) to accept or (False, "error for the LLM") to reject. Test it two ways: as a plain function over a truth table of outputs, and inside a crew to check the retry. A guardrail that keeps failing raises Task failed guardrail validation after N retries after guardrail_max_retries retries (default 3). The truth-table and give-up tests are in example-tests.md section 1.

# tests/test_guardrails.py
from pydantic import BaseModel

from crewai import Agent, Crew, Task
from crewai.tasks.task_output import TaskOutput
from stub_llm import StubLLM, prompt_text


class Brief(BaseModel):
    title: str
    bullets: list[str]


def max_three_bullets(output: TaskOutput):
    try:  # output.pydantic can be None here, so fall back to parsing raw
        brief = output.pydantic or Brief.model_validate_json(output.raw)
    except ValueError:
        return (False, "Return JSON matching the Brief schema.")
    if len(brief.bullets) > 3:
        return (False, f"Use at most 3 bullets, you used {len(brief.bullets)}.")
    return (True, output.raw)  # a string: crewai re-runs the pydantic conversion


def brief_crew(llm):
    writer = Agent(role="Writer", goal="Write briefs", backstory="Concise.", llm=llm)
    task = Task(description="Write a brief", expected_output="Title and up to 3 bullets",
                agent=writer, output_pydantic=Brief,
                guardrail=max_three_bullets, guardrail_max_retries=2)
    return Crew(agents=[writer], tasks=[task])


def test_retry_feeds_the_error_back():
    llm = StubLLM(responses=[{"title": "T", "bullets": ["a", "b", "c", "d"]},
                             {"title": "T", "bullets": ["a"]}])
    out = brief_crew(llm).kickoff()
    assert out.pydantic == Brief(title="T", bullets=["a"])
    assert "Use at most 3 bullets, you used 4." in prompt_text(llm.calls[1])

Two traps:

• output.pydantic may be None inside the guardrail. When a task has a guardrail and the LLM returned text, crewai skips the pydantic conversion before the first guardrail call. An LLM returns text whenever the agent has tools, or when it is a custom LLM that returns strings. Parse output.raw yourself.
• return (True, output) can silently lose out.pydantic. If output.pydantic was None, crewai keeps that TaskOutput as-is. Return (True, output.raw) and crewai converts the string. Then assert out.pydantic is not None at the boundary.

7. Testing a flow

Flow methods are plain methods, so call them directly to test one step. Use kickoff(inputs=...) to test routing end to end. kickoff returns the return value of the last method that ran, and inputs keys overwrite state fields before @start runs.

# tests/test_flow_routing.py
from pydantic import BaseModel

from crewai.flow import Flow, listen, or_, router, start


class State(BaseModel):
    topic: str = ""
    log: list[str] = []


class TopicFlow(Flow[State]):
    @start()
    def normalize(self):
        self.state.topic = self.state.topic.strip().lower()

    @router(normalize)
    def choose(self):
        return "has_topic" if self.state.topic else "no_topic"  # labels differ from method names

    @listen("has_topic")
    def research(self):
        self.state.log.append("research")

    @listen("no_topic")
    def skip(self):
        self.state.log.append("skip")

    @listen(or_(research, skip))
    def finish(self):
        return f"done:{self.state.topic or '-'}"


def test_step_directly():
    flow = TopicFlow()
    flow.state.topic = "  Bees "
    flow.normalize()
    assert flow.state.topic == "bees"
    assert flow.choose() == "has_topic"


def test_both_branches_via_kickoff():
    flow = TopicFlow()
    assert flow.kickoff(inputs={"topic": " Bees "}) == "done:bees"
    assert flow.state.log == ["research"]

    flow = TopicFlow()
    assert flow.kickoff(inputs={"topic": "   "}) == "done:-"
    assert flow.state.log == ["skip"]

When a flow step builds a crew itself, as the crewai create flow scaffold does with ContentCrew().crew(), there is no parameter to pass a stub through. monkeypatch.setattr the crew class's crew method with a wrapper that calls the original and then sets agent.llm = stub on every agent. The full test is in example-tests.md section 2.

Flow is a pydantic model. If you own the flow code, declaring llm: Any = None on the flow class and passing MyFlow(llm=stub) is a cleaner seam than patching. A @listen("x") on a method named x raises when the flow is instantiated, so a test that only builds the flow catches it.

8. What crewai test actually does

crewai test is a model-graded evaluation, not a unit-test runner.

Use it for a periodic prompt-quality check against a real model. Do not use it as CI, because it costs money and its scores are not deterministic. Crew.test accepts a BaseLLM as eval_llm, so with stubs for both the crew and the judge it runs offline. That checks only the wiring (example-tests.md section 4).

9. Event listeners for debugging

Every kickoff emits typed events on crewai_event_bus: crew, task, agent, LLM call, tool, flow and memory events. Import them from crewai.events. crewai.utilities.events no longer exists.

Subclass BaseEventListener, implement setup_listeners(self, bus), and register handlers inside it with @bus.on(EventClass). Each handler has the signature handler(source, event). Instantiating the listener registers its handlers. example-tests.md section 3 has a debug listener (ToolUsageFinishedEvent, LLMCallCompletedEvent, TaskFailedEvent) and a recorder test.

In tests, register listeners inside with crewai_event_bus.scoped_handlers(): so they are removed afterwards. Call crewai_event_bus.flush() before asserting, because handlers run on worker threads.

• To debug a real run, put the listener in its own module and import it from main.py, so it is instantiated once. Other useful events: LLMCallFailedEvent, MemorySaveFailedEvent, CrewKickoffStartedEvent, FlowFinishedEvent.
• LLMCallCompletedEvent.response is the text, or for a native tool call the provider's list of tool-use blocks. event.model and event.usage carry the model and token counts.
• A custom BaseLLM must emit its own LLM call events. The built-in providers emit LLMCallStartedEvent / LLMCallCompletedEvent. A subclass that only returns text emits nothing, and the trace listener subscribes to the same events, so listeners and traces both miss every LLM call.

10. Tracing and telemetry switches

They are two separate systems. Telemetry is anonymous usage stats sent to CrewAI. Tracing is the execution trace (prompts, inputs, outputs, tool calls) you view in CrewAI AMP.

Tracing precedence: the tracing= argument, then CREWAI_TRACING_ENABLED, then the consent saved by crewai traces enable|disable. Saved consent is stored per project directory name (or per CREWAI_STORAGE_DIR), so renaming the checkout resets it. crewai traces status shows the env var (false when unset) and the saved consent, not whether you are logged in.

Rules:

• Disable observability only in tests, and only when the user chooses to. Put the switches in tests/conftest.py or the CI test job's environment. Never put them in the project's .env, main.py, or a deployment's env vars, because that blinds production debugging.
• Spell the values true / false. true is the only spelling every disable switch fully honors, and CREWAI_TRACING_ENABLED ignores yes/no/on/off.
• To debug one run, use CREWAI_TRACING_ENABLED=true crewai run or tracing=True. Where the trace goes depends on whether a CrewAI credential exists: CREWAI_USER_PAT, or a saved crewai login (no PAT needed).
  • With a credential: kickoff first requests a trace grant, then uploads the trace to CrewAI AMP under that login, with or without a terminal. 1.15.23 then prints a View traces: link; 1.15.22 prints none. If the grant request fails, kickoff raises TraceGrantError before the crew runs.
  • Without a credential, 1.15.23, interactive terminal: turning tracing on counts as consent. The trace uploads as an ephemeral trace with no prompt and an Ephemeral Execution Traces link is printed. Nothing is saved as consent.
  • Without a credential, 1.15.22, interactive terminal: crewai asks Share this execution trace with CrewAI? [y/N] after the run.
  • Without a credential, no terminal (CI, pytest, < /dev/null): the trace is discarded, on both releases.
  • No switch at all, first run: crewai auto-collects locally, asks Share this execution trace with CrewAI? [y/N] (20s timeout) in a terminal, and saves the answer as consent.
• From 1.15.23, crewai wraps every BaseLLM subclass's call in a rate-limit retry: a stub that raises an error mentioning "rate limit" or 429 is called 3 times over about 3 s before the error surfaces. To test your own rate-limit handling, raise a different error type, or assert on the retried call count.
• Traces contain task text, inputs and outputs. Use synthetic data in traced runs unless your data may leave the machine.

Common mistakes

Checklist

• tests/stub_llm.py copied from references/stub_llm.py; pytest added as a dev dependency
• conftest.py sets an absolute CREWAI_STORAGE_DIR before importing crewai and blocks the network; the observability switches are uncommented only if the user chose them
• Every agent's LLM is injectable (factory parameter, swap after .crew(), or patched crew class)
• Each structured task gets valid JSON from the stub, and tests assert out.pydantic is not None
• At least one test asserts on prompt text (inputs, context, tool listing)
• Each guardrail has a truth-table test plus a retry test
• Each flow router branch has a kickoff test; key steps are also called directly
• Real-model tests are marked credentialed and excluded from default CI
• Observability switches appear only in test config, never in .env or deployment env vars

References

• stub_llm.py - the offline deterministic StubLLM, plus tool_call() and prompt_text() helpers. Copy into tests/.
• Example tests - guardrail truth tables and retry exhaustion, a scaffolded flow that builds its crew inside a step, an event-listener test, and an offline Crew.test wiring check
• Public docs: https://docs.crewai.com/en/concepts/event-listener, https://docs.crewai.com/en/observability/tracing, https://docs.crewai.com/en/telemetry

For related skills:

• check-crewai-api - current Agent/Task/Crew API, structured output fields, guardrail contract
• build-flow - flow state, routers, or_/and_, persistence
• connect-tools-and-mcp - building the tools your stub drives
• design-task - writing guardrails and expected_output worth testing
• ask-docs - query the live docs when a version-sensitive row may have changed