Skip to main content
testpilot test [path/to/plan.pilot.yaml | path/to/directory/with/pilot/plans] [flags]

Examples

# Run all tests in the current directory
testpilot test

# Run tests from a specific pilot plan file
testpilot test ./my-test.pilot.yaml

# Run tests from a specific directory
testpilot test ./tests/

# Run only tests with names containing "login"
testpilot test --filter login

# Run tests and save reports to a custom directory
testpilot test --outdir ./test-results

# Run tests in shard 1 of 3 total shards
testpilot test --shard 1/3

Options

OptionDescription
-f, --filter stringonly run tests with names containing the given string
-h, --helphelp for test
--concurrency intnumber of tests to run concurrently (default 10)
--driver stringdriver to use for the test (playwright or appium) (default “playwright”)
—launch-vars stringToStringEnvironment variables passed when launching the application under test. Currently only applies to iOS apps and tests (default [])
--os stringoperating system to run tests on (local, linux, macos, windows) (default “local”)
--os-version stringoperating system version to run tests on (e.g., “13”, “18.3”)
--outdir stringDirectory in which to save reports (default “testpilot-out”)
--runtime stringruntime environment to execute the test in (chromium, firefox, webkit, native) (default “chromium”)
--shard valueSplit tests into n shards and only run tests in shard x
--udid valuespecify the UDID of the iOS device you want to test. Note: this cannot be used along with --os-version

Advanced Options

The following options are available but not commonly used:
OptionDescription
--show-browsershow the browser
--viewport intsscreen dimensions to use for your web browser (playwright driver only)

Environment Variables

The following environment variables are required or affect the behavior of the test command:
VariableRequiredDescription
OPENAI_API_KEYYesOpenAI API key for LLM interactions
ANTHROPIC_API_KEYNoAnthropic API key (optional)
OPENAI_BASE_URLNoCustom OpenAI API base URL

SEE ALSO

I