Skip to main content

​
Step 1: Write a Simple Test​

Testpilot runs tests that are written in a pilotplan format. For this example we’ll use YAML, but Testpilot can also recognize pilot files written in TOML or JSON: Here is a simple test that will go to Hacker News, search for a topic, and click the first result. Copy these contents to a yaml file titled example-hn-search.pilot.yaml. 
name: "Hacker News Search Test"

# List of Cases to Execute
cases:
  - id: "testcase_01jy2m0vh8f05vv5qmkkxf6zxt"
    name: "Search for Devbox on Hacker News"
    description: "Navigate to news.ycombinator.com, search for Devbox, and click the first search result"
    url: "https://news.ycombinator.com"
    
    # List of Test Steps
    steps:
      - "Use the search bar on the front page to search for Devbox"
      - "Verify search results for 'Devbox' are displayed"
      - "Click on the first search result link"
      - "Verify the clicked post or article has loaded correctly"
Tests are provided to Testpilot as a list of cases, where each case executres one or more steps. When Testpilot executes a test case, it opens the url of the test case in a new browser or device context, and begins executing the steps.

​
Step 2: Running your Test​

You can run your test files using testpilot test <path_to_file>. Testpilot runs your tests in headless mode by default, but you can use --show-browser to watch what Testpilot is doing live: 
testpilot test example-hn-search.pilot.yaml --show-browser
Testpilot will launch the browser and begin executing your testplan live. You can view the LLM’s actions and reasoning in your terminal: 
❯ testpilot test example-hn-search.pilot.yaml
Info: Found 1 pilot plan
-> Test Case: Search for Devbox on Hacker News
Starting step 1/4: Use the search bar on the front page to search for Devbox
  📸 Action: screenshot
        Reasoning: Scrolling to find search bar
  ↕️ Action: scroll at [513, 742]
        Reasoning: Clicking search bar, entering "Devbox"
  🖱️ Action: left_click at [524, 742]
        Reasoning: Typing "Devbox" in search bar
  ⌨️ Action: type 'Devbox'
        Reasoning: Initiating search for "Devbox"
  🎹 Action: keypress
        Reasoning: Displaying successful search results for Devbox
  📸 Action: screenshot
Finished step 1/4: Use the search bar on the front page to search for Devbox
  🏁 Action: Conclude Step (Success)
Starting step 2/4: Verify search results for 'Devbox' are displayed
  📸 Action: screenshot
        Reasoning: Confirming successful search for Devbox
  📸 Action: screenshot
  🏁 Action: Conclude Step (Success)
Finished step 2/4: Verify search results for 'Devbox' are displayed
Starting step 3/4: Click on the first search result link
  📸 Action: screenshot
        Reasoning: Clicking first search result link
  🖱️ Action: left_click at [184, 81]
        Reasoning: Accessed Devbox page, confirming content
  📸 Action: screenshot
Finished step 3/4: Click on the first search result link
Starting step 4/4: Verify the clicked post or article has loaded correctly
  🏁 Action: Conclude Step (Success)
  📸 Action: screenshot
        Reasoning: Confirming page content relevance, successful.
  📸 Action: screenshot
  🏁 Action: Conclude Step (Success)
Finished step 4/4: Verify the clicked post or article has loaded correctly

​
Step 3: Viewing your Test Results​

Once your test is finished, a static HTML report with the results, screenshots, and video of the test will be written to testpilot-out. You can view the results by opening the index.html at the root of the folder: 
open testpilot-out/index.html
Test Output Example