Skip to main content

Testbot

Testbot is an LLM agent in the Tesmon platform designed to streamline both front-end UI flows and back-end processes, including API, Kafka, and database interactions. By learning from user interactions during test recordings, Testbot creates assertion-free tests. It trains models based on the sequence of events within these sessions, significantly reducing the manual effort required for test creation and maintenance, thus pushing towards the goal of near-zero effort testing.

How Testbot Works​

During a test recording session, Testbot captures the sequence of user interactions and the associated data flow. When a test is initiated, Testbot utilizes a model that has been trained on these tasks to predict and verify the outcomes, automating the validation process. Here’s what happens during this process:

  • Model Training: Testbot trains on the sequences of tasks performed during the recording to understand the expected input and output.
  • Test Execution: Upon running a test, Testbot uses the trained data to automatically verify both inputs and outputs, looking for discrepancies.
  • Error Handling: If a test encounters scenarios that were not covered during training, Testbot will report an error and provide guidance on how to re-train or adjust the test.
  • Handling Changes: When changes occur, such as updates to API responses or data models, Testbot can prompt the user to accept these changes through a simple interface, usually with a single click to "Accept Changes". This feature is particularly useful for adapting tests to expected changes without manual intervention.

Testbot Vision: Beyond Pixel Comparison​

Unlike traditional visual testing tools that rely on pixel-based comparisons, Testbot incorporates advanced vision capabilities to analyze the content of screenshots taken during tests. This feature, known as Testbot Vision, allows it to understand and validate visual elements contextually, marking a significant leap in visual testing.

Dynamic Field Handling​

Testbot is adept at recognizing and managing fields that frequently change, such as IDs, timestamps, or specific headers. To prevent these dynamic fields from causing unnecessary test failures, Testbot is configured to ignore changes in these areas, ensuring that fluctuations do not affect the overall test outcomes.

Handling Non-Dynamic Field Changes​

For fields that are static and expected to remain consistent, Testbot maintains strict assertions. Any changes in these non-dynamic fields during testing are flagged as failures, ensuring that only deliberate and verified changes are accepted. This approach helps maintain the integrity and reliability of your test results.

Example Test​

tasks:
  - action: actions.http-client-1716866461
    testbot: false
    testbotSpec: body.id should be exactly equal to 100
    validate: assert 1 == 1

Enable/Disable Testbot​

The Testbot can be enabled or disabled at either the test level or the task level. Set it to true or false accordingly. If disabled at the test level, it will override any true setting at the task level.

testbot: false

Example​

testbot: false
tasks:
  - action: actions.http-client-1716866461
    testbot: false
    testbotSpec: body.id should be exactly equal to 100
    validate: assert 1 == 1

Instructing Testbot​

You can define specific rules for the Testbot to determine whether a test passes or fails. Specify these requirements for a task as follows:

tasks:
  - action: actions.http-client-1716866461
    testbotSpec: body.id should be exactly equal to 100