Evaluations

Use evaluations to assess how agents and agentflows perform against a dataset. Add one or more evaluators to measure result quality. Run the evaluation, review each evaluation instance, and identify areas for improvement. Evaluations provide a consistent way to test an agentflow across multiple test cases. You can use the results to confirm expected behavior, identify responses that need improvement, and check the impact of changes made to an agentflow.

When to use evaluations

Use an evaluation when you want to:

• Check whether an agentflow returns the expected result.
• Measure response quality against defined criteria.
• Test multiple inputs without reviewing each response manually.
• Identify test cases where an agentflow does not perform as expected.
• Test an agentflow after changing its instructions, actions, or configuration.
• Compare results from different evaluation runs.

An evaluation includes the following components:

• Agentflow: The agentflow you want to test. For more information, see Agents.
• Dataset: A collection of test cases used to test the agentflow. For more information, see Dataset.
• Evaluator: The method used to assess the response for each test case. For more information, see Evaluations.
• Evaluation instance: A record of an evaluation run and its results. For more information, see Evaluation instances.

Access the evaluations page

The evaluation list is displayed on the Evaluations page.

1. Go to Agents > Evaluations.

2. Click the Evaluations tab.
   The page displays the evaluations available to you and information about the most recent run.

Create an evaluation

Create an evaluation to test an agentflow against the test cases in a dataset.

Before creating an evaluation, make sure the agentflow you want to test is available. You must also select an existing dataset or create a dataset during the evaluation setup.

1. Go to Agents > Evaluations.

2. Click Create evaluation.

3. In Evaluation name, enter a name for the evaluation.
   Include information such as the purpose, version, date, or experiment name to help distinguish the evaluation from other evaluations. For example: Q4 quality check - customer support.

4. In Description, enter information about the purpose of the evaluation.

5. In Agentflow, search for and select the agentflow you want to evaluate.

6. In Dataset, search for and select the dataset that contains the test cases. To create a dataset, click Create dataset. For more information about creating a dataset, see Dataset.

7. Under Evaluators, select one or more evaluators. For more information, see Select evaluators.

8. Complete the required settings for each selected evaluator.

9. Click Create evaluation.
   The evaluation is added to the Evaluations page.

Select evaluators

Select evaluators based on the type of response you want to assess. You can add up to 10 evaluators to an evaluation.

LLM as a judge

Use LLM as a judge to assess responses against criteria that you define. The evaluator reviews the selected inputs, actual outputs, and expected outputs, and then determines whether the response meets the criteria. Use this evaluator when different responses can still be correct or when you need to assess qualities such as accuracy, relevance, completeness, clarity, or reasoning.

For example, a support ticket agentflow might generate a summary using different words from the expected summary. LLM as a judge can assess whether the generated summary still describes the correct issue, includes the important details, and avoids unsupported information.

Use clear criteria that describe what the response must include or avoid.

Configure an LLM as a judge evaluator

After selecting LLM as a judge, configure how the evaluator assesses the agentflow responses.

1. In Evaluator name, enter a name that describes what the evaluator measures. For example: Decision accuracy evaluator, Summary quality.

2. In Description, enter a brief explanation of the evaluator’s purpose and method.

3. In Provider, select the AI model provider.

4. In Connection, select the connection used to access the provider.

5. In Evaluation prompt, describe how the evaluator must assess each response.

   Include clear requirements, such as decision accuracy, reasoning quality, hallucination detection, or completeness.

6. In Input columns, select the dataset columns that contain the information provided to the agentflow.

7. In Actual outputs, select the columns that contain the responses produced by the agentflow.

8. In Expected outputs, select the dataset columns containing the expected response. Expected outputs provide reference information that the evaluator can use when assessing the agentflow response.

9. Set the Pass rate threshold. This is the minimum score required for the evaluation to pass.

10. Under Criteria, define at least one of the following:

    • Pass criteria: Describe the conditions that the response must meet to pass.

    • Fail criteria: Describe the conditions that cause the response to fail.

11. To assess another aspect of the response, click Add evaluator and configure the additional evaluator.

Exact match

Use Exact match to compare the agentflow response with the expected response in the dataset. You can compare responses using an exact or case-insensitive match.

Use this evaluator for responses that should return a specific value, such as:

• Categories
• Labels
• Identifiers
• Status values
• Yes or no responses

Exact match is not suitable for responses where different wording, punctuation, spacing, or formatting can still be correct. Use LLM as a judge for these responses. Use Exact match only when the output must follow a defined value or format. Run the evaluation with representative test cases and review the results before setting the final pass rate threshold. A 100% threshold is appropriate only when every response must match.

Configure an Exact match evaluator

After selecting Exact match, configure the columns and pass rate used to compare the responses.

1. In Evaluator name, enter a name that describes the value being compared.

2. In Description, enter a brief description of what the evaluator measures.

3. In Expected Output Column, select the dataset column containing the expected value.

4. In Actual Output Column, select the column where the agentflow stores the generated value.

5. Set the Pass Rate Threshold. This is the minimum percentage of matching responses required for the evaluation to pass.

6. To add another evaluator, click Add evaluator.

Manage evaluations

You can search, open, and manage evaluations from the Evaluationspage. Each evaluation in the list has an options menu that displays the actions available for that evaluation. To open the options menu, find the evaluation you want and click on the right.

Search for an evaluation

1. Go to Agents > Evaluations.
2. Click the Evaluations tab.
3. In Search evaluations, enter the evaluation name.
   The list displays evaluations that match your search.

Open an evaluation

1. Go to Agents > Evaluations.
2. Click the Evaluationstab.
3. Click the name of the evaluation you want to open.

Refresh the evaluation list

To display the latest evaluation information:

1. Go to Agents > Evaluations.
2. Click Refresh.

View datasets

To view the datasets available for evaluations:

1. Go to Agents > Evaluations.
2. Click the Datasets tab.

View evaluation instances

To view records of evaluation runs:

1. Go to Agents > Evaluations.
2. Click the Evaluation instances tab.