Self-Reflection Tutorial¶

Overview¶

Self-Reflection is a feature for the Utility Agents, which allows the agent to evaluate and iteratively improve its response before responding to the user query, thereby enhancing the overall quality and accuracy of the final output.

Goals¶

This tutorial will guide you through the following steps:

Get an overview of the Self-Reflection mechanism and its usage.
Create or modify a YAML configuration file.
Add Self-Reflection-related configuration to the Utility Agent you want to enable.
Develop your assistant and observe how the Self-Reflection mechanism:
- Evaluates responses,
- Iteratively improves them,
- And enhances the final output before responding to the user.

Self-Reflection Workflow¶

The workflow for Self-Reflection consists of the following steps:

Response Evaluation
The response generated by the Utility Agent is evaluated using a set of predefined criteria, such as correctness, completeness, and relevance. If the response meets the desired quality standard, it will be directly returned to the user.
Response Regeneration
If the response does not meet the standard, the Utility Agent will reflect on the evaluation feedback and regenerate a new response.
Iterative Refinement
Steps 1 and 2 are repeated up to a maximum number of Self-Reflection attempts which is predefined by the user, or until the response meets the quality standard.
Final Response Output
If a high-quality response is generated before reaching the maximum number of attempts, it will be directly returned to the user. If the maximum number of attempts is reached, the Utility Agent will select a final output based on one of the following user-defined modes:
- best: Selects the single best response from all generated attempts based on LLM evaluation.
- aggregate: Aggregates useful information from multiple previous responses into one.
- auto: Lets the LLM choose the final response generation strategy between best and aggregate.

Usage¶

The Self-Reflection feature can be enabled in a Utility Agent by adding the following fields under the self_reflection_config section of each Utility Agent entry in your project YAML file:

self_reflection: A boolean value (true or false) to turn Self-Reflection on or off.
max_attempts: The maximum number of times the agent can reflect and regenerate its response if it fails to meet the quality standard.
response_selection_mode: The strategy used to generate the final response if the maximum number of attempts is reached. Options include best, aggregate, and auto.
return_internal_reflection_msg: If set to true, the internal responses before Self-Reflection ends will also be returned to the user.

The above configurations are all optional. When the Self-Reflection config is not explicitly provided in the YAML file, the values will fall back to the default settings.

Currently Supported Utility Agent¶

Analytics Agent
Critical Thinker Agent
Planning Agent
Research Agent
Search Agent
Base Utility Agent

Each agent has different default values for Self-Reflection configs. Please refer to the detailed Utility Agent page for each supported agent listed above to check.

Steps¶

1. Configuration file¶

To enable Self-Reflection on a supported Utility Agent, you need to create a YAML file with the Self-Reflection configuration. Here's a sample configuration file config.yaml which adds Self-Reflection to the Search Agent:

orchestrator:
  agent_list:
    - agent_name: "Search Agent"

utility_agents:
  - agent_class: SearchAgent
    agent_name: "Search Agent"

    config: 
      self_reflection_config: # Optional
        self_reflection: True # Optional, a boolean value (true or false) to turn Self-Reflection on or off
        max_attempts: 3 # Optional, the maximum number of times the agent can reflect and regenerate its response if it fails to meet the quality standard
        response_selection_mode: "auto" # Optional, the strategy used to generate the final response if the maximum number of attempts is reached. Options include "best", "aggregate", and "auto".
        return_internal_reflection_msg: True # Optional, if set to true, the internal responses before Self-Reflection ends will also be returned to the user.

2. Python file¶

Now, you can start the development of your assistant using these lines of code:

from dotenv import load_dotenv
from air import login, DistillerClient
import os

load_dotenv() # This loads your ACCOUNT and API_KEY from your local '.env' file

login(
    account=str(os.getenv("ACCOUNT")),
    api_key=str(os.getenv("API_KEY")),
    oauth_server=os.getenv("OAUTH_SERVER", ""),
)

# Initialize the Distiller client
distiller_client = DistillerClient()

project = "search_agent_self_reflection"

# Create the project using the YAML file prepared in Step 1
distiller_client.create_project(
    config_path="config.yaml",
    project=project
)

# Start an interactive session with the project just created
response = distiller_client.interactive(
    project=project,
    uuid="test_user",
)

Sample Queries¶

Here are some sample queries for testing, along with example outputs generated by the Search Agent with Self-Reflection enabled, using the YAML configuration defined in Step 1:

⚠️ Note: The responses you get may vary each time you run the same query due to the inherent randomness of large language models (LLMs).

Query: On what date, month, and year was the Jonas Mekas Visual Arts Center opened by avant-garde filmmaker Jonas Mekas with its premiere exhibition entitled "The Avant-Garde: From Futurism to Fluxus"?
(Correct answer: November 10, 2007)

  [SELF-REFLECTION - Search Agent] Initial Response Before Self-Reflection
  Searching over Web Search
  # Jonas Mekas Visual Arts Center Opening Date

  The Jonas Mekas Visual Arts Center was opened by avant-garde filmmaker Jonas Mekas in November 2007. Its premiere exhibition, entitled 'The Avant-Garde: From Futurism to Fluxus,' marked the beginning of the center's mission to return heritage Lithuanian émigré art to Lithuania and create an environment for Lithuanian artists to be part of European and global culture.


  [SELF-REFLECTION - Search Agent] Attempt 1 / 3
  Searching over Web Search
  # Jonas Mekas Visual Arts Center Opening Date

  The Jonas Mekas Visual Arts Center was opened by avant-garde filmmaker Jonas Mekas on November 10, 2007. Its premiere exhibition, entitled 'The Avant-Garde: From Futurism to Fluxus,' marked the beginning of the center's mission to return heritage Lithuanian émigré art to Lithuania and create an environment for Lithuanian artists to be part of European and global culture.

  ## References
  [1] https://jonasmekas.com/diary/m=201210
  [2] https://www.sfaq.us/2019/01/jonas-mekas-conversation-with-marianne-shaneen/
  [3] https://mikrotext.de/book/jonas-mekas-bum-ba-conversations-with-john-lennon-yoko-ono/
  [4] https://www.govilnius.lt/visit-vilnius/places/jonas-mekas-visual-arts-centre/
  [5] https://jonasmekas.com/bio.php


  [SELF-REFLECTION - Search Agent] Final Response After Self-Reflection
  # Jonas Mekas Visual Arts Center Opening Date

  The Jonas Mekas Visual Arts Center was opened by avant-garde filmmaker Jonas Mekas on November 10, 2007. Its premiere exhibition, entitled 'The Avant-Garde: From Futurism to Fluxus,' marked the beginning of the center's mission to return heritage Lithuanian émigré art to Lithuania and create an environment for Lithuanian artists to be part of European and global culture.

  ## References
  [1] https://jonasmekas.com/diary/m=201210
  [2] https://www.sfaq.us/2019/01/jonas-mekas-conversation-with-marianne-shaneen/
  [3] https://mikrotext.de/book/jonas-mekas-bum-ba-conversations-with-john-lennon-yoko-ono/
  [4] https://www.govilnius.lt/visit-vilnius/places/jonas-mekas-visual-arts-centre/
  [5] https://jonasmekas.com/bio.php

Query: On what day, month, and year did the Brazilian mathematician Leopoldo Luis Cabo Penna Franca marry Ana Cristina Leonardos?
(Correct answer: July 28, 1983)

  [SELF-REFLECTION – Search Agent] Initial Response Before Self-Reflection
  Searching over Web Search
  Unfortunately, the retrieved documents do not contain any relevant information related to your query.

  [SELF-REFLECTION – Search Agent] Attempt 1 / 3
  Searching over Web Search
  # Marriage Date of Leopoldo Luis Cabo Penna Franca and Ana Cristina Leonardos

  Unfortunately, the retrieved documents do not contain any relevant information related to your query.

  [SELF-REFLECTION – Search Agent] Attempt 2 / 3
  Searching over Web Search
  # Marriage Date of Leopoldo Luis Cabo Penna Franca and Ana Cristina Leonardos

  Unfortunately, the retrieved documents do not contain any relevant information related to the marriage date of Leopoldo Luis Cabo Penna Franca and Ana Cristina Leonardos. However, it is mentioned that Leopoldo Luis Cabo Penna Franca and Ana Cristina Leonardos were a couple who divorced in 1988.

  [SELF-REFLECTION – Search Agent] Attempt 3 / 3
  Searching over Web Search
  # Marriage Date of Leopoldo Luis Cabo Penna Franca and Ana Cristina Leonardos

  Leopoldo Luis Cabo Penna Franca married Ana Cristina Leonardos on 28 July 1983.

  [SELF-REFLECTION – Search Agent] Triggering 'auto' mode to generate final response

  [SELF-REFLECTION – Search Agent] Final Response After Self-Reflection
  # Marriage Date of Leopoldo Luis Cabo Penna Franca and Ana Cristina Leonardos

  Leopoldo Luis Cabo Penna Franca married Ana Cristina Leonardos on 28 July 1983.