A Simple Agent With Guidance and Local LLM

When trying to build a ReAct agent with Langchain, I found it hard to make it run correctly all the time. The model often distracts and does not follow the instruction strictly. This happens more in small models like Alpaca/Vicuna 3B-7B. The execution stops because of regex syntax errors. Making it strongly follow our instructions is really hard and you may take lots of time for optimizing your prompt. Therefore, we need to tool for controlling the output of LLMs. And that is exactly what guidance does!

Okay, let’s go into detail

Guidance is a tool from Microsoft that is described as “A guidance language for controlling large language models”. It allows you to control the output of LLM so which makes it easy to follow instruction prompts. For GPT3.5–4, we may see it works well with most instructions. But, if we are using small-size local LLM like LLaMa and its variants (Alpaca, WizardML, etc.), they might not always give a correct response. That is a big problem. Some frameworks like ReAct require a specific instruction format for response. Also, it will be hard to handle if your LLM sometimes provides JSON format data with a syntax error.

Let’s take an example:

valid_dish = ["Pizza", "Noodles", "Pho"]

# define the prompt
order_maker = guidance("""The following is a order in JSON format.
```json
{
    "name": "{{name}}",
    "age": {{gen 'age' pattern='[0-9]+' stop=','}},
    "delivery": "{{#select 'delivery'}}Yes{{or}}No{{/select}}",
    "order": "{{select 'order' options=valid_dish}}",
    "amount": {{gen 'amount' pattern='[0-9]+' stop=','}}
}```""")

# generate a character
order_maker(
    name="Alex",
    valid_dish=valid_dish
)

**Output**
The following is a order in JSON format.
```json
{
    "name": "Alex",
    "age": 25,
    "delivery": "Yes",
    "order": "Noodles",
    "amount": 10
}```

You can see, we can easily make a prompt, pre-define several things, and force the LLM to follow them strictly. You set the corrected regex for the age field with {{gen ‘age’ pattern=’[0–9]+’ stop=’,’}} , means it only takes numbers and ends with ,. The food type is also limited with a valid_dish by “{{select ‘order’ options=valid_dish}}”. For more details, I recommend you take a look at the official Github link.

A ReAct agent with Guidance and Wizard-Mega-13B-GPTQ

Note: you can find code in this section on this Github: https://github.com/QuangBK/localLLM_guidance

Before running, you need to have an LLM on your local PC. I use the wizard-mega-13B-GPTQ in this case. Feel free to try other models. First, we load the model and let Guidance use it.

model_para = 'YOUR_MODEL_DIR'
checkpoint_para = 'YOUR_MODEL_FILE'
model = load_quant(model_para, checkpoint_para, 4, 128)
model.to(DEV)
tokenizer = AutoTokenizer.from_pretrained(model_para)

llama = guidance.llms.Transformers(model=model, tokenizer=tokenizer, device=0)
guidance.llm = llama

Let’s test with a simple prompt. I use the ReAct format to check whether it works correctly or not.

valid_answers = ['Action', 'Final Answer']
valid_tools = ['Google Search']

prompt_template = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
Answer the following questions as best you can. You have access to the following tools:

Google Search: A wrapper around Google Search. Useful for when you need to answer questions about current events. The input is the question to search relavant information.

Strictly use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [Google Search]
Action Input: the input to the action, should be a question.
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question

For examples:
Question: How old is CEO of Microsoft wife?
Thought: First, I need to find who is the CEO of Microsoft.
Action: Google Search
Action Input: Who is the CEO of Microsoft?
Observation: Satya Nadella is the CEO of Microsoft.
Thought: Now, I should find out Satya Nadella's wife.
Action: Google Search
Action Input: Who is Satya Nadella's wife?
Observation: Satya Nadella's wife's name is Anupama Nadella.
Thought: Then, I need to check Anupama Nadella's age.
Action: Google Search
Action Input: How old is Anupama Nadella?
Observation: Anupama Nadella's age is 50.
Thought: I now know the final answer.
Final Answer: Anupama Nadella is 50 years old.

### Input:
{{question}}

### Response:
Question: {{question}}
Thought: {{gen 'thought' stop='\\n'}}
Action: {{select 'tool_name' options=valid_tools}}
Action Input: {{gen 'actInput' stop='\\n'}}
Observation:{{search actInput}}
Thought: {{gen 'thought2' stop='\\n'}}
Final Answer: {{gen 'final' stop='\\n'}}"""

def searchGoogle(t):    
    return search.run(t)

prompt = guidance(prompt_template)
result = prompt(question='Is Eminem a football player?', search=searchGoogle, valid_answers=valid_answers, valid_tools=valid_tools)

Wow, it works like a charm! Now, hope we can build a ReAct agent with it.

We define a tool list. In this case, I have one: searchGoogle. You can add more tools like calculator, SQL retriever, etc.

def searchGoogle(key_word):    
    return search.run(key_word)

dict_tools = {
    'Google Search': searchGoogle
}

Okay, now make a CustomAgentGuidance as:

class CustomAgentGuidance:
    def __init__(self, guidance, tools, num_iter=3):
        self.guidance = guidance
        self.tools = tools
        self.num_iter = num_iter

    def do_tool(self, tool_name, actInput):
        return self.tools[tool_name](actInput)
    
    def __call__(self, query):
        prompt_start = self.guidance(prompt_start_template)
        result_start = prompt_start(question=query, valid_answers=valid_answers)

        result_mid = result_start
        
        for _ in range(self.num_iter - 1):
            if result_mid['answer'] == 'Final Answer':
                break
            history = result_mid.__str__()
            prompt_mid = self.guidance(prompt_mid_template)
            result_mid = prompt_mid(history=history, do_tool=self.do_tool, valid_answers=valid_answers, valid_tools=valid_tools)
        
        if result_mid['answer'] != 'Final Answer':
            history = result_mid.__str__()
            prompt_mid = self.guidance(prompt_final_template)
            result_final = prompt_mid(history=history, do_tool=self.do_tool, valid_answers=['Final Answer'], valid_tools=valid_tools)
        else:
            history = result_mid.__str__()
            prompt_mid = self.guidance(history + "{{gen 'fn' stop='\\n'}}")
            result_final = prompt_mid()
        return result_final['fn']

The code is pretty clear. You start the prompt prompt_start_template with the instruction. Then, always check if the model can give the “Final Answer”. if it needs more information, try to search and provide related information with prompt_mid_template. After a maximum of 3 iterations, we force the agent to stop finding and give the “Final Anwer”. Let’s test with some questions:

custom_agent = CustomAgentGuidance(guidance, dict_tools)

list_queries = [
    "How much is the salary of number 8 of Manchester United?",
    "What is the population of Congo?",
    "Where was the first president of South Korean born?",
    "What is the population of the country that won World Cup 2022?"    
]

final_answer = custom_agent(list_queries[0])

**Output**
### Input:
How much is the salary of number 8 of Manchester United?

### Response:
Question: How much is the salary of number 8 of Manchester United?
Thought: I need to find out the number 8 of Manchester United.
Action: Google Search
Action Input: Who is the number 8 of Manchester United?
Observation: Bruno Fernandes is the current incumbent of the famous No. 8 shirt at Manchester United – but how does he rank among his predecessors?
Thought: I should find out Bruno Fernandes's salary.
Action: Google Search
Action Input: What is Bruno Fernandes's salary?
Observation: Bruno Fernandes joined Manchester United in January 2020 from Sporting CP, instantly increasing his salary from $59,717 to $289,418 per week. Apr 3, 2023
Thought: I now know the final answer.
Action: Google Search
Action Input: How much is Bruno Fernandes's salary?
Observation: Bruno Fernandes joined Manchester United in January 2020 from Sporting CP, instantly increasing his salary from $59,717 to $289,418 per week. Apr 3, 2023
Thought: I now know the final answer.
Final Answer: Bruno Fernandes's salary is $289,418 per week.

final_answer = custom_agent(list_queries[1])

**Output**
### Input:
What is the population of Congo?

### Response:
Question: What is the population of Congo?
Thought: First, I need to find the population of Congo.
Action: Google Search
Action Input: What is the population of Congo?
Observation: 95.89 million (2021)
Thought: Now, I should check the source of this information.
Action: Google Search
Action Input: Is 95.89 million the correct population of Congo?
Observation: From 1960 to 2021 the population of the Democratic Republic of the Congo increased from 15.25 million to 95.89 million people. This is a growth of 528.9 ... Democratic Republic of the Congo: Population size, in millions: For that indicator, ... minimum of 15.28 million in 1960 and a maximum of 95.89 million in 2021. the DR Congo 2020 population is estimated at 89,561,403 people at mid year according to UN data. the DR Congo population is equivalent to 1.15% of the total ... Missing: 95.89 | Must include:95.89. Population, Total. 95.89 million. (as of 2021) ; GNI, Atlas Method. 52.54 billion. (in current US$ as of 2021) ; GNI per capita, Atlas Method. 550. (in current US ... The Republic of Congo (Congo-Brazzaville), located in Central Africa, covers 342,000 km2. Its population of 5.7 million people is largely young, ... Missing: 95.89 | Must include:95.89. With a current population of about 89.56 million, DR Congo is projected to surpass 100 million in 2024 and double its population by 2047. DR Congo has one of ... Missing: 95.89 | Must include:95.89. Country Characteristics ; Population, Total. 95.89 million. (as of 2021) ; GNI, Atlas Method. 52.54 billion. (in current US$ as of 2021) ; GNI per capita, Atlas ... The Democratic Republic of the Congo (DRC) is the second largest country in Africa and has a total population of 67.8 million people, of which ... Missing: 95.89 | Must include:95.89. By land area, the DRC is the second-largest country in Africa, after Algeria, and the 11th-largest in the world. With a population of around 112 million, the ... Missing: 95.89 | Must include:95.89.
Thought: I now know the final answer.
Action: Google Search
Action Input: What is the final answer to the population of Congo?
Observation: The Republic of Congo (Congo-Brazzaville), located in Central Africa, covers 342,000 km2. Its population of 5.7 million people is largely young, ... DRC is among the five poorest nations in the world. In 2022, nearly 62% of Congolese, around 60 million people, lived on less than $2.15 a day. The Democratic Republic of the Congo (DRC) is the second largest country in Africa and has a total population of 67.8 million people, of which ... The number of internally displaced persons (IDPs) has been steadily rising in recent years with the current IDP Population at 6.17 million people (31 March 2022) ... The Pew Research Center estimates 95.8 percent of the population is Christian, 1.5 percent Muslim, and 1.8 percent report no religious ... The population is nearly 85 million, with over 12 million people living in the capital city, Kinshasa. The DRC is endowed with large mineral resources and ... The DRC has the largest population of internally displaced persons (IDPs) in Africa—nearly 6.2 million people as of February 2023—and the highest number of ... The current population of the DR Congo is 97,372,326 as of Wednesday, May 17, 2023, based on Worldometer elaboration of the latest United Nations data. the DR ... Missing: answer | Must include:answer. Populations in large swathes of eastern DRC have been living with conflict ... WFP's emergency crisis response currently covers seven of the most populous ... The Democratic Republic of the Congo (DRC) is a country located in Central Africa, and has a population of about 69.6 million people. DRC is endowed with ...
Thought: I now know the final answer.
Final Answer: The final answer to the population of Congo is 97,372,326 as of Wednesday, May 17, 2023.

Well, now you don’t have to worry about optimizing your prompts. Just give it to Guidance, and it will follow the instruction.