Composability in LLM Apps

Exploring prompt templates and chaining with Slack and OpenAI

The LLM world is moving super fast right now and lots of teams are building tools and scaffolding for this new flavor of applications. While the out-of-the-box results you get with GPT-3 and ChatGPT are incredible, for anything but the most simple applications, we need composability. At a minimum, the ability to inject dynamic data into prompts and chain LLM processes together.

The example I’m going to dive into is a chatbot-style onboarding experience powered by AI. The AI assistant converses with new users, learns more about them, and then dynamically connects them to resources within the community — in this case the different channels in Slack.

conversational flow

The stack we’re using is OpenAI’s GPT-3 and the new ChatGPT API, and the Slack API which has robust support for conversations, bots, social graphs, and channels.

The code for the project is on GitHub here (pull requests welcome!)

ChatGPT and the new gpt-3.5-turbo model.

OpenAI’s general release of the ChatGPT API last week was another great catalyst for this ecosystem. The new gpt-3.5-turbo model can be used in the ChatCompletion endpoint, and right from the get-go it's pretty great. By taking the most recent messages in a Slack channel you can easily create context for ChatGPT.

response = slack_web_client.conversations_history(channel=channel_id, limit=20, include_all_metadata=False)
for m in response["messages"]:
   if ("subtype" in m): 
      continue
   role = "assistant" if (m["user"] == bot_user['user_id']) else "user"
   prompt_messages.insert(0, {"role":role, "content":m["text"].rstrip()} )

We need to provide instructions to the bot using the “system” role passed in as part of the messsages array. The instructions set up the basic context, which includes among other things the fact that the bot is running in Slack.

The ACME Network is a member, invite only communinity for special events.
You are an AI assistant that helps onboard new members, and guide them in navigating the Network, bringing a smart entrepreneurial spirit to everything you say. 
You should prompt new members for their interests, professional and personal so you can know them to serve them.
You should make introductions to other members, and reference them in responses where appropriate.
You should be open to engaging in conversation about relevant business topics.
If it seems like the member is still composing a thought or the conversation is over, you should remain silent or respond that you're still listening. 
You happen to be integrated with Slack so can use emojis such as :raised_hands: and :wink:.

Make the call to the new ChatCompletion endpoint with the messages array argument.

channel_context = load_prompt_context("onboarding")
prompt_messages.insert(0, {"role":"system", "content":channel_context})
response = openai.ChatCompletion.create(
   model="gpt-3.5-turbo",
   messages=prompt_messages
)

And you’re up and running with good conversational continuity.

A bit verbose, but pretty cogent.

Dynamic Prompts

This is interesting, but pretty much the same thing you’d get from a vanilla ChatGPT interaction. What if you want to integrate the bot with your own app and dynamic data?

A simple next step is to make the prompts dynamic by injecting data at runtime. In this demo, in order to provide more relevant recommendations to users, we’re pulling the channels from the Slack API and injecting them into the system prompt.

The updated prompt adds additional instructions to the completion, and Jinja2-style markup to inject the values pulled from the Slack API.

available_channels = slack_web_client.conversations_list(types="public_channel", exclude_archived=True)
prompt_context = jinja2.Template(prompt_context).render(channels=available_channels["channels"])
prompt_messages.insert(0, {"role":"system", "content":prompt_context})

response = openai.ChatCompletion.create(
  model="gpt-3.5-turbo",
  messages=prompt_messages
)

The prompt is updated using templating markup, with the channel data replaced by the Slack channels loaded from the API.

You should refer users to the relevant channels defined below ("CHANNELS").
Do not make up or hallucinate channels.

CHANNELS:
{% if channels %}
{% for c in channels %}
<#{{c.id}}|{{c.name}}>:{{c.purpose.value}}
{% endfor %}
{% endif %}

Now in the responses, we’re getting intelligent recommendations on channels included along with direct links properly formatted for the Slack API.

GPT-3 speaks JSON

Another element of composability is the ability of the GPT-3 completion outputs to be in machine-readable form so these outputs can be wired into other processes. In our case, we want to use GPT-3 to generate profiles for users in our community based on their conversations in the channel. Why not have GPT-3 directly generate the profiles as JSON so we can then store them in memory or some memory cache database?

The prompt we’re using specifies in a few-shot learning style the exact schema for the JSON output we want. Again, we use Jinja2 templating to inject the actual chat messages loaded from the Slack API.

Generate profiles about each user based on their chat history.
Use the description field in the JSON object to summarize their interests and anything else you can glean about them from what they say.

CHAT:
<@U50101>: I'm a CTO at a large fitness brand based in Europe. Interested in all things sports, technology, and fitness tracking. When I can find the time, I'm out cycling and mountain climbing in the Alps.
<@U5B232>: I've been looking into investments in China and Asia more broadly. Primarily Investment banking. Also interested in Emerging tech, blockchain and crypto.
<@AI>: We'll make a note of that. Where are you based?
<@U5B232>: I'm based in Washington, DC, but I travel often to Hong Kong and London.

PROFILES:
```json
{
    "U50101": 
    {
        "description":"CTO based in Europe whose professional interests include Sports, Technology and Finance. Personal interests include cylcing and mountain climbing."
    },
    "U5B232":
    {
        "description":"Interests include China and Asia, Investment Banking, Emerging tech, blockchain, and crypto. Based in Washington, D.C. but travels to London and Hong Kong"
    }
}
```

CHAT:
{% if messages %}
{% for m in messages %}
<@{{m.user}}>:{{m.text}}
{% endfor %}
{% endif %}

PROFILES:

Now GPT-3 knows the schema to return the response in, and has the dynamically injected chat history. Magically, it generates impressive descriptions for each user, and directly into the desired JSON schema.

{
    "U04N6U1E7FA": 
    {
        "description": "Interests include climate tech, and sustainability. Based in Washington, D.C. and interested in connecting with other members in the ACME Network community. Professional and open to forming personal connections."
    }
}

Next Steps

I feel like this is a new way of thinking about coding apps and my mental model is still evolving. The only way to do that is to build and push against the unknowns.

Next up for me will be looking at some of the tools out there designed to facilitate this kind of composability, like langchain and Dust.tt. One of the reasons I didn’t use these frameworks in this first go is the underlying APIs are changing fast, and like any higher-order abstraction framework, they’re going to be a step behind (eg. as I was building this, the ChatGPT API was made publically available, but not yet supported in either of these tools).

Other things I’m looking at are using embeddings to do things like search against profiles, as well as moving from Slack to the discord API.

#buildinpublic

Thanks to Omar Pera, Rollen Gomes, and Michael Piccuirro for their feedback on this post.

I’m CTO at R/GA and a partner at R/GA Ventures, where I help companies big and small build and future-proof digital businesses.