Chat History

Example Code

Examples from this module can be found on GitHub

Chatting with Jupyter

We can also use the OpenAI SDK for Python in a Jupyter Notebook. Create a new notebook as described in Python Development. In the first cell, import the os module and the OpenAI class from the openai package:

import os
from openai import OpenAI

In the next cell, retrieve the GH_MODELS_PAT (or the name you used for the secret in the previous module)

GH_MODELS_PAT = os.getenv("GH_MODELS_PAT")

In a new cell, create an instance of the OpenAI client:

client = OpenAI(
    base_url="https://models.github.ai/inference",
    api_key=GH_MODELS_PAT
)

In the next cell, create a function that accepts a prompt and returns the response from the client:

def ask_client(prompt):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}]
    )
    return response.choices[0].message.content

In the next cell, ask the client what year Microsoft was founded:

```python
ask_client("What year was Microsoft founded?")

The model should respond with something like “Microsoft was founded in 1975.”

In the next cell, ask what year the company went public:

ask_client("What year did the company go public?")

Implementing Chat History

The response has no idea what company you’re referring to! Prompts to an LLM are a lot like HTTPS requests. In other words, they are stateless. The model doesn’t keep track of previous requests. So we have to do that ourselves. Remember that you can pass a list of messages to the messages parameter of the create function. As you saw in the previous module, each message is a dictionary with a role and content key. You also saw the system and user roles for the system prompt and the user prompt. There’s also an assistant role for responses from the LLM. We can create a list of messages with the system prompt. As we chat with the model, we can append the user prompts and assistant responses to the list of messages. Then each time we chat with the model, we can pass the entire history of messages. This is how the model can use the context from the conversation to provide more relevant responses.

First, in a new cell, create a list of messages with the system prompt for a tech industry analyst from 05_chat_history/tech_analyst_prompt.txt:

with open("05_chat_history/tech_analyst_prompt.txt") as f:
    system_prompt = f.read()

history = [
    {
        "role": "system",
        "content": system_prompt
    }
]

In the next cell, create a new function, ask_client_with_history. The function should accept a prompt and append it to the list of messages. Then call the create function and pass the history to the messages keyword argument. Get the response from the model and append it to the history with the assistant role. Next, return the response from the model:

def ask_client_with_history(prompt):
    history.append({
        "role": "user",
        "content": prompt
    })
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=history
    )
    assistant_message = response.choices[0].message.content
    history.append({
        "role": "assistant",
        "content": assistant_message
    })
    return assistant_message

In a new cell, ask the model what year Microsoft was founded, using the ask_client_with_history function:

ask_client_with_history("What year was Microsoft founded?")

The model should have a similar response to the previous time you asked.

Now in a new cell, print the history variable to see the list of messages:

print(history)

The next time you call ask_client_with_history, this is the list of messages that will be passed to the model. Now ask the model about Microsoft going public:

ask_client_with_history("What year did the company go public?")

The model will be able to understand the “the company” is referring to “Microsoft” because of the context provided by the message history. It will say something about 1986 being the year Microsoft went public.

A Web ChatBot

The Jupyter Notebook gives you an idea of how the chat history can be implemented. In a real world application, you’re not going to expect the user, especially non-technical users, to interact with a Jupyter Notebook. Most users who interact with a chatbot are used to a conversational UI like you might find in ChatGPT.

In this section, we will implement a backend REST API with FastAPI that accepts a prompt and return the response from the LLM. The backend will also manage the chat history. Then I have provided a simple React frontend that can be used to send prompts and display the responses.

Note:

If you want to learn more about Fastapi, check out the FastAPI QuickStart which is part of the Python QuickStart.

In the folder 05_chat_history, there is a fastapigpt/backend folder with the starter code for the FastAPI backend in api.py. Most of this code is boilerplate for setting up the FastAPI application. The GITHUB_TOKEN constant will be retrieved from GitHub Secrets using the os module. The MODEL and SYSTEM_PROMPT you can change or use the values provided.

For the first TODO comment, initialize the OpenAI client just like you did in the Jupyter Notebook, using the GITHUB_TOKEN for the api_key and the same base_url as before.

client = OpenAI(
    base_url="https://models.github.ai/inference",
    api_key=GITHUB_TOKEN
)

The next TODO comment is a placeholder for the chat history. Again, create variable called chat_history and that a list of dict that have keys and values as strings. Then initialize it with a message for the system prompt:

chat_history: list[dict[str, str]] = [
    {
        "role": "system",
        "content": SYSTEM_PROMPT
    }
]

The final set of TODO comments is a placeholder for the impkementation of the chat function.

@app.post("/api/chat", response_model=ChatResponse)
def chat(request: ChatRequest) -> ChatResponse:

Notice that the chat function is decorated with the FastAPI @app.post("/api/chat") decorator. This tells FastAPI to invoke the chat function whenever a POST request is made to the /api/chat endpoint. The chat function expects a JSON payload with a prompt key which will be transformed into a ChatRequest object to be used inside of the function. Also notice the chat function returns a ChatResponse object which has a field for the response from the model. FastAPI will transform the ChatResponse object into a JSON response to be sent back to the client.

Now for the implementation. First, get the user prompt from the ChatRequest objeect and append it to the chat_history list in a dict with the role set to user:

user_message = {"role": "user", "content": request.prompt}
chat_history.append(user_message)

Next, call the client.chat.completions.create function and pass it the MODEL and the chat_history list. Then get the content from the response.

response = client.chat.completions.create(
    model=MODEL,
    messages=chat_history
)
content = response.choices[0].message.content

Then append the model’s response to the chat_history list in a dict with the role set to assistant:

assistant_message = {"role": "assistant", "content": content}
chat_history.append(assistant_message)

Finally, return the model’s response wrapped in a ChatResponse object:

return ChatResponse(response=content)

Note:

You can find the complete code for api.py in the 05_chat_history/finished/fastapigpt/backend folder.

Try Out the API

Save the starter.py file. Run the API with the command:

uvicorn starter:app --reload

The server will start on localhost and listen on port 8000. You can test the API using the built in API docs at http://localhost:8000/docs.

Expand the POST /api/chat endpoint and click the Try it out button.

The textarea contains JSON with a prompt key, the same as the ChatRequest object in the code. Change the value of the prompt key to “What year was Microsoft founded?” and click the Execute button.

Scroll down to the Server response in the Responses section. You will see the response from the model in the response key of the JSON payload. This matches the structure of the ChatResponse object in the code.

Execute the endpoint again with a different prompt, “What year did the company go public?”. The response will be something about the year 1986, when Microsoft went public. The model is able to infer that “the company” is referring to “Microsoft” because the chat history is being aggregated and sent with each call to the create function for the endpoint.

Connecting the Conversational UI (In Progress)