LangChain: The LLM Application Framework

LangChain: The LLM Application Framework#

LangChain, an open-source library, empowers developers by providing a standardized and structured interface for building and integrating various components of an LLM Application. Its model-agnostic nature allows for compatibility with models from multiple LLM providers, including OpenAI, HuggingFace, and others.

Using Langchain allows us to build (“like a chain”) reusable components as part of complex multi-step LLM-based applications clearly and succinctly.

You can learn about different LangChain components here.

This tutorial will focus on a few LangChain components and learn about chaining, one of its powerful features.

Prompt templates#

Prompt Templates provides templates for designing prompts fed as inputs to the LLM models. It helps us design templates with multiple inputs that are parameterized and reusable.

Note

For this tutorial, we will only cover the use of String PromptTemplates as this gives us a finer control over the template string structure, unlike the ChatPromptTemplate.

Below is an example of how to use a prompt template. We can import this class from the langchain-core package.

The langchain-core package contains base abstractions of different components and ways to compose them together.

from langchain_core.prompts import PromptTemplate

String PromptTemplates#

The String PromptTemplates are used to format a string input. By default, the templates take Python’s f-string format. There are currently 2 choices of template_format available: f-string and jinja2. Later we will see the use of jinja2 format. In the example below, we will use the f-string format.

prompt_template = PromptTemplate.from_template(
    "{planet_name} in the solar system is the "
)

prompt_template.format(planet_name="Mars")

Let’s instantiate our OLMo model like in the previous section of the tutorial with llama-cpp-python.

from llama_cpp import Llama
from ssec_tutorials import download_olmo_model
from ssec_tutorials.scipy_conf import parse_text_generation_response

OLMO_MODEL = (
    download_olmo_model()
)  # It won't actually download again if it's already there
olmo = Llama(model_path=str(OLMO_MODEL), verbose=False)

Now that we have our model ready to go, let’s try different prompt templating starting from the previous prompt template with an input of planet_name.

model_response = olmo(
    prompt=prompt_template.format(planet_name="Mars"),
    temperature=0.2,
    max_tokens=8,
    echo=True,
)  # Generate a completion, can also call olmo.create_completion

print(parse_text_generation_response(model_response))

# Another example
prompt_template = PromptTemplate.from_template(
    "{entity_1} of the planet {entity_2} is "
)
prompt_template.format(entity_1="Size", entity_2="Earth")

model_response = olmo(
    prompt=prompt_template.format(entity_1="Size", entity_2="Earth"),
    temperature=0.2,
    echo=True,
)

print(parse_text_generation_response(model_response))

Your turn 😎#

Create a String PromptTemplate that outputs some text generation prompt, for example, “Sun is part of galaxy …”.

Feel free to experiment with the built in Python f-string for the prompt input argument to the model.

# Write your prompt_template and model_response code here

LLM Interface#

LangChain has implemented a Runnable protocol that allows us to create custom “chains”. This protocol has a standard interface for defining and invoking various LLMs, PromptTemplates, and other components, enabling reusability. For more details, go to LangChain’s Runnable documentation.

Note

In this tutorial, you will see the use of the .invoke method on various LangChain objects. This is essentially using that standard interface for the Runnable protocol.

Loading the model via LangChain’s LlamaCpp abstraction enables us to use the chaining feature. This class is part of the langchain-community package, which contains third party integrations that are maintained by the LangChain community.

from langchain_community.llms import LlamaCpp

olmo = LlamaCpp(
    model_path=str(OLMO_MODEL),
    temperature=0.8,
    verbose=False,
)

As you can see below, we now have a LlamaCpp Langchain object rather than the Llama llama-cpp-python object from previous sections.

type(olmo)

We learned above about the Runnable protocol. Let’s see how we can invoke the model using the standard interface compared to how we originally invoked the model with llama-cpp-python.

answer = olmo.invoke("What is the meaning of life?")

print(answer)

If you’d like to access the base Llama object from the llama-cpp-python package, you can access it via the .client attribute of the LlamaCpp object.

type(olmo.client)

With access to the underlying Llama object, you can directly retrieve any metadata information. In this example, we are retrieving OLMo’s tokenizer chat template we saw in the previous notebook to setup a String PromptTemplate.

The built in model’s chat template is using jinja2 templating syntax, which is a popular templating engine for Python.

prompt_template = PromptTemplate.from_template(
    template=olmo.client.metadata["tokenizer.chat_template"], template_format="jinja2"
)

The PromptTemplate object has 2 main attributes that are very useful to explore the built-in prompt template of the model:

input_variables: This is a list of all the input variables that the prompt template expects.
template: This is the actual template string that the model uses.

prompt_template.input_variables

For this particular template, we can see that it expects add_generation_prompt, eos_token and messages. But what are the variable types for these inputs? What do they mean?

We can answer the questions above by looking at the template string itself. The template string is using the jinja2 templating engine syntax, so it may look confusing at first, but at the end of the day it’s essentially just some python code in a template string.

print(prompt_template.template)

As we can see above, the template reads as follows:

eos_token is a string that is added at the top of the resulting string after the prompt is formatted. You can also see that eos_token is used to append content string values from an assistant role. You can find this value by going to the Model’s tokenizer_config.json file and looking for the eos_token key. Unfornately, this is currently the only way to get this information, you can go to ggerganov/llama.cpp#5040 for more details. In our case, the eos_token is <|endoftext|>.
messages is a list of dictionary that is iterated over. As you can see these dictionaries should contain role and content keys.
add_generation_prompt is a boolean that is used to determine whether to add a generation prompt or not. In this case, when it’s the last message and add_generation_prompt is True, it will add the <|assistant|> string to the end of the prompt.

Now that we know what the template expects we can create the final prompt string by passing in the expected input variables. This time, instead of using the .format method, let’s see what happens if we use the .invoke method on the PromptTemplate object.

prompt_template.invoke(
    messages=(
        {
            "role": "user",
            "content": "You are a helpful assistant. Tell me a joke about cats",
        }
    ),
    add_generation_prompt=True,
    eos_token="<|endoftext|>",
)

As you can see, this results to an error if we pass in the input variables directly. This is because the .invoke method expects an input argument called input that is a dictionary of the input variables, which will be passed into the runnable. Also, there’s a config input argument that is a RunnableConfig object, however, this is optional and can be omitted, and you will see that we will use this later when invoking the model.

?prompt_template.invoke

Let’s try again, this time with the correct input type.

prompt_value = prompt_template.invoke(
    input=dict(
        messages=[
            {
                "role": "user",
                "content": "You are a helpful assistant. Tell me a joke about cats",
            }
        ],
        add_generation_prompt=True,
        eos_token="<|endoftext|>",
    )
)

You can see below that we get a StringPromptValue object this time as the output rather than a pure string. But we can still get the string value by calling the .to_string method on the StringPromptValue object.

prompt_value.to_string()

The output string above contains the necessary signifier tokens for the OLMo Model to understand what the user input is and where the model should put generated responses. This whole string output will then become the full prompt for the model.

Note

For the rest of the tutorial, we won’t be using .invoke method on the PromptTemplate object, but rather we will use the .format method to get the final prompt string. This is more straightforward and easier to understand. The walkthrough above is just to show you how to use the .invoke method.

Chain in LangChain#

Chaining allows us to combine multiple components, as described above, in series or parallel to develop a multi-step LLM pipeline. As shown in the image below, any number of components can be linked together to form a chain.

LancChain Chain

Image Source: www.analyticsvidhya.com

Internally, the chain works like below:

STEP 1: Dictionary is processed as an input to the prompt template.
STEP 2: Prompt Template reads the variables to form the prompt text as output - “What are stars and moon?”
STEP 3: The prompt is given as input to the LLM model.
STEP 4: LLM Model produces output.
STEP 5: The output goes through StrOutputParser that parses it into a string and gives the result.

We can use the pipe operator (“|”), which is part of the LCEL(Lang Chain Expression Language). The pipe operator sequentially arranges each component, similar to the above image.

llm_chain = prompt_template | olmo

When we check the type of the resulting chain, it’s just a RunnableSequence! So, essentially, it’s a series of runnables that are executed in sequence.

type(llm_chain)

llm_chain

Like other Runnable type, it has an invoke method that expects the same input and config arguments as we’ve seen before with the LLM and PromptTemplate objects.

?llm_chain.invoke

Just like the example above, we’ll need to pass in the input variables as a dictionary.

# Construct the prompt as expected by OLMo
llm_chain.invoke(
    {
        "messages": [
            {
                "role": "user",
                "content": "You are a helpful assistant. Tell me a joke about cats",
            }
        ],
        "add_generation_prompt": True,
        "eos_token": "<|endoftext|>",
    }
)

Instead of having to invoke llm_chain repeatedly with add_generation_prompt and eos_token, we can update our prompt_template.

# Create a prompt template using OLMo's tokenizer chat template we saw earlier, but this time use partial variables.
prompt_template = PromptTemplate.from_template(
    template=olmo.client.metadata["tokenizer.chat_template"],
    template_format="jinja2",
    partial_variables={"add_generation_prompt": True, "eos_token": "<|endoftext|>"},
)

llm_chain = prompt_template | olmo

Let’s stream the output instead of waiting for OLMo to generate and display the text. We can use Callbacks to subscribe to various events in your LLM application pipeline. Check this out for a list of events.

Below, we will use the StreamingStdOutCallbackHander to stream the output to the console. To do this, we can pass in a dictionary to the config argument of the invoke method, with a callbacks key that contains a list of callback handlers, to see all the options, checkout the RunnableConfig documentation.

from langchain_core.callbacks import StreamingStdOutCallbackHandler

llm_chain.invoke(
    {
        "messages": [
            {
                "role": "user",
                "content": "You are a helpful assistant. Tell me a joke about cats",
            }
        ]
    },
    config={"callbacks": [StreamingStdOutCallbackHandler()]},
)

We will cover more LangChain concepts in upcoming notebooks.

Your turn 😎#

Try different message values and see how the output changes. But remember to follow the template structure. The dictionary keys must contain role and content and the allowed role values are only user and assistant.

# Write your llm_chain.invoke code here, feel free to also, create your own template and try partial_variables

LangChain: The LLM Application Framework

Contents

LangChain: The LLM Application Framework#

Prompt templates#

String PromptTemplates#

Your turn 😎#

LLM Interface#

Chain in LangChain#

Your turn 😎#