How to Monitor LLMs: Implementing Traceability and Observability Using OTEL + Langfuse

There are 3 different entities that you should know about when you are monitoring your LLMs.

Langfuse is used to capture LLM specific metadata —

• Token counts
• Model parameters
• Output
• Scores

Langfuse is used to show —

• Prompt level performance analysis
• A/B test comparisons
• Quality metrics (hallucination rate, relevance scores)

You will need this to visualize your traces, spans, metrics, logs etc in a nice graphical user interface. Exports sends these to tools like Jaeger, Langfuse cloud or to a console as a JSON.

In this article, we will try to create all 3 kinds of exporters and see how we can trace and observe the metrics on dashboards.

Softwares needed:

• Python
• Ollama — to run LLM locally
• Docker Desktop — to run Jaeger locally
• Langfuse account — to export to Langfuse cloud, its free with some limitations

Step 1: Create inference file

Create a python file, console_langchain_inference.py as shown,

Steps involved

• Uses langchain to initialize llama3 model running locally
• Create a prompt and call the model
• Uses trace_llm decorator — This is the next file we are going to create

Step 2: Create console exporter file

Create a python file, otel_console_exporter.py and copy the below code,

Steps involved

• Create a simple decorator
• Use OTELs console span exporter
• Add the required attributes to show — input, prompt and output

Explanation

• Prompt response is also printed
• The entire trace, span, metrics etc is output as JSON
• You can see the input, output and prompt related attributes

Excellent, we can see the trace displayed in the console now, but hold on, is it user friendly ? Not for everyone. This is where we need a nice graphical user interface to visualize the contents of the JSON. But, what is the use of console exporter, then ? Console exporters are especially useful where you have built your own custom visual interface using JSON as input or if you want to send the JSON to any other system for further processing and analysis.

Lets modify our code to use Langfuse’s observe decorator instead of our custom decorator.

Navigate to cloud.langfuse.com, create an account and create a sample project and keep it ready. Once created, go to “Settings -> API Keys” copy the below to your .env file

Now, modify your console_langfuse_inference.py file, as below

Explanation

• Custom decorator is replaced with Langfuse observe decorator
• dotenv is initialized to make sure our code retrieves the secret key, public key and host url variables from .env file

Lets run again with the same command and see the output,

This time, the output contains only the response of LLM as we are printing it in the code.

Navigate to cloud.langfuse.com, and you will see the trace is exported to Langfuse cloud.

You can trace the request under “Observability -> Tracing”

There will be various dashboards created by default, you can explore all under “Dashboards”

In my next article, we will create an Jaeger exporter and we will see how we can trace a request end to end, until then, Happy generating!