# Monitoring AI/LLM Applications with New Relic: Token Usage, Cost, and Model Comparison

> What I learned running AI monitoring workshops: how to instrument a Python OpenAI app with New Relic, what metrics matter in production LLM apps, and how to compare GPT models on cost and quality.

*Published July 13, 2026* · Categories: newrelic, observability, ai

Source: https://www.kimpel.com/2026/07/13/monitoring-ai-llm-applications-with-new-relic/


Over the past year I've been running a hands-on workshop called **"New Relic AI Monitoring Mastery"** at developer events. The premise is simple: attendees build a small AI chatbot from scratch using Python, Flask, and the OpenAI API, then instrument it with New Relic to see exactly what's happening inside. This post captures the key ideas and lessons from those sessions — what metrics matter, how to get visibility quickly, and why model choice has a bigger operational impact than most people expect.

## Why monitoring AI apps is different

When you deploy a traditional web application, your observability checklist is well-established: response time, error rate, throughput, CPU/memory. For an AI application those metrics still apply at the infrastructure layer, but there's an entirely new class of concerns sitting above them:

- **Token usage** — LLM APIs bill by the token. A single misbehaving prompt or a feedback loop in your chat history can quietly drain your budget.
- **Response quality** — Unlike a SQL query that either returns a row or doesn't, an LLM response can be technically successful (HTTP 200, valid JSON) but semantically wrong. You need a way to see the actual prompt and completion, not just the status code.
- **Latency per model** — A gpt-4o call takes noticeably longer than gpt-3.5-turbo. Under load that difference compounds, and you need data to make the right model/latency/cost tradeoff.
- **Cost attribution** — When you have multiple services, users, or features hitting the same OpenAI account, you need to know which parts of your application are responsible for spend.

None of this is captured by traditional APM alone. That's the gap New Relic AI Monitoring fills.

## The sample application: a Python Flask chatbot

The workshop application is a minimal Python Flask app that wraps the OpenAI Chat Completions API. The core setup is straightforward — install the `openai` package, set an environment variable for the API key, and wire it into a Flask route:

```bash
pip install openai
export OPENAI_API_KEY=your_key_here
```

```python
from openai import OpenAI
from flask import Flask, request, jsonify

app = Flask(__name__)
client = OpenAI()

@app.route("/chat", methods=["POST"])
def chat():
    user_message = request.json.get("message")
    response = client.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=[{"role": "user", "content": user_message}]
    )
    return jsonify({"reply": response.choices[0].message.content})
```

That's the entire "AI app." The interesting work starts when you instrument it.

## Instrumenting with New Relic

Adding New Relic to a Python Flask application takes two things: a license key and the `newrelic` Python agent. Configure both in a `newrelic.ini` file:

```ini
[newrelic]
license_key = YOUR_LICENSE_KEY
app_name = ai-chat-workshop
```

Then run the app through the New Relic admin wrapper:

```bash
NEW_RELIC_CONFIG_FILE=newrelic.ini newrelic-admin run-program flask --app app.py run --host 0.0.0.0 --port 5000
```

That single change gives you the full APM experience — response times, throughput, error rates, distributed traces, and logs in context. The New Relic Python agent automatically detects OpenAI API calls and feeds them into **AI Monitoring** without any additional code changes.

## What you see in New Relic AI Monitoring

Once traffic is flowing, navigate to **APM & Services → your app → AI Responses**. The overview page shows three headline metrics:

- **Total responses** — how many LLM completions your app has served
- **Response time** — end-to-end latency per completion
- **Token usage per response** — average prompt + completion tokens, which maps directly to cost

Below those headline numbers is a table of individual requests with the actual prompt text and the model's response. This is where the real debugging happens. When a user reports a bad answer, you can find the exact prompt that produced it, see the token count, check the latency, and trace it back through your application stack.

Clicking into any individual trace shows the full distributed trace — your Flask handler, the outbound OpenAI API call as a span, and any downstream calls. The span for the OpenAI call includes the model name, token counts, and finish reason, so you can tell at a glance whether a response was cut short by a max-tokens limit or completed normally.

**Logs in context** is particularly useful here. Because the New Relic agent correlates your application logs with the trace, you can jump directly from a slow or failed AI response to the surrounding log lines without any manual cross-referencing.

## Advanced: Node.js, Pinecone, and a chat service with games

The second half of the workshop steps up the complexity. Instead of a simple single-turn chatbot, we switch to a Node.js service that integrates **Pinecone** (a vector database) with OpenAI to power a stateful multi-turn chat experience. The service exposes an API that lets you list "activities" (pre-loaded game prompts stored as embeddings in Pinecone), create a chat session, and then exchange messages using a persistent `guid`.

```bash
# Start a new chat with a specific game prompt
export LLM_PROMPT=$(curl "$BASE_URL/activities/search?activity=higher+or+lower" | jq .prompt)
curl -X POST -H 'Content-Type: application/json' \
  -d "{ \"message\": $LLM_PROMPT }" \
  $BASE_URL/chat | jq

# Continue an existing session
curl -X PUT -H 'Content-Type: application/json' \
  -d '{ "message": "42" }' \
  $BASE_URL/chat/$LLM_CHAT_GUID | jq
```

The New Relic instrumentation follows the same pattern. In New Relic you can now see not just the OpenAI calls but also the Pinecone vector lookups as spans in the same trace, giving you a complete picture of the retrieval-augmented generation (RAG) flow — how long the embedding lookup took, what the vector search returned, and how that fed into the final completion.

## Model comparison: gpt-3.5-turbo vs gpt-4-turbo vs gpt-4o

One of the most impactful exercises in the workshop is running the same prompts through three different models by switching a single git branch:

```bash
git checkout gpt-4-turbo  # or gpt-4o, or main (gpt-3.5-turbo)
```

With New Relic collecting data from all three runs, you can compare them directly in the AI Responses view. The differences are significant:

| Model | Avg response time | Avg tokens/response | Relative cost |
|---|---|---|---|
| gpt-3.5-turbo | ~1s | ~350 | baseline |
| gpt-4-turbo | ~4–6s | ~450 | ~10× |
| gpt-4o | ~2–3s | ~400 | ~5× |

These numbers vary by workload, but the pattern holds: gpt-4o sits between the other two on both latency and cost, while delivering meaningfully better responses than gpt-3.5-turbo. For most production applications gpt-4o is the right default — better quality than 3.5 without the latency hit of 4-turbo.

What New Relic makes visible that you can't see from the API response alone is the *distribution* of token usage. Some prompts are cheap; some, especially those that involve long chat histories or complex reasoning, use 3–4× more tokens than the average. Spotting those outliers before they cause a surprise bill is exactly the kind of thing AI Monitoring is built for.

## Bonus: image generation with DALL-E 3

The workshop ends with a short optional challenge — a Flask app that calls the OpenAI Images API using the `dall-e-3` model to generate images from text prompts. This part is purely for fun (and to demonstrate the breadth of the OpenAI API surface), but it also makes an important operational point: **not all OpenAI capabilities are equally observable yet**. At the time of writing, New Relic AI Monitoring focuses on chat completions and embeddings; image generation endpoints produce different telemetry and aren't yet surfaced in the same AI Responses view. That's a gap worth knowing about if your application uses DALL-E in production.

## What to take away

Running this workshop across multiple events, a few things stand out consistently:

1. **Instrumentation is trivial; the insight is not.** Adding the New Relic agent takes minutes. Understanding the token/cost/latency tradeoffs in your specific application requires real traffic data — you can't reason your way to it.

2. **Look at individual responses, not just aggregates.** The average response time and average token count will look fine while a small percentage of expensive, slow, or low-quality completions quietly drag down the user experience and inflate your bill. The AI Responses table is where you find those.

3. **Model choice is an operational decision, not just a quality decision.** gpt-4o costs more than gpt-3.5-turbo and takes longer. Whether that's worth it depends on your use case, your latency budget, and what your users actually notice — and the only way to answer that confidently is with observability data.

If you want to explore this yourself, reach out to your New Relic account team (if you are an existing customer with New Relic) or reach anyone in the [New Relic Developer Relations team](https://www.linkedin.com/showcase/new-relic-developers). You'll need your own OpenAI API key and a free New Relic account to run through the full flow.