Monitoring AI/LLM Applications with New Relic: Token Usage, Cost, and Model Comparison

Monitoring AI/LLM Applications with New Relic: Token Usage, Cost, and Model Comparison

Over the past year I’ve been running a hands-on workshop called “New Relic AI Monitoring Mastery” at developer events. The premise is simple: attendees build a small AI chatbot from scratch using Python, Flask, and the OpenAI API, then instrument it with New Relic to see exactly what’s happening inside. This post captures the key ideas and lessons from those sessions - what metrics matter, how to get visibility quickly, and why model choice has a bigger operational impact than most people expect. ...

July 13, 2026 · 7 min · 1387 words · Harry Kimpel
Getting Started with OpenTelemetry: Zero-Code Instrumentation, Custom Signals, and the Collector

Getting Started with OpenTelemetry: Zero-Code Instrumentation, Custom Signals, and the Collector

“Instrumentation” is one of those words that gets thrown around constantly in the observability world - but what does it actually mean, and how does it work in practice? I’ve run this as a hands-on workshop several times, and what I find most effective is starting from first principles: build a tiny app, make it produce telemetry, and watch what comes out. By the end you understand why the pieces fit together the way they do. ...

January 18, 2024 · 8 min · 1690 words · Harry Kimpel