Monitoring AI/LLM Applications with New Relic: Token Usage, Cost, and Model Comparison
Over the past year I’ve been running a hands-on workshop called “New Relic AI Monitoring Mastery” at developer events. The premise is simple: attendees build a small AI chatbot from scratch using Python, Flask, and the OpenAI API, then instrument it with New Relic to see exactly what’s happening inside. This post captures the key ideas and lessons from those sessions — what metrics matter, how to get visibility quickly, and why model choice has a bigger operational impact than most people expect. ...