What does a mature LLM observability stack look like?

Traditional APM tools weren't built for non-deterministic systems. We're capturing latency and token counts but missing what matters: answer quality drift over time, prompt regression, retrieval quality by query type. What's actually in your observability stack — tools, custom metrics, alert thresholds — for an LLM-heavy product in production?