P-010AI Observability Platform

Build a self-hosted AI observability platform that traces LLM calls, tracks costs, monitors latency, evaluates quality, and alerts on anomalies. Integrates with OpenTelemetry for vendor-neutral instrumentation and provides a dashboard for real-time monitoring. Think of it as a mini Langfuse — purpose-built for your LLM applications.

Skills

Tech Stack

Deploy To

What You'll Learn

• ✓Build a trace ingestion API that accepts OpenTelemetry spans
• ✓Implement per-request cost calculation with model pricing tables
• ✓Create a real-time monitoring dashboard with cost, latency, and quality metrics
• ✓Build an automated evaluation pipeline using LLM-as-judge
• ✓Set up configurable alerts for cost spikes, latency regressions, and quality drops
• ✓Design a trace viewer with span hierarchy visualization