Designing a Private, High‑Security Observability Stack for AI Evaluation Platforms

Why a Self‑Hosted Observability Stack Matters

AI evaluation platforms handle highly sensitive data, from user prompts to model outputs. Exposing these systems to the public internet creates significant risk, yet many teams still begin with public dashboards and generic tooling. A dedicated, private observability pipeline guarantees that telemetry stays within controlled boundaries while delivering the same level of insight needed to monitor performance and compliance.

Key Advantages

• Full control over data residency and encryption policies
• Fine‑grained access based on company email domains and one‑time passcodes
• Isolation of production traffic from development and testing environments
• Scalable architecture that can grow with model complexity and user load

The Core Architecture

The implemented path is designed to keep the core AI services hidden behind an encrypted tunnel while exposing a clean web interface for trace and evaluation data.

Browser          HTTPS
Cloudflare Access     Email OTP & Domain Filter
Cloudflare Tunnel      Outbound‑only to Azure VM
Azure VM          Caddy Reverse Proxy
—Caddy Routes—     Arize Phoenix UI & Evaluation API
Evaluation API     Generates OpenTelemetry Spans
OpenTelemetry Collector  Redacts Sensitive Fields
Arize Phoenix     Stores and Visualizes Traces

Each component is hardened to meet strict security criteria, from TLS 1.3 everywhere to role‑based access on the Azure Key Vault.

Live Features Today

• Self‑hosted Arize Phoenix instance for trace visibility
• Edge‑level authentication via Cloudflare Access with company‑email OTP
• Outbound‑only Cloudflare Tunnel to an Azure VM
• Caddy reverse proxy routing UI and API traffic locally
• Evaluation API that emits safe OpenTelemetry spans
• Collector that applies redaction rules before sending data to Phoenix
• Private container registry through Azure Container Registry
• Azure Managed Identity for secure workload access
• Azure Key Vault for secret retrieval

What Is Planned Next

Future releases aim to add full job execution pipelines, CI gating, and multi‑tenant support. The current observability layer provides a robust foundation for these additions.

Security Considerations

Data Security

• All data in transit uses TLS 1.3; data at rest is encrypted with rotating keys.
• PII detection tools flag and redact sensitive fields before data leaves the API.
• Least‑privilege RBAC controls access across services.

Platform Hardening

• Operating systems are hardened, unnecessary ports closed, and root login disabled.
• VPC segmentation limits inter‑service communication to essential paths.
• Container images are minimal, signed, and scanned for vulnerabilities.

Operational Safeguards

• Continuous auditing of all platform actions.
• Regular vulnerability scans integrated into CI/CD.
• Environment isolation between staging, testing, and production.
• Compliance mapping against regulations such as the EU AI Act and NIST AI RMF.

Recommended Open‑Source Tooling

Choosing the right stack can accelerate deployment while maintaining flexibility:

• Logs: ELK Stack or Loki for efficient storage and searching.
• Metrics: Prometheus with Grafana dashboards.
• Traces: OpenTelemetry Collector feeding Jaeger or Zipkin.
• AI‑Specific Observability: Arize Phoenix, Langfuse, Evidently AI, and Helicone for LLM monitoring.

Implementation Best Practices

• Instrument from the earliest development stages.
• Define clear ownership and responsibility matrices for each model.
• Automate security patches, vulnerability scans, and compliance reporting.
• Deploy observability as core operational infrastructure, not an add‑on.

Building a secure, self‑hosted observability pipeline is not just a technical task—it is a strategic investment that protects sensitive AI data, ensures compliance, and scales with organizational growth.