Inside Booking.com's AI Observability Pipeline: Kafka, PII Redaction, and Arize

The interesting part of Booking.com's AI observability architecture is not the vendor at the end of the diagram. It is everything before it: a durable Kafka buffer so producers and consumers can fail independently, and PII redaction built as its own service instead of a policy someone is supposed to follow. Robert Ta walks through an Arize case study on how Booking.com scales observability across its GenAI applications and traditional ML models, and pulls out the design decisions that matter for anyone responsible for enterprise AI architecture.

The reference architecture

The diagram on screen comes from the Arize case study "How Booking.com scales AI observability with Arize." It splits telemetry flow into three numbered stages:

Ingress and sampling. N GenAI applications emit traces via standard OTLP endpoints using OpenTelemetry plus OpenInference conventions. Traces include spans for LLM calls, tools, retrieval, and guardrails. An AI telemetry collector groups traces by attributes (application, use case, environment) and applies per-use-case sampling that reduces volume while preserving complete traces, so no spans get dropped inside a trace.
PII-aware redaction. Sampled traces land on a secure message bus, then pass through a dedicated PII redaction service before export.
Export to Arize and warehouse. Redacted traces go to Arize in near real time and persist to warehouse tables for offline analytics.

The diagram footer states the three design principles: reliability (the message bus buffers outages and spikes so downstream systems can catch up), privacy-by-design (all traces sent to Arize are centrally redacted by approved PII categories), and flexibility (multiple consumers reuse the same standardized traces without custom logging per destination).

Kafka as the durable buffer

Booking.com runs the message bus on managed Kafka. Ta frames it the same way the diagram does: a durable buffer that absorbs spikes and decouples producers from consumers. GenAI applications can keep emitting traces during a downstream outage, and consumers catch up later. That decoupling is what lets the same trace stream feed both real-time monitoring in Arize and batch warehouse ingestion without either path affecting the other.

PII redaction as a service, not a policy

Ta calls this the most interesting piece for anyone dealing with compliance, regulation, and privacy in enterprise software. Booking.com's architecture includes a service that detects and masks PII in span attributes and text fields based on approved categories, and emits redacted traces. Because it sits in the pipeline between Kafka and every export target, redaction is enforced centrally rather than left to each team's logging discipline. Compliance built into the pipeline is the difference between an AI system that runs in production and a demo legal shuts down.

The traditional ML side: prediction logs to datasets

A second diagram covers the traditional ML path. Model serving (ranking models, fraud detection, other classic ML) writes prediction logs to a unified prediction table in the warehouse. From there: sampling and enrichment per model and hour with metadata from the model registry, per-model transformations into the schema Arize expects, and import jobs that continuously ingest from the warehouse. The output is model-specific datasets, such as a ranking model dataset and a fraud detection dataset, that teams can use for analysis and domain-specific fine-tuning.

What observability catches in production

The case study text shown on screen includes two concrete wins. An Arize monitor fired when latency for a specific agent spiked several standard deviations above baseline; the team traced it to a model deployed as an experiment running without a service tier enabled, and fixed it by coordinating with the provider. In another case, evaluation scores dropped across multi-turn conversations because a chatbot agent fed tool outputs back into conversation history and the context ballooned over time. Visibility during prototyping let the team intervene before that configuration reached production.

The takeaway for architects

Ta's closing point: anyone deploying AI architectures at scale needs to chunk out services this way. None of the hard parts in this diagram are AI parts. Ingestion, buffering, redaction, and warehouse persistence are standard data engineering, applied deliberately so that observability, privacy, and analytics each get their own decoupled component.

Key Takeaways

Buying an observability tool is not an architecture. The pipeline in front of the vendor is where the design work lives.
A managed Kafka message bus decouples trace producers from consumers, absorbs spikes, and lets multiple destinations consume the same standardized stream.
PII redaction runs as its own service in the pipeline, masking span attributes and text fields by approved categories before anything reaches external tools.
Standardized telemetry (OTLP with OpenTelemetry plus OpenInference conventions) covers agents with traces and traditional ML with warehouse prediction logs, routed to one platform.
Sampling is per use case and preserves complete traces; no spans are dropped within a kept trace.
Production monitoring caught real issues: an agent latency regression from a misconfigured deployment and context bloat degrading a chatbot's evaluation scores.

Resources

How Booking.com scales AI observability with Arize -- the Arize case study shown in the video, with both architecture diagrams.
Arize AI -- the AI observability and evaluation platform at the end of the pipeline.
OpenInference -- Arize's OpenTelemetry-complementary conventions for tracing AI applications, used at the ingress layer.
OpenTelemetry -- the open standard behind the OTLP trace ingestion endpoints.
Apache Kafka -- the message bus technology behind Booking.com's durable trace buffer.
Full video from @therobertta_ -- the longer YouTube version linked in the TikTok description.