Logo Event-Driven Architecture & Industrial Edge EN ES

CDC Pipeline Architecture

Real-Time Data Streaming

Real-time streaming moves data as sources generate it, so you can analyze and process events immediately.

Common data sources:

  • Website activity

  • Banking transactions

  • E-commerce purchases

  • Player activity in gaming

  • IoT sensors

Change Data Capture (CDC)

CDC identifies INSERT, UPDATE, and DELETE operations in a database and publishes them as events to other systems in real time.

Why CDC?

Traditional approach CDC with Debezium

Periodic polling against the DB

WAL-based capture (write-ahead log)

High latency (seconds/minutes)

Sub-second latency

Extra load on the DB

Minimal impact (reads the transaction log)

May miss changes between polls

Captures all changes, including deletes

Pipeline Architecture

Diagram

Detailed data flow

  1. A user inserts or updates a row in PostgreSQL

  2. PostgreSQL writes the change to the Write-Ahead Log (WAL) with wal_level=logical

  3. Debezium (running as a KafkaConnect plugin) reads the WAL via a replication slot

  4. Debezium serializes the event as JSON (optionally Avro via Apicurio Registry)

  5. Debezium publishes the event to the Kafka topic cdc.public.customers

  6. Apache Camel consumes the event from the topic

  7. Camel transforms the event and sends an email notification via Mailpit

  8. Grafana monitors the pipeline and Kiali shows the Service Mesh traffic

How it Works

The CDC pipeline operates as a reactive chain of five stages, where each component acts independently and asynchronously:

  1. Capture — PostgreSQL writes every transaction to the Write-Ahead Log (WAL) before committing it. Debezium opens a replication slot and reads the WAL using PostgreSQL’s native logical replication protocol, without additional queries against the tables.

  2. Serialize — Debezium turns each change into a structured JSON event that includes: the before value (previous row state), after (new state), op (operation type: c`reate, `u`pdate, `d`elete, `r`ead), and `source (metadata: table, schema, WAL LSN, WAL timestamp).

  3. Stream — Debezium publishes the event to the Kafka topic (cdc.public.<table>). Kafka persists it to disk, replicates it to 3 brokers (RF=3), and acknowledges the write only when at least 2 replicas persist the data (min.insync.replicas=2).

  4. Process — Apache Camel consumes from the topic using a consumer group. Kafka assigns each partition to a single consumer (partition affinity), preserving order by primary key. Camel deserializes the JSON, applies content-based routing ($.payload.op), and transforms the payload.

  5. React — The Camel processor runs the appropriate action: HTTP POST to Mailpit for notifications, production to DLQ topics for errors, or Prometheus metric emission. Everything is non-blocking — if a destination fails, the message goes to the Dead Letter Queue.

Typical end-to-end pipeline latency is sub-second from the PostgreSQL commit to the notification. The bottleneck is not Kafka (which persists in microseconds) but the HTTP POST to external services.

Namespace and resources

Argo CD deploys all CDC pipeline components in the kafka-cdc namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: kafka-cdc
  labels:
    istio.io/dataplane-mode: ambient
    istio-discovery: enabled

The istio.io/dataplane-mode: ambient label enrolls the namespace in the Service Mesh, so Kiali shows all traffic without sidecars.

High Availability

The Helm chart runs Kafka, KafkaConnect, and Camel with multiple replicas and persistent storage for fault tolerance.

Kafka Brokers — Persistent Storage

Brokers use persistent-claim instead of ephemeral to retain data across restarts:

spec:
  kafka:
    replicas: 3
    storage:
      type: persistent-claim
      size: 10Gi
      deleteClaim: false

  • size: 10Gi — capacity per broker (adjust based on data volume)

  • deleteClaim: false — OpenShift keeps the PVCs when you scale or delete the cluster

KafkaConnect — 2 replicas

Argo CD deploys KafkaConnect with 2 replicas for load balancing and connector failover:

spec:
  replicas: 2

Kafka Connect automatically distributes tasks across available replicas. If one replica fails, the other takes over the tasks.

Camel CDC Processor — 2 replicas

The Helm chart runs 2 Camel processor replicas. Kafka balances partitions across consumers in the same group:

spec:
  replicas: 2

With 3 partitions per topic and 2 consumers, each replica processes ~1.5 partitions on average.

Official Documentation