Logo Event-Driven Architecture & Industrial Edge EN ES

Developer Hub — Kafka Plugin

Kafka Plugin in Red Hat Developer Hub

The @backstage-community/plugin-kafka plugin lets you view Kafka information directly on Developer Hub entity pages.

What does it show?

  • Topic offsets: current production position per partition

  • Consumer group offsets: consumption position per consumer group

  • Consumer lag: difference between production and consumption position

  • Dashboard URL: direct link to Grafana/Streams Console

Configuration

The plugin is enabled as a dynamic plugin in Developer Hub:

# Backend — connectivity to the Kafka cluster
- package: ./dynamic-plugins/dist/backstage-community-plugin-kafka-backend-dynamic
  disabled: false
  pluginConfig:
    kafka:
      clientId: backstage
      clusters:
        - name: cdc-cluster
          brokers:
            - cdc-cluster-kafka-bootstrap.kafka-cdc.svc:9092

# Frontend — UI on entity pages
- package: ./dynamic-plugins/dist/backstage-community-plugin-kafka
  disabled: false
  pluginConfig:
    dynamicPlugins:
      frontend:
        backstage-community.plugin-kafka:
          mountPoints:
            - mountPoint: entity.page.overview/cards
              importName: EntityKafkaContent
              config:
                if:
                  anyOf:
                    - hasAnnotation: kafka.apache.org/consumer-groups

Annotation on catalog entities

For the Kafka plugin to appear on an entity page, add the following annotation to the component’s catalog-info.yaml:

metadata:
  annotations:
    kafka.apache.org/consumer-groups: cdc-cluster/camel-cdc-consumer

This makes Developer Hub show offsets and lag for the consumer group camel-cdc-consumer directly in the component view.

How it Works

Plugin backend: connection to the Kafka cluster

The Kafka plugin in Developer Hub does not use third-party REST APIs — it connects directly to the Kafka cluster:

  1. The plugin backend (plugin-kafka-backend-dynamic) establishes an AdminClient connection to the cluster using the configured brokers (cdc-cluster-kafka-bootstrap.kafka-cdc.svc:9092).

  2. It uses the AdminClient.describeConsumerGroups() API to get consumer group state.

  3. It queries AdminClient.listConsumerGroupOffsets() for current offsets and AdminClient.listOffsets(OffsetSpec.latest()) for high watermarks.

  4. It computes lag per partition and exposes it to the frontend via Backstage’s internal API.

Plugin frontend: entity-driven UI

The frontend is mounted conditionally on entity pages:

  1. When a user navigates to a component in Developer Hub, the router checks whether the entity has the kafka.apache.org/consumer-groups annotation.

  2. If the annotation exists, the EntityKafkaContent card renders at the configured mountPoint (entity.page.overview/cards).

  3. The card fetches from the plugin backend, which returns consumer group data as JSON.

  4. The frontend shows a table with: topic, partition, current offset, high watermark, lag, and consumer state (active/idle).

Software Templates: auto-annotation

Developer Hub Software Templates can automatically inject the Kafka annotation into generated catalog-info.yaml, so every new component that consumes from Kafka has lag visibility from the first commit.

Streams for Apache Kafka Console

In addition to the Developer Hub plugin, the Streams Console (StreamsHub) provides a dedicated UI for Kafka management:

Access: https://kafka-console-kafka-cdc.

Streams Console features

Feature Description

Cluster Overview

Cluster-wide view: brokers, partitions, replication factor

Topics

List topics, view messages in real time, search by offset

Consumer Groups

View consumer group state, lag per partition, reset offsets

Kafka Nodes

State of each broker, disk and network metrics

Kafka Connect (Tech Preview)

Manage connectors directly from the console

Verification

To verify that the Kafka plugin works:

  1. Open Developer Hub: https://backstage-developer-hub-developer-hub.

  2. Navigate to a component that has the kafka.apache.org/consumer-groups annotation

  3. On the Overview tab, you should see the Kafka card with offsets and lag

  4. Click the consumer group to see per-partition detail

Official Documentation