Vault and secrets management

⏱ ~10 min

Secrets in this pattern flow from HashiCorp Vault through External Secrets Operator (ESO) into Kubernetes Secrets. No credentials are committed to Git.

Vault Secret Injection Flow

How the secrets pipeline works

Step What happens

1. Template

values-secret.yaml.template defines secret names and fields. Fields with onMissingValue: generate are auto-created by Vault.

2. Vault load

make load-secrets (CLI) or manual vault kv put (console install) stores secrets in Vault at secret/hub/<name>.

3. ESO sync

ExternalSecret CRs watch Vault paths via ClusterSecretStore vault-backend and project them into Kubernetes Secrets.

4. Pod consumption

Deployments mount the K8s Secret as environment variables or volumes.
Vault path secret/hub/maas-credentials syncs to Kubernetes Secret openshift-ai-maas-credentials in namespace maas-workshop (see Module 03 — OpenShift AI). The Vault KV name and the K8s Secret name differ by design.

Inspect Vault status (platformadmin)

oc exec vault-0 requires pods/exec permission. Workshop users can verify ESO sync below instead. Platform admins run: 
hub-login guest (register first)
oc exec vault-0 -n vault -- vault status 2>/dev/null | grep -E "Initialized|Sealed"
Expected output
Initialized     true
Sealed          false

List secrets stored in Vault (platformadmin)

oc exec vault-0 -n vault -- vault kv list secret/hub 2>/dev/null
Expected output
Keys
ai-gateway-platform-keys
developer-hub-secrets
gitlab-credentials
keycloak/realms/cv/backstage-provisioner
keycloak-realm-clients
maas-credentials
minio-credentials
rhbk-credentials
workshop-registration
Vault KV v2 paths use prefix secret/data/hub/<name> in ESO remoteRef.

The values-secret.yaml.template

Key secrets for this workshop:

# values-secret.yaml.template (excerpt)
secrets:
  - name: ai-gateway-platform-keys
    fields:
      - name: platformApiKey
        onMissingValue: generate

  - name: developer-hub-secrets
    fields:
      - name: oidc-client-secret
        onMissingValue: generate
      - name: session-secret
        onMissingValue: generate
      - name: gitlab-token
        onMissingValue: generate

  - name: minio-credentials
    fields:
      - name: accesskey
        onMissingValue: generate
      - name: secretkey
        onMissingValue: generate

  - name: workshop-registration
    fields:
      - name: adminToken
        onMissingValue: generate

  - name: maas-credentials
    fields:
      - name: api-key
        value: "optional upstream MaaS key (legacy)"

  - name: keycloak-realm-clients
    fields:
      - name: cv.user1.clientSecret
        onMissingValue: generate

  - name: keycloak/realms/cv/backstage-provisioner
    fields:
      - name: clientSecret
        onMissingValue: generate

Check ExternalSecret sync status

Workshop users can verify the chain without Vault exec:

hub-login guest (register first)
oc get externalsecret -A | head -20
oc get secret keycloak-client-cv-guest (register first) -n keycloak-system -o name
oc get externalsecret platform-ai-gateway-key -n ai-gateway-system 2>/dev/null
Expected output
NAMESPACE         NAME                  STORE           STATUS         READY
keycloak-system   keycloak-client-cv-user1  vault-backend  SecretSynced   True
secret/keycloak-client-cv-user1

If any shows SecretSyncedError, force ESO to re-sync:

oc annotate externalsecret -n keycloak-system --all force-sync=$(date +%s) --overwrite

Console install vs CLI install

Method Secrets loading Action needed

./pattern.sh make install

make load-secrets runs automatically

Fill ~/values-secret-ia-computer-vision.yaml before install

Pattern CR (OCP Console)

Secrets are NOT loaded automatically

After Vault initializes (wave 2), load secrets manually with vault kv put

Verify the full chain (without Vault exec)

echo "=== ExternalSecrets ==="
oc get externalsecret -A --no-headers 2>/dev/null | awk '{printf "%-25s %-30s %s\n", $1, $2, $6}' | head -15

echo "=== Keycloak client secret (OIDC lab) ==="
oc get secret keycloak-client-cv-guest (register first) -n keycloak-system -o jsonpath='{.data.clientId}' 2>/dev/null | base64 -d
echo

What you learned

  • Secrets flow: values-secret.yaml.template → Vault → ESO → K8s Secret → Pod

  • ai-gateway-platform-keys auto-generates the platform AI Gateway key per install

  • keycloak/realms/cv/backstage-provisioner supplies the scaffolder OIDC provisioner secret

  • minio-credentials backs model/object storage used by inference workloads

  • Workshop users verify ESO via oc get externalsecret and read synced secrets in allowed namespaces

  • oc annotate externalsecret …​ force-sync=…​ forces ESO to re-read from Vault

Next

Continue with Module 07 — Review and next steps.