ocp-datalake
Databricks → OpenShift AI
Governance runs on the cluster: validate, register, serve.
The Databricks registry is simulated (PVC + model.json, not an MLflow bundle).
OpenShift AI Models-as-a-Service is the governed LLM gateway (CPU simulator; the L4 stays on Llama).
Desired state is OpenShift GitOps (Argo CD). Captures from the OpenTLC sandbox, 19 Sep 2026.
model.json (coefficients), not an MLflow directory. The PVC stands in for the registry. MaaS is a second governed path on the same cluster.
/predict. MaaS: Gen AI studio → LLMInferenceService → Connectivity Link → /v1/chat/completions. The L4 stays on Llama.
databricks-ml-registry stands in for ODF/S3. MinIO and MariaDB are DSPA run history only (dev).
notebook-to-openshift-zxjz9 Succeeded in 45 s. Same promotion as KFP, started with oc create -f manifests/08-pipelinerun.yaml.
pull task validates keys on our JSON (weights, bias, threshold), not an MLmodel signature.
InferenceService/churn-score Ready True. UBI Python runtime, one replica, no GPU request.
/predict and /healthz skip login in this PoC. OpenTLC hostnames rotate.
llama-32-3b-instruct in my-first-model holds the only GPU. The linear model does not contend for it.
notebook-to-openshift (8 runs) and churn-score Ready. Single-model serving enabled.
verify7: seed-model → validate-model → register-model → rollout-inference. SA pipeline-runner-dspa.
ocp-datalake-registry (HTTPS :8443). churn-score latest version 1.
pvc://databricks-ml-registry/models/churn-score/1.
ocp-datalake → Workbenches. CPU HardwareProfile default-profile. Image Jupyter | Data Science | CPU | Python 3.12 tag 3.5 (Latest). The L4 stays on Llama. Open uses Gateway API (inject-auth).
inference:4180/predict: churn: true, probability 0.5987, source databricks-mlflow.
sk-oai- key (prefix only in the capture), lists facebook/opt-125m, then /v1/chat/completions. Usage in the payload: prompt tokens 6. Do not commit full keys.
churn: true (0.5987). The consumer never talks to Databricks.
Facebook OPT 125M (Simulated) is Ready as an AI asset (opendatahub.io/genai-asset=true). Use case LLM. The GPU is not on this model.
https://maas.<apps-domain>/. Badge Model as a Service. Pick a subscription, then mint an sk-oai- key. GET / is 404 by design; talk to /v1/chat/completions.
Simulator Access Ready: authenticated users may call the simulator. Connectivity Link enforces the policy on the gateway.
facebook-opt-125m-simulated in llm, Ready True. Labels include opendatahub.io/genai-asset=true so it shows in the dashboard. CPU simulator; Llama keeps the L4.
openshift-ingress, service maas-default-gateway-openshift-default, TLS passthrough, Admitted. Host maas.apps.<cluster>.
fast) with curl against the inference Route. Native /predict and KServe v1 instances[] return the same churn probability.
sk-oai- key with your OpenShift token, list models, then call /v1/chat/completions on the simulator. Key prefix only in the capture — do not commit full keys.
facebook/opt-125m in llm: 44 requests after the workbench chat cell and maas-usage-load.sh. P90 270 ms, error 0%, GPU 0%, CPU cores on the simulator. Llama keeps the L4.
bash scripts/maas-usage-load.sh.
SparkApplication spark-to-churn-score (Apache Spark 3.5.8). Driver and executor stay Running only while SPARK_HOLD_SECONDS sleeps after POST /predict. This capture is that window: churn true, probability 0.5987. Replay with SPARK_HOLD_SECONDS=1500 bash scripts/run-spark-score.sh. When the sleep ends, Stackable deletes the pods.
score.py, Java SparkPi from the image JAR, Tekton-built ChurnScoreJob, and Streams for Apache Kafka → spark-from-kafka-churn. All score via POST /predict. Source: docs/_diagram-spark.html. SparkApplications stay opt-in (not Argo).
bash scripts/run-spark-pi.sh prints Pi is roughly 3.1423511423511425. bash scripts/run-spark-java-churn.sh builds the custom JAR image with OpenShift Pipelines and runs spark-java-churn-score: PREDICT_HTTP=200, churn true, probability 0.5987.
gitops/. Applications: churn-score (includes the CPU workbench), Model Registry, MaaS layers, Stackable Spark operators, Streams for Apache Kafka (streams-kafka-operators, streams-kafka-cluster), kubelet maxPods: 500. CheCluster and SparkApplications are opt-in (bash scripts/enable-devspaces.sh, bash scripts/run-spark-score.sh, run-spark-pi.sh, run-spark-java-churn.sh, run-spark-kafka.sh). prune is off. Install: bash scripts/enable-gitops.sh.
main. SparkApplication CRs are not synced by Argo.
← → to move · F for fullscreen · click a capture or diagram to enlarge
Models as a Service auth path
Red Hat Connectivity Link attaches one AuthPolicy to Gateway maas-default-gateway. Every child HTTPRoute (/maas-api and the simulator) is covered. The CRD API is still kuadrant.io/v1; the product name is Connectivity Link. Source: docs/_diagram-authpolicy.html.
X-MaaS-* headers. Health checks skip auth.Quick commands
Resolve the Route from the cluster you are logged into. OpenTLC FQDNs rotate.
ROUTE=https://$(oc get route inference -n ocp-datalake -o jsonpath='{.spec.host}')
curl -sk -H 'Content-Type: application/json' \
-d '{"tenure":12,"charges":70,"support_tickets":3}' \
"$ROUTE/predict"
MAAS_URL=https://maas.$(oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}')
TOKEN=$(oc whoami -t)
API_KEY=$(curl -sk -X POST "$MAAS_URL/maas-api/v1/api-keys" \
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d '{"name":"poc","subscription":"simulator-free","expiresIn":"1h"}' \
| python -c "import json,sys; print(json.load(sys.stdin).get('key',''))")
curl -sk "$MAAS_URL/v1/chat/completions" \
-H "Authorization: Bearer $API_KEY" -H "Content-Type: application/json" \
-d '{"model":"publishers/llm/models/facebook/opt-125m","messages":[{"role":"user","content":"Hello"}],"max_tokens":16}'
Databricks compatibility
Today: high conceptual fit, low technical fit. The flow (registry → promotion → serving under cluster policy) is modeled well. The artifact contract is not MLflow: it is a coefficients JSON, not an MLmodel + model.pkl directory. Do not point a production registry at this PoC.
| Fit | What lands |
|---|---|
| High | Linear model exported by hand (coef_ / intercept_) into this JSON. That is churn-score v1, not “fetch the model from the registry”. |
| Medium | sklearn / XGBoost / LightGBM: change the ServingRuntime (MLServer + mlflow, or ONNX + OVMS/Triton). Do not patch server.py. Watch pickle vs UBI 3.12. |
| Low | Spark ML, Feature Store / Unity Catalog at inference, DBFS paths or UC volumes. |
Artifact contract and model-type fit only. For the real pull path and product mapping, see Databricks bridge below or the README.
Databricks bridge
OpenShift never enters the Databricks workspace. In a real integration the cluster pulls a versioned artifact, validates it, registers it, and serves it under the same policies as the rest of the workloads. Databricks stays the training and catalog plane; OpenShift is deployment, identity, and inference.
What would change in this repo
Additive only — keep the JSON seed path as an offline fallback.
| Today (PoC) | Real bridge |
|---|---|
Task seed copies apps/model/model.json onto PVC databricks-ml-registry | New task databricks-pull: mlflow.artifacts.download_artifacts() against databricks-uc (models:/catalog.schema.churn_score/1). JSON seed remains fallback. |
| No Databricks SDK or credentials | Databricks service principal OAuth M2M (not a PAT in git). Secret via External Secrets or cluster-local store. OpenShift side stays RHBK + service accounts. |
NetworkPolicy with no egress | Allow egress to the Databricks control plane and artifact bucket (S3). |
| PVC stand-in for registry | Prefer ODF NooBaa ObjectBucketClaim; scripts/s3_model.py becomes the live helper. KServe storageUri from S3. |
validate checks JSON keys | Validate MLmodel signature vs HTTP contract (tenure, charges, support_tickets). |
UBI Python server.py linear runtime | MLServer + mlserver-mlflow (native MLflow directory) or ONNX + OVMS/Triton. Do not unpickle into server.py. Watch sklearn/numpy versions: Databricks wheels vs UBI 3.12. |
Model Registry URI pvc://… | Register in ocp-datalake-registry with UC source URI; promotion still Tekton/KFP + GitOps. |
Out of scope: Spark ML in the predictor, Feature Store lookups at inference time, DBFS or UC volume paths embedded in the artifact.
Flow: Databricks UC → OAuth M2M pull (Tekton/KFP) → ODF or PVC → RHOAI Model Registry → KServe (MLServer) → Route /predict.
Databricks capabilities vs Red Hat and community
No 1:1 parity. This table is honest about what this PoC already runs and what would replace each Databricks surface.
| Databricks capability | This PoC | Red Hat | Community | Gap note |
|---|---|---|---|---|
| Notebooks / workspace | OpenShift AI workbench ocp-datalake-demo (CPU default-profile) | OpenShift AI workbenches, OpenShift Dev Spaces | — | No GPU on the workbench. The L4 stays on Llama. Example notebook calls /predict and MaaS chat. |
| Jobs / Workflows | Tekton + KFP promotion only | OpenShift Pipelines, Data Science Pipelines | Argo Workflows | GitOps installs; it does not run training jobs. |
| Apache Spark batch | One-shot SparkApplication spark-to-churn-score (opt-in) | — | Stackable Spark (commons + secret + listener + spark) | PySpark DataFrame map → churn-score /predict. productVersion 3.5.8. |
| Apache Spark Java | SparkApplication spark-pi (scripts/run-spark-pi.sh) | — | Same Stackable operator; org.apache.spark.examples.SparkPi from the image JAR | Demo only; no /predict. Driver log must show Pi is roughly. |
| Apache Spark Java custom | Tekton spark-java-churn → spark-java-churn-score (scripts/run-spark-java-churn.sh) | OpenShift Pipelines (Maven → Buildah → ImageStream) | Stackable Spark + custom JAR in apps/spark-churn-java | Parity with Python batch → /predict. SparkApplication applied by Tekton, not Argo. |
| Streams for Apache Kafka | Single-node KRaft churn + Console + topic churn-events | Streams for Apache Kafka (amq-streams + amq-streams-console) | — | Plain listener 9092, no auth. Outside ocp-datalake quota (ns kafka). |
| Spark ← Kafka | Streaming SparkApplication spark-from-kafka-churn (scripts/run-spark-kafka.sh) | — | Stackable Spark + spark-sql-kafka | Structured Streaming → /predict. Pods stay Running until the CR is deleted. |
| MLflow Tracking | Not connected | KFP run metadata, DSPA history (dev MinIO here) | Self-hosted MLflow Tracking server | No Red Hat–branded MLflow Tracking product. |
| Model Registry (legacy + UC models) | PVC + RHOAI ocp-datalake-registry | OpenShift AI Model Registry | MLflow Registry (if self-hosted) | UC grants and databricks-uc URI are on the Databricks side. |
| Unity Catalog | Simulated naming only | OpenShift RBAC, GitOps, Model Registry | OpenLineage/Marquez, OpenMetadata | No full UC equivalent. Lineage is separate tooling. |
| Feature Store | Not used | Feature Store (tech preview in some RHOAI releases — verify 3.5) | Feast | Remote Databricks Feature Store lookups at inference do not work off-platform. |
| Delta / lakehouse / DBFS | PVC only (no ODF on sandbox) | OpenShift Data Foundation (NooBaa/S3) | Apache Iceberg, Delta connectors, Trino | scripts/s3_model.py is ready for OBC; not wired in Tekton yet. |
| Model Serving | KServe churn-score + MaaS LLM path | KServe, Models-as-a-Service, Connectivity Link | vLLM, Triton (via RHOAI runtimes) | Databricks Model Serving ≈ KServe/MaaS, not the registry. |
| SQL warehouses | Not used | — | Trino, Spark SQL | No Red Hat SQL warehouse 1:1. |
| GenAI / Mosaic | MaaS CPU simulator; L4 on Llama KServe | MaaS, OpenShift AI serving | vLLM, Hugging Face stacks | Llama Stack is Removed on this cluster by design. |
| Identity / SCIM | RHBK + oauth-proxy on Route | Red Hat Build of Keycloak, OpenShift OAuth | Keycloak upstream patterns | Databricks SP OAuth M2M is separate from cluster IdP. |
| Asset Bundles / IaC | GitOps app-of-apps | OpenShift GitOps (Argo CD) | Helm, Kustomize (already in repo) | Cluster desired state is git; Databricks bundles stay in Databricks. |
Enterprise architecture views
Source HTML: docs/_diagram-bridge-enterprise.html, docs/_diagram-replace-enterprise.html (render with python scripts/render-diagrams.py).
Bridge vs replace: Bridge — Databricks trains and registers; OpenShift pulls, governs, and serves (this document). Replace — RHOAI + ODF + Pipelines + Feast/Iceberg on OpenShift; you lose UC and the managed Spark runtime. This PoC demonstrates the bridge story.