Przejdź do treści

Databricks connector

What this connector reads

The Databricks connector enumerates MLflow registered models and Model Serving endpoints in a single Databricks workspace. We see registered model names, latest versions, creation timestamps, and serving endpoint configurations (model name, deployment state, served-model identifiers). We do not read notebook contents, table data, or training data.

This connector is per-workspace. If you have multiple Databricks workspaces, install one connection per workspace.

What you (the customer) need to do

You will need a workspace admin to create a Service Principal and authorise it for OAuth machine-to-machine (M2M) access.

Why not a Personal Access Token (PAT)? Databricks PATs are bound to a human user — revoking the user revokes the connection, and a PAT's effective permissions match the user's. Service Principal + OAuth M2M is the modern path Databricks recommends for machine integrations: scoped workspace permissions, no human dependency, and rotation is a single "Generate secret" click. PAT remains supported by Databricks but we deliberately don't accept it.

  1. In your Databricks workspace, go to Settings → Identity and access → Service principals → Add service principal. Choose OAuth secret as the credential type.
  2. Generate a client ID + client secret pair. Store the secret securely — Databricks shows it only once.
  3. Grant the Service Principal the following workspace-level privileges:
  4. CAN_VIEW on workspace resources (so it can list clusters; we use clusters/list as a token health check).
  5. CAN_VIEW or higher on the MLflow registry (so it can list registered models).
  6. Read access to Model Serving endpoints (Workspace setting → Compute → Serving → Service Principal access).
  7. In Secruna, open Connections → Connect Databricks. (screenshot placeholder TODO)
  8. Paste:
  9. Workspace URL — e.g., https://adb-1234567890.0.azuredatabricks.net
  10. Client ID — the Service Principal client ID
  11. Client secret — the Service Principal OAuth secret
  12. Click Connect. We immediately exchange the credentials for a workspace token via POST /oidc/v1/token and validate by calling GET /api/2.0/clusters/list. If validation passes, the connection moves to active.

We re-exchange the OAuth credentials on every discovery run, so short-lived access tokens are never persisted — only the client_id + client_secret are stored encrypted in Azure Key Vault.

Permissions / scopes requested

Databricks OAuth M2M does not use granular scopes. Workspace permissions on the Service Principal control what the connector can see. The endpoints Secruna actually exercises are:

Endpoint Why
POST /oidc/v1/token Mint short-lived workspace token from client_id + client_secret.
GET /api/2.0/clusters/list Token health check on connect and on every run.
GET /api/2.0/mlflow/registered-models/list (paginated) List MLflow registered models.
GET /api/2.0/serving-endpoints List Model Serving endpoints.

We never call:

  • Job, run, or workflow APIs
  • Workspace file or notebook APIs
  • SQL warehouse / Unity Catalog data APIs
  • Any endpoint that mutates state

What we do NOT touch

  • Notebook contents. We do not list, read, or fetch notebooks.
  • Table data via Unity Catalog or DBFS. We do not query data.
  • Training datasets or stored model artefacts (the binary files produced by mlflow.log_model). We list registered model metadata; we never download model artefact files.
  • Cluster configurations beyond exists. The cluster list call is purely a health check.
  • Workflow / job histories.
  • User identities in the workspace.

How to revoke access

  1. In Databricks, Settings → Identity and access → Service principals, find the Service Principal you provisioned for Secruna and either Revoke the OAuth secret or Delete the Service Principal. Either action invalidates our credentials immediately.
  2. In Secruna, open Connections and click Disconnect on the Databricks row. We delete the encrypted secret from Key Vault on disconnect.
  3. Within 90 days every artefact collected through this connection is purged. See data retention policy.

Troubleshooting

Error you see Likely cause Fix
OAuth credentials rejected by workspace (401/403) client_id or client_secret typo, or Service Principal disabled. Re-paste both. Confirm the SP exists and has not been disabled.
Service Principal cannot list clusters — grant it CAN_VIEW on workspace resources before retrying. The SP has the OAuth secret but no workspace-level read. In Settings → Identity and access → Service principals → [your SP] → Permissions, grant CAN_VIEW.
Connection active but 0 registered models or endpoints The SP doesn't have read on the MLflow registry / serving namespace. Grant the SP read on each MLflow experiment / registered model and on the Serving endpoints page.
Databricks token exchange failed: 5xx Workspace is in maintenance, or sovereign-cloud Databricks deployment uses a different OIDC base URL. Retry. If persistent, email support.
  • Subprocessor list — your Databricks workspace is controller-controlled; Databricks is your subprocessor, not ours.
  • DPA template — Annex II details the metadata categories produced by this connector.