Databricks

Query data from completed runs through Delta Sharing

Kadoa configures the integration for you through the Support Center.

Before you begin

Databricks workspace

• A Unity Catalog-enabled workspace with compute for queries.
• Permission to create a catalog from a share: metastore admin, or CREATE CATALOG with USE PROVIDER or provider ownership.

Get your sharing identifier in the workspace where you will query the data:

select current_metastore();

Send the full value, including the cloud and region: <CLOUD>:<REGION>:<UUID>.

Open Delta Sharing client

You need a compatible Delta Sharing client and a contact who can receive credentials securely. A Databricks workspace is not required.

Request the integration

Send these details through the Support Center:

Detail	Value
Access method	Databricks workspace or open Delta Sharing client
Sharing identifier	Workspace only: <CLOUD>:<REGION>:<UUID>
Credential contact	Open client only: contact for the activation link
Activity log	On (default) or off

Activity sharing includes available historical events.

Import the share

Databricks workspace

1. Find the share

   Open Catalog > gear icon > OpenSharing > Shared with me. Older workspaces call it Delta Sharing. Select the Kadoa provider and the share whose name starts with KADOA_SHARE_. A new share can take up to one minute to appear.

2. Create a catalog

   Select Mount to catalog > Create a new catalog. Enter a name such as kadoa. Select Create.

3. Grant access

   Give readers USE CATALOG on the catalog, USE SCHEMA on its schemas, and SELECT on the shared tables.

Open Delta Sharing client

Open the activation link. Select Download Credential File. You can download the file only once. Kadoa does not store it. Store the file securely. Restrict access to authorized readers.

Tokens expire within 365 days of creation. Contact Support before expiry. If you lose the file, request a replacement from Support.

For Python, install delta-sharing. List the shared tables using the path to your credential file:

import delta_sharing

profile = "<PATH_TO_CREDENTIAL_FILE>"
client = delta_sharing.SharingClient(profile)
print(client.list_all_tables())

Use the returned share, schema, and table names to read a table:

df = delta_sharing.load_as_pandas(
    f"{profile}#<SHARE_NAME>.<SCHEMA_NAME>.<TABLE_NAME>"
)

For large tables, use a Spark client instead of loading the table into pandas.

Verify the first delivery

Allow about 5 minutes after a workflow run completes.

1. Find your workflow_id in WORKFLOW_RUNS. Select the delivered run and note its job_id.
2. Match its workflow_id, schema_version, and schema_fingerprint in WORKFLOW_SCHEMA_VERSIONS. Read versioned_table_name.
3. Read that table with a job_id filter. Compare the row count with the same run in the Kadoa dashboard.

In Databricks, select your catalog and the schema whose name starts with TEAM_. Run this query:

select count(*) as row_count
from <VERSIONED_TABLE_NAME>
where job_id = '<JOB_ID>';

With the Python client, use the earlier example to read WORKFLOW_RUNS and WORKFLOW_SCHEMA_VERSIONS. Then load the versioned table and count the run’s rows:

print(df.loc[df["job_id"] == "<JOB_ID>"].shape[0])

Query the shared tables

Table	Contains
WORKFLOW_RUNS	Delivered runs, schema versions, row counts, and timestamps
WORKFLOW_SCHEMA_VERSIONS	Workflow IDs and their generated table names
WORKFLOW_SCHEMA_FIELDS	Field names, types, and positions in field_index
WF_<GENERATED_ID>__V<VERSION>	Workflow data for one schema version
WF_<GENERATED_ID>__LATEST	Workflow data for the latest schema version

Get exact table names from WORKFLOW_SCHEMA_VERSIONS. Do not substitute a workflow ID into a table name.

Each data table can contain multiple runs. Filter by job_id to query one run. Older schema versions remain available in versioned tables.

Other tables include ACTIVITY_LOG when enabled and TPL_<GENERATED_ID>__LATEST for shared template data.

What’s next

Snowflake

Query shared secure views.

How data integrations work

Review integration behavior.