Databricks Data Export Connector

Visier can send data through this connector if the following requirements are met.

Overview

This connector uses Iceberg REST to send data to Databricks. For a guide that walks you through Databricks set up, see Push Data From Visier to Databricks.

Caution: Visier manages the destination table. When the linked data export changes, Visier updates the table schema. The data export connector job may fail if you apply security policies, such as row-level security and column masking, directly to the destination table. To secure the exported data, create a view on the destination table that filters and masks the data. Give users access to the view instead of the table.

Service account

Create a service account in your target system and assign it the appropriate security access so that Visier can send data through this user.

The service account must use OAuth 2.0 client ID and client secret authentication.

  1. Set up a service principal for OAuth 2.0 machine-to-machine (M2M). For more information, see Create service principal.
  2. Grant the necessary privileges to the service principal. For more information, see Grant a principal Unity Catalog privileges.
Copy
Example: Service principal access
-- Example service principal ID: 123e4567-e89b-12d3-a456-426614174000

-- Allow service principal to create and write data to tables in a provided catalog and schema
GRANT EXTERNAL USE SCHEMA ON SCHEMA <catalog name>.<schema name> TO `<service principal ID>`
GRANT USE CATALOG ON CATALOG <catalog name> TO `<service principal ID>`;
GRANT CREATE TABLE ON SCHEMA <catalog name>.<schema name> TO `<service principal ID>`;
GRANT USE SCHEMA ON SCHEMA <catalog name>.<schema name> TO `<service principal ID>`;

-- Allow service principal ability to create external metadata for lineage
GRANT CREATE EXTERNAL METADATA ON METASTORE TO `<service principal ID>`;

Connector credentials

Provide the following credentials in Visier. For more information, see Add data export connector credentials in Visier.

Credential

Description

Example

Workspace URL

The URL of the Databricks workspace to send data to.

example.cloud.databricks.com

Catalog name

The catalog to send data to.

data_export_connector

Schema name

The schema to send data to.

example_schema

Client ID

The OAuth 2.0 client ID.

abcdef-123456

Client secret

The OAuth 2.0 client secret.

**************