Databricks Data Export Connector

Visier can send data through this connector if the following requirements are met.

Overview

This connector uses Iceberg REST to send data to Databricks. For a guide that walks you through Databricks set up, see Push Data From Visier to Databricks.

Prerequisites

• AWS: If your data is stored in an S3 bucket, provide the bucket name and bucket region so Visier can access the data.
• Azure: If your data is stored in a blobstore, provide the blobstore URL so Visier can allow data access through the firewall.

Service account

Create a service account in your target system and assign it the appropriate security access so that Visier can send data through this user.

The service account must use OAuth 2.0 client ID and client secret authentication.

1. Set up a service principal for OAuth 2.0 machine-to-machine (M2M). For more information, see Create service principal.
2. Grant the necessary privileges to the service principal. For more information, see Grant a principal Unity Catalog privileges.

1
2
3
4
5
6
7
8
9
10
-- Example service principal ID: 123e4567-e89b-12d3-a456-426614174000

-- Allow service principal to create and write data to tables in a provided catalog and schema
GRANT EXTERNAL USE SCHEMA ON SCHEMA <catalog name>.<schema name> TO `<service principal ID>`
GRANT USE CATALOG ON CATALOG <catalog name> TO `<service principal ID>`;
GRANT CREATE TABLE ON SCHEMA <catalog name>.<schema name> TO `<service principal ID>`;
GRANT USE SCHEMA ON SCHEMA <catalog name>.<schema name> TO `<service principal ID>`;

-- Allow service principal ability to create external metadata for lineage
GRANT CREATE EXTERNAL METADATA ON METASTORE TO `<service principal ID>`;

Connector credentials

Provide the following credentials in Visier. For more information, see Add data export connector credentials in Visier.