Send Data to Visier

Learn how to use the Direct Data Intake API to retrieve an objects data load schema and send data to Visier.

Introduction

The Direct Data Intake API provides you with the ability to send data to Visier in a transactional format.

In this tutorial, you will learn how to load data into objects in your Visier solution using the Direct Data Intake API. We will first walk you through how to discover the data load schema associated with a data target that you want to load, and then show you how to start, upload, and commit a data load transaction.

To demonstrate we'll take you through a hypothetical scenario where we use the Direct Data Intake API to answer a business question.

You will learn how to:

  • Call the Direct Data Intake API to retrieve an objects data load schema
  • Send data to Visier in a text/csv format

Prerequisites

Before beginning this tutorial, you will need:

  • A Visier account with a profile that has the following capabilities. If you don't have an account, contact your administrator.
    • Data: Write (Access Level), Simple (View Level)
    • Model: Write (Access Level), Simple (View Level)

    • Request Auth Token

    • Manage Jobs
    • Data Upload
    • Direct Publish
  • Enable the deltaJob feature toggle in your Visier tenant. To enable this feature, contact Visier Technical Support.
  • API key, and security token. See API Authentication.
  • An API client such as Postman.
  • Foundational understanding of Visier’s data model.
  • Transformed data to send to Visier.

Scenario

As a new Visier customer, we are excited to get started using the application. However, we currently store our data in a data warehouse and have established procedures for data transformation. We would like to continue leveraging our existing tools for data transformation and subsequently load the transformed data into Visier.

Steps

Retrieve data load schema associated with a target object

The Direct Data Intake (DDI) API treats Visier objects as direct data targets, each with a specific structure that defines how the object receives data, this is known as a data load schema.

Before we send our data to Visier, we need to retrieve the data load schema associated with the target object we want to load data into. This will help us verify that the columns in our data file match with the data target's load schema.

Knowing we want to load employee exit data, an event type object, we also need to load data for the subject object associated with it, in this case the employee object.

Note: To access the data loaded into a event object type, for example Employee_Exit, you must load data for the associated subject Employee as well. If no data is loaded into the associated subject object, the event object data will not be available once loaded.

Based on the data we want to load, we need to retrieve the data load schema for the Employee and Employee_Exit target objects.

Let's send our first request to retrieve the data load schema for Employee.

In each request in this tutorial we will replace the {vanity_name} with our tenant name jupiter and include the API key and security token as a header.

Copy

cURL sample request: Retrieve data load schema for Employee

curl -X GET --url 'https://jupiter.api.visier.io/v1/data/directloads/prod/schemas/Employee' \
-H 'apikey:12345' \
-H 'Cookie:VisierASIDToken=abc123'

The response will include the following fields:

Field

Description

name

The field’s column name. Column names are case sensitive.

dataType

The column’s data type.

formats

The column’s accepted formats, such as date formats like “yyyy-MM-dd”.

isMandatory

If true, the field must contain a value to successfully load data into the object.

emptyValuesAllowed

If true, the value may be empty.

Once we retrieve the data load schema for Employee, we can verify that the column headings in our data file match the data load schema.

Retrieve data load schema associated with an additional target object

Let's send our second request to retrieve the data load schema for Employee_Exit.

Copy

cURL sample request: Retrieve data load schema for Employee_Exit

curl -X GET --url 'https://jupiter.api.visier.io/v1/data/directloads/prod/schemas/Employee_Exit' \
-H 'apikey:12345' \
-H 'Cookie:VisierASIDToken=abc123'
Copy

Sample response: The Employee_Exit object’s data load schema

{
    "schema": [
        {
            "name": "EmployeeID",
            "dataType": "TEXT",
            "isMandatory": true
        },
        {
            "name": "EventDate",
            "dataType": "DATE",
            "formats": [
                "yyyy-MM-dd",
                "MM/dd/yyyy",
                "dd-MM-yyyy"
            ],
            "isMandatory": true
        },
        {
            "name": "Employer_Post_Termination",
            "dataType": "TEXT",
            "emptyValuesAllowed": true
        },
        {
            "name": "Termination_Reason-Termination_Detailed_Reason",
            "dataType": "TEXT",
            "emptyValuesAllowed": true
        },
        {
            "name": "Termination_Reason-Termination_Reason",
            "dataType": "TEXT",
            "emptyValuesAllowed": true
        },
        {
            "name": "Termination_Type",
            "dataType": "TEXT",
            "emptyValuesAllowed": true
        }
    ]
}

Once we retrieve the data load schema for Employee_Exit, we can verify that the column headings in our data file match the data load schema.

By looking through the response, we can see the following lines include mandatory fields.

  • Line 6 shows the data name EmployeeID isMandatory
  • Line 16 shows the data name EventDate isMandatory

This tells us the EmployeeID and EventDate column in our data file must contain values to successfully load data into the object. If a mandatory field is empty, the data load will fail.

Start a transaction

After verifying our data files match the data load schema for Employee and Employee_Exit, we can start a transaction for loading data into the objects.

Note: A single transaction can hold multiple data files.

Copy

cURL sample request: Start a transaction

curl -X GET --url 'https://jupiter.api.visier.io/v1/data/directloads/prod/transactions' \
-H 'apikey:12345' \
-H 'Cookie:VisierASIDToken=abc123'
Copy

Sample response: The transaction’s transaction ID

{
    "transactionId": "18fdccb1-c0be-48e2-9bc5-36be51d606e6"
}

Line 2 shows that the transaction ID is 18fdccb1-c0be-48e2-9bc5-36be51d606e6. Let’s copy the transaction ID to use in our request to upload a data file.

Upload the data file

Now that we have our transaction ID, we can upload our data file file1.csv for our target object Employee.

Note:  

  • To upload multiple files to the same transaction, you must make multiple PUT calls.
  • You can upload the following file types:
    • Text/csv: One CSV file per call.
    • Application/zip: One ZIP folder containing a single CSV file per call.

      If you use application/zip, the filename of the CSV file must match the name of the target object. The ZIP filename can be any name. For example, if uploading data for Employee_Exit with a ZIP file, the ZIP filename is anyname.zip and the CSV filename is Employee_Exit.csv.

Note: The data files must be in a location that your API application can access. If using Postman, the data files you want to upload must be in Postman’s files folder on your local device. For more information, see “Working directory” in Setting up Postman.

Copy

cURL sample request: Upload the data file for the employee target object

curl -X GET --url 'https://jupiter.api.visier.io/v1/data/directloads/prod/transactions/18fdccb1-c0be-48e2-9bc5-36be51d606e6/Employee' \
-H 'apikey:12345' \
-H 'Cookie:VisierASIDToken=abc123' \
-F files=@/path/to/data/file1.csv
Copy

Sample response: A successful upload

{
  "transactionId": "18fdccb1-c0be-48e2-9bc5-36be51d606e6",
  "uploadStatus": "SUCCEEDED"
}

Lets make another request to upload our data file file2.csv for our target object Employee_Exit in the same transaction.

Copy

cURL sample request: Upload the data file for the employee_exit target object

curl -X GET --url 'https://jupiter.api.visier.io/v1/data/directloads/prod/transactions/18fdccb1-c0be-48e2-9bc5-36be51d606e6/Employee_Exit' \
-H 'apikey:12345' \
-H 'Cookie:VisierASIDToken=abc123' \
-F files=@/path/to/data/file2.csv
Copy

Sample response: A successful upload

{
  "transactionId": "18fdccb1-c0be-48e2-9bc5-36be51d606e6",
  "uploadStatus": "SUCCEEDED"
}

Roll back the transaction

After we’ve uploaded our data files, the transaction remains uncommitted and the files have not started to process. During this stage, we have the option to roll back a transaction to remove all uploads in the transaction. This can be helpful if we need to correct any files we’ve uploaded.

Note: After rolling back a transaction, the transaction ID cannot be used again.

Copy

cURL sample request: Roll back transaction

curl -X DELETE --url 'https://jupiter.api.visier.io/v1/data/directloads/prod/transactions/18fdccb1-c0be-48e2-9bc5-36be51d606e6' \
-H 'apikey:12345' \
-H 'Cookie:VisierASIDToken=abc123'

This is optional, and in our case, we do not need to roll back the transaction and will move to the next step.

Commit the transaction

Now that we’ve successfully uploaded our data files for Employee and Employee_Exit, we can commit the transaction. This starts the processing job to load the data files into Visier.

Note: After committing a transaction, you cannot upload additional files to the transaction.

Copy

cURL sample request: Commit transaction

curl -X POST --url 'https://jupiter.api.visier.io/v1/data/directloads/prod/transactions/18fdccb1-c0be-48e2-9bc5-36be51d606e6' \
-H 'apikey:12345' \
-H 'Cookie:VisierASIDToken=abc123'
Copy

Sample response: A successful commit

{
  "transactionId": "18fdccb1-c0be-48e2-9bc5-36be51d606e6",
  "uploadStatus": "SUCCEEDED",
  "message": "ReceivingDataJob scheduled"
}

Check the status of the processing job

After we’ve successfully committed the transaction, we can check the status of the underlying processing job.

Copy

cURL sample request: Check job status

curl -X GET --url 'https://jupiter.api.visier.io/v1/data/directloads/prod/transactions/18fdccb1-c0be-48e2-9bc5-36be51d606e6 \
-H 'apikey:12345' \
-H 'Cookie:VisierASIDToken=abc123'
Copy

Sample response: The job has successfully completed

{
  "tenantCode": "WFF_j2r",
  "transactionId": "18fdccb1-c0be-48e2-9bc5-36be51d606e6",
  "jobId": "ef054370-c533-4574-bf96-59e0b75f33a2",
  "status": "SUCCEEDED"
}

Now you know how to discover the data load schema associated with a data target and how to start, upload, and commit a data load transaction using the Direct Data Intake API.

As a result, we can continue to utilize our existing data transformation tools and incorporate Visier into our data transformation workflows to then load the transformed data into Visier.

Resources