Data Load Deletions and Corrections

Learn how to delete and correct subject data, profile records, and events.

Overview

Project data can be deleted or corrected to ensure your data is accurate and up-to-date. The following examples show the recommended methods and examples for data deletion and correction for specific data types within your project.

Restating data

Often, the easiest method to delete data and capture retroactive source system changes is to resend your data in full and exclude all previously-sent data.

This method must be adopted if you have no means of correcting data in your source systems, or you are able to correct data in your source systems but are not able to identify the effective date of the previously sent records. For more information about sending your full history see Send Data to Visier.

Common fields

The following attributes are required per source for data deletion and corrections:

• TenantCode: The unique tenant code for the data being provided. To send data to an administrating tenant, use its tenant code. For example "WFF_e2b3d". To send data to an analytic tenant use its tenant code. For example "WFF_e2b3d~1234". The tenant code makes it easier to identify which tenant a row of data belongs to. This ensures data ends up in the right tenant, reducing the risk of data leakage.
• LoadTimestamp: A datetime with a minimum of seconds precision is included as a column to support incremental loads. For data connectors that wish to leverage delta extractions, this date and time is referenced during the extract and determines which records are pulled. The format is "yyyy-mm-dd hh:mm:ss". For example, "2022-04-30 13:02:59". This value is independent of the EffectiveDate attribute, as Visier supports the ability to back-date a change to indicate that this record has been updated recently. For more information, see Data In Connectors.
• EffectiveDate: The date the attribute value is valid. For example, if a JobName attribute changes from Consultant to Senior Consultant on April 1, 2022 the EffectiveDate is "2022-04-01 13:00:30" (including time precision) for the Senior Consultant row.
• Action: An identifier to determine what action should be taken for the data row. A value of "A" dictates that a corrected value should be inserted. A value of "D" dictates that a record should be deleted. The Action identifier is not required when modifying the entire history of the subject's data.

The attribute titles in your schema may differ from what is above, but the attributes must be included in your source data. Contact your Customer Success Manager for help defining your schema if necessary.

Deleting subject data

To delete the entire history of a subject (SubjectID) without excluding prior files and restating data, you can send a separate file specifying which SubjectIDs to delete.

This file must contain the full history of all SubjectIDs requested to be deleted as the revised file is treated as the single source of truth and all previous files will be ignored. This also allows you to correct a mistake if a SubjectID should no longer be deleted.

Before sending the data to Visier, create and manage an internal process to remove IDs from this file for any employees who should not be deleted and need to be reloaded.

The table below shows an example of the file schema to delete records with the following columns:

• The LoadTimestamp is the date that the data was extracted from source.
• The EffectiveDate can be set to the same as the LoadTimestamp, as the EffectiveDate is not always required to complete the changes.
• The SourceIdentifier is a means for Visier to identify which analytic object to delete the subject member from. This is a necessary attribute and must be in your analytic object.

TenantCode	LoadTimeStamp	EffectiveDate	SubjectID	SourceIdentifier
WFF_e2b3d~TenantA	2022-01-25 00:00:00	2022-01-25 00:00:00	EID123456	Employee
WFF_e2b3d~TenantA	2022-01-25 00:00:00	2022-01-25 00:00:00	EID999982	Employee
WFF_e2b3d~TenantA	2022-01-25 00:00:00	2022-01-25 00:00:00	1234567	Skills
WFF_e2b3d~TenantA	2022-01-25 00:00:00	2022-01-25 00:00:00	FGER234	Requisition

Sending the table above to Visier will remove a subject, not an event. To remove an event see below.

Correcting subject records

You can correct subject data by resending the corrected records along with a full history for the SubjectID similar to the table shown above. For more information, see Correct File Errors With Restatement.

Deleting profile records

Records cannot be deleted directly. Instead, this process is better described as changes to properties of a subject where the result of these changes do not produce the attribute information in the application.

To clear a subject attribute value, insert a record with the same timestamp (EffectiveDate) and key (EmployeeID) previously sent, but leave the Value blank. Starting from this insertion timestamp, the attribute appears as "Unknown" in the application. If you need to remove multiple attribute values, insert a separate blank record for each value.

For example, if you want to remove an employee from your headcount, you can remove the employee’s hire record. When the hire record is removed, an entity is set automatically as inactive and is not counted in the application.

The following table is an example of a section from an original file sent to Visier.

TenantCode	LoadTimeStamp	EffectiveDate	EmployeeID	Attribute	Value	Action
WFF_e2b3d~TenantA	2022-04-30 13:02:59	2022-04-01 00:00:00	1234567	JobName	Senior Consultant	A
WFF_e2b3d~TenantA	2022-04-30 13:02:59	2022-04-01 00:00:00	1234567	Gender	Female	A
WFF_e2b3d~TenantA	2022-04-30 13:02:59	2022-04-01 00:00:00	1234567	Location 0	EMEA	A
WFF_e2b3d~TenantA	2022-04-30 13:02:59	2022-04-01 00:00:00	1234567	Location 1	Germany	A

The following table is an example section from the full history for employee 1234567. Sending this revision in the full table to Visier will delete the JobName and Location 1 records. Note there is no data provided in the Value column.

TenantCode	LoadTimeStamp	EffectiveDate	EmployeeID	Attribute	Value	Action
WFF_e2b3d~TenantA	2022-05-02 12:42:43	2022-04-01 00:00:00	1234567	JobName		A
WFF_e2b3d~TenantA	2022-05-02 12:42:43	2022-04-01 00:00:00	1234567	Location 1		A

Correcting profile records

The following scenarios describe how Visier corrects profile records. The examples follow the standard record format, though your actual format may vary. Each example table focuses only on the columns relevant to the scenario and does not represent the full dataset.

In the following scenarios, you only need to alter new data that has been added since the last load, rather than all of the table's data. As long as the EmployeeID, EffectiveDate, and Attribute match an older record, the new record corrects the old record by replacing it in the application.

Scenario 1

The attribute value sent previously was incorrect and needs to be updated.

In the example scenario a JobName value of Senior Consultant should have been Consultant effective April 1, 2022. In your source system you have overridden the value but retained the same EffectiveDate.

The original record sent to Visier:

TenantCode	LoadTimeStamp	EffectiveDate	EmployeeID	Attribute	Value	Action
WFF_e2b3d~TenantA	2022-04-30 13:02:59	2022-04-01 00:00:00	1234567	JobName	Senior Consultant	A

New record to send to Visier:

TenantCode	LoadTimeStamp	EffectiveDate	EmployeeID	Attribute	Value	Action
WFF_e2b3d~TenantA	2022-05-30 16:22:21	2022-04-01 00:00:00	1234567	JobName	Consultant	A

The JobName attribute value changed from Senior Consultant to Consultant, but the EffectiveDate remains the same as the previous record. The latest record sent overrides the previous record. Note that the LoadTimeStamp is a later date, indicating that this record has been updated.

Scenario 2

The EffectiveDate and the attribute value sent previously are incorrect and need to be updated with a new EffectiveDate that is after the original EffectiveDate.

The original record sent to Visier:

TenantCode	LoadTimeStamp	EffectiveDate	EmployeeID	Attribute	Value	Action
WFF_e2b3d~TenantA	2022-04-30 13:02:59	2022-04-01 00:00:00	1234567	JobName	Senior Consultant	A

Desired result:

TenantCode	LoadTimeStamp	EffectiveDate	EmployeeID	Attribute	Value	Action
WFF_e2b3d~TenantA	2022-04-30 13:02:59	2022-05-01 00:00:00	1234567	JobName	Consultant	A

New records to send to Visier:

TenantCode	LoadTimeStamp	EffectiveDate	EmployeeID	Attribute	Value	Action
WFF_e2b3d~TenantA	2022-05-30 16:22:21	2022-04-01 00:00:00	1234567	JobName		A
WFF_e2b3d~TenantA	2022-05-30 16:22:21	2022-05-01 00:00:00	1234567	JobName	Consultant	A

Note the blank value in the first record and the change in EffectiveDate of the second record. Send the original record with a blank value for the attribute to null the original value.

Scenario 3

The EffectiveDate and the Value sent previously are incorrect and need to be updated. The new EffectiveDate is before the original EffectiveDate.

The original record sent to Visier:

TenantCode	LoadTimeStamp	EffectiveDate	EmployeeID	Attribute	Value	Action
WFF_e2b3d~TenantA	2022-04-30 13:02:59	2022-04-01 00:00:00	1234567	JobName	Senior Consultant	A

Desired result:

TenantCode	LoadTimeStamp	EffectiveDate	EmployeeID	Attribute	Value	Action
WFF_e2b3d~TenantA	2022-04-30 13:02:59	2022-03-01 00:00:00	1234567	JobName	Consultant	A

New records to send to Visier:

TenantCode	LoadTimeStamp	EffectiveDate	EmployeeID	Attribute	Value	Action
WFF_e2b3d~TenantA	2022-05-30 16:22:21	2022-04-01 00:00:00	1234567	JobName	Consultant	A
WFF_e2b3d~TenantA	2022-05-30 16:22:21	2022-03-01 00:00:00	1234567	JobName	Consultant	A

In the case of an incorrect attribute Value being previously sent, you need to send the original EffectiveDate with the corrected value and the new EffectiveDate with the corrected value.

Deleting and correcting an event

At this point, all methods for deleting and correcting features in your Visier application have been leveraging a regular mapping workflow. Deleting and correcting an event in your application is a separate correction workflow and will differ from the methods outlined above. If you configure your system to always send the full history for an event when something changes for a given ID, the standard restatement behavior will automatically handle corrections and deletions.

However, as there is no data to restate in a full removal, an actual deletion is required. Each example table focuses only on the columns relevant to the scenario and does not represent the full dataset.

To delete an event, send a new event record with the same schema and EffectiveDate with the Action column value set to “D”. If the deletion record does not match the existing record, the deletion will not occur. All properties on the original event must be present with matching values.

Scenario 1

The existing event in Visier:

TenantCode	LoadTimeStamp	EffectiveDate	EmployeeID	StartType	StartReason	Action
WFF_e2b3d~TenantA	2022-04-30 13:02:59	2022-04-01 00:00:00	1234567	Hire	Growth	A

To correct the StartType and StartReason, send the following information to Visier. Note the change in the StartType, StartReason, and Action values:

TenantCode	LoadTimeStamp	EffectiveDate	EmployeeID	StartType	StartReason	Action
WFF_e2b3d~TenantA	2022-05-30 16:22:21	2022-04-01 00:00:00	1234567	Hire	Growth	D
WFF_e2b3d~TenantA	2022-05-30 16:22:21	2022-04-01 00:00:00	1234567	Rehire	Other	A

Scenario 2

The existing event in Visier:

TenantCode	LoadTimeStamp	EffectiveDate	EmployeeID	StartType	StartReason	Action
WFF_e2b3d~TenantA	2022-04-30 13:02:59	2022-04-01 00:00:00	1234567	Hire	Growth	A

To correct or update the event's EffectiveDate, send the following information to Visier:

TenantCode	LoadTimeStamp	EffectiveDate	EmployeeID	StartType	StartReason	Action
WFF_e2b3d~TenantA	2022-05-30 16:22:21	2022-04-01 00:00:00	1234567	Hire	Growth	D
WFF_e2b3d~TenantA	2022-05-30 16:22:21	2022-05-01 00:00:00	1234567	Hire	Growth	A