File Retention

Specify data retention policies in data files.

File retention allows you to automate data deletion requests from your customers. You can configure a rolling window to exclude files from data versions after a set amount of days, and optionally delete excluded files.

Automatic retention types

For each source in your tenant, you can set automatic file exclusion based on days since upload time or days since latest date in a column. Automatic file exclusion excludes the file from data versions that are generated after the exclusion date.

  • None: File retention is not set for this source. The data files in this source aren't automatically excluded from data versions.
  • Upload Time: The data files in this source are excluded based on file upload times. If selected, you can set the number of days after the Upload Time to exclude files from future data versions.
  • File Content Aware: The data files in this source are excluded from data versions based on the latest date in a selected column. If selected, you can select a source column and set the number of days after the latest date in that column to exclude files from data versions.

Example: Upload Time automatic retention

Let’s say that you want your Employee source to exclude data files that were uploaded over 1 year ago.

  1. In Data > File Retention, find the Employee source.
  2. In Automatic Retention Type, select Upload Time.
  3. In Exclude files older than (days), type 365.

In this example, the platform will calculate file retention by adding 365 days to the upload time on all data files in the Employee source. Any files that are older than 365 days since Upload Time will be automatically excluded from future data versions.

Example: File Content Aware automatic retention

Let’s say that you want your Applicant source to exclude data files that have an EventDate older than 1 year.

  1. In Data > File Retention, find the Applicant source.
  2. In Automatic Retention Type, select File Content Aware.
  3. In Column, select EventDate.
  4. In Exclude files older than (days), type 365.

In this example, the platform will calculate file retention by adding 365 days to the latest date within the EventDate column on all data files in the Applicant source. Any files with more than 365 days since the latest EventDate will be automatically excluded from future data versions.

Set up file retention

Note: To enable this feature, contact Visier Technical Support.

In the global workspace, navigate to Data > File Retention, as shown in the following screenshot.

  1. Search: Search for a source by display name.
  2. Source: The display name of the source.
  3. Automatic Retention Type: The automatic retention type for this source. For more information, see Automatic retention types.
  4. Column: The column to calculate file retention. This is only available with the File Content Aware retention type.
  5. Exclude files older than (days): The number of days after the file upload time or column latest date to exclude files.

  6. Delete excluded files: If selected, excluded files are also deleted. Visier deletes files from our systems approximately twice per week.

    Warning: Deleting the excluded files is permanent and cannot be reversed. We don't recommend selecting Delete excluded files when setting up file retention for the first time. After you set up file retention, validate that the expected files are excluded and then select Delete excluded files, if desired.

  7. Refresh: View the progress of the file retention job.
  8. Save: Save your changes and run a file retention job.

To set up file retention:

  1. In the global workspace, navigate to Data > File Retention.
  2. On the source that you want to set up file retention, select an Automatic Retention Type. For more information, see Automatic retention types.
  3. If you selected Upload Time in step 2:
    • In Exclude files older than (days), type an integer. This is the number of days after the file was uploaded to exclude the file. For example, if you type 365, the file is excluded from future data versions 365 days after the file was uploaded.
  4. If you selected File Content Aware in step 2:
    1. In Column, select a date column. This is the column to calculate file retention with. For example, if you select the EventDate column in the Employee source, the latest date in the EventDate column is used to calculate whether or not the data should be excluded.

      Note: The list includes all columns in the source that use the DATE data type.

    2. In Exclude files older than (days), type an integer. This is the number of days after the latest date in the column to exclude the file. For example, if you type 365, the file is excluded from future data versions 365 days after the latest date in the selected column.
  5. Optional: If you want the source files to be deleted after they are excluded, select Delete excluded files. Visier deletes files from our systems approximately twice per week.

    Warning: Deleting the excluded files is permanent and cannot be reversed. We don't recommend selecting Delete excluded files when setting up file retention for the first time. After you set up file retention, validate that the expected files are excluded and then select Delete excluded files, if desired.

  6. Repeat steps 2 to 5 for each source that you want to set up file retention.

  7. When finished, click Save.

    Result: A job starts running. To view the progress of the job, click Refresh. If the job is successful, the file retention configuration is implemented. If the job fails, the automatic retention type for the configured sources are reverted to None and you must try again.

    Note: One file retention job can run at a time. While a file retention job is running, you cannot configure any sources in the File Retention room.