Source Settings

Learn about the settings you can change for a source.

Overview

Each source has a set of configurable settings, such as the file regex and cleansing options. Source settings are changed in the global workspace and once saved, changes cannot be rolled back.

To find the source's settings:

1. On the global navigation bar, click Data > Sources.
2. Select a source.
3. Navigate to Settings.

File regex

Records supplied through a file require a regular expression (regex) to select the file in a folder.

Recommended: Write as specific an expression as possible to avoid mistakenly accepting the wrong file. Keep in mind that regular expressions are not shell wildcards and the dot (.) character is a wildcard.

Narrow the expression by including the file extension. For example, use \.txt for text and \.xlsx for Microsoft Excel®. Include the item the source represents with strings like .*Employee.* or .*Exit.*.

You may broaden the expression with the wildcard (.) and quantifications: ?, +, *, or {n}. As well, prepending (?i) makes the expression case insensitive.

Sheet name

If the source originates from an XLS or XLSX extension containing multiple sheets, the file regex includes the Sheet name setting. The sheet name is an exact name match to the name of the sheet from which the source was generated. This allows the source to load records in future data transfers if there is an exact sheet name match in the transfer.

If the sheet name is blank but the schema of the sheet matches the source, the source will still load records from the data transfer. This allows the source to recognize the correct source sheet without requiring an exact name match, however, it will attempt to load all sheets to a source that match the original regex and may load too many sheets.

Delimiter

A delimiter is a character used to separate individual fields in a record. You can either select or type a delimiter.

Escape character

If the delimiter is part of a string and is to appear in the data, use an escape character to stop Visier from splitting the value in two. For example, if double quote (") is the escape character and comma (,) the delimiter, the platform renders the following input spring as a unitary value: "Director, PMO".

Encoding

The encoding schema for the characters in a source must be set.

Visier supports many different encodings in families such as IBM, ISO, JIS, Windows, x-IBM, x-Mac and x-Windows. A complete list is shown in the solution. Common encodings are ISO-8859-1, UTF-8, UTF-16LE, and Windows-1252. Our preferred encoding is UTF-8.

There are some practical aspects when working with encodings:

• Visier converts all accepted encodings to UTF-8.
• You can validate any selected encoding against the source.
• Incorrect encoding may lead to an error message that the source has no input data.
• The encoding must stay consistent over time from one load to another. Inconsistent file encodings result in load failures.

  Note: If you change the encoding and that change is consistent going forward, you can adjust the source setting for encoding with no impact to the previously received and loaded files in the source.
  

Skip lines

You can configure the source settings to skip a specific number of lines at the beginning or end of a source file. Skip line settings are applied to every file added to the source; ensure that any uploaded files are consistent and require the configured skip line settings.

• Skip beginning lines: The first line of a source file should include a header listing the column names. Some source files include a preamble above the header such as a notice of confidentiality or provenance. You can skip a specified number of lines at the beginning of the source file, such as the header and any preamble lines, by setting the Skip beginning lines value.
• Skip end lines: Some source files contain trailing lines that aren’t needed in Visier, such as lines for metadata records. You can skip unnecessary lines at the end of your source file by setting the Skip end lines value.

Cleansing options

In cleansing a source, Visier (the loader) moves the records, checks they are in proper format, and changes the encoding. The loader strips out blank lines and applies other cleansing tasks specified for the source.

Use these settings to specify how to remove unneeded elements from the source. For example, the loader may strip new lines within a cell, but new lines at the end of a record are honored. If your source includes long cell values, you can set the loader to ignore or truncate cell content in excess of 4096 characters.

The following table describes the elements involved in cleansing a source.

Element	Description
Run scripts	You may have a cleansing script to address errors in your record formats. If enabled, the loader searches for a script: /esldata/scripts/<Tenant_Name>_pre_process.sh For example, a preprocessing script for X0X tenant is: WFF_X0X_pre_process.sh
Strip line breaks	Sometimes data has fields that contain line-breaks to create text wrapping. These line breaks, which are wrapped in escape characters and placed in a value, break our loader. If enabled, the loader removes line breaks. Note: The loader always strips blank lines from files including a full row with all empty cells, and blank lines with no delimiters.
Truncate large cells	The loader has an internal limit for how many characters may be in a cell. If enabled, the loader shortens any cells in excess of Visier's limit.

In this section