Visier Extraction Language (VEL)

Use the Visier Extraction Language to extract and transform data in Visier.

Overview

Use the Visier Extraction Language (VEL) to extract or transform records from your source data in Visier. VEL formulas are written as a single expression that extract or transform records in sources. In Visier, you can use the VEL in the following places:

  • In a mapping’s record filter to select or ignore records in a source.

  • In formulas used to map properties from source data.

  • In data transfers to filter row data.

Basic syntax

Read column values from your data source so that you can alter them when necessary.

column

Returns the value of a column.

You can optionally specify whether the column is optional with the isOptional parameter, where true is optional and false is mandatory. Default is false. This is useful if you have non-mandatory columns defined on the source.

  • If isOptional is true and the column doesn't exist in the source file, the rule returns none for each row because the column is optional.
  • If isOptional is false and the column doesn't exist in the source file, the rule does nothing because the column is mandatory.

dateColumn

Returns a date value based on the format defined in the source column settings.

You can optionally specify whether the column is optional with the isOptional parameter, where true is optional and false is mandatory. Default is false. This is useful if you have non-mandatory columns defined on the source.

  • If isOptional is true and the column doesn't exist in the source file, the rule returns none for each row because the column is optional.
  • If isOptional is false and the column doesn't exist in the source file, the rule does nothing because the column is mandatory.

doubleColumn

Returns the value of a double column.

You can optionally specify whether the column is optional with the isOptional parameter, where true is optional and false is mandatory. Default is false. This is useful if you have non-mandatory columns defined on the source.

  • If isOptional is true and the column doesn't exist in the source file, the rule returns none for each row because the column is optional.
  • If isOptional is false and the column doesn't exist in the source file, the rule does nothing because the column is mandatory.

intColumn

Returns the value of a integer column.

You can optionally specify whether the column is optional with the isOptional parameter, where true is optional and false is mandatory. Default is false. This is useful if you have non-mandatory columns defined on the source.

  • If isOptional is true and the column doesn't exist in the source file, the rule returns none for each row because the column is optional.
  • If isOptional is false and the column doesn't exist in the source file, the rule does nothing because the column is mandatory.

booleanColumn

Returns the value of a Boolean column.

You can optionally specify whether the column is optional with the isOptional parameter, where true is optional and false is mandatory. Default is false. This is useful if you have non-mandatory columns defined on the source.

  • If isOptional is true and the column doesn't exist in the source file, the rule returns none for each row because the column is optional.
  • If isOptional is false and the column doesn't exist in the source file, the rule does nothing because the column is mandatory.

jsonColumn

Returns the value from a JSON column.

Note: Valid in column formulas; not supported in record or row filters.

You can optionally specify whether the column is optional with the isOptional parameter, where true is optional and false is mandatory. Default is false. This is useful if you have non-mandatory columns defined on the source.

  • If isOptional is true and the column doesn't exist in the source file, the rule returns none for each row because the column is optional.
  • If isOptional is false and the column doesn't exist in the source file, the rule does nothing because the column is mandatory.

doubleJsonColumn

Returns a double value from a JSON column.

Note: Valid in column formulas; not supported in record or row filters.

You can optionally specify whether the column is optional with the isOptional parameter, where true is optional and false is mandatory. Default is false. This is useful if you have non-mandatory columns defined on the source.

  • If isOptional is true and the column doesn't exist in the source file, the rule returns none for each row because the column is optional.
  • If isOptional is false and the column doesn't exist in the source file, the rule does nothing because the column is mandatory.

intJsonColumn

Returns an integer value from a JSON column.

Note: Valid in column formulas; not supported in record or row filters.

You can optionally specify whether the column is optional with the isOptional parameter, where true is optional and false is mandatory. Default is false. This is useful if you have non-mandatory columns defined on the source.

  • If isOptional is true and the column doesn't exist in the source file, the rule returns none for each row because the column is optional.
  • If isOptional is false and the column doesn't exist in the source file, the rule does nothing because the column is mandatory.

booleanJsonColumn

Returns a Boolean value from a JSON column.

Note: Valid in column formulas; not supported in record or row filters.

You can optionally specify whether the column is optional with the isOptional parameter, where true is optional and false is mandatory. Default is false. This is useful if you have non-mandatory columns defined on the source.

  • If isOptional is true and the column doesn't exist in the source file, the rule returns none for each row because the column is optional.
  • If isOptional is false and the column doesn't exist in the source file, the rule does nothing because the column is mandatory.

dateJsonColumn

Returns a date value based on the format defined in the source column settings.

Note: Valid in column formulas; not supported in record or row filters.

You can optionally specify whether the column is optional with the isOptional parameter, where true is optional and false is mandatory. Default is false. This is useful if you have non-mandatory columns defined on the source.

  • If isOptional is true and the column doesn't exist in the source file, the rule returns none for each row because the column is optional.
  • If isOptional is false and the column doesn't exist in the source file, the rule does nothing because the column is mandatory.

jsonPathColumn

Returns a value described by a JSONPath.

You can optionally specify whether the column is optional with the isOptional parameter, where true is optional and false is mandatory. Default is false. This is useful if you have non-mandatory columns defined on the source.

  • If isOptional is true and the column doesn't exist in the source file, the rule returns none for each row because the column is optional.
  • If isOptional is false and the column doesn't exist in the source file, the rule does nothing because the column is mandatory.

intJsonPathColumn

Returns a integer value described by a JSONPath.

You can optionally specify whether the column is optional with the isOptional parameter, where true is optional and false is mandatory. Default is false. This is useful if you have non-mandatory columns defined on the source.

  • If isOptional is true and the column doesn't exist in the source file, the rule returns none for each row because the column is optional.
  • If isOptional is false and the column doesn't exist in the source file, the rule does nothing because the column is mandatory.

doubleJsonPathColumn

Returns a double value described by a JSONPath.

You can optionally specify whether the column is optional with the isOptional parameter, where true is optional and false is mandatory. Default is false. This is useful if you have non-mandatory columns defined on the source.

  • If isOptional is true and the column doesn't exist in the source file, the rule returns none for each row because the column is optional.
  • If isOptional is false and the column doesn't exist in the source file, the rule does nothing because the column is mandatory.

booleanJsonPathColumn

Returns a Boolean value described by a JSONPath.

You can optionally specify whether the column is optional with the isOptional parameter, where true is optional and false is mandatory. Default is false. This is useful if you have non-mandatory columns defined on the source.

  • If isOptional is true and the column doesn't exist in the source file, the rule returns none for each row because the column is optional.
  • If isOptional is false and the column doesn't exist in the source file, the rule does nothing because the column is mandatory.

dateJsonPathColumn

Returns a date value described by a JSONPath.

You can optionally specify whether the column is optional with the isOptional parameter, where true is optional and false is mandatory. Default is false. This is useful if you have non-mandatory columns defined on the source.

  • If isOptional is true and the column doesn't exist in the source file, the rule returns none for each row because the column is optional.
  • If isOptional is false and the column doesn't exist in the source file, the rule does nothing because the column is mandatory.

delimitedMultiValueColumn

Create separate rows for each delimiter-separated value.

Note:  

  • You can use functions that split rows like delimitedMultiValueColumn once per mapping.
  • Valid in column formulas; not supported in record or row filters.

You can optionally specify whether the column is optional with the isOptional parameter, where true is optional and false is mandatory. Default is false. This is useful if you have non-mandatory columns defined on the source.

  • If isOptional is true and the column doesn't exist in the source file, the rule returns none for each row because the column is optional.
  • If isOptional is false and the column doesn't exist in the source file, the rule does nothing because the column is mandatory.

identity

Returns the default value as untransformed input. This is most useful in conjunction with replace whose output is none if the input is present but not matching; if identity is used, non-matching input passes through.

Note: This cannot be used with date rules, orElse, or rules that take a collection of inputs.


none

Applies the value of none when the expression fails to resolve.

Note: Using the value none as a default value is the same as not specifying a default.


lookup

Returns a single column value from a file based on matching keys. You can optionally return a defaultValue as a fallback value.


file

Defines the file in a lookup.

Note: Valid in column formulas; not supported in record or row filters.


meltColumns

Returns the column names.


rowNumber

Returns the index of the row being processed.


filePath

Returns the path to the input file.


fileName

Returns the name of the input file.


if

Keyword for starting a conditional statement. This is used with else.


else

Keyword to define the fallback for a conditional statement. This is used with if.


Conversion functions

Convert column values to specific data types so that you can perform additional operations. For example, convert a string to an integer so that you can perform additional mathematical calculations and produce new derived values.

toString

Converts a value to a string. You can optionally return a defaultValue when the conversion fails, otherwise none is returned.


toDouble

Converts a value to a double. You can optionally return a defaultValue when the conversion fails, otherwise none is returned.


toInt

Converts a value to an integer. You can optionally return a defaultValue when the conversion fails, otherwise none is returned.


toBoolean

Converts a value to a Boolean. You can optionally return a defaultValue when the conversion fails, otherwise none is returned.


toDate

Converts a value to a date. You can optionally return a defaultValue when the conversion fails, otherwise none is returned.


intToDouble

Converts an integer to a double. You can optionally return a defaultValue when the conversion fails, otherwise none is returned.


doubleToInt

Converts a double to an integer. You can optionally return a defaultValue when the conversion fails, otherwise none is returned.


dateToString

Converts a date to a string in the specified format.


dateToInt

Converts a date to an integer.


utcEpochToDate

Converts an epoch timestamp to a date.


Rule functions

Define the column values to keep, replace, or skip.

keep

Return a value that matches a member of the collection provided, otherwise return none.


replace

Replaces values that match members in a collection.


skipNone

Skips extracted none values.


takeNotEmpty

Accepts a collection of columns and returns the first column value that isn't a none value. You can optionally return a defaultValue when all the columns return none, otherwise none is returned.


keepSuccessOnly

Skip the incorrect or empty cell value if the default value is not none.


Math operators

Use math operators to perform mathematical calculations.

+

Adds numbers.


-

Subtracts numbers.


*

Multiplies numbers.


/

Divides numbers.


Comparison operators

Use comparison operators to determine how two values compare.

=

Compares two values and determines whether they are equal.


!=

Compares two values and determines whether they are not equal.


>

Compares two values and determines whether the left value is greater than the right value.


<

Compares two values and determines whether the left value is less than the right value.


>=

Compares two values and determines whether the left value is greater than or equal to the right value.


<=

Compares two values and determines whether the left value is less than or equal to the right value.


Logical operators

Use logical operators to compare expressions and return a Boolean value.

and

Compares two expressions and returns true if both are true. This is interchangeable with &&.


&&

Compares two expressions and returns true if both are true. This is interchangeable with and.


or

Compares two expressions and returns true if either is true. This is interchangeable with ||.


||

Compares two expressions and returns true if either is true. This is interchangeable with or.


not

Compares two expressions and returns true if not true. This is interchangeable with !.


!

Compares two expressions and returns true if not true. This is interchangeable with not.


inSet

Compares a column to a set of values and returns true if there is a match.


Date functions

Use date functions to create date objects, format date objects, shift dates and get points in time related to your data source and tenant.

date

Converts a string to a date object. This is interchangeable with toDate. You can optionally return a defaultValue when the dateString is invalid.


dateToUTC

Converts the date to the UTC equivalent. You can optionally return a defaultValue when the dateString is invalid.

Note: If specified, defaultValue is used as-is, and not converted to UTC.


dateFormat

Specifies the format for a date object. You can optionally return a defaultValue when the dateString is invalid.

Note: Date formats are based on the Java SimpleDateFormat specification.


shiftMillis

Shifts a date by a number of milliseconds. You can optionally return a defaultValue when the dateString is invalid.


shiftSeconds

Shifts a date by a number of seconds. You can optionally return a defaultValue when the dateString is invalid.


shiftMinutes

Shifts a date by a number of minutes. You can optionally return a defaultValue when the dateString is invalid.


shiftHours

Shifts a date by a number of hours. You can optionally return a defaultValue when the dateString is invalid.


shiftDays

Shifts a date by a number of days. You can optionally return a defaultValue when the dateString is invalid.


shiftWeeksWithCustomCalendar

Shifts a date by a number of weeks. You can optionally return a defaultValue when the dateString is invalid.


shiftMonths

Shifts a date by a number of months. You can optionally return a defaultValue when the dateString is invalid.


shiftMonthsWithCustomCalendar

Shifts a date by a number of months. You can optionally return a defaultValue when the dateString is invalid.


shiftYears

Shifts a date by a number of years. You can optionally return a defaultValue when the dateString is invalid.


shiftYearsWithCustomCalendar

Shifts a date by a number of years. You can optionally return a defaultValue when the dateString is invalid.


firstDateInMonth

Shifts the date to the beginning of the month. You can optionally return a defaultValue when the dateString is invalid.


firstDateInMonthWithCustomCalendar

Shifts the date to the beginning of the month. You can optionally return a defaultValue when the dateString is invalid.


firstDateInQuarterWithCustomCalendar

Shifts the date to the beginning of the quarter. You can optionally return a defaultValue when the dateString is invalid.


firstDateInWeek

Shifts the date to the beginning the week. You can optionally return a defaultValue when the dateString is invalid.

Note: This rule considers Monday to be the first day of the week. If you require a different day, consider using shiftDays.


firstDateInWeekWithCustomCalendar

Shifts the date to the beginning the week. You can optionally return a defaultValue when the dateString is invalid.

Note: This rule considers Monday to be the first day of the week. If you require a different day, consider using shiftDays.


lastDateInMonth

Shifts the date to the end the month. You can optionally return a defaultValue when the dateString is invalid.


lastDateInWeek

Shifts the date to the end the week. You can optionally return a defaultValue when the dateString is invalid.

Note: This rule considers Sunday to be the last day of the week. If you require a different day, consider using shiftDays.


lastDateInWeekWithCustomCalendar

Shifts the date to the end the week. You can optionally return a defaultValue when the dateString is invalid.

Note: This rule considers Sunday to be the last day of the week. If you require a different day, consider using shiftDays.


lastDateInMonthWithCustomCalendar

Shifts the date to the end the month. You can optionally return a defaultValue when the dateString is invalid.


lastDateInQuarterWithCustomCalendar

Shifts the date to the end the quarter. You can optionally return a defaultValue when the dateString is invalid.


customCalendarDayName

Returns the day name. You can optionally return a defaultValue as a fallback value.


customCalendarWeekName

Returns the week name. You can optionally return a defaultValue as a fallback value.


customCalendarMonthName

Returns the month name. You can optionally return a defaultValue as a fallback value.


customCalendarQuarterName

Returns the quarter name. You can optionally return a defaultValue as a fallback value.


customCalendarYearName

Returns the year name. You can optionally return a defaultValue as a fallback value.


dateFromCustomCalendar

Returns the first calendar date that falls within the constraints provided. You can optionally return a defaultValue as a fallback value.


sourceTimestamp

Returns the "Snapshot time (UTC)" for the uploaded file.


tenantStartDate

Returns the start date for your tenant.


String functions

Use string functions to work with string values.

leftPadding

Adds a specified character to the left of the designated string until the length of the string reaches the specified length. In cases where the designated string exceeds the specified length, the defaultValue is used.


rightPadding

Adds a specified character to the right of the designated string until the length of the string reaches the specified length. In cases where the designated string exceeds the specified length, the defaultValue is used.


+

Concatenates string values.

Note: Using + to concatenate strings with none values is not supported and generates an error. If you need to concatenate values that can be none, consider using concat instead.


concat

Concatenates string values. You can optionally return a defaultValue when concatenating with none values, otherwise returns none.

Note: Using concat with any columns that have a none value results in a final value of none.


sha256

Computes the SHA-256 hash of a given string, returning the hash as a hexadecimal string.


toLowerCase

Converts a string to lower case. You can optionally return a defaultValue when the conversion fails, otherwise returns none.


toUpperCase

Converts a string to upper case. You can optionally return a defaultValue when the conversion fails, otherwise returns none.


trimCharactersLeft

Removes characters from the start of a string. You can optionally return a defaultValue when the sourceString is invalid, otherwise returns none.


trimCharactersRight

Removes characters from the end of a string. You can optionally return a defaultValue when the sourceString is invalid, otherwise returns none.


stringLeft

Returns a subset of characters from the left-side of a string. You can use the result of other functions like indexOf as the number of characters you want to include in your substring. Additionally, you can optionally return a defaultValue when the sourceString is invalid, otherwise returns none.


stringRight

Returns a subset of characters from the right-side of a string. You can use the result of other functions like indexOf as the number of characters you want to include in your substring. Additionally, you can optionally return a defaultValue when the sourceString is invalid, otherwise returns none.


stringDropLeft

Removes characters from the start of a string. You can use the result of other functions like lastIndexOf as the number of characters you want removed. You can optionally return a defaultValue when the sourceString is invalid, otherwise returns none.


stringDropRight

Removes characters from the end of a string. You can use the result of other functions like lastIndexOf as the number of characters you want removed. You can optionally return a defaultValue when the sourceString is invalid, otherwise returns none.


stringSubstitute

Searches and replaces in a string. You can optionally return a defaultValue when the sourceString, searchString, or replaceString is invalid, otherwise returns none.

Note: This function returns a none value when any of the parameters are set to a none value.


stringReplace

Replaces a substring. You can optionally return a defaultValue when the sourceString or replaceString is invalid, otherwise returns none.


indexOf

Returns the index of the first matched substring. Otherwise, returns -1 if no match is found. Index starts at 0. You can optionally return a defaultValue when the sourceString is invalid, otherwise returns none.


lastIndexOf

Returns the index of the last matched substring. Otherwise, returns -1 if no match is found. Index starts at 0. You can optionally return a defaultValue when the sourceString is invalid, otherwise returns none.


length

Returns the length of a string.


capitalizeAll

Converts the first character of all words in a string to upper case, and converts all other characters to lower case.


capitalizeFirstChar

Converts the first character of the first word in a string to upper case. This does not modify other characters.


format

Replaces the %s placeholder value in a string with the first argument provided. You can optionally return a defaultValue when the sourceString is invalid, otherwise returns none.


Number functions

Use number functions to work with numbers.

min

Returns the lesser of the two values.

Note: Both values must be of the same type.


max

Returns the greater of the two values.

Note: Both values must be of the same type.


mod

Returns the modulus of two values.

Note: Both values must be of the same type.


abs

Returns the absolute of a number.


floor

Returns the floor of a number.


ceiling

Returns the ceiling of a number.


round

Rounds a number.


doubleToDecimal

Rounds a double to a specified number of decimal places.


Regex functions

Use regular expression rules to manipulate values.

regexCapture

Returns the part of a string that matches the specified regular expression. You can optionally return a defaultValue when the sourceString or regexString is invalid, otherwise returns none.


regexMatch

Returns a string if it matches the specified regular expression. You can optionally return a defaultValue when the sourceString or regexString is invalid, otherwise returns none.


regexSubstitute

Searches and replaces using regular expressions.