Filter Script

The importer allows to filter the data provided by the data source. This optional step allows to only import a subset of the rows, if the data source does not provide filter mechanisms on its own (e.g. import from CSV).

For filtering, a filter script is needed, which can be defined in the Transformation section after fetching the initial preview. They can be written in Groovy. It is applied once per row and must return a boolean value whether a specific row should be processed further or not. Therefore, the example return false would skip all rows, and return true would keep all rows (same behaviour as with no script at all).

Skipped rows are handled the same way as if it was not provided by the data source in the first place. For example within an asset importer, any skipped asset row will not be imported. Additionally, if the asset has been imported before, the asset will be deleted.

The script has access to the values of the current row via the row object. It provides access to the values of the other columns via getter methods (see API section below).

If a script fails to execute on a certain row, the error is logged within the importer log and the row is skipped.

In the data preview you can see the result of the filter script. The column Filtered? tells you whether the row will be imported or not. You can also use a filter script in combination with other data transformation steps, such as the data enhancer.

In the example below, the column Instance Name.dataCenterKey, added by the data enhancer in the first transformation step is used to filter the dataset by data center. In the data preview, you can see that only columns with the data center key "azure_europe-west" are imported.

Examples

Filter out local ip addresses:

return !row.getString('ip').startsWith('127') && !row.getString('ip').startsWith('192.168')

Only process rows from Austria:

return row.getString('country') == 'AT'

Skip all rows where a deleted flag is set:

return row.isNull('deleted') == false