Scripting Utilities

There are several places in Txture where user-specified scripts can be employed for customization purposes. Those include:

Filter Scripts and Dynamic Columns in Importers
Rule Bodies in Change Propagation Rules

All scripts in Txture are written in Groovy, more precisely in the statically typed subset of Groovy. In order to ease the creation and maintenance of such scripts for common tasks, Txture provides a number of utilities which are available to all scripts.

General Utilities

To ensure data consistency, particularly in scenarios like link importer configurations, the cleanString method is designed to sanitize and normalize strings. This method includes the following operations:

Trimming leading and trailing whitespace.
Removing all null bytes ('\u0000').
Eliminating carriage return ('\u000D') characters.
Replacing tab characters ('\t') with a space (' ').
Converting non-breakable spaces ('\u00A0') into regular spaces.
Removing all non-printable, non-whitespace characters.
Standardizing various dash characters to a simple hyphen ('-').
Replacing all whitespace characters with a single space (' ').
Condensing multiple consecutive whitespace characters into a single space.

cleanString

cleanString(String inputData)

Example

def incorrectData = " my  wrongly formatted asset "
def cleansed = cleanString(incorrectData)  
return cleansed // This will remove trailing and leading whitespace and condense multiple spaces into one, returning "my wrongly formatted asset".

Cleanses the provided string by performing a series of normalization and sanitization steps. It trims whitespace, removes specific unwanted characters, and standardizes spaces and dashes. The result is a more uniform and cleaner string, suitable for consistent data handling.

Parameters:

inputData The string to be cleansed. Typically, this is a variable retrieved from a column in the source data.

Returns:

A cleansed and normalized string.

Working with CSV Data

In all scripts, CSV Data can be parsed with the methods parseSingleLineCSV and parseMultiLineCSV. Both methods have the following properties in common:

All lines in the input which are either empty or consist only of whitespace (tabs, spaces, ...) will be ignored.
For each individual cell, all leading and trailing whitespace will be removed (i.e. the values will be trimmed)
The delimiter character is always an optional parameter and defaults to comma (,).
The double-quote (") is used as the quote symbol. If the delimiter appears inside a quote, it will not end the cell value. A quote character can be escaped by using another quote character.

parseSingleLineCSV

Map<String, String> parseSingleLineCSV(String csvData, char delimiter)

Example

def csvData = " Hello, World, , CSV is fun   "
def parsed = parseSingleLineCSV(csvData)  // note that we omit the delimiter, defaults to ','
// ... or:   parseSingleLineCSV(csvData, ',')

def entry0 = parsed["0"] // gives 'Hello' (without quotes; note that the leading whitespace has been removed)
def entry1 = parsed["1"] // gives 'World' (without quotes)
def entry2 = parsed["2"] // gives null (note the comma in the input data)
def entry3 = parsed["3"] // gives 'CSV is fun' (without quotes; note that the surrounding whitespace has been removed)
def size = parsed.size() // counts the entries in the map, gives 4 in this case

Parses the given string in CSV format, focusing only on the first line. Empty lines and blank lines will be ignored. All leading and trailing whitespace will be removed from all individual entries.

Parameters:

`csvData`	The raw CSV content to parse. May contain multiple lines, only the first non-blank line is considered, all others are ignored. Empty and blank lines are skipped.
`delimiter`	Optional parameter, defaults to comma (`,`). Determines the character which separates the values in a single row.

Returns:

The parsed CSV record, as a map. The map key is the String representation of the 0-based column index (e.g. "0", "1", "2"...), the map value is the value of this column.

List<Map<String, String>> parseMultiLineCSV(String csvData, boolean hasHeader, char delimiter)

Parses the given string in CSV format, considering each row. Blank or empty rows will be ignored. The first non-blank row can be treated as header by setting hasHeader to true. All leading and trailing whitespace will be removed from all individual entries.

Example 1 (no headers):

def csvData = '''
1, John, Doe, Austria
2, Jane, Doe, Austria

3, John, Smith
'''

def parsedLines = parseMultiLineCSV(csv)
def lineCount = parsedLines.size() // counts the number of parsed lines, gives 3 (note that the blank line has been ignored)

def record1 = parsedLines[0]  // get the first parsed record
def record1ID        = record1["0"] // gives "1"
def record1FirstName = record1["1"] // gives "John"
def record1LastName  = record1["2"] // gives "Doe"
def record1Country   = record1["3"] // gives "Austria"

def record2 = parsedLines[1] // get the second parsed record
def record2ID        = record2["0"] // gives "2"
def record2FirstName = record2["1"] // gives "Jane"
def record2LastName  = record2["2"] // gives "Doe"
def record2Country   = record2["3"] // gives "Austria"


def record3 = parsedLines[2] // get the third parsed record
def record3ID        = record3["0"] // gives "3"
def record3FirstName = record3["1"] // gives "John"
def record3LastName  = record3["2"] // gives "Smith"
def record3Country   = record3["3"] // gives null (source data has no value for this column)

Example 2 (with headers):

def csvData = '''
ID, FirstName, LastName, Country
1, John, Doe, Austria
2, Jane, Doe, Austria

3, John, Smith
'''

def parsedLines = parseMultiLineCSV(csv, true) // note that we set the 'hasHeaders' parameter to 'true' here.
def lineCount = parsedLines.size() // counts the number of parsed lines, gives 3 (note that the blank line has been ignored)

def record1 = parsedLines[0]  // get the first parsed record
def record1ID        = record1["ID"]        // gives "1"
def record1FirstName = record1["FirstName"] // gives "John"
def record1LastName  = record1["LastName"]  // gives "Doe"
def record1Country   = record1["Country"]   // gives "Austria"

def record2 = parsedLines[1] // get the second parsed record
def record2ID        = record2["ID"]        // gives "2"
def record2FirstName = record2["FirstName"] // gives "Jane"
def record2LastName  = record2["LastName"]  // gives "Doe"
def record2Country   = record2["Country"]   // gives "Austria"

def record3 = parsedLines[2] // get the third parsed record
def record3ID        = record3["ID"]        // gives "3"
def record3FirstName = record3["FirstName"] // gives "John"
def record3LastName  = record3["LastName"]  // gives "Smith"
def record3Country   = record3["Country"]   // gives null (source data has no value for this column)

Parameters:

`csvData`	The raw CSV content to parse. May contain multiple lines.
`hasHeader`	Optional parameter, defaults to `false`. Determines if the first row of the CSV contains column headers or not.
`delimiter`	Optional parameter, defaults to comma (`,`). Determines the character which separates the values in a single row.

Returns:

The parsed CSV records, as a list of maps. Each entry in the list is one row, which in turn is a map. If no headers are used, the map key is the String representation of the 0-based column index (e.g. "0", "1", "2"...). If headers are used, the header content is the key. The map value is the value of this column.

CSV in Filter Scripts and Dynamic Columns

When working with a row object in a Dynamic Column or Filter script, you can fetch the content of a cell and parse it at the same time.

Map<String, String> row.getSingleLineCSV(String name, char delimiter)

For the current row, get the content of the cell with the given name, and parse it as a single-line CSV (see above).

Parameters:

`name`	The name of the column to get the cell value for.
`delimiter`	Optional parameter, defaults to comma (`,`). Determines the character which separates the values in a single row.

Returns:

The parsed CSV record, as a map. The map key is the String representation of the 0-based CSV column index (e.g. "0", "1", "2"...), the map value is the value of this column.

List<Map<String, String>> row.getMultiLineCSV(String name, boolean hasHeader, char delimiter)

For the current row, get the content of the cell with the given name, and parse it as a multi-line CSV (see above).

Parameters:

`name`	The name of the column to get the cell value for.
`hasHeader`	Optional parameter, defaults to `false`. Determines if the first row of the CSV contains column headers or not.
`delimiter`	Optional parameter, defaults to comma (`,`). Determines the character which separates the values in a single row.

Returns:

The parsed CSV records, as a list of maps. Each entry in the list is one row, which in turn is a map. If no headers are used, the map key is the String representation of the 0-based CSV column index (e.g. "0", "1", "2"...). If headers are used, the CSV header content is the key. The map value is the value of this column.

Working with JSON Data

There are three ways to parse JSON data:

Parse JSON content from a string and work directly on the JSON Node model.
Parse JSON content from a string and work with lists and maps.
Parse JSON content and apply a JSON Path specification to find a certain entry.

Parsing JSON Nodes

The most general way to parse a JSON string is parseJsonNode. This method will work for all valid JSONs, however other (more specific) methods may be more convenient to use.

JsonNode parseJsonNode(String jsonContent)

Parses the given JSON string into a JSON Node.

Returns: The parsed JSON, as a JSON Node.

Example

def json = '{ "hello": "world" }'
def node = parseJsonNode(json)

// here we use the JSON Node API
node.get("hello").asText()      // returns "world"

Parameters:

jsonContent

The content to parse. If null or blank, the method will return null.

Utility Methods to parse Maps and Lists from JSON

While the JSON node model is very powerful and universal, it may be cumbersome to work with. As an alternative, you can parse JSON into regular (potentially nested) maps and lists.

Map<String, Object> parseJsonMap(String jsonContent)

Parses the given JSON string into a map. Each map key is a field in the input JSON, the map value corresponds to the value of the JSON field.

This method assumes that the top-level element in your JSON content is an object (curly brackets { }).

Example

// this is just a constant example json for demonstration purposes. 
// Your JSON data can come from import rows, property values...
def json = '''
    {
        "busStop": {
            "name": "Main Station",
            "buses": [ "A", "B", "C1" ]
        }
    }
'''

// parse the JSON into a structure of nested maps and lists.
def rootMap = parseJsonMap(json)

// note that in the lines below, we have to use Groovy's "as" operator
// to indicate which type of object we expect.

// JSON objects will be parsed into nested maps
def busStop = rootMap["busStop"] as Map<String, Object>

busStop.keySet() // will result in a set of [ "name", "buses" ]

// primitive fields (numbers, text...) are parsed into Groovy primitives
busStop["name"] as String // results in "Main Station"

// JSON arrays will be parsed into lists.
def buses = busStop["buses"] as List<String>
buses[0] // results in "A"
buses[1] // results in "B"
buses[2] // results in "C1"
// you can use other list methods as well
buses.size() // results in 3

Parameters:

jsonContent

The JSON string to parse. This method will return null if this value is null or blank.

Returns:

The parsed JSON object, as a map.

List<Object> parseJsonArray(String jsonContent)

Parses the given JSON string into a list. Each list entry can be a primitive value (for JSON primitive nodes), a map (for JSON object nodes) or another list (for JSON array nodes).

This method assumes that the top-level element in your JSON content is an array (square brackets [ ]).

Example

def json = '''
    [
        "hello",
        [ "apple", "orange" ],
        {
            "firstName": "John",
            "lastName": "Doe"
        }
    ]
'''


def rootArray = parseJsonArray(json)

rootArray.size() // results in 3

def greeting = rootArray[0] as String  // results in "hello"

def fruits = rootArray[1] as List<String>
fruits.size() // results in 2
fruits[0]     // "apple"
fruits[1]     // "orange"

def person = rootArray[2] as Map<String, Object>
person.keySet() // results in a set of [ "firstName", "lastName" ]

def firstName = person["firstName"] as String  // "John"
def lastName = person["lastName"] as String    // "Doe"

Parameters:

jsonContent

The JSON string to parse. This method will return null if this value is null or blank.

Returns:

The parsed JSON object, as a list.

Finding JSON entries with JSON Path

In many cases, you know in advance which path in your JSON structure you want to navigate through in order to extract a certain value. In such cases, using JSON Paths can be very helpful. Txture provides a number of extraction methods which allow you to specify the target as a path.

Escaping $ in Groovy

The majority of JSON paths start with the dollar ($) character, representing the root JSON node. However, in Groovy the $ character is also used for string templating. For JSON paths, you therefore need to escape the $ character with a backslash (\).

For example:

Wrong: '$.firstName' (will not compile)
Correct: '\$.firstName'

JsonNode extractJsonNode(String jsonContent, String path)

Extracts the sub-node from the given JSON content by following the specified JSON path.

Example

def json = '''
    {
        "company": {
            "name": "My Company",
            "location": {
                "city": "Innsbruck",
                "country": "Austria"
            }
        }
    }
'''

def node = extractJsonNode(json, "\$.company.location.city")

node.asText()  // returns "Innsbruck"

Parameters:

`jsonContent`	The content to parse. If `null` or blank, the method will return `null`. Instead of a `String` you can also pass in a `JsonNode`.
`path`	The JSON path expression.

Returns:

The JSON node at the end of the path, or null if nothing was found.

Map<String, Object> extractJsonMap(String jsonContent, String path)

Extracts the sub-node from the given JSON content by following the specified JSON path, and extracts it as a map.

This method assumes that the JSON element at the end of the given path is a JSON object (curly brackets { }).

Example

def json = '''
    {
        "company": {
            "name": "My Company",
            "location": {
                "city": "Innsbruck",
                "country": "Austria"
            }
        }
    }
'''

def location = extractJsonMap(json, '\$.company.location')

location["city"]    // "Innsbruck"
location["country"] // "Austria

Parameters:

`jsonContent`	The JSON content to operate on. Can also be a `JsonNode`.
`path`	The JSON Path to evaluate.

Returns:

The JSON element at the specified path location, as a map.

List<Object> extractJsonArray(String jsonContent, String path)

Extracts the sub-node from the given JSON content by following the specified JSON path, and extracts it as a list.

This method assumes that the JSON element at the end of the given path is a JSON array (square brackets [ ]).

Example

def json = '''
    {
        "busStop": {
            "name": "Main Station",
            "buses": [ "A", "B", "C1" ]
        }
    }
'''

def buses = extractJsonArray(json, '\$.busStop.buses')

buses.size() // 3
buses[0] // "A"
buses[1] // "B"
buses[2] // "C1"

Parameters:

`jsonContent`	The JSON content to operate on. Can also be a `JsonNode`.
`path`	The JSON path to evaluate.

Returns:

The JSON element at the specified path location, as a list.

String extractJsonString(String jsonContent, String path)

Extracts the sub-node from the given JSON content by following the specified JSON path, and extracts it as a string.

This method assumes that the JSON element at the end of the given path is a JSON string.

Example

def json = '''
    {
        "busStop": {
            "name": "Main Station",
            "buses": [ "A", "B", "C1" ]
        }
    }
'''

extractJsonString(json, '\$.busStop.name') // results in "Main Station"

Parameters:

`jsonContent`	The JSON content to operate on. Can also be a `JsonNode`.
`path`	The JSON path to evaluate.

Returns:

The JSON element at the specified path location, as a string.

Boolean extractJsonBoolean(String jsonContent, String path)

Extracts the sub-node from the given JSON content by following the specified JSON path, and extracts it as a boolean.

This method assumes that the JSON element at the end of the given path is a JSON string.

Example

def json = '''
    {
        "busStop": {
            "name": "Main Station",
            "buses": [ "A", "B", "C1" ],
            "active": true
        }
    }
'''

extractJsonBoolean(json, '\$.busStop.active') // results in true

Parameters:

`jsonContent`	The JSON content to operate on. Can also be a `JsonNode`.
`path`	The JSON path to evaluate.

Returns:

The JSON element at the specified path location, as a boolean.

Short extractJsonShort(String jsonContent, String path)

Extracts the sub-node from the given JSON content by following the specified JSON path, and extracts it as a short.

This method assumes that the JSON element at the end of the given path is a JSON short.

Example

def json = '''
    {
        "busStop": {
            "name": "Main Station",
            "buses": [ "A", "B", "C1" ],
            "number": 42
        }
    }
'''

extractJsonShort(json, '\$.busStop.number') // results in 42 (short)

Parameters:

`jsonContent`	The JSON content to operate on. Can also be a `JsonNode`.
`path`	The JSON path to evaluate.

Returns:

The JSON element at the specified path location, as a short.

Integer extractJsonInt(String jsonContent, String path)

Extracts the sub-node from the given JSON content by following the specified JSON path, and extracts it as an integer.

This method assumes that the JSON element at the end of the given path is a JSON integer.

Example

def json = '''
    {
        "busStop": {
            "name": "Main Station",
            "buses": [ "A", "B", "C1" ],
            "number": 42
        }
    }
'''

extractJsonInt(json, '\\$.busStop.number') // results in 42 (integer)

Parameters:

`jsonContent`	The JSON content to operate on Can also be a `JsonNode`.
`path`	The JSON path to evaluate.

Returns:

The JSON element at the specified path location, as an integer.

Long extractJsonLong(String jsonContent, String path)

Extracts the sub-node from the given JSON content by following the specified JSON path, and extracts it as a long.

This method assumes that the JSON element at the end of the given path is a JSON long.

Example

def json = '''
    {
        "busStop": {
            "name": "Main Station",
            "buses": [ "A", "B", "C1" ],
            "number": 42
        }
    }
'''

extractJsonLong(json, '\$.busStop.number') // results in 42 (long)

Parameters:

`jsonContent`	The JSON content to operate on. Can also be a `JsonNode`.
`path`	The JSON path to evaluate.

Returns:

The JSON element at the specified path location, as an integer.

Float extractJsonFloat(String jsonContent, String path)

Extracts the sub-node from the given JSON content by following the specified JSON path, and extracts it as a float.

This method assumes that the JSON element at the end of the given path is a JSON float.

Example

def json = '''
    {
        "busStop": {
            "name": "Main Station",
            "buses": [ "A", "B", "C1" ],
            "number": 9.75
        }
    }
'''

extractJsonFloat(json, '\$.busStop.number') // results in 9.75 (float)

Parameters:

`jsonContent`	The JSON content to operate on. Can also be a `JsonNode`.
`path`	The JSON path to evaluate.

Returns:

The JSON element at the specified path location, as a float.

Double extractJsonDouble(String jsonContent, String path)

Extracts the sub-node from the given JSON content by following the specified JSON path, and extracts it as a double.

This method assumes that the JSON element at the end of the given path is a JSON double.

Example

def json = '''
    {
        "busStop": {
            "name": "Main Station",
            "buses": [ "A", "B", "C1" ],
            "number": 9.75
        }
    }
'''

extractJsonDouble(json, '\$.busStop.number') // results in 9.75 (double)

Parameters:

`jsonContent`	The JSON content to operate on. Can also be a `JsonNode`.
`path`	The JSON path to evaluate.

Returns:

The JSON element at the specified path location, as a double.

JSON in Filter Scripts and Dynamic Columns

When working with a row object in a Dynamic Column or Filter Script, you can fetch the content of a cell and parse it as JSON at the same time.

JsonNode row.getJsonNode(String column)

Gets the content of the cell at the given column, and parses it into a JSON node.

See also: parseJsonNode(String jsonContent)

Parameters:

column

The column to parse the JSON from.

Returns:

The JsonNode.

Map<String, Object> row.getJsonMap(String column)

Gets the content of the cell at the given column, and parses it as JSON. Returns the map representation of the JSON structure.

This method assumes that the top-level element of the JSON structure is a JSON object (curly brackets { }).

See also: parseJsonMap(String jsonContent)

Parameters:

column

The column to parse the JSON from.

Returns:

The parsed map. Map keys are the field names of the JSON, map values are the corresponding parsed values.

List<Object> row.getJsonArray(String column)

Gets the content of the cell at the given column, and parses it as JSON. Returns the list representation of the JSON structure.

This method assumes that the top-level element of the JSON structure is a JSON array (square brackets [ ]).

See also: parseJsonArray(String jsonContent)

Parameters:

column

The column to parse the JSON from.

Returns:

The parsed list. List values can be primitives, maps (for JSON object nodes) or nested lists (for JSON array nodes).

JsonNode row.getJsonAndExtractNode(String column, String path)

Gets the content of the cell at the given column, and parses it as JSON. Evaluates the given JSON path expression on the resulting JSON structure, and returns the JSON node at the end of the path.

See also: extractJsonNode(String jsonContent, String path)

Parameters:

`column`	The column to parse the JSON from.
`path`	The JSON path to follow.

Returns:

The extracted node.

Map<String, Object> row.getJsonAndExtractMap(String column, String path)

Gets the content of the cell at the given column, and parses it as JSON. Evaluates the given JSON path expression on the resulting JSON structure, and returns the JSON node at the end of the path, in map representation.

This method assumes that the JSON node at the end of the path is a JSON object (curly brackets { }).

See also: extractJsonMap(String jsonContent, String path)

Parameters:

`column`	The column to parse the JSON from.
`path`	The JSON path to follow.

Returns:

The extracted map. Map keys are the field names of the JSON, map values are the corresponding parsed values.

List<Object> row.getJsonAndExtractArray(String column, String path)

This method assumes that the JSON node at the end of the path is a JSON array (square brackets [ ]).

Returns: The extracted list. List values can be primitives, maps (for JSON object nodes) or nested lists (for JSON array nodes).

See also: extractJsonArray(String jsonContent, String path)

Parameters:

`column`	The column to parse the JSON from.
`path`	The JSON path to follow.

Boolean row.getJsonAndExtractBoolean(String column, String path)

This method assumes that the JSON node at the end of the path is a JSON boolean (true or false).

Returns: The extracted boolean.

See also: extractJsonBoolean(String jsonContent, String path)