Updates
Pipeline Processor versions
The Pipeline Processor determines how your data is processed. Updating to a newer version may change your pipeline results.
Test how the changes affect your data by creating a duplicate pipeline and updating the version.
Updating the Pipeline Processor version
The latest Pipeline Processor version is applied to newly created pipelines automatically.
For existing pipelines, an icon appears in the top right indicating that a new version is available.
Clicking the update button will re-interpret your uploaded data using the new processor version.
Updating may change your uploaded data and affect your pipeline results.
Before updating a pipeline, it's recommended to test how the changes will affect your data by creating a duplicate of the pipeline and updating its processor version.
Datasets linked from the Data Repo will not be automatically updated to the new version.
To update a dataset, navigate to the dataset and follow the instructions
here.
Version 2
November 2019
Version 2 contains updates in three areas:
1. Interpreting data in .xlsx files
The way data in uploaded .xlsx files is handled has been changed so that values generated by formulae are treated in the same way as regular values.
The changes are:
1.
Columns generated by formulae are now interpreted as values rather than as formatted text.
2.
Dates created by formulae are now interpreted as ISO rather than as formatted text.
3.
Results of booleans (e.g. IF statements) are now interpreted ‘true’ / ‘false’ rather than ‘1’/’0’.
4.
Results of formulae with text in quotes now retain the quotes rather than removing them.
Example
|
|
|
|
|
|
|
|
$10.00
|
01/10/2019
|
1
|
string
|
|
|
10
|
20191001
|
true
|
“string”
|
2. Arithmetic equations treat blank as zero
In Calculate and Validate, arithmetic equations ( + - / * ^ ) and some functions (ABS, ROUND) now interpret blank values as zero. The functions MIN and MAX now ignore non-numeric values.
Previously, if these functions encountered a blank value they would return ‘NaN’ (‘Not a Number’), which could cause problems with downstream processes.
Now the functions will always give a valid numeric result when encountering blanks.
Inequalities ( < <= > => ) will still produce a NaN if encountering a blank.
Example
|
|
|
|
|
|
|
2 + ‘blank’ = NaN
|
MIN(‘blank’,1) = NaN
|
MIN(‘blank’,-1) = NaN
|
|
|
2 + ‘blank’ = 2
|
MIN(‘blank’,1) = 1
|
MIN(‘blank’,-1) = -1
|
3. Flexible interpretation of column values
In Calculate and Validate a numeric value can be the result of a text function, and vice-versa.
Previously the whole calculation would be ignored with the result column not being created/updated.
Example
|
|
|
|
|
|
|
1
|
IF('Age'="", "", 'Age' + 1) = Column not created
|
ABS( CONCATENATE("-", "1")) = Column not created
|
|
|
1
|
IF('Age'="", "", 'Age' + 1) = 2
|
ABS( CONCATENATE("-", "1")) = 1
|
Version 1
Original Pipeline Processor version.
