Append metadata allows dataset and pipeline metadata to be appended to the output data as new columns.
The fields that can currently be appended are:
Pipeline ID
The unique reference Quantemplate gives to every pipeline and is used in the pipeline’s URL.
This is useful for audit purposes where it is necessary to prove that a dataset originated from a specific pipeline.
Pipeline name
The name of the pipeline at run time. This is useful to augment the Pipeline ID above.
Pipeline run date
The date and/or time of the start of the pipeline run.
Date-time will be recorded in the timezone set in this operation, which may be different from the user’s local timezone.
This ensures consistent results when working with teams across multiple time zones, or across daylight-saving time periods.
The field can optionally include the date, time and timezone in the same column.
Pipeline run number
The pipeline run number which generated the output dataset.
This is useful to link a dataset to a specific run number, enabling review of validations and seeing time of run.
Stage input filename
The name of the input file entering the stage.
This is useful for tracing the input file or extracting a company name or date from the file name.
If there are more metadata fields which would be useful to you, please contact support.
Using the Append Metadata operation
To add and configure the Append Metadata operation:
From the operations menu, select the Append Metadata operation.
Select the metadata fields to be added.
If needed, edit the names of the new columns which will be created.
Click the plus button to add another metadata field. Each field can only be added once.
To remove a metadata field, click the cross button which appears on hovering a field block.
Example
Append the filename to multiple stage inputs, then union the files.
Input name: Acme-bdx-2022-08
Policy No
Premium
H24788
100
H84729
120
H89722
130
Input name: NewCo-bdx-2022-08
Policy No
Premium
8723XYZ
140
8738GHI
150
8264GUI
160
Append Metadata operation
Field: Filename
Column name: Input file
Union stage
Feed both outputs from Append Metadata into a Union stage.
Output from Union stage
Policy No
Premium
Input file
H24788
100
Acme-bdx-2022-08
H84729
110
Acme-bdx-2022-08
H89722
130
Acme-bdx-2022-08
8723XYZ
140
NewCo-bdx-2022-08
8738GHI
150
NewCo-bdx-2022-08
8264GUI
160
NewCo-bdx-2022-08
Note: the Append Metadata operation replaces the Append Filename operation.
Append Filename will continue to be supported in existing pipelines but should not be added to new pipelines.