Data

Data semantics in Quantemplate comprises automated data type detection, data profiling, data type validation and column header mapping.

Whenever you create or import a dataset to Quantemplate, it analyses the contents of every column, identifying its semantic type and presenting a summary chart to the user. Clicking on the chart opens the profiling popup, containing a detailed analysis of the data and its quality.

Data typing

Quantemplate automatically detects dozens of unique data types across multiple industries. See the full reference table of auto-detected types below.

Typing informs the Map Column Headers matching algorithm, giving stronger suggestion weightings to matches that have the same type.

Typing is also used for automated type validation. In Map column headers, define in the Master schema by adding a dataset where types have been detected, then enable validations for the fields you want to validate. When the pipeline runs, it checks the incoming data meets the expected type and creates an entry in the validation report detailing exceptions.

Note: Type detection applies to columns containing at least one row of data. Blank columns are not typed. The detected type is based on the first 1000 rows.

Data profiling

Data profiling helps you rapidly understand the data and spot any exceptions. Every data grid in Quantemplate (in data repo, pipeline outputs or feeds) displays a data type and profile chart at the top of the column. The profile chart shows:

• Distribution histogram showing the spread of values in the column, for numeric types and free text.
• Category bar chart showing the proportion of rows in each category, for categorical data.
• Matches vs mismatches when a specific ID format has been detected
• Format exceptions indicated by a red line below the chart, showing the proportion of values which may not meet the detected format.

Profile popup

Click on the chart to open the profile popup.

The profile popup shows:

• Detailed statistics such as min, max, mean, std, dev, variance, etc.
• Count of blanks and unique values
• Distribution chart
• Preview of the first 1000 rows
• First 1000 unique values
• Proportion of mismatching values, where the specified type is not met

Profiling is performed across the whole dataset.

Auto-detected data types

General types

Specific industry types

Related articles

• Data type validation
• Map Column Headers