Updated November 10, 2020
When it comes to bringing insight into data, where it comes from and how it is used, data lineage is often put forward as a crucial feature. However, it is important to note there is technical lineage and business lineage, and both are meant for different audiences and difference purposes. In this post, we’ll clarify the differences between technical lineage and business lineage, which we also call traceability.
Technical lineage shows facts, a flow of how data moves and transforms between systems, tables and columns. Often these technical lineage diagrams produce end-to-end flows that non-technical users find unusable. This is because these diagrams show ‘as built’ transformations, staging tables, look ups, etc. This is great for technical purposes, but not for business users looking to answer questions like
- Where does my data come from?
- What policies were used?
- What standards are applied?
What is the purpose of traceability?
First of all, a traceability view is made for a certain role within the organization. Policy managers will want to see the impact of their security policy on the different data domains — ideally before they enforce the policy. Analysts will want to have a high level overview of where the data comes from, what rules were applied and where it’s being used. An auditor might want to trace a data issue to the impacted systems and business processes.
Traceability views can also be used to study the impact of introducing a new data asset or governance asset, such as a policy, on the rest of the business.
How is traceability achieved?
Any traceability view will have most of its components coming in from the data management stack. Systems, profiling rules, tables, and columns of information will be taken in from their relevant systems or from a technical metadata layer. Where the true power of traceability (and data governance in general) lies, is in the information that business users can add on top of it.
As an example, envision a program manager in charge of a set of Customer 360 projects who wants to govern data assets from an agile, project point-of-view. By building a view that shows projects and their relations to data domains, this user can see the data elements (technical) that are related to his or her projects (business).
Summing it up
Good technical lineage is a necessity for any enterprise data management program. It does not, however, fulfill the needs of business users to trace and link their data assets through their non-technical world. The right solution will curate high quality and trustworthy technical assets and allow different lines of business to add and link business terms, processes, policies, and any other data concept modelled by the organization.
Enabling customizable traceability, or business lineage views that combine both business and technical information, is critical to understanding data and using it effectively and the next step into establishing data as a trusted asset in the organization.
