In the ever-evolving landscape of data management, ensuring the accuracy and completeness of your data lineage is crucial. Custom technical lineage is a valuable tool for organizations looking to bridge the gaps in their lineage tracking. With the introduction of the new Custom Technical Lineage Batch Definition, this essential feature has been supercharged to provide even greater functionality and ease of use. In this blog post, we’ll explore why you need custom technical lineage and the benefits of this new version.
Why you need custom technical lineage
Custom technical lineage is the ideal method to create lineage for data sources that the lineage harvester does not support or to add functionality that is not supported out of the box. It empowers you to fill any gaps in your lineage, ensuring that consumers of the lineage data have a complete and accurate view.
Here are a few key reasons why custom technical lineage is essential:
- Support for unsupported data sources: The lineage harvester might not support all the data sources you use in your organization. Custom technical lineage allows you to track data from these sources effectively.
- Enhanced accuracy: By using custom technical lineage, you can ensure that every relationship and data flow is accurately represented, leading to better decision-making and compliance.
Benefits of the Custom Technical Lineage – Batch Definition
While custom technical lineage has been a valuable feature, the batch definition takes it to the next level. Here are some of the notable enhancements:
-
- Multiple file support: In the past, you were limited to working with a single file for custom lineage. Now you can provide multiple files, eliminating the complexity to merge different files. This simplifies the process and enhances flexibility.
- Streamlined asset definition: Previously, you had to define every asset used in lineage relations separately, which was both time-consuming and error prone. Now, you can do this all at once.
- Support for any asset type: Custom Technical Lineage Batch Definition is not limited to specific asset types.
- Native support for indirect and table-level lineage: This new version offers native support for indirect and table-level lineage.
- Scalability: Custom Technical Lineage Batch Definition is designed with scalability in mind, ensuring that it can handle your evolving data landscape effectively.
- Multiple file support: In the past, you were limited to working with a single file for custom lineage. Now you can provide multiple files, eliminating the complexity to merge different files. This simplifies the process and enhances flexibility.
How to get started
If you’re already using custom technical lineage, upgrading to version 2.0 is a smart move. You can take advantage of the migration script provided on our marketplace to convert your existing files to the new format seamlessly.
If you’re new to custom technical lineage, here’s how to get started:
- Explore documentation: Dive into our documentation, which contains all the details you need to get started. It includes comprehensive explanations and examples to guide you through the process.
- Python scripts: To simplify the creation of your custom lineage, you can leverage the Python scripts available on our marketplace. These scripts provide re-usable instances and helper functions and contain examples how they can be used.
Custom technical lineage is a vital tool for organizations aiming to maintain robust and accurate data lineage. With Custom Technical Lineage Batch Definition, you have access to enhanced functionality and usability, making it easier than ever to ensure your data lineage is complete and reliable.
