Lineage data kinship

[ChenpiDog]

This is a very important and meaningful function that can be prioritized. Generally, the data blood relationship of table level can meet the needs of most enterprises.
This function mainly has two important points:

1. Visualization: Display the blood relationship between tables in the form of DAG.
2. Metadata: The data blood relationship needs to be stored for use in other places. The graph database is very suitable for storing such information.

[saul-data]

@ChenpiDog - this has come up with enterprise users of Dataplane and I'd like to understand it more. My basic understanding of lineage is that it helps to understand if we make an upstream change, we will understand the knock on impacts downstream. Are there any other benefits that can be articulated?

From a product design perspective, would you be available to develop some wireframes in Figma to show how this could best work if we implemented these features?

[ChenpiDog]

It can be used for data quality monitoring customized by some enterprises. If a table has data missing or other data quality problems, I need to quickly locate the problem link, whether it is its own problem or the table it uses. Of course, the check logic is implemented by the user through SQL or Python, but the platform needs to provide data blood relationship, so that the whole data link can be traced.

[ChenpiDog]

It is convenient for users to trace the source of data problems. Users can see the entire data context through a visual interface, or they have to look at the code. Obviously, the former is more efficient.

[ChenpiDog]

Does "enterprise users" mean that this function will not be open to the community?

[ChenpiDog]

There are two main aspects to consider in visualization:

1. Automatically generate Lineage data kinship from online code (such as SQL);
2. It is necessary to consider the views and materialized views of the database, which are generally stored in the database meta information table.

Note: The global Lineage data kinship needs to combine the above two points. For example, the source table used in my SQL is a view.

[saul-data]

Thanks this is very helpful and great feedback. I think with this request, if there is support from the community to help design the feature, build and maintain, I think it makes sense to open source. I do find lineage important, it is just matter of when and whats the best way to give these features ongoing support.