Add processor type

[ChenpiDog]

Dataplane can consider adding SQL and HTTP processor types, especially SQL, which is quite common in ETL.

[ChenpiDog]

Dataplane can provide a data source management (#453 is built on it), manage and reuse various data connections, such as MySQL, GreenPlum, etc. When using SQL processor, you can select data connection.

[ChenpiDog]

For ETL related processors, lineage data kinship should be considered. #451

[ChenpiDog]

HTTP processor is mainly used for serverless functions (such as OpenFaaS), which is also related to data processing.

[saul-data]

Hi @ChenpiDog, thank you for all the good feedback on lineage, data management and processors.

For SQL statements, I am busy working on a Dataplane python library that will be easy to use SQL to pull data, perform transformations and push data. Is there a reason why you wouldn't use SQL inside python? I think it makes sense with SQLAlchemy because of all the adapters are available to the different databases including Greenplum for example: https://pypi.org/project/sqlalchemy-greenplum/

For HTTP processor, using requests in Python can achieve that if it is a short running process (see example below). If it is a long running process, I agree we need a delayed response i.e. send a request to the serverless function with a callback hook to resume the to the next step in the pipeline once the process is complete. I can definitely add that in our roadmap and I think quite powerful capability in general.

import requests
x = requests.get('https://w3schools.com/python/demopage.htm')
print(x.text)

[ChenpiDog]

The pure SQL method may be more convenient for lineage data kinship. In addition, some data engineers are not familiar with Python. If SQL processor is provided, more users will be attracted.
Many big data tools provide SQL data processing methods, even if they are not databases. In my personal opinion, data processing is best implemented in SQL (even machine learning can be implemented in SQL), and Python is used for other logical judgments (of course, do not limit the function of Python).

[ChenpiDog]

SQL and Python are the two most used methods in data processing. I think at least the native support for these two methods is better. Especially in the field of data warehouse, SQL is used much more than Python.

[saul-data]

I have been giving this a lot of thought. I can see where you are coming from. One way or another, we would need a coding language to authenticate with the database and send the SQL code. This may be done using SQLAlchemy which has many database connection types already available. I think doing it in Golang would be harder to maintain. I do like the idea and I will look product design with some wireframes - I can invite you to look/feedback on the wireframes before we implement.

[ChenpiDog]

Well, as the main developer of the code, you need to consider the technical implementation details. I offer another idea. Is it possible to provide a more convenient way to write SQL directly on the UI page, while the underlying layer still uses the original technology stack.