[Feature] Create Ingestion Pipeline to Scrape the Data And store into the Milvus VectorDB

[kolhesamiksha]

Research on stratergies or framework to reduce time and efficiently store data into Milvus VectorDB.

mainly focus on below efficient workflows, mostly used by the high workload environments.

• Sample python + Kubeflow

• Pypsark

• Dask

• Test which pipeline requires lesser time and implement best practices to manage CPU resources.