PDF File Ingest Performance prepdocs.ps1

[PatrickGallucci]

I have 25K 1-5 page PDF files (65kb avg size) that I am uploading and it is taking about 30 seconds per file. At this pace it will take 8 days to process. I have bumped up my search service to standard with 18 search units (3x6) and there was no change in duration. Is there something that I can do to increase the performance of prepdocs.ps1?

Request: Can this be multi-threaded to run in parallel against multiple "data" folders?

This issue is for a: (mark with an x)

• bug report -> please search issues before submitting
• feature request
• documentation issue or request
• regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce
Copy 26K PDF files to .\data\ folder
execute prepdocs.ps1

[cforce]

It’s to slow because all is processing in sequence instead of of parallel . It requires Thread worker pool

[simonfisher-ss]

Would that be a massive job to implement? I'm having this general problem too

[tonybaloney]

Try this branch, #2553

The time saving would be different for the size + shape of the files in your data/ directory. Most of the prepdocs time is spent waiting for the embeddings API to respond to the huge batches and not CPU activity. This is why I did asyncio.gather instead of a parallel pool like a MP processing pool. CPU is still quite idle this whole time.