Web Scraper: Adaptive mode / Switch of modes

[nicklamonov]

Similarly to Adaptive mode in WCC: If we see a way to make things cheaper for the user, we can suggest it.

From technical standpoint it's not feasible to make a true adaptive mode.
So a switch of the mode or another solution needs to be evaluated.

[B4nan]

We already discussed this, we can't do this for quite a lot of reasons. Web scraper runs code in browser, and uses puppeteer, adaptive crawler uses playwright and runs code in node, plus it has a limited crawling context. Those things are pretty different, quite incompatible. Yes, we can add a playwright adaptive mode, but it feels like a very bad idea to me, the page function would have to differ a lot, both API wise (puppeteer vs playwright, plus restricted crawling context), and context wise (running code in browser vs node, different API available, ...), plus image size would increase due to bundling both puppeteer and playwright.

If we want to have some adaptive crawler, it should be a new one, or we can add adaptive mode to the existing playwright scraper, it doesnt make much sense to do it in web scraper.

[nicklamonov]

Yes, I remember the discussion about technical feasibility.

In a recent discussion with Mara there was still an idea about some combination of various scrapers, maybe with some kind of manual switch between modes.
So I tracked it for now just to explore other options (just updated the title and description). We may discard even all other options in the future too, but for now I'd keep it for possible future discussions.