Crawler vs LinkContentFetcher

[laskas1234]

Hi,
I have a set of URLs that I want to extract the text from. I am trying to understand what is the difference between the Crawler and the LinkContentFetcher and if there are any advantages to using one versus the other.

To my understanding the Crawler will start from a link, extract the text and then move on to other "neighboring" URLs and extract those as well. While the LinkContentFetcher will only extract the text from the URLs provided. Is there any benefit to using the Crawler when I am only interested in specific URLs?

I was also reading through the Haystack 2.0 docs and I was not able to find any mention of Crawler. Does that mean that it will not be in the new version of Haystack?

Thank you for your amazing work I really love what you have done so far!

[ZanSara]

To my understanding the Crawler will start from a link, extract the text and then move on to other "neighboring" URLs and extract those as well. While the LinkContentFetcher will only extract the text from the URLs provided. Is there any benefit to using the Crawler when I am only interested in specific URLs?

That's correct! If you already have a list of URLs, LinkContentFetcher is what you need.

I was also reading through the Haystack 2.0 docs and I was not able to find any mention of Crawler. Does that mean that it will not be in the new version of Haystack?

It's not yet ported, but this doesn't mean we won't add it later. It's just not very high on the priority list right now but sooner or later we will probably add it again.