Methods for large corpora?

[gnewton]

Sort of related to #8...

You have methods in the API, like in your example, that take an array of strings (docs).

matrix, _ := vectoriser.FitTransform(testCorpus...)

I'd like to use this for very large corpora, with 10s or 100s of millions of (not tiny) documents. Putting these all into a single array of strings does not sound optimal.
Any chance the methods that now have a string array parameter for the documents could be altered to take in a function or interface that could allow iteration to get all the docs? (Or new methods that support this?)

Thanks,
Glen

[james-bowman]

Thanks, this is on the agenda. Was thinking something like a FitPartial() method and/or adding support for more generalisable input streams.