perf: Improve search performance by avoiding DFA intersection if possible. #158
Description
Request
Currently all query substrings are assigned a token using the DFA intersection. While DFA intersection is a very powerful tool, specifically when considering wildcards, there are definitely more efficient techniques for substrings that don't contain wildcards or only contain leading/trailing wildcards.
Possible implementation
When possible switch from using the intersection to a forward/reverse lexer (no central wildcards). For anything without a wildcard or with a trailing wildcard a forward lexer will work. For anything with a leading wildcard a reverse lexer is required.