fix CLIPTokenizer skipping underscores

[TyrianOtter]

The current CLIPTokenizer regex uses [^\s\w]+, which causes it to skip underscores since \w matches them. Checking the bundled vocab, it seems underscores are only ever part of a word that has no alphanumerics (aside from a couple mojibaked words), so the underscore fits in with the [^\s\w]+ part.

Changing [^\s\w]+ to (?:[^\s\w]|_)+ allows tokenizing underscores, which should better match other tokenizer implementations.

[TyrianOtter]

Looks like I didn't actually check this specific regex pattern works (must not have saved the file). It errors on the prompt 'a_b'.

Using [^a-zA-Z0-9\s]+ instead does works on the inputs I've tried it on. But \w includes unicode that a-zA-Z doesn't, making that a bigger change to the tokenizer than just additionally matching underscores.

Haven't looked into what's causing issues yet.

[TyrianOtter]

I forgot to make the group non-capturing in my commit, (even though I didn't forget in my PR description? 🙃).

Works fine on my inputs now.

[catwell]

Squash the commits before merging, otherwise good find, thanks!

[TyrianOtter]

Actually, it looks like it's possible to get near parity with openai/clip (and thus transformers) without the regex dependency by using <\|startoftext\|>|<\|endoftext\|>|'s|'t|'re|'ve|'m|'ll|'d|[^\W\d_]+|\d|(?:[^\s\w\d]|_)+. I think this only differs on inputs with Unicode characters in the "Nl" (Letter Number) and "No" (Other Number) categories, and some niche whitespace (specifically U+001C to U+001F). It matches the numbers as letters (i.e. in [^\W\d_]+) when it should be matched as part of (?:[^\s\w\d]|_)+.

As far as I can tell, this should agree with openai/clip on more inputs, while only failing to agree on any inputs that already agree using the current implementation or the change in the PR when the input uses whitespace from U+001C to U+001F.

Should I update the PR with this new pattern?

[catwell]

Yes, we meant something like this with the comment above. I'd prefer if we had some test coverage for a change like this though.

Do you know if there are some "standard" tests for this tokenizer somewhere we could use?

[TyrianOtter]

Guess I'll leave the Unicode alone for now. Differing supported Python versions use different Unicode versions, all of which are themselves different from regex's Unicode version. Seems like a huge headache to get test coverage on.