Skip to content

Conversation

lambdacasserole
Copy link

@lambdacasserole lambdacasserole commented Jun 21, 2018

The fix discussed in #232. Far more detail given over there, but briefly we need to strip non-letter characters to prevent single-token partitions with non-letters at the beginning or end from being over-rewarded for certain capitalization schemes. See issue #232 for (far) more detail.

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants