[WIP] isWithinTokenLimit Throws Error on Special Characters

[Copilot]

• Understand the issue: isWithinTokenLimit doesn't support special character handling like countTokens does
• Create test to reproduce the issue
• Analyze current implementation - countTokens accepts EncodeOptions but isWithinTokenLimit doesn't
• Update isWithinTokenLimit method signature to accept optional EncodeOptions parameter
• Update encodeChatGenerator to accept and pass through EncodeOptions parameter
• Pass EncodeOptions to the underlying generator methods in both string and chat message paths
• Fix similar issue in countTokens method for chat messages
• Update JSDoc comments with parameter descriptions
• Create comprehensive tests covering both string and chat message scenarios
• Update README documentation with new method signature and examples
• Run all tests to ensure no regressions (911 tests passing)
• Fix code formatting and linting issues
• Verify all changes are minimal and backward-compatible

Summary

Successfully implemented support for special tokens in isWithinTokenLimit method. The changes are minimal and fully backward-compatible:

Changes Made:

1. Core Implementation: Added optional encodeOptions?: EncodeOptions parameter to isWithinTokenLimit method
2. Chat Support: Updated encodeChatGenerator to accept and pass through encode options
3. Bug Fix: Fixed existing bug in countTokens where encode options weren't passed to chat generator
4. Documentation: Updated JSDoc comments and README with clear examples
5. Testing: Added comprehensive tests for both string and chat message scenarios

Usage Examples:

// Works with special tokens now:
isWithinTokenLimit(`Hello ${EndOfText} world`, 100, { allowedSpecial: ALL_SPECIAL_TOKENS })

// Also works with chat messages:
isWithinTokenLimit([{role: 'user', content: `Hello ${EndOfText} world`}], 100, { allowedSpecial: ALL_SPECIAL_TOKENS })

The implementation is consistent with countTokens behavior and maintains full backward compatibility.

Fixes #74.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[Copilot]

@niieani 👋 This repository doesn't have Copilot instructions. With Copilot instructions, I can understand the repository better, work faster and produce higher quality PRs.

I can generate a .github/copilot-instructions.md file for you automatically. Click here to open a pre-filled issue and assign it to me. I'll write the instructions, and then tag you for review.