Tokenizer Web Application

This project provides a simple web interface to a tokenizer, with a Node.js backend and some nice visual effects.

Prerequisites

• Node.js and npm
• Python (for the tokenizer.py script, if used)
• Anthropic API key (The tokenizer endpoint of the API is free to use, but it requires a valid API key. This doesn't need to be funded an empty one will do.) Go to the anthropic console (https://console.anthropic.com) and create one to get started.

Setup

1. Clone the repository:

   git clone https://github.com/KillAllTheHippies/tokenizer-webapp.git
   cd tokenizer-webapp

2. Install Node.js dependencies:

   npm install

3. Set up environment variables: Create a .env file in the root of the project by copying the example file:

   cp .env.example .env

   Open the .env file and add your Anthropic API key where it says YOUR_KEY_HERE:

   ANTHROPIC_API_KEY="YOUR_KEY_HERE"
   

Running the Application

Windows

1. Start the backend server:

   launch_tokenizer_backend.bat

   or

   node server.js

2. Open the frontend: Open the tokenizer.html or TOKENIZERV1.html file in your web browser.

macOS / Linux

1. Start the backend server:

   node server.js

2. Open the frontend: Open the tokenizer.html or TOKENIZERV1.html file in your web browser.

   open tokenizer.html # on macOS
   xdg-open tokenizer.html # on Linux