Tokenizer

The Project is an assignment 1 for GenAI Cohort. The Project have following files:

/project-root
│
├── index.html       # Main HTML page
├── index.css        # Stylesheet
├── app.js           # Main application logic (imports  Tokenizer)
├── tokenizer.js     # Tokenizer class module
└── README.md        # This documentation

It have following features:

1. Build a vocab
2. Encode a text
3. Decode Tokens
4. Visually see the result.

Tokenizer.js

This Tokenizer class provides a simple way to convert text into numerical token IDs and back. It helps in building a vocabulary from input text and encoding/decoding sequences of words.

Methods

constructor() Initializes empty mappings and sets the starting token ID to zero.

buildVocab(text) Takes a string of text, splits it by spaces, and builds a vocabulary by assigning a unique ID to each word.

encode(text) Converts a text string into an array of token IDs. If a word is not in the vocabulary, it adds it dynamically.

decode(tokens) Converts an array of token IDs back into a text string. Unknown token IDs are converted to 'UNK'.

How to run

1. Clone the Repo.
2. Use Live Server to run this Project.

Visit Live Link