Correction in Section 1.5: Tokenization and Embeddings

[mohammadp1001]

Hi,

In Chapter 1, Section 5
, the text currently says:

“GPT-2 uses byte pair encoding (BPE) to tokenize words and generate a token embedding.”

This phrasing is a bit misleading.

BPE is used to tokenize text into tokens (subwords).

After tokenization, each token is mapped to an ID.

The model’s embedding layer then converts these IDs into token embeddings.

So the embedding is not generated directly by BPE, but rather by the embedding lookup.

I suggest rephrasing it to something like:

“GPT-2 uses byte pair encoding (BPE) to tokenize words into tokens. Each token is mapped to an ID, which is then converted into a token embedding by the embedding layer.”

This should make it clearer for learners who are just getting started.

Thanks for the great course! 🙌