RESEARCH

Imagine we gave the unknowns the last 100 rows instead. When we see an unknown word, we'll hash its word form, and mod the result to pick a row for it. Each unknown word will share its vector with lots of others, but its vector-mates are probably rare.

→ View original post on X — @honnibal

Resizing the embedding table for fine-tuning is pretty awkward. We'd rather have some way of making sure that novel words can get a unique representation, even from a fixed-size embedding table. The hashing trick achieves this.

→ View original post on X — @honnibal

This is kind of weird, if you think about it. The unknown words together will be more frequent than the 9998th most frequent term. So the fidelity of representation isn't being distributed well. How can we give the unknowns more vectors?

→ View original post on X — @honnibal

For language modelling and translation, word pieces are the standard way to address this. But for smaller datasets, especially where the decisions need to be made on a per-token basis, it's very helpful to learn representations for words, rather than characters or word pieces.

→ View original post on X — @honnibal

The basic motivation is that vocabularies are unbounded, but embedding tables can only be a fixed size. If you know your training data matches up to your test data well, this isn't such a big deal for most applications. If it's rare at training time, it'll be rare at test time.

→ View original post on X — @honnibal