Because we sum, the gradients for each token get backpropagated to all the rows that were used for it. So with enough training, the model ends up at a good compromise. Frequent words are naturally prioritised in this, because they'll simply have more gradients.
@honnibal
-
Hash Embedding Performance on Small Datasets Analysis
By
–
However, the hash embedding doesn't do so well if the datasets are too small. That's one of the results that I found really useful in this tech report. It's great to have this all explored more thoroughly, and we look forward to publishing more experiments like this in future.
-
spaCy’s Hashing Approach for Word Representations Learning
By
–
This is almost what we do in spaCy — it's just that, instead of reserving the first 9900 rows for common words, we just use the hashing. Given that the assignment of words to shared rows is arbitrary, how can the model learn good representations?
-
Hashing Unknown Words to Shared Vector Rows for Rare Words
By
–
Imagine we gave the unknowns the last 100 rows instead. When we see an unknown word, we'll hash its word form, and mod the result to pick a row for it. Each unknown word will share its vector with lots of others, but its vector-mates are probably rare.
-
Hashing Trick for Fixed-Size Embedding Tables in Fine-Tuning
By
–
Resizing the embedding table for fine-tuning is pretty awkward. We'd rather have some way of making sure that novel words can get a unique representation, even from a fixed-size embedding table. The hashing trick achieves this.
-
Embedding Table Strategy: Frequent Words and Shared Vectors
By
–
Let's say we have 10k rows in the embedding table. In a standard embedding table, you give each of the 9999 most frequent words their own vector, and have all the others share the last vector.
-
Improving Vector Representation for Unknown Words in NLP
By
–
This is kind of weird, if you think about it. The unknown words together will be more frequent than the 9998th most frequent term. So the fidelity of representation isn't being distributed well. How can we give the unknowns more vectors?
-
Word representations learning for smaller datasets and per-token decisions
By
–
For language modelling and translation, word pieces are the standard way to address this. But for smaller datasets, especially where the decisions need to be made on a per-token basis, it's very helpful to learn representations for words, rather than characters or word pieces.
-
spaCy Trained Pipelines: Domain Flexibility and Fine-tuning Capabilities
By
–
But for spaCy, we want to give people trained pipelines and have them be as applicable as possible across different domains. We also want people to be able to fine-tune the models on their data if it's not working ideally.
-
Unbounded Vocabularies and Fixed-Size Embedding Tables Explained
By
–
The basic motivation is that vocabularies are unbounded, but embedding tables can only be a fixed size. If you know your training data matches up to your test data well, this isn't such a big deal for most applications. If it's rare at training time, it'll be rare at test time.