AGI DEBATE "The Debate The World Is Waiting For" Fri, Dec 23, 2022 Website: https://
agidebate.com It takes a village . . . Four panels of the world's most distinguished researchers and experts on AGI. #AGI #AIDebate #AGIDebate
RESEARCH
-
AGI Debate: World’s Distinguished Researchers Panel Discussion
By
–
-
MMR Example Selector for Optimal Contextual Diversity
By
–
Max Marginal Relevance (MMR) Example Selector Based on this preprint by @xiye_nlp et al, the MMR example selector chooses diverse, in-context examples H/t @johnjnay for surfacing this paper Docs: https://langchain.readthedocs.io/en/latest/examples/prompts/prompt_management.html Original Paper: https://x.com/xiye_nlp/status/1603821850592628738?s=20&t=wQPcRd8IJ8URbmytuYP7QA
-
Gradient Backpropagation and Frequent Word Prioritization in Training
By
–
Because we sum, the gradients for each token get backpropagated to all the rows that were used for it. So with enough training, the model ends up at a good compromise. Frequent words are naturally prioritised in this, because they'll simply have more gradients.
-
Hash Embedding Performance on Small Datasets Analysis
By
–
However, the hash embedding doesn't do so well if the datasets are too small. That's one of the results that I found really useful in this tech report. It's great to have this all explored more thoroughly, and we look forward to publishing more experiments like this in future.
-
spaCy’s Hashing Approach for Word Representations Learning
By
–
This is almost what we do in spaCy — it's just that, instead of reserving the first 9900 rows for common words, we just use the hashing. Given that the assignment of words to shared rows is arbitrary, how can the model learn good representations?
-
Hashing Unknown Words to Shared Vector Rows for Rare Words
By
–
Imagine we gave the unknowns the last 100 rows instead. When we see an unknown word, we'll hash its word form, and mod the result to pick a row for it. Each unknown word will share its vector with lots of others, but its vector-mates are probably rare.
-
Hashing Trick for Fixed-Size Embedding Tables in Fine-Tuning
By
–
Resizing the embedding table for fine-tuning is pretty awkward. We'd rather have some way of making sure that novel words can get a unique representation, even from a fixed-size embedding table. The hashing trick achieves this.
-
Improving Vector Representation for Unknown Words in NLP
By
–
This is kind of weird, if you think about it. The unknown words together will be more frequent than the 9998th most frequent term. So the fidelity of representation isn't being distributed well. How can we give the unknowns more vectors?
-
Word representations learning for smaller datasets and per-token decisions
By
–
For language modelling and translation, word pieces are the standard way to address this. But for smaller datasets, especially where the decisions need to be made on a per-token basis, it's very helpful to learn representations for words, rather than characters or word pieces.
-
The Hashing Trick in spaCy Models: An Underexplored Technique
By
–
The hashing trick is an old technique from sparse linear models, well known in toolkits like Vowpal Wabbit. It's one of the unusual things I did in @spacy_io 's models that I've always felt was quite neat, and needed more experiments and write up.