I know this result from google's GTR paper (
https://
arxiv.org/pdf/2112.07899
.pdf
…) they scale their embedding model and it gets a little better, but not much
@jxmnop
-
Google’s GTR Paper: Embedding Model Scaling Results
By
–
-
Why Aren’t We Training Massive Embedders?
By
–
why aren't we training massive embedders? some guesses:
– contrastive loss isn't the right loss function for embeddings
– not enough good paired data
– unclear what the use case is (retrieval? clustering? classification?)
– no principled scaling laws
– diminishing returns -
GTR: Google’s Neural Retrieval Model for Information Retrieval
By
–
barely though this is from GTR, the Google retrieval model (
https://
arxiv.org/pdf/2112.07899
.pdf
…) -
Concurrent Research on AI Paper Released Earlier
By
–
actually @ZhongRuiqi was working on this at the same time, and released his paper a few days earlier, I just checked 🙂 https://
arxiv.org/abs/2104.04670 -
No Free Lunch in Convex Optimization: Momentum Limitations
By
–
lol I received several messages telling me to learn more about optimization so I might have to do that and think about it more my thought process was that there's no free lunch in convex optimization so even if eg momentum works in some cases, it doesn't work in others, which is
-
Exploring AI Career Path: A Serious Consideration
By
–
few enough for me to seriously consider this career path
-
PhD dropout considers piano tuning career over computer science
By
–
thinking of quitting my computer science phd program to become a professional piano tuner. many apparent benefits
– get to work with hands (touch grass every day)
– solve interesting problems (tuning pianos)
– make more money (many pianos, few tuners, basic supply and demand) -
Refusal Classifier Training with Supervised Data
By
–
@wzhao_nlp is the expert on this and she said she thinks they have a classifier trained for refusal, probably just with some supervised data it would still work well even if it were pretty small (<1B params)