yeah i was thinking, learning from real data never looks like this …
@jxmnop
-
Off-by-One Error in Autoregressive Model Training
By
–
hahah I’ve made this mistake too; it’s pretty easy to do in autoregressive model-training. just an off-by-one error: predict token T from token T instead of T+1
-
Model Training Bug Resets AGI Performance to Baseline
By
–
yesterday my model was training and its performance was so good that twitter ppl threatened to blow up my house in case i was about to release AGI then, i fixed one line of code. now my model performance is exactly the same as the baseline ah, the ups and downs of science
-
Loss Function Behavior in Deep Learning Training
By
–
also i think it would be pretty hard to artificially get the loss to jump like that after so many steps
-
Synthetic Data Quality Issues in AI Model Training
By
–
oh it wasn't fake because of this reason it's "fake" because some of the inputs are 1s not real data
-
Shuffling Within Batch in Contrastive Learning Practices
By
–
well i do shuffle; it's not standard to shuffle *within the batch*, for contrastive learning
-
Batch Shuffling Causes Accuracy Regression to Baseline
By
–
shuffled each batch and accuracy regressed down near the baseline
-
Contrastive Learning Bug Discovery in AGI Development
By
–
ok before one of you tries to assassinate me for building AGI, i figured out the bug. at least it's an interesting one so we're doing contrastive learning, which is a matching task between (query, document) pairs query 1 matches to document 1, query 2 matches to document
-
Potential Bug or Text Retrieval Breakthrough Discovery
By
–
okay what 99.99% chance there's a bug in my code, but
0.01% chance i just solved text retrieval