path to AGI 1. train large language model (M1) on human text (T1)
2. generate synthetic text (T2) from M1
3. train large language model (M2) on T2
4. generate synthetic text (T3) from M2
5. train large language model (M3) on T3
6. generate synthetic text (T4) from M3
7. train la
@jxmnop
-
Iterative AGI Training Through Synthetic Data Generation
By
–
-
GPU Batch Size Memory Utilization Performance Analysis
By
–
actually the batch size is relatively low. memory about half full. do you think that would explain ~30% utilization?
-
Optimizing Transformer Training on Single GPU with HuggingFace
By
–
what are some easy ways for a normal person like me to speed up training a transformer on a single GPU? i'm using huggingface T5 implementation + defaults. i know about FlashAttention and BetterTransformer and torch compile? will these all work together? is there anything else?
-
Neural Networks Impressive But Human Brain Underrated
By
–
sure, neural networks make for impressive demos, but the human brain is seriously underrated
-
Training tiny character RNN for Office script generation
By
–
update: training a tiny char RNN to write lines from The Office (no BOS token yet so these samples always start with M; using ancestral sampling for now)
-
Building a Simple Neural Language Model with MLP for Education
By
–
what’s the simplest neural language model i can code for my little class? maybe just an MLP on some word embeddings? doing an RNN feels like it could be overkill
-
Research Advances in Text Embeddings Language Models and Prompting
By
–
a tiny summary of my research projects of 2023: Text Embeddings Reveal (Almost) As Much As Text EMNLP 2023, http://
arxiv.org/abs/2310.06816 Language Model Inversion ICLR 2024, http://
arxiv.org/abs/2311.13647 TreePrompting EMNLP 2023, http://
arxiv.org/abs/2310.14034