Step-By-Step LLM Engineering Projects Roadmap – Build a tokenizer
– Learn embeddings
– Implement RoPE / ALiBi
– Hand-wire attention
– Build MHA
– Build a Transformer block
– Train a mini-former
– Compare objectives
– Build sampling
– Speculative decoding
– KV cache
– MQA / GQA /

→ View original post on X — @theahmadosman