Introducing the Diff Transformer, which calculates attention scores as the difference between two separate softmax attention maps. This improves NLP tasks like long-context modeling, in-context retrieval, and mitigating hallucinations! Join the discussion with @ytz2024 here!