MACHINE LEARNING

→ View original post on X — @_akhaliq

Accelerating Long-Context LLMs with Massive Input Token Reduction

By

@_akhaliq

–

27 September 2024 4h46

Discovering the Gems in Early Layers Accelerating Long-Context LLMs with 1000x Input Token Reduction discuss: https://
huggingface.co/papers/2409.17
422
… Large Language Models (LLMs) have demonstrated remarkable capabilities in handling long context inputs, but this comes at the cost of increased

Programming Every Example: AI-Powered Data Refinement Framework

By

–

27 September 2024 4h30

We introduce Programming Every Example (ProX), a novel framework that treats data refinement as a programming task, enabling models to refine corpora by generating and executing fine-grained operations, such as string normalization, for each individual example at scale.

Small Language Models Refine Pre-training Data Quality at Scale

By

–

27 September 2024 4h30

Programming Every Example: Lifting Pre-training Data Quality like Experts at Scale https://
huggingface.co/papers/2409.17
115
…
we demonstrate that even small language models, with as few as 0.3B parameters, can exhibit substantial data refining capabilities comparable to those of human experts.

MLRCopilot: Autonomous ML Research Framework with LLM Agents

By

–

27 September 2024 4h22

a new systematic framework, autonomous Machine Learning Research with large language models (MLRCopilot), designed to enhance machine learning research productivity through the automatic generation and implementation of research ideas using Large Language Model (LLM) agents.

MLR-Copilot: Autonomous Machine Learning Research with LLM Agents

By

–

27 September 2024 4h22

MLR-Copilot: Autonomous Machine Learning Research based on Large Language Models Agents https://
github.com/du-nlp-lab/MLR
-Copilot
… https://
arxiv.org/pdf/2408.14033

→ View original post on X — @jeffdean

Backpropagation Support and Use in Neural Network Research

By

@jeffdean

–

27 September 2024 2h56

But it both supported and used backprop, as it said right in the paper.