Training loss is evaluated over the batch, i.e. 0.5M tokens. It's noisy but this is expected, you could be iterating through easy or hard documents in the training data. The validation loss is averaged over 20 batches of 0.5M tokens (this is a hyperparameter), so it is smoother.
RESEARCH
-
Conservative hyperparameters with potential for 10X speedup
By
–
Agree!! I'm using very conservative settings for a lot of the hyperparameters (following GPT-3 paper when possible) and haven't tried to speed this up at all yet, but I expect a 10X multiplier here should be possible.
-
Reproduce GPT-2 124M in llm.c for $20 in 90 Minutes
By
–
# Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 The GPT-2 (124M) is the smallest model in the GPT-2 series released by OpenAI in 2019, and is actually quite accessible today, even for the GPU poor. For example, with llm.c you can now reproduce this model on one 8X
-
OpenAI Begins Training Next Major Language Model
By
–
El misterio de la jornada es el texto escrito por OpenAI en un nuevo blog post en el que anuncian que recientemente han comenzado a entrenar su próximo gran modelo… ¿GPT-5? ¿Tan tarde? ¿GPT-5o? ¿GPT-6?
-
Building New Models Makes Obsolete Existing Reality
By
–
You never change things by fighting the existing reality. To change something, build a new model that makes the existing model obsolete. —Buckminster Fuller.
-
ECCV 2024 Workshop on Audio-Visual Generation and Learning
By
–
Initial CfP advertisement for the ECCV 2024 workshop "AVGenL: Audio-Visual Generation and Learning". It will cover a wide range of topics about audio-visual generation and learning. Paper submission deadline is 15 Jul 2024. #ECCV2024 #ECCV @eccvconf
-
Fine-tuning Faster R-CNN for Sea Rescue Detection
By
–
Fine-tuning Faster R-CNN for Sea Rescue 🌊https://t.co/U1G0q7YrSk
— Satya Mallick (@LearnOpenCV) 28 mai 2024
Our research enhances Faster R-CNN to detect people in distress using the SeaDroneSee dataset. By preprocessing images into patches, we significantly improve detection accuracy.
Dive into our findings and see… pic.twitter.com/YQ5KOfvUClFine-tuning Faster R-CNN for Sea Rescue https://
learnopencv.com/fine-tuning-fa
ster-r-cnn/
… Our research enhances Faster R-CNN to detect people in distress using the SeaDroneSee dataset. By preprocessing images into patches, we significantly improve detection accuracy. Dive into our findings and see -
Domino Flows Launches Advanced Life Sciences Analytics Platform
By
–
Domino Data Lab unveils Domino Flows for life sciences with top security and compliance. We're also launching the SCE Coalition to advance clinical analytics. Learn more: https://
domino.buzz/44ZA8dV #AI #LifeSciences #Innovation #RevX24 -
LeCun vs Musk: Open Research Scientists Drive Progress
By
–
I would pick @ylecun over @elonmusk every single day of the week. Despite getting much less $$, recognition & visibility than entrepreneurs, the scientists who publish their groundbreaking research openly are the cornerstone of technological progress & massively contribute to
-
OpenAI Begins Training New Flagship AI Model, Creates Safety Committee
By
–
OpenAI begins training its new flagship AI model, a process that could take nine months or more, and creates a Safety and Security Committee to explore risks (@cademetz / New York Times) nytimes.com/2024/05/28/techn… 📫 Subscribe: techmeme.com/newsletter?from… techmeme.com/240528/p5#a2405…