@jimkxa talks to @wccftech about the future of #ai, #cpu and @risc_v in #Japan
RESEARCH
-
Training YOLOv8 Segmentation Models on Trash Can Dataset
By
–
Get ready to dive into our latest video!https://t.co/wnlOVn7m3k
— Satya Mallick (@LearnOpenCV) 26 juin 2023
We're exploring the 'trash can' dataset and putting it to work, training Ultralytics YOLOv8 segmentation models. #deeplearning #AI #computervision pic.twitter.com/PDEHzPtEOhGet ready to dive into our latest video! https://
youtube.com/watch?v=6J2Pvz
hO_Mk
… We're exploring the 'trash can' dataset and putting it to work, training Ultralytics YOLOv8 segmentation models. #deeplearning #AI #computervision -
AI Research Integrity Crisis: Academic Fraud Parallels Emerge
By
–
What a hustle!
I feel sorry for the students on the paper for starting their journey in such a careless, low-integrity environment. The last time AI hit a bubble was ~2014, and this paper reminds me of Baidu cheating on the Imagenet competition. Similar playbook — threw a https://
x.com/ML_PhDer/statu
/ML_PhDer/status/1672750801234857984
… -
Collaborative LLM Project with Falcon and LLaMA Teams
By
–
This was a really cool project with @clefourrier @nathanhabib1011 @osanseviero and a lot of help from @slippylolo from the Falcon team and the amazing LLaMA team, in particular Xavier Martinet, Aurélien Rodriguez and Sharan Narang!
-
MMLU Benchmark Updated in Open LLM Leaderboard
By
–
25/ THE END PS: Please, come comment/discuss at https://
huggingface.co/spaces/Hugging
FaceH4/open_llm_leaderboard/discussions/82
… PPS: During the writing of this post, @Ollmer & the community have done an amazing work updating MMLU in the Harness to make it similar to the original. We're thus now updating the leaderboard based on it! -
Open Standardized Benchmarks Essential for AI Model Evaluation
By
–
24/ That's why open, standardized, reproducible benchmarks such as the EleutherAI Harness https://
github.com/EleutherAI/lm-
evaluation-harness/
… or Stanford HELM https://
github.com/stanford-crfm/
helm/
… are invaluable to the community. Without them comparing results across models/papers would be impossible, stifling research! -
MMLU Scores Incomparable: Evaluation Implementation Details Matter
By
–
23/ …not at all comparable even if they're both called MMLU & evaluated on same dataset Takeaway? Evaluations are strongly tied to implementations–down to minute details. A mere indication of "MMLU score" gives almost no information about how you can compare these numbers
-
LLaMA 65B Evaluation Discrepancies Between Implementations
By
–
22/ Say you've trained a perfect LLaMA 65B reproduction & evaluated it with EAI harness (score 0.488). Comparing it to the published number (evaluated w. original implementation, score 0.637), it's a 30% difference so you're likely thinking "Oh no " But these numbers are…
-
Model Evaluation Methods Significantly Impact Scores and Rankings
By
–
21/ Now let’s compare the model scores on these three possible ways to evaluate the models We can see that for the same dataset, both absolute scores and model rankings are very sensitive to the evaluation method we decide to use.
-
Probability-based sequence generation normalization in language models
By
–
19/ In this case, we use probabilities again but this time we compute the probability of generating the full answer sequence, not just the letter: we sum the log of the probabilities and compute a normalization by dividing by the number of tokens to not penalize longer sequences.