Our @FT weekend profile of Geoff Hinton, and why he’s speaking up, with
@RichardWaters
. Feat. long-time collaborators/peers Yoshua Bengio and Stuart Russell. https://
on.ft.com/3pbpVtR
RESEARCH
-
Geoff Hinton Speaks Out on AI Safety Concerns
By
–
-
Top Technical Talent in AI Industry Operating at Incredible Levels
By
–
heavily related: the level at which the top technical people in the world are operating is incredible to watch and makes me feel very fortunate to be in this industry!
-
Cross-sector Technology Vision Enables Competitive Innovation Advantage
By
–
a huge amount of value comes from being able to see what’s coming over the next decade in many different areas of technology and put it together. you can find solutions others can’t. going deep is good but don’t forget to look wide too!
-
Data Quality Emerges as Key Differentiator in AI Development
By
–
Hidden amongst the zingers in the leaked Google memo there's this: data quality is what is making the difference now. This is an important shift away from internet-scale garbage dump datasets that have been running the show for years.
-
Open Source AI Progress Reaching Critical Inflection Point
By
–
it feels like an inflection point for open source — do you get that sense? (i'm very much an outsider but it appears to me, at least, that the last year or so we've seen so much progress in AI coming from the open-source community.)
-
Proper Task Prompts Essential for Fair Model Evaluation
By
–
Best is to evaluate the models with a prompt where the understand the task. If a model is 20% below with a prompt compared to another, clearly you are not evaluating the model properly and it makes no sense to report this number or use it in a comparaison.
-
Prompt engineering investigation: minimal sensitivity to formatting changes
By
–
We spent a lot of time investigating what others papers did (Chinchilla, GPT-3, PaLM) but very few of them actually provide any prompt so we just implemented what made sense to us. And we did not observe 20% differences by adding or removing a space in the prompt.
-
Custom Prompts and Benchmark Evaluation Standards for LLMs
By
–
You mention our "custom prompt" like if there was an official way of prompting (yours?). Most benchmarks were created before this concept of LLM eval with prompting even exists, and for many of them there is no official prompt or way to evaluate them with LLMs.
-
Formatting Alone Cannot Explain Performance Gap with LLaMA
By
–
I don't think that formatting alone can explain such a gap. You could maybe try to reproduce LLaMA numbers given the available model. It is likely that fixing these differences for LLaMA will also improve the numbers you report for your model.
-
LLaMA evaluation metrics concern and measurement discrepancies
By
–
It is true that accuracy on some metrics can be quite sensitive to the prompt, however this is not normal that all metrics reported for LLaMA here are systematically (and significantly) below what we measured. There may be an issue in how LLaMA was evaluated.