To those in the replies who say 'but opus 4.8 is weaker so without fallback, the score would be even higher': this is not necessarily true because of how any benchmark works – which is an average of queries – and what is called 'the x.com/ClementDelangu…'
@clementdelangue
-
The erroneous reasoning about fallback and benchmarks
By
–
To those in the replies who say "but opus 4.8 is weaker so without fallback, the score would be even higher": this is not necessarily true because of how any benchmark works – which is an average of queries – and what is called "the x.com/ClementDelangu…"
-
Refutation of the argument that a weaker model increases the score
By
–
To the people in the replies who say "but opus 4.8 is weaker, so without fallback, the score would be even higher": this is not necessarily true because of how any benchmark works – which is an average of queries – and what is called "the x.com/ClementDelangu…"
-
Benchmarks: why a weaker model does not necessarily improve the score
By
–
To those in the replies who say 'but opus 4.8 is weaker so without fallback, the score would be even higher': this is not necessarily true because of how any benchmark works – which is an average of queries – and what is called 'the x.com/ClementDelangu…'
-
Debate on the impact of fallback in benchmarks
By
–
To the people in the replies who say 'but opus 4.8 is weaker so without fallback, the score would be even higher': that is not necessarily true because of how any benchmark works – which is an average of queries – and what is called 'the x.com/ClementDelangu…'
-
Refutation on benchmarks and the lack of fallback
By
–
To the people in the replies who say "but opus 4.8 is weaker so without fallback, the score would be even higher": this is not necessarily true because of how any benchmark works – which is an average of queries – and what is called "the x.com/ClementDelangu…"
-
Refutation of the argument on fallback in benchmarks
By
–
To those in the replies who say 'but opus 4.8 is weaker so without fallback, the score would be even higher': this is not necessarily true because of how any benchmark works – which is an average of queries – and what is called 'the'
-
Proposal to evaluate Omni and build a routing system
By
–
Yes maybe we should evaluate Omni @victormustar or https://openrouter.ai/docs/guides/routing/provider-selection … @alexatallah? Or does someone want to build a routing system and evaluate on AA?
-
The graph illustrates the bias of AI evaluations toward closed APIs
By
–
This graph illustrates what is flawed in AI evaluations: they structurally favor closed APIs that can route, switch, ensemble and optimize behind the scenes without any transparency. No offense, @ArtificialAnlys, but how to compare a
