When humans generate text (articles, posts, papers, etc) they spend very different amount of time per token, create intermediate work, make edits, etc. Very different from GPTs that just go chunk chunk chunk. But there seem to be enough puzzle pieces out and about to remedy.
@karpathy
-
ChatGPT lacks time to think unlike humans
By
–
The deepest unintuitive disconnect w.r.t. psychology of ChatGPT is that it doesn't get "time to think". It has a small, fixed amount of thought for each output token. A bit like human forced to speak very fast. Asking them to produce more text is giving them more time to think.
-
Diffusion Models: A Powerful Alternative to Autoregressive Generation
By
–
(diffusion is a new class of generative models, an alternative to the autoregressive generative modeling framework, independent of transformers. Feels intuitively more pleasing, flexible and powerful)
-
Recent Examples of AI and Machine Learning Developments Shared
By
–
– https://
x.com/sedielem/statu
s/1597401529920520192
…
– https://
x.com/du_yilun/statu
s/1597618021342023680
…
– https://
x.com/ai_fast_track/
status/1594158623059574785
…
– https://
x.com/tingchenai/sta
tus/1580574210631426048
…
– https://
x.com/poolio/status/
1575576632068214785
…
– https://
x.com/huggingface/st
atus/1597248942353584131
…
among only a few of the recent examples -
GeLU, LayerNorm and Mathematical Operations in Neural Networks
By
–
A lot of fun in the Appendix, e.g. how GeLU can be used for multiplication / bypassing it as identity, use of LayerNorm for division, or bypassing that as identity, etc.
-
Transformer Equations as Low-Level Microcode and Assembly Instructions
By
–
Nice! Like the track of work. Equations of Transformer are a bit like low-level microcode, this track tries to "go up" to uncover an implied assembly instruction set (e.g. RAW "read-arithmetic-write" operator?), and implemented algorithms on top of that for e.g. ridge regression.
-
New Neural Network Zero to Hero YouTube Series in Development
By
–
(I think he means my new NN: Zero to Hero series https://
youtube.com/playlist?list=
PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ
… , which I'm still building out) -
Comparing Stable Diffusion 1.5 vs 2.0 Image Generation Quality
By
–
easy to compare a lot of images from both models on http://
stableboost.ai , e.g. "cute dog cooking tacos, photorrealistic", grid of boosted images from 1.5 (left) and 2.0 (right). 2.0 looking more distorted, cartoony, simpler, ignores text more. may need more prompt engineering -
Stable Diffusion 2.0 Shows Quality Decline Compared to 1.5
By
–
plot twist: stable diffusion 2.0 looks quite a bit worse on the few prompts i've tried so far compared to 1.5 (even not including celebrities/artists). Running theory seems to be this is due to an aggressive data sanitization campaign since the original release (?).
-
AI Performance Limitations on Unconstrained Web Search Tasks
By
–
It works well when it’s force constrained to sites like reddit twitter etc. it just can’t be trusted to find good sites