my notes are here for those following along https://
github.com/sw-yx/ai-notes
/blob/main/TEXT.md#chatgpt
…
RESEARCH
-
AI Notes on ChatGPT and Related Technologies
By
–
-
Model with 8192 token context, double others, no RLHF tuning
By
–
It also has a context window of 8192 tokens, which is 2x any of the instruction-following models available in the API. Also the model is internally labelled “text-davinci-002-render”, which would seem to imply it doesn’t include RLHF tuning (unique to 003).
-
Fine-tuning process and 8192 token context window
By
–
It’s clearly a very different fine-tuning process, which they talk about in some detail in the announcement. Also the context window is 8192 tokens, double any of the text models available through the API.
-
Deep RL for Real Robots: Limited Success in 2016
By
–
Deep RL for real robots didn’t really work so well back in 2016.
-
Deep RL at Peak Hype in 2016, AGI Hopes
By
–
Deep RL was at peak hype during NIPS2016. Everyone thought training a ConvNet to play a few Atari games using Q-Learning would lead to AGI.
-
Reconnecting at NeurIPS 2022: Students and Research Groups
By
–
I have missed conference dinners. #NeurIPS2022 reunion. Great to see former students grow their research groups. Great to have students presenting at their first-ever conference. Great to connect everyone together.
-
Optical Illusion Engineering: Straight Line Ball Motion
By
–
Beautiful #engineering
— Pascal Bornet (@pascal_bornet) 3 décembre 2022
All balls are moving in straight lines… Can you believe it?#science #brain #optical
Credit: P. Jain pic.twitter.com/ewMRGeEYMeBeautiful #engineering All balls are moving in straight lines… Can you believe it? #science #brain #optical
Credit: P. Jain -
AI knows URLs from training data, cannot browse web
By
–
I assure you it can't browse the web, it just knows the URLs for many, many websites. It's seen many URLs in its training data, and can both recite real ones and hallucinate plausible fake ones.
-
ChatGPT uses a simple sliding window of 8192 tokens
By
–
I now believe ChatGPT uses a simple sliding window with a context window of 8192 tokens: