Also, my personal opinion is that both public pretraining (with certain constraints/limitations) as well as private training from scratch both have roles to play in the space. They of course can't be held to the same accuracy standard though.
LLMS
-
How GPT Obtains Its Ability: Tracing Emergent Abilities
By
–
It’s from here: https://
yaofu.notion.site/How-does-GPT-O
btain-its-Ability-Tracing-Emergent-Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1
… -
ChatGPT: Tool Not Threat to Humans, Like Thermomix for Cooks
By
–
ChatGPT is not human rather it has been taught to write in grammatically correct and speak sounding human. It’s a threat to Wikipedia or automated telephone menus but not real people. A Thermomix is not a threat to a cook even when it nails a recipe every time @TVNaga01
-
Control Text Generation Model Creativity: Temperature Top-k Top-p
By
–
To recap, you can adjust 3 parameters to control how creative a text generation model’s output are:
1. Temperature
2. Top-k
3. Top-p Lower values = more predictable.
Higher values = more creative. More on this blog: https://
txt.cohere.ai/generative-ai-
part-1/
… -
Top-p Sampling Technique for Language Models Explained
By
–
3. Top-p – is similar to top-k, except the choice scoping is not by the token count, but by the sum of the token probabilities. By the way, you can try out the prompt here: https://
os.cohere.ai/custom-preset?
ref=Startup-Idea-Generator-nyrnln&e=generate
… -
Top-k Sampling Control in AI Model Generation
By
–
Let’s take the same example of startup ideas brainstorming. Now, set the top-k to 1. This means the model can choose from exactly 1 option. So, no matter how many times you generate, or what temperature value you set, the output will be the same every time, as below.
-
Temperature vs Top-k/Top-p: Token Selection Explained
By
–
How is it different from temperature? In a nutshell:
• Temperature – how to choose between tokens
• Top-k/top-p – how many tokens to choose from As a result,
• Lower top-k/top-p = more predictable
• Higher top-k/top-p = more creative -
Top-k and Top-p: Controlling Model Token Selection
By
–
2. Top-k (also top-p) – controls the list from which the model can make its choices. Increasing the top-k increases the number of tokens that the model can choose from, and vice versa.
-
Temperature Controls Model Token Generation Creativity
By
–
1. Temperature – controls how the model chooses from its next choice of tokens. Increasing the temperature makes generating tokens with lower likelihoods more probable, and vice versa. As a result:
• Lower temperature = more predictable
• Higher temperature = more creative -
Understanding Token Likelihood Parameters in Language Models
By
–
It does this by giving a “likelihood” number to each token. So, for the phrase “I like to bake …”, “cookies” has a higher likelihood than “chairs.” “chairs” can still appear, but “cookies” has a much higher chance. But we can change this behavior with those 3 parameters.