That starts a llamafile running Llama 3 70 B on localhost port 8080 – then talk to it from LLM like this: llm install llm-llamafile
llm -m llamafile "3 neat characteristics of a pelican"
@simonw
-
Running Llama 3 70B locally with llamafile and LLM
By
–
-
Running Meta-Llama-3-70B Locally with 64GB RAM
By
–
If you have 64GB RAM:
curl -L -o Meta-Llama-3-70B-Instruct.Q4_0.llamafile 'https:''//huggingface.co/jartine/Meta-Llama-3-70B-Instruct-llamafile/resolve/main/Meta-Llama-3-70B-Instruct.Q4_0.llamafile?download=true'
chmod 755 Meta-Llama-*
./Meta-Llama-3-70B-Instruct.Q4_0.llamafile -
Groq Llama 3 70B API Setup Guide and Installation
By
–
For the ludicrously fast Llama 3 70B API from @GroqInc
, get an API key from https://
console.groq.com/keys and then: llm install 'https:''//github.com/lexh/llm-groq/archive/ba9d7de74b3057b074a85fe99fe873b75519bd78.zip'
llm keys set groq
# paste API key here
llm -m groq-llama3-70b 'hi' -
Running Meta-Llama-3-8B Local Model on Your Laptop
By
–
Here's the super-short version. For a ~4GB local model that should run on most laptops: brew install llm # or pipx install llm
llm install llm-gpt4all
llm -m Meta-Llama-3-8B-Instruct "Three great names for a pet emu" -
Llama 3 LLM Plugins: Local Hosting and API Access Guide
By
–
I pulled together notes on all of the LLM plugins that have worked for me for Llama 3 – both for hosting locally (I've run 8B and 70B on my 64GB M2) and access via APIs (Groq is SO FAST for that)
— Simon Willison (@simonw) 22 avril 2024
Options for accessing Llama 3 from the terminal using LLMhttps://t.co/OuG7EqCI4F pic.twitter.com/xmBisMH7IoI pulled together notes on all of the LLM plugins that have worked for me for Llama 3 – both for hosting locally (I've run 8B and 70B on my 64GB M2) and access via APIs (Groq is SO FAST for that) Options for accessing Llama 3 from the terminal using LLM https://
simonwillison.net/2024/Apr/22/ll
ama-3/
… -
BLOB Columns: Embeddings Storage in LLM Datasette
By
–
BLOB columns: https://
llm.datasette.io/en/stable/embe
ddings/storage.html
… -
Community Standards for AI Systems and OpenAI Conventions
By
–
Yeah, I totally understand that! Promising not to change anything would be an unreasonable burden But apparently the community /really/ wants some kind of standard here – and is making do with the OpenAI conventions in the absence of something more appropriate
-
Meta-Llama-3-70B-Instruct Q4 Quantized Model Implementation
By
–
I used the Meta-Llama-3-70B-Instruct.Q4_0.llamafile (quantized at Q4) file from here:
-
Running Llama-3-70B-Instruct locally on M4 Mac
By
–
Pleased to report that the Llamafile release of Llama-3-70B-Instruct works on my 64GB M4 – I had to download the 37GB file, quit a whole bunch of apps to free up RAM, then chmod 755 it, run ./Meta-Llama-3-70B-Instruct.Q4_0.llamafile and visit it on port 8080 on localhost
-
Meta-Llama-3-70B runs on M2 Mac with 64GB RAM
By
–
OK, can confirm that it works on an M2 with 64GB of RAM I had to quit Firefox and VS Code to free up enough RAM for it to run – Meta-Llama-3-70B-Instruct.Q4_0.llamafile now uses 38GB of RAM and runs at about 7.5 tokens a second