Feels like a lot of fertile ground is left in managing the "attention" of an LLM during its training via a meta-learning policy, instead of the typical "memorize dataset uniformly at random" strategy. And giving it a calculator and a scratch pad.
AI
-
Training Strategies: Skimming, Filtering Noise, and Revisiting Content
By
–
More generally a few remarkable strategies people use during their training:
1) skim text because they already know it
2) ignore text because it's clearly noise (e.g. they won't memorize SHA256 hashes. LLMs will.)
3) revisit parts that are learnable but not yet learned -
Examples vs. Presentations: Spaced Repetition in LLM Training
By
–
Is it the number of examples that matters or the number of presentations to the model during training? E.g. humans used spaced repetition to memorize facts but there are no equivalents of similar techniques in LLMs where the typical training regime is uniform random.
-
Datasette Now Runs in WebAssembly Browser Environment
By
–
… update to this thread: I got Datasette working in WebAssembly! https://
lite.datasette.io and -
Decentralized AI Through Web3 Agents and AI Nodes
By
–
Blockchain.AGI.Eth Web3 AI Agents X AI Nodes = Decentralized AI Preliminary Presentation: https://
montreal.ai/aiagents.pdf #AGI #AINodes #AIAgents #AIToken #MontrealAI -
CR-SQLite WASM Implementation Now Available
By
–
Oh wow that thing's already working in WASM too? Exciting! https://
github.com/vlcn-io/cr-sql
ite/tree/main/pkg/wasm-esm/crsqlite
… -
Stanford HAI Spring Conference 2023: AI, Creativity, Policy
By
–
A sneak preview of the @StanfordHAI Spring Conference 2023 – AI, generative models, arts, creativity, policy, regulation, and more! Stay tuned and sign up to our mailing list: https://
hai.stanford.edu/subscribe-hai-
mailing-list
… -
Datasette Lite adds GitHub JSON loading capability
By
–
Added a new feature to http://
lite.datasette.io – it can now load JSON data directly from GitHub (or other CORS-friendly hosting) using the new ?json= URL parameter Documentation here: https://
github.com/simonw/dataset
te-lite#loading-json-data
… -
Takahe: Open Source Decentralized Social Protocol Project
By
–
I'm keeping a close eye on https://
github.com/andrewgodwin/t
akahe
… -
Lightweight Mastodon-compatible blogging software for static hosting
By
–
I'm really hoping we can get some much more lightweight Mastodon-compatible blogging software working – there are WordPress plugins and suchlike, but something more compatible with static hosting and a bit of serverless function magic would be fantastic