decoding (tokens -> string) is just lookup table and string concat. encoding (string -> tokens) is a pain. For sentencepiece I *think* llama2.c has a simple implementation that probably works but I'm not 100% sure: https://
github.com/karpathy/llama
2.c/blob/master/run.c#L452
… For tiktoken-style, the problem is the
OPEN SOURCE
-
Token Encoding and Decoding: Asymmetric Complexity in LLMs
By
–
-
Neural Network Projects with Python for Data Science
By
–
Neural Network Projects with #Python! #BigData #Analytics #DataScience #AI #MachineLearning #IoT #IIoT #RStats #TensorFlow #Java #JavaScript #ReactJS #GoLang #CloudComputing #Serverless #DataScientist #Linux #Books #Programming #Coding #100DaysofCode https://
geni.us/N-N-Projects-W
ith
… -
Julia Lang Programming Projects for Data Science and AI
By
–
#JuliaLang Programming Projects! #BigData #Analytics #DataScience #AI #MachineLearning #IoT #IIoT #Python #RStats #TensorFlow #Java #JavaScript #ReactJS #GoLang #CloudComputing #Serverless #DataScientist #Linux #Books #Programming #Coding #100DaysofCode https://
geni.us/JuliaLang-Pro-
Proj
… -
Python Libraries Cheat-Sheets for Data Science and Machine Learning
By
–
Cheat-Sheets! #Python Libraries! #BigData #Analytics #DataScience #AI #MachineLearning #IoT #IIoT #PyTorch #Python #RStats #TensorFlow #Java #JavaScript #GoLang #CloudComputing #Serverless #DataScientist #Linux #Programming #Coding #100DaysofCode https://
geni.us/Python-Cheat_Sh -
25 Essential GitHub Repositories for Data Science and AI
By
–
25 Amazing GitHub Repos to Check. #BigData #Analytics #DataScience #AI #MachineLearning #IoT #IIoT #Python #RStats #TensorFlow #Java #JavaScript #ReactJS #GoLang #CloudComputing #Serverless #DataScientist #Linux #Programming #Coding #100DaysofCode https://
geni.us/25-Github-Check -
ScrapeGraphAI: LLM-Powered Web Scraping Pipeline Library
By
–
We'll use ScrapeGraphAI, a Python library that uses LLMs to create smart scraping pipelines. It involves four main steps:
1. Fetch: Retrieve the data
2. Parse: Analyze and structure the data
3. RAG: Processes the parsed data to find relevant information
4. Generate Answer -
Specialized Open-Source Models Outperform Generalist Models
By
–
Specialized models based on open-source > generalist models for most tasks (+ cheaper, faster, easier to run/maintain/build)! https://
reddit.com/r/LocalLLaMA/c
omments/1cp2h1v/3b_model_beating_gpt4_on_medical_summarisation
… -
Open Source Impact on Science and Technology Development
By
–
Responsible development and use of Open Source projects and software have changed the world and propelled science and technology in public and private sectors, benefitting developers, researchers, students, entrepreneurs, businesses, etc – Human Genome, Linux, ImageNet, PyTorch,
-
Accessible LLM Studio App for Local Deployment Without Quantization
By
–
LLMs should be able to be used for everyone, locally too Any way all I'm saying is there's probably opportunity for a noob-friendly LM Studio type app See the top models rn, auto download, run locally No quantization questions
-
Open Source Usage Rights and OSS Philosophy Debate
By
–
Huh? He didn’t steal, and didn’t claim to invent it. The whole point of OSS is so that others can use it. This take is way off-base.