it's not really doing the same thing, they are doing encoding only while the tokenizer lib does a lot of work keeping track of offset between the original string and the final tokens, truncating, etc. this is needed to support various model hub task like QA, etc
@thom_wolf
-
AI Detector Monetization Strategy Business Model
By
–
maybe they plan to sell/monetize a detector? could make sense from a business point of view I guess
-
OpenAI’s Missing GPT Detector Tool Release
By
–
I'm a bit surprised OpenAI didn't release an updated GPT detector together with ChatGPT. Seems like something that would be useful for the impacted sectors, no?
-
ChatGPT Prompt Dataset Released in Datasets Library
By
–
ChatGPT prompt dataset just landed in the datasets lib
-
GPT-3 Training Data: C4 Dataset Technical Details
By
–
(Except that GPT3 is not trained on C4 per se @Francis_YAO_ but it’s a small tweak)
-
100 Puppies Trained with RL Deep Learning on Hub Platform
By
–
there are now over 100 puppies trained to play fetch shared on the hub thanks to the new rl deep learning course! So many good bois we should get all of them to play together in a space of the hub
-
Virtual Agents and Conversational Interfaces Report Coverage
By
–
what does « virtual agent or conversational interface » cover in the report?
-
Academic Research in AI: Key Considerations and Applications
By
–
I was more thinking in terms of academia research but some of your points are also valid there actually
-
NLP Research Direction Practical Applications Focus
By
–
I fully agree on my side but there is also an entire part of the NLP research community which is of the opinion that thinking in terms of practical applications is the only way to have a sensible direction and reason for why we are building what we build