wrote some faster dataloading code for HuggingFace datasets – sped up datasets.load_from_disk() from 17 minutes to around 30s the problem I was running into is that our virtual disk at school is really slow. HF loads datasets in a single thread and doesn't always memory-map.

→ View original post on X — @jxmnop