(Initial version supported CPUs. Later versions added support for GPUs and TPUs).
MACHINE LEARNING
-
DistBelief trained 10000+ models across architectures
By
–
Distbelief was used internally for training 10,000+ distinct models of all kinds of architectures (convnets, RNNs, LSTMs, feed forward networks, sparse MoEs, etc), with all kinds of different training objectives (supervised, unsupervised, RL, etc).
-
Deep Learning v2: What Topics Need Better Coverage?
By
–
Along the way, deep learning has become a lot more popular, with many other sources to learn about it. But are there topics you’d want covered in v2 that aren’t covered well in other sources?
-
Deep Learning Evolution From Academic Niche To Commercial Dominance
By
–
For me at least it’s partly that the need isn’t as great now as in 2013 when we planned v1. Deep Learning has gone from a niche academic workshop topic to the most publicized product of more than one > trillion dollar company.
-
Why Deep Learning Textbook Lacks Code Examples
By
–
Sometimes people ask me why there isn’t code in the Deep Learning textbook and I say “well, the code would’ve been in pylearn2,” and they say “what’s pylearn2?” and I say “exactly.”
-
RAG for Diffusion Models to Reduce Knowledge Requirements
By
–
i wonder if we can do rag for diffusion models to get rid of the knowledge requirement. 0.6b params is great
-
DistBelief: Distributed Training and Knowledge Distillation Framework
By
–
Or https://
arxiv.org/abs/1503.02531, which describes how distbelief was used to train the baseline model (a quite large model for the circa 2014/2015 time frame), and later describes using the framework to train specialists and then distill them into a single model. -
Theano and cuda-convnet: Early deep learning frameworks history
By
–
Yoshua’s lab was all Theano. I actually wrote Theano bindings for Alex’s cuda-convnet before ImageNet came out. He was working on the code as an open source library and had good results on CIFAR quite a while before the ImageNet results were public.
-
Supervised ConvNets Scale Better With Smaller Parameter Count
By
–
Supervised convnets worked just fine with disbelief (in fact they scale better because of the smaller parameter count). For example: https://
papers.nips.cc/paper_files/pa
per/2013/hash/7cce53cf90577442771720a370c3c723-Abstract.html
…