Keras code compiles to JAX, TensorFlow, or PyTorch code, so anything you can do with one of these frameworks you can do with Keras. I recommend using JAX as your backend, as it is suitable for very-large scale computation and can be easily parallelized across many devices.
MACHINE LEARNING
-
GPU acceleration and XLA compilation limitations in framework integration
By
–
But then you would not be able to run it on GPU, nor compile to XLA for a ~5x speedup, nor use it inside a TensorFlow codebase or a PyTorch codebase.
-
Keras ops API: NumPy syntax across JAX PyTorch TensorFlow
By
–
A neat feature of Keras is that its `keras.ops` API follows the NumPy API syntax exactly. And since it runs on JAX, PyTorch, TensorFlow, and NumPy itself, you can use Keras to write GPU-accelerated NumPy code that runs with any framework.
-
Dealing with random MPI hangs in large model training
By
–
I thought I didn't have to deal with these, but already the 350M model (14 hours of 8 GPUs working) sometimes randomly hangs with a cryptic MPI error once in a while. So I have to put the whole optimization into a `while 1` loop and a script that watches the log file and sends
-
Critique of flawed paper on model interpolation definition
By
–
I've seen a paper that argued that, but it was utterly idiotic since it used "L2 interpolation in encoding space" as its definition for "interpolation". Effectively, it was saying, "models that aren't linear aren't linear".
-
Regularization as a Prior: Simplicity Without Extrapolation
By
–
Regularization is a prior towards simplicity. It is not a domain-specific prior and does not enable extrapolation on its own.
-
Single Node Training vs Large-Scale Multi-GPU Distributed Runs
By
–
But those were also much much bigger runs, so it's a lot more impressive. This was on a single node so you don't need to deal with any cross-node interconnect. It starts to get a lot more fun when you have to keep track of O(10,000) GPUs all at once. For a very specific
-
Why 10B Tokens Suffice for GPT Training Performance
By
–
Great question yes I was surprised that 10B seemed enough. I believe GPT-2 was trained on somewhere ~100B tokens. The reason we reach this performance in 10B tokens I think may be the following: 1. FineWeb could just be higher quality than WebText, on a per-token basis. This was
-
GPT-3 Hyperparameters Analysis and Model Scaling Expectations
By
–
ah ok these are gpt3 hparams. it sounds like this alone would be enough to beat GPT2, and its unknown if 290B more tokens would get it to match or beat GPT-3 Small looking forward to the 1.5b – fascinating to see these all documented and taught live!!
-
CNNs remain essential for real-world AI deployment efficiency
By
–
Entertainment aside, getting rid of CNNs for real-world AI deployment is almost impossible. Even if you went for a ViT architecture, you must process the input using local patches with shared weights for efficiency and generalization. It is even more the case for processing