I didn’t really appreciate that, because they operate at under one volt, high end GPUs are continuously flowing over 500 amps through the die at full load.
AI HARDWARE
-
CoreWeave Raises $2.3B Debt Collateralized by Nvidia Chips
By
–
https://
reuters.com/technology/cor
eweave-raises-23-billion-debt-collateralized-by-nvidia-chips-2023-08-03/
… cc @aashaysanghvi_ -
Smart GPU Financing: Collateral and Depreciation Strategies
By
–
When actual finance guys get into AI infra, they think of actually smart ways to finance their growth re: don't sell equity for GPUs. "We negotiated with them to find a schedule for how much collateral to go into it, what the depreciation schedule was going to be versus the
-
NVIDIA’s 55x Growth Since 2015: GPUs Dominating AI Market
By
–
Overheard in the hallowed hallways of AI compute: "I have been hearing GPUs are not designed for AI since 2015 – during that time NVIDIA 10x (at least) its market cap" I actually checked. It's not 10x since 2015, it's…55x!
-
Optimized LLM Training Framework Reduces Host Memory Overhead
By
–
If you use an optimized #LLM training framework like https://
pbase.ai/3DHqnE5, you can get the host memory overhead back down to a more reasonable 7 * 4 = 28 GiB of host memory even when training on multiple GPUs. -
Loading Pretrained Checkpoints: Multi-GPU Memory Challenge
By
–
Before you even get to multi-GPU training with model parallel frameworks like #Deepspeed, you need to load the pretrained checkpoint into memory. To make matters worse for machines with multiple GPUs, you need to load the checkpoint into host memory once for each GPU in your job!
-
Llama2 7B Model Training Memory Requirements on Multiple GPUs
By
–
Now training your 7B parameter #Llama2 model in float32 with 8 GPUs requires 7 * 4 * 8 = 224 GiB of host memory just to load it onto the GPUs.
-
Main Problem Teams Face When Fine-tuning LLMs: Out of Memory
By
–
What’s the #1 problem teams encounter when #finetuning an #LLM? The dreaded "Out of memory" error. No, not a CUDA OOM, just a regular host out of memory error.
-
Tenstorrent’s AI chip designs secure major funding from Hyundai Samsung
By
–
"Lead by acclaimed chip architect @jimkxa
, Tenstorrent has a track record of not only general purpose AI processor designs based on the open source #riscv architecture, but also highly-tuned, bespoke #AI chips and board level accelerators." – @Forbes https://
forbes.com/sites/davealta
villa/2023/08/02/confidence-in-ai-startup-tenstorrent-spurs-100m-funding-round-by-hyundai-and-samsung/?sh=6d1f7aeb4c5a
…