#AskDatabricks is back on October 18! This time we’re diving into Spark Structured Streaming — feature updates, best practices, and more Join @MrSiWhiteley + Databricks PM Ray Zhu and bring your questions. https://
bit.ly/44NppBC
CODE
-

Spark Structured Streaming Features and Best Practices Discussion
By
–
-
GPT-4 Self-Improving Code Through Recursive Scaffolding Programs
By
–
The core idea begins with an initial seed 'improver' scaffolding program that utilizes the language model to improve a solution to some downstream task. They demonstrate that GPT-4 is capable of writing code that can call itself to improve itself. /15
-
INDEX and MATCH Functions for Advanced Data Lookup
By
–
9. INDEX and MATCH: INDEX and MATCH can be used together to look up a value in a table and return a value from another column based on a match. This is a more powerful alternative to VLOOKUP and HLOOKUP in some cases.
-
HLOOKUP Function: Horizontal Data Search in Spreadsheets
By
–
6. HLOOKUP: Similar to VLOOKUP, HLOOKUP searches for a value in a table but searches horizontally instead of vertically.
-
VLOOKUP: Dynamic Data Lookup and Reporting Automation
By
–
5. VLOOKUP: VLOOKUP is used to search for a value in a table and return a corresponding value from another column. It's helpful for creating dynamic reports.
-

Copy Suppression: Understanding Attention Head Mechanisms
By
–
Copy Suppression: Comprehensively Understanding an Attention Head McDougall et al.: https://
arxiv.org/abs/2310.04625 #Artificialintelligence #DeepLearning #MachineLearning -
Understanding Wave Quantization Effect in AI Systems
By
–
Sorry what's the wave quantization effect?
-
FP8 Training: Understanding Gradient and Weight Data Types
By
–
Why does it matter for fp8? Are grads and weights different data types in that case? (Sorry if it's a dumb question – I've never done any fp8 training)
-
Fast.ai Model Achieved 100% Accuracy in Classification Task
By
–
We used this example in http://
fast.ai class years ago and even then it got 100% correct! -

FSDP vs DDP Communication Overhead Derivation Explained
By
–
Anyone know how to derive this '1.5x' communication overhead between FSDP vs DDP (from the FSDP paper)?