I gave a talk at WSDM 2009 that talks about some of the early search systems engineering that gives a bit of a flavor of some of the developments through multiple generations of the search system.
SYSTEMS
-
Search Infrastructure: MapReduce, BigTable, Spanner, Protocol Buffers
By
–
Much of the search system scaling work led to infrastructure creation of things like MapReduce, BigTable, Spanner, protocol buffers, and other things.
-
Unpublished Web Index Scaling Techniques and Data Storage Innovation
By
–
We never published most of the things we did in order to accomplish this scaling. Lots of cool things here like storing data on outside of disk platters for better read bandwidth, better index compression, to then storing the entire web index in RAM (in each serving cluster!)
-
Scaling Search Index from Millions to Billions of Pages
By
–
As our search traffic grew, we were desperately trying to avoid melting every Tuesday around noon (peak traffic for the week), while simultaneously improving the service. Making the index bigger (30M->70M->200M->500M->1B pages and beyond in a short span), …
-
Collective Intelligence Transforms Deep Learning Systems Beyond Static Design
By
–
Collective Intelligence for Deep Learning https://
doi.org/10.1177/263391
37221114874
… “The way we are building modern neural network systems is similar to the way we are building bridges and buildings, which are designs that are not adaptive. Natural systems, in contrast, adapt and become part of -
AI Safety Systems Prevent Fatal Car Accidents
By
–
It’s giving ‘Nah, you ain’t gonna kill me’ 😂🤣👇🏾🎥 pic.twitter.com/tE4WAOGhGE #driving #SafetyFirst #car
— Catherine Adenle (@CatherineAdenle) 3 septembre 2023It’s giving ‘Nah, you ain’t gonna kill me’ #driving #SafetyFirst #car
-
AI transforms SRE practices and technical debt management
By
–
#AI: A Game-Changer for SRE Work-Sharing and #TechnicalDebt https://
devops.com/ai-a-game-chan
ger-for-sre-work-sharing-and-technical-debt/
… via @devopsdotcom #ArtificialIntelligence #DevOps #CloudNative #cloud #tech #lowcode #NoCode #DevSecOps #AIOps #MLOps #k8s #Kubernetes #Docker #github #Engineering #technology #OpenSource #bot -
AI Transforms SRE Anti-Fragility in Cloud Infrastructure
By
–
Embracing Chaos With AI: Reinventing SRE’s Anti-Fragility Practices! https://
devops.com/embracing-chao
s-with-ai-reinventing-sres-anti-fragility-practices/
… via @devopsdotcom #chaosengineering #DevOps #CloudNative #cloud #tech #lowcode #NoCode #DevSecOps #AIOps #MLOps #k8s #Kubernetes #Docker #github #Engineering #technology #OpenSource -
Responsible AI System Design: Values Over Speed
By
–
&…specific goals. Then you can…ask questions like “What values do we want the system to reinforce?” “What data makes sense to use here?” It’s the opposite of the “move fast and break things” philosophy. It’s…hard to advocate for that kind of work because it’s not flashy"
-
Adaptive Learning of System Values from Observable Behavior
By
–
Not necessarily. For example, you might learn that value adaptively based on observed behavior of the system (and therefore respond to dynamic changes), or learn different settings of that value for different machines on which your system is being run.