Seems great, right? Public data is plentiful online, we can just download tons of it, pretrain our models with this public data, and do fine-tuning privately! Privacy is solved! Of course not, and we highlight three (orthogonal) considerations for these settings. 5/n
@thegautamkamath
-
Private Fine-Tuning of LLMs Shows Modest Utility Loss
By
–
Story is similar for language models. In some prior works at #ICLR2022 (by Yu et al https://
arxiv.org/abs/2110.06500 and @lxuechen et al https://
arxiv.org/abs/2110.05679), it was shown that privately fine-tuning (publicly) pretrained LLMs suffers only a modest utility loss. 4/n -
Pre-training ImageNet Fine-tuning CIFAR-10 Privacy
By
–
The good news is, public data can help dramatically! Pre-training on ImageNet (without privacy constraints) and fine-tuning on CIFAR-10 (privately) gets us up to 95%+ (figure from the same paper). Takes it from bad (and in some cases, unusable) to pretty good! 3/n
-
Differential Privacy in ML: CIFAR-10 Performance Gap Challenge
By
–
Differentially private ML is hard. Even basic "solved" tasks in the non-private setting are very hard to do with privacy. Figure from a nice paper by @sohamde_ @LeonardBerrada et al (
https://
arxiv.org/abs/2204.13650), showing SOTA results on CIFAR10.. 60-80%, versus 99%+ non-privately 2/n -
Differential Privacy Challenges in Large-Scale AI Pretraining
By
–
New paper w Nicholas Carlini & @florian_tramer
: "Considerations for Differentially Private Learning with Large-Scale Public Pretraining." We critique the increasingly popular use of large-scale public pretraining in private ML. Comments welcome. https://
arxiv.org/abs/2212.06470 1/n -
NeurIPS ICML Experiment: Data Collection on Decision-Making Process
By
–
To make it absolutely clear, like I said, this is an EXPERIMENT, similar to past experiments at NeurIPS and ICML. It will not have a significant impact on the actual decisions, more just to gather data. I have heard there have been emails complaining… x.com/thegautamkamat…
-
AI Chatbot Designed for Refund Processing
By
–
Just use your AI chatbot designed to get you a refund in response.
-
Lupe Fiasco Discusses Complexity Theory at MIT
By
–
Lupe Fiasco (
@LupeFiasco
) giving a talk at MIT in front of a blackboard covered in complexity theory https://
youtube.com/watch?v=zBHRsY
hYb-o
… -
Self-ranking method improves academic paper evaluation system
By
–
This is an obvious critique, which the paper deals with. It makes you rank your own papers against each other. This turns out to make things work out. Check out the paper!