Not necessarily overfitting, but lack of transparency and easy editing are the major things. See previous thread:
@alexjc
-
Hybrid AI Systems More Practical Than Full LLM Retraining
By
–
I love the dramatic futurism here! But it's more likely they will be hybrid systems. You don't want to retrain a LLM everytime a Wikipedia page is changed, or a news item is published. (Plus, Google would be in the best place to deliver this new system, incrementally.)
-
T5 and CLIP Embeddings: Superior Combined Performance Analysis
By
–
They ablate the embeddings in the paper and found that T5 and CLIP together are superior to either on their own. Not clear just how much difference it makes on text though… Imagen is bigger than SD, hard to disentangle the size from embedding on text. Good hypothesis though!