Fascinating work — I noticed repetition elicits base-model-like generations in GPT-3 last year but assumed they were fully hallucinated. Kicking myself now. As of August you could also do token repeats within a prompt to get base-like completion of grammatically incomplete text: