FUTURE WORK – deduplication even though i varied the random seed and used temperature, a lot of the outputs are highly redundant it would be prudent to deduplicate, i bet there are only 100k or fewer mostly-unique examples here
Deduplicating Redundant AI-Generated Output Data
By
–