For the echo merge, the idea is that self-merges might be slightly "insane" because replicating blocks of several layers disrupts the information flow. By only replicating individual layers in a distributed way, I'm trying to minimize the input/output difference they introduce.
Echo Merge: Optimizing Layer Replication in AI Models
By
–