Freezing all layers except these: – ^lm_head.weight$ – ^model.embed_tokens.weight$ – model.layers.4[4-9]+.block_sparse_moe.gate – model.layers.4[4-9]+.block_sparse_moe.experts – model.layers.5[0-5]+.block_sparse_moe.gate – model.layers.5[0-5]+.block_sparse_moe.experts
Selective Layer Freezing for Language Model Fine-tuning Strategy
By
–