Gradient descent is a powerful tool for optimization spaces that verify the manifold hypothesis, but the space of reasoning is discrete and combinatorial. GD fails in cliff-like landscapes where a single discrete change (a logical step) alters the entire outcome. Unless…???

→ View original post on X — @fchollet