Can language models invent solutions to new problems?

A commentator disputes a quoted claim that language models cannot invent solutions to new problems, pointing to von Neumann as an example of strong memory coexisting with creativity. They argue that training shapes a model’s representations rather than simply storing facts. In a follow-up reply, they say the network’s information lies in relationships between numerical weights, while describing the limits of those relationships’ expressivity as unclear.