Andrej Karpathy recommends removing the memory from models to focus on the algorithms for thought and the idea of an experiment.
I'd love to have them have less memory so that they have to look things up, and they only maintain the algorithms for thought, and the idea of an experiment, and all this cognitive glue of acting.