Eight years ago I had an idea. Four weeks ago I decided to start implementing that idea. Last weekend I started running an experiment to validate this idea.
This is real data: The model has a fixed set of 5.6B parameters. It does not expand parameters. It does not rely on an external memory system or replay buffers. This chart is the result of running four continuous learning sessions with the pa