All the memories belong to the false self, not to the true self, which is formless.
Every mind [universe] merges into Consciousness and remains not as mind, but as Consciousness without the experience. When the consciousness becomes aware of its true nature then there is no duality it nondual existence, which is ultimate reality.
In ignorance or illusion, one first forgets the soul [Consciousness], which is the true self, and views and judges the worldview on the false self/ego. All the memories belong to the false self, not to the true self, which is formless. The false self is dependent on true self for its existence. Therefore, on the standpoint of the true self [soul] the memory is mere illusion because it is based on the false self within the false experience.