I have the opposite problem from Gary D's. That is, I am a poor reader but a good play by ear player.
I think you are drawing the wrong conclusion. Before I pulled away from public performance altogether at about age 30, I performed long programs in public, entirely from memory, and my memory was rock solid.
The point was that I did not have to memorize in order to play many things well. Not that I couldn't.
While there are some very lucky people who have trick memories (they seem to memorize anything instantly because, among other things, they have photographic memory and may be reading the score, but in their head), most people find memorizing purely for the sake of memorization to be an added step IF they read very, very well.
However, while teaching students things that they want to learn and I've never seen or heard before, I demonstrate how I can use the same techniques I'm teaching them to learn the music faster to memorize the music while watching them, without touching the keyboard myself.
There is a huge difference between not having to memorize in order to play something well and not being able to.
The point is that people who read very poorly have no choice but to memorize anything they want to sound good. Otherwise it is painfully slow, full of holes, stumbles, and so on.
