I'm not so convinced you're in a small minority in preferring Three out of the first two trio albums.
It was a very impressionistic comment that I made. Just a sense I get from hearing and reading stuff over the years. In particular, there's a long thread on this site about ATTWT. I remember reading through the comments and being struck by how many people were critical of the album.
I absolutely agree with you that we all hear things differently. We also all hear music in different circumstances and contexts. My affection for ATTWT is probably tied up with the fact that it was the first Genesis album that I heard as a young lad. It was music for grown-ups but it was also accessible enough for me to get into. By the time I heard Duke, on the other hand, I was heavily into all that 'classic' '70s stuff, so Duke felt like a let-down.