Yeah, I've always made a similar argument to fans of punk music over the years, met with reluctant acceptance through to "**** off mate, no way!"
Numerous punk icons from John Lydon, Hugh Cornwell and Jaz Coleman expressed a liking for some progressive bands; The likes of Hawkwind, VDGG, Amon Duul, Neu, Soft Machine and even some early Genesis. Phil Collins was said to have been approached by a nervous Rat Scabies from The Damned in an airport, who claimed to be a great admirer of his talents as a drummer.
Punk does have something in common with prog IMO. It was just ironic that the latter was so unceremoniously replaced by the former.