I find it no harder listening to Morrissey following his comments than I do listening to Stewart Lee after hearing him make the sound of eating a popadom for half an hour. They both had their reasons for wanting to break something, I guess.
As Stewart Lee was a Fall fan (and I was until MES started turning up with his granny on bongos around 1998) - yes, why did he remain in thrall to MES after 1983 (when MES by his own admission stopped voting Labour and could always be relied upon to upset the politically correct agenda), why do paedophile and murderers' records remain in his collection, if those are his rules as to what makes a record listenable?
Do I have to retrospectively scuttle off and research the beliefs and actions of everyone I own music by? No, I don't. I think people who have been raised on social media have this obsession, as does anyone still trying to turn the clock back by remaining in touch with that generation. It was never going to remotely affect Mark E Smith, that's for sure.
Morrissey has, whether by accident or design, highlighted this difference in the rules by which he is judged. Sorry there are still stragglers who think of him as their special artist, that's their problem. My advice to them is think of him an artist making better than average music - not so politically correct, nor yet a paedophile nor a murderer - no need to destroy records, no need to stop listening to them.