The few times ive been confronted I normally just say to them I have a blue badge and am as entitled to park here just as much as you are, sue me if your not happy and walk off leaving them muttering to themselves.
On one occasion some time ago the situation got really heated and close to a scuffle was on holiday to see the Blackpool lights, when this couple was not happy because I had just parked in a disabled bay they wanted. They got out of their car giving it all in carnation they had more of a right to park there than me and the bloke went to let a tyre down! Luckily a couple of PCSO’s were walking past at the time and dealt with the situation.