Brydo,
Even then it is not simple, as since 2019 people with hidden disabilities can apply for and receive a blue badge:
So, your ‘Mr Watcher’ has no way at all knowing who has and hasn’t a disability.
For your part, you have no reason to have to explain to him your situation. Say, for example, your good lady had an embarrassing hidden disability, would you feel the need to explain that to a stranger?
Speaking personally, unless I am wearing shorts, or walking (or my form of walking anyway), it is difficult for people to know I have a prosthetic leg if I am stood still!
If someone did ask me (other than Police, parking wardens etc) I would tell them it is none of their business – if they can understand my speech that is.