There is also the very grey area of prescription medicine. If you are on something as simple as pain killers (e.g. co-codamol) it could be a difficult thing to justify you were/are fit to drive.
And don’t rely on a GP to back you up in court as they will not be able to say at a particular point in time, if you were fit to drive.