Removing points would be a minefield. I think more what tends to happen here is that the criteria shift over time so whilst our conditions are likely lifelong (and will probably get worse, not better) they may not score the same if the reviews are a significant time apart.
My advice around critiquing the assessors report is to try and do it with an open mind and to be as unemotional about it as possible – stick to facts and provide supporting documents where possible to put the assessment into doubt if you feel it is wrong.