Yeah, for the between detents (5°) adjustments you need to align the 1° marks with the vernier and forgo the detent.
For fine adjustments like that (and tuning for a perfect 90), I'll just place some tape layers or other shim stuff between the fence and bracket on one side, between the screw and fence.
Tape being generally 0.004" or so gives you really great control when that 0.0x degree tuning is needed.