Yeah, template cutting on oak is challenging!
I do about the same as you Ross. Cut close with the bandsaw, then make small passes without bearing contact to nibble away at the difficult sections. A top/bottom bearing bit that is sharp is a must. I'll use whatever top bearing I have that has the length for a full cut and borrow a bearing and collar so I can turn it into a top/bottom. Pencil marks where I'll flip the part to stay "downhill".
Those scarf areas are a worst case! I'll try to nibble uphill to peck away what I can with a kung-fu grip!
Sanding the end grain noses is a great idea!