It's always worth spending time on the control rods & the bearings that they rotate in. Often the rods are not dead straight but can be corrected.
I think the OEM plain bearing components (top hat, olive etc) were not just nylon, but an oil impregnated material, which is far less likely to swell like the cr@p replacements.
If you are not already aware, the saddle clamps that retain the olives can have a washer fitted beneath them to act as a shim to prevent any binding. Best fitted with SHCS's which are M4 though if the threads are dodgy, a stud with nylocs will do.
I know you haven't got there yet, but the screws that pull the top onto the bottom headset are M5. I have used SHCS's for decades with a ball ended hexagonal key & that makes the job so easier & 'non-slip' as far as the tool goes. Yes, I know the holes are countersunk rather than counterbored, but the SHCS's never come loose......