Old swing arm rubber - replace or not

If the tubes the stud goes through are still in a straight line, I would just ride the bike.

Much easier said than done. A long time since I attempted to remove the silentbloc bushes, gave up in the end. Can't even remember now what drove me to even try!.Others have written that they've taken a blowlamp or gas torch to burn them out.
That said, others have written of damage to the rubbers when having the swinging arm powder coated due to the bake temperatures that can be involved, the consensus on that being choose and inform your paintshop appropriately.
Given that you write it is not a very high mileage bike, as Michael B writes above, providing the spindle is OK, leave well alone.

Generally the old bushes will need a 12 ton press to push them out of the Swinging Arm.
They should be an interference fit so getting replacements in place is an equally challenging game.  Also......be prepared for the need of a new spacer to go between the bushes.

Unlike tyres and engine seals the rubber in the SilentBloc bushes appears to be everlasting and you will meet other Norton owners with mega mileages under their wheels and absolutely no sign of wear in the bushes. 
The bushes appear to tolerate being baked during powder coating as well as being coated with oil, grease and Paraffin. Petrol is a well known enemy of rubber and I suspect Ethanol and some  chemical cleaning fluids would do them no favours either.
An easy test for swinging arm bush wear is to grab the end of the wheel and heave it forward and backwards parallel to the S/A spindle.. If there is clunking or serious  sideways movement of more than 1mm total then take action as either your wheel bearings are knackered or the Silentblocs are worn.

My 1960 ones are still fine and when in the mood I can corner with anything !.

My November 1956 assembled ones are fine too.

To be fair, they may not have done much work until spring of 1957.

You could convert your swinging arm to the Manx type with phosphor bronze bushes. 

And at how much cost? Then there is the modification to the rest of the frame, then there is the up keep of oiling.

hello, i have replaced swing arm bushes before now using a press .never had much success with threaded bar and heat to remove them.
using a socket or tool of similar size as the bush to press it through the tube. i pressed the bush one side until it bangs and moves then turn over do the same for the other side. then just press the whole lot through  from one side until its all out. i replaced the bushes because i noticed when i pushed my hand down on the brake pedal,when the bike was on its lift. i noticed a lot of movement in the swing arm and slackening of the chain when i pressed the brake pedal down.

barry