Should I be really concerned about this rust on the block? It feels really smooth

 

As that's in the gasket sealing surface, yes, you should get the block decked to clean it up.

 

If you can't feel it I would use a gasket surface cleaning pad, scrub it thoroughly and not worry about it.

To deck the block you are looking at full disassemble and cleaning of the block afterwards, which means cam bearings and rod resizing and bearings, cost is going to skyrocket FAST.

 

What about using a rust remover like evapo-rust or naval jelly? Is it safe to use it there?