Tips for pan filter seal removal

The thing is fused to the port, how do I dislodge it and is it a big deal if the alloy gets scratched?
It's not the best idea to scratch the aluminum especially on the ID of the hole.
To reinstall the new one I use one inch solid steel bar stock and a hammer.
The problem with using a socket is that the seal partially falls in it.

