General Maintenance & Repairs - A/C Compressor Grinding when disengaged

I have seen a couple of threads on this, but no real answer for it. The compressor grinds when it is disengaged and immediately quiets down when it is engaged.
I have traced it to the clutch. Putting my finger on it in the right location while it is making noise quiets it down. I have a piece of thin cardboard wedged into it now to silence it.
Anybody experience this or know what it takes to fix it?

I pulled my ac belt off because it was bothering me so much, so please someone help us out. Thanks