That takedown button has always been hard for me on mine as well but never to the point I had to break out the mallet. My first thought was you are on the right track a rough spot or burr etc. There isn't much resistance on the take down spring as I recall so that shouldn't be an issue.
Have to ponder this some, feeling plumb poorly today so guess I got time to ponder.
In Memory of Paul "Dietrich" Stines.
Dad: Say something nice to your cousin Shirley
Dietrich: For a fat girl you sure don't sweat much.
Cue sound of Head slap.
RIP Muggsy & TMan
"If you are a warrior legally authorized to carry a weapon and you step outside without that weapon, then you become a sheep, pretending that JOCKO will not come today."