Exactly.
If it were my pistol, I would contact Jay and request a return label. Explain the fact that your CM40 was intended to be your EDC pistol but you now don't have enough confidence in it for that.
Kahr Arms knows very well that this is a problem with their pistols, especially the .40 cal. models.
It's only your life your betting on by only replacing the followers.