I wondered this myself. I also have a young child at home. Less than a year old. I used to unload when I came home. I read about setback. Noticed it on my glock. I probably chambered that round 50 times. It fired no problem. I now have 2 handguns. A glock 30 and cm9. I leave the glock at home, loaded, in my holster, and in a locked safe for home defense. Wife and I have combo. The cm9 rides with me or in the safe. Never any exceptions (other than cleaning). I found that loading and unloading seemed more dangetous than not. I have a few rules. Two of them are: an unloaded gun is useless. Alsoi dont take my gun out of the holster until im ready to shoot it.
I just finished reading an article in a special handgun edition of guns and ammo on this very subject and they discuss setback. They state about this very subject and suggest every time you cycle a round just throw that round in a box and load a new round and you'll have plenty of range ammo to practice with.