So you carry an unloaded gun?
The average gun fight lasts 3 seconds, and happens at 3 feet, the other guy will also likely all ready have his gun out, and chambered at the start of the fight, do you really think you will have the time, or ability to rack the slide in that scenario?
It would seem that way.
With any quality manufactured modern firearm in usable condition it is safe, and very important, to carry with the firearm ready to go. My M&P was designed with no manual safeties and that is the only way to safely (your safety) to carry that pistol as an EDC.
During a fight (which apparently averages 3 seconds, 3 feet) you will probably need that other hand and extra time for things you cannot have already prepared. What if a dog attacks you out of nowhere with no warning (happens all the time), do you just open your vital organs up to an attack just because of an unprecedented fear? During most attacks your non dominant hand would be better served doing something else.
Sig, Smith, Remington, Wilson, Keltec and even Hi-points does not want the firearms they manufacture to just spontaneously discharge or the liability of it. Trust me on that. A good law suit and some bad publicity could put even the big names out of business. They want all the money from you that they can get.
I think the fear of carrying any weapon other than a paper weight is comparable to the fear of clowns, it is very un-rational and no sane adult should hold them.
Carry your firearm like you expect something to happen, pray that it never comes to that.
That is my advice.