Thankfully, I did buy a magazine loader. Not sure I could have loaded the mags without one!! I will probably pick up some Hornady Critical Defense ammo Wednesday. That's what I use in my other guns.
Once you break in the mag spring, it will get easier to load.
Also, depending on the size of your hands, you might want to consider putting on a Pearce Extension to the short mag. I have found that carrying with the extended mag gets a bit cumbersome all for 3 more rounds. Here's a pic of mine with the extension.