Down and to the left...

[lrahm]

I've done the same thing. Hard habit to break. You probably grab the trigger in between the two joints of your trigger finger. Try grabbing it by the pad of your finger. It has a tendancy to make you pull straight back instead of pulling it low and to the left.