While playing Annie, if you start stacking up your passive, then use your Q ability (Disintegrate? dont remember the name), and whatever you are attacking dies while the projectile is in the air (I.E. it hits nothing), you lose all stacks of your passive no matter how many you originally had.

I kept wondering why I wouldnt have passive stacks up occasionally when I knew I was supposed to, but I'm pretty confident this is whats causing me to lose stacks.

Step 1) Choose Annie
Step 2) Get any amount of passive stacks up that you want without becoming "Energized" (1,2,3, and 4 all work)
Step 3) Cast Q on a creep as it is dying so that you still cast your projectile, but it hits nothing
Step 4) Lose all your passive stacks