The drop statistic of a story is only created through people who put your story on a reading list marked "dropped". Most likely to filter your story out or remember it as "not for them".
Honestly, those are only a handful of people. The majority of people who'll drop your story (or have long dropped it) are invisible. You might realize it when your overall reader count drop. Or when you get a 5 readers and the count only goes up by 3. But otherwise, there'll always be people who leave your story and some who'll start reading it... I probably saw a hundred or so people drop my story just over the weeks of hiatus or updating statistics by chance. The story is still growing and never really had any dip in the reader count.
If you want to panic, look at your chapter views, comments, or the like. If you see a big drop there, that's a bigger problem... :D