Basically, cats perceive not looking directly at them when they don't know you well as like, the polite/non-aggressive thing to do, so they're like, "ah, finally someone with manners-- I shall introduce myself to them."
Generally, it's bad to stare at smaller animals if you want to get along...