If the kitten was looking frightened or overwhelmed it might have been. No matter how cute, most things don't want to be touched by strangers while having a freak out. (this is especially true if it got confused and thought you were someone known)

Otherwise it's ususally considered an insult not to pet if it appears welcome.
I'm allergic. And most people wont take "no thank you my eyes will swell shut" at face value.