You really have to apply a filter in the workplace. You cannot relax and think that you're among friends. I think violent thoughts about annoying people all the time. I enjoy thinking of extended scenarios in my head to deal with irritation, and sometimes I share those thoughts with people who I know won't be horrified by them. It's obviously just a way of dealing with stress, since I have not done and would not do these things, but some people find admitting to such thoughts to be a sign of instability. (Unless of course your fantasies involve sprog protection, in which case you get a free pass on
imagining hitting people at a market when they haven't done anything aggressive or annoying to you.)
Looking at the news that we bash here reminds me of how most of the world thinks about natalist issues, and keeps me from saying anything "too outragous" at the workplace. I will readily admit that I don't enjoy spending time around children and find them annoying, but other thoughts I save for Bratfree.
It may have been forgotten, or it may have subtly changed his opinion of you. I know my opinion of someone was forever changed when he let slip that he thought I was selfish for not reproducing. He'd never bingoed me or shown any curiosity about my lack of children for the five years or so I'd known him, but then suddenly this. I didn't talk about it, but I will never look at him the same way.
I don't know if it is possible to repair the damage, but you could consider a subtle campaign of rebranding yourself. That would require casually mentioning a positive encounter with a child at some point in the future, say 2-3 months from now, when it fit in the conversation. For instance, if you have an anecdote that somewhat relates to the topic of discussion, you could mention it. It need not be something about your lurve for children, it could just be an unrelated story that happens to involve a friend who has a child (where it is relevant to mention that the child exists). This says "I am not a monster who hates all children and people with children" to the average person. Most of us have had at least a few positive interactions with children, so it's not impossible to work in these references. We just normally don't deliberately try to mention them, and that's what I'm suggesting you have in the back of your mind. Don't overdo it though.