It depends on how you define atheist.
If atheist just means "someone who doesn't believe in a god," then yes, babies are atheists. But, you know what? So is your dog. And every fetus. And every corpse lying in the ground. They don't believe in God, because they don't think about these things. So if that's your definition, then who really cares?
You might as well say the baby doesn't like sports. No, it doesn't, but only because it has no idea what you're talking about. My wife doesn't like sports, but it's because she thinks they're boring. The key words there being: "She thinks"
I promise you no atheist group is saying, "You just had a baby? Look like we have a new member!" Because a baby's atheism means nothing.
So how should we define atheist? At least in context here, we should treat it as active disbelief -- someone who has come to the conclusion that he doesn't believe in God. Because if you haven't thought about it, it means nothing.
Taking credit for someone's beliefs when they've never even really thought about it isn't something atheists ought to be doing. The church does that already. Between childhood baptisms and indoctrination classes and people who think you're religious because you happen to be born to religious parents, there are enough people who think children need to have a religious label.
Let's not be like them. Next time someone says a baby is an atheist, challenge their definition. Even if they're technically right, it's a meaningless victory.