it's one of many inconsistencies in the books, and JK did admit to the error.
as a baby, he probably wouldn't have registered that his mum had been killed. I'm not so sure about the Quirrel case.