It's Petunia's sisters son. Regardless how much they hate him and stuff, Harry is always going to be her nefiew. It's that annoying "Doesn't make since it just happens" kind of thing. Plus... in the 5th book you find out that Dumbldor told her that if Harry doesn't stay with them that he would die... that would make it her fault if she chooses not to let him stay.