It was cruel as Harry didn't deserve any of it, and how they kept blaming him for everything, like in the first book with Dudley and the Zoo, and when he went to Hogwarts they became afraid he would hurt them so they allowed him more freedom. Also Petunia couldn't send him away because when Petunia took him in she sealed the pact that they had to house him till he was 17 or something like that, as it was written in the letter Dumbledore laid with Harry at the beginning