I believe he is referring to the letter he left with Harry, and in that letter, didn't it say something about how Harry would be protected in the place he called home? I think that's what it's referring to, and Petunia realised that even if she hated him, she had to allow him safety, at least for a little while.