I think basically he is saying "Remember my last words to you", meaning the letter that he left on the night he left Harry on the Dursley's doorstep. It is this that makes Petunia quickly change and insist that Harry stays, because of what was said in the letter about the protection charm placed upon Harry until he is 17.