I'm not sure about his worst memories, but I think his boggart would be Dumbledore telling him that he's a wizard.
Even the most spoiled of us has bad and/or sad memories. Maybe he would be forced to relive all the times he broke something or the rare occasions when his parents didn't give in to his every whim. It sounds silly to us, but that might scare a bloke like Dudley.