Well, when I think of Voldermort, I image someone who is too horrible to look upon (I'm referring back to book 4 when people i.e. the garden, die just by the site of him). Although, Voldermort use to be a handsome Tom Riddle, I do believe that Rowling makes it clear that the face of evil is eventually protrayed by the evil doer. Thus, I believe that all of Tom's beauty has been sold for power. In CoS, Tom could still be thought of as the handsome and clever boy named Tom. Keep in mind, however, that we were going back into the past- Tom's past. In GoF, Tom is Lord Voldermort, and to be Voldermort, is to be merceless, powerful, and to live with a divided soul. Rowling never portrays a beautiful Lord Voldermort to the reader, but gives us a fowl image.