Is it the sun? museums? famous landmarks? the reviews online?
For me it depends on what time of year I go on holiday so if it's around the summer time I look for somewhere that's hot, has a nice beach. I know Greece is good for the beaches, famous landmarks and for the sun too which is why I decided to go there last year.
Anyway, what sort of things attract you when looking for a place to go to?