Anyone who visits NYC simply has to visit the Flushing meadows/Corona Park in Queens. The park is full of incredibly ugly, seemingly abandoned concrete buildings from the 1939/40 and 1964/65 World's fairs. It is a stunning place that many tourists (and New Yorkers) seem to be unaware of.
My personal favorite is the Queens Museum of Arts, situated right in the middle of the park, just next to the big world globe fountain. My friend Anders recently blogged about the museum's NYC panorama.