Try to avoid the typical touristy areas.
If you like history, though, take a trip down to lower Manhattan, in the Wall Street area. Federal Hall, Fraunces Tavern and on the other side of Broadway is a very old church, where George Washington went to pray when he was first inaugurated.
Some other suggested Fraunces Tavern, too. Looks like we’ll be fairly close from our hotel.