Or after dark depending where you are.
Speaking of Harlem, I'm in the 125th Street area (near the Metro North tracks) quite often during the day. Never have any issues. One night, I decided to walk to that station after a Yankees game instead of standing in line to get onto the crowded subway cars. I survived but I'm NEVER doing that again. Totally different neighborhood after working hours.
My dad was NYPD for 23 years and served about 10 years in the 23rd precinct at 123rd St and St Nick when he was a detective, other side of Harlem from you.
That was when Harlem was really a craphole (through the 80’s). Whenever i was going into the city I’d always park the car at the precinct, walk up a few blocks and take the A or D train downtown. Never had a problem.
It all about attitude- if you have the “don’t F with me look” they will leave you alone