Trayvon was supposedly in a hurry, and kept questioning Rachel about whether the 'game' was on yet. Rachel said that Trayvon told her he was in his 'dads' back yard. He had to leave it to encounter Zimmerman.
P.S. I could be wrong about the babysitting part. No one knows if ‘Chad’ was even there. Even if he was, then Brandy’s sister (the one with a drug dealer arrest record) might have been doing the babysitting. Brandy may not have even been living in the townhouse at the time.