It seems as if you aren't on before the top of the hour, you have to wait 'til a few minutes after. Guess its folks starting work or getting off and crowding the net. Happens a lot when I start work in the morning.
I have noticed that at times the whole net seems to crawl and it would indicate that a high flow of people, probably due to what you say. Shift changes, etc. Generally happens between 5:30 and 6:30 EST.