It's not infinite. There is a theoretical end to the known universe.
It is expanding like a balloon expands, everything goes farther out, and individual galaxies move farther away from each other as the universe expands. The essential qualities of space include: gravity, the strong force, the weak force and electromagnatism. Without these forces, time and space cannot exist.
So what's on the other side? You know, the "non-universe" side?