That is not silly in the least. There is absolutely no way I would consider living in such a house. Even if it wasn’t an omen, whenever anything went wrong, you’d always wonder if it was because of “the number”.
I also had visions of neighborhood goth kids doing animal sacrifices on Halloween, or something along those lines.