It is very much uncharted territory.... legally, the DNC can still select a new candidate with a national meeting of the DNC called by its chairman. But ballots are already printed in all of the states, and some states have already started early voting.
They could go for Kaine or Sanders or someone not previously in the picture, such as Biden or Warren (even Kerry!), but they would have to decide whether to try to change names on ballots in all 50 states, which would be a logistical and legal nightmare (every state administers its own voting procedures and ballots).
The DNC could decide to leave Clinton/Kaine on the ballots, to avoid massive problems, and tell voters that DNC electors meeting in Dec. will be instructed to vote for “X” rather than Clinton... but it would be an enormous mess and would probably doom the Democrats to defeat, no matter who they might name.