Perhaps you have a different Didache? The one I am familiar with says nothing about infant baptism one way or another.
Chapter 7. Concerning Baptism. And concerning baptism, baptize this way: Having first said all these things, baptize into the name of the Father, and of the Son, and of the Holy Spirit, in living water. But if you have no living water, baptize into other water; and if you cannot do so in cold water, do so in warm. But if you have neither, pour out water three times upon the head into the name of Father and Son and Holy Spirit. But before the baptism let the baptizer fast, and the baptized, and whoever else can; but you shall order the baptized to fast one or two days before.
Now before you get all indignant and preachy define "fasting" and how it does and does not apply to infants. Most interpretations refer to abstinence from solid foods and none specifically address or prohibit mothers milk. (Besides, there is that whole household thing in Acts).
Nice try but no cigar...So you 'interpret' it to mean something other than what it says...Now you can justify yourself...
Problem with your own definition is a three or four year old would have a rough time on only mother's milk for a couple of days...