There's a pretty simple explanation for most of the mess described in the article. Back in the 90's the venture capitalists developed their own version of "crowd funding", namely that numerous deep pockets investors would back a startup and that startup would become the winner in some new area when it went public. The investors would get all their investments back (early money time 100, later money times 10) and they would start over and do it again. That is of course oversimplified and there were competitions regardless of the cooperation, but the big money in the markets could support several competitors.
Come 9/11 (and even before 9/11), a ton of taxpayer money got sunk into "counterterrorism" and associated data gathering. The data was gathered in huge volumes with needs for new sources and new ways of handling it. The key is that big taxpayer (now Fed printed) bucks were available for those companies. They would have open discussions like the ones pointed out in the article to better align the tech sector ideas with government needs all in an effort to tap into the gravy train.
It is the new way things are done. There are analytical tools at the forefront which have a giant thirst for new data sources and comb "open source" like the web including forums like this one for tidbits to add to the database. The current market dynamics are a little different than described in the article. There are huge commercial markets for these analytical tools and those markets are the ultimate driver of venture funding although the government largess is still a big part of it.
Samaritan is not too far off in our futures.....