[powerset]

One way Twitter could have covered up their bias is outlined below.

Look at the full article for live links and lots of comments.

P.S. Schneier is not a conservative, but he is committed to online privacy, computer security and unrestricted availability of encryption. And he is opposed to government attempts to subvert these things. (He also is a well-known cryptographer, whose algorithms have been in widespread use.)

- - - - - - - - - - - - - - - - - - - - - - - - - -

The following is excerpted from:

https://www.schneier.com/blog/archives/2022/05/manipulating-machine-learning-systems-through-the-order-of-the-training-data.html

Most deep neural networks are trained by stochastic gradient descent. Now “stochastic” is a fancy Greek word for “random”; it means that the training data are fed into the model in random order.

So what happens if the bad guys can cause the order to be not random? You guessed it—all bets are off. Suppose for example a company or a country wanted to have a credit-scoring system that’s secretly sexist, but still be able to pretend that its training was actually fair. Well, they could assemble a set of financial data that was representative of the whole population, but start the model’s training on ten rich men and ten poor women drawn from that set,­ then let initialisation bias do the rest of the work.

Does this generalise? Indeed it does. Previously, people had assumed that in order to poison a model or introduce backdoors, you needed to add adversarial samples to the training data. Our latest paper shows that’s not necessary at all. If an adversary can manipulate the order in which batches of training data are presented to the model, they can undermine both its integrity (by poisoning it) and its availability (by causing training to be less effective, or take longer). This is quite general across models that use stochastic gradient descent.

[glorgau]

If an adversary can manipulate the order in which batches of training data are presented to the model, they can undermine both its integrity (by poisoning it) and its availability (by causing training to be less effective, or take longer). This is quite general across models that use stochastic gradient descent.

Nice find!

I'm not a mathematician by any means. ;-)

But I've been led to believe that neural nets don't find optimal solutions, they find local minima within a "space". It sounds like that scheme sets the bounds of the "locality" and then trains the net to create rules within that space. Interesting stuff.