HN Theater @HNTheaterMonth

In many unbalanced situations you simply can't have an unbiased decision that's fair across all measures, no matter if the decision is done by a model, a human or a deity; you have to trade off between different types of unfairness. Equality of opportunity for individuals will result in unequal results for groups; equality of outcomes requires unequal opportunities if the historical circumstances have resulted in socioeconomic unequalities.

In your financial evaluations example, many biases and disadvantages would remain even if you solely reduce the decision to relevant financial facts for a specific individual, because a poorer individual with lower socioeconomic status and less opportunities actually has a higher risk of non-payment, and a disadvantaged group will have disproportionally more such individuals. Should we accept that? Should we require the other groups to subsidize their non-payment? Both options are unfair in some aspect and fair in another, you can't have your cake and eat it too, and it's not the fault of the model you use - the only difference with a human is that they can better hide the factors they use, lie about the influences (perhaps also lie to themselves) and rationalize/invent factors to justify their decision.

For this topic, perhaps this talk "AI Ethics, Impossibility Theorems and Tradeoffs" https://www.youtube.com/watch?v=Zn7oWIhFffs or its slides https://www.chrisstucchio.com/pubs/slides/crunchconf_2018/sl... might be interesting for you, it has some flaws but is a decent exploration of the problem space.

> I think it's also important to remember a lot of human models were abandoned because of the inherent bias

The industry needs to stop misusing the term bias this way. Virtually every attempt to find this supposed human bias has failed. Latest public example was Amazon and hiring[1]

Bias is the tendency to consistently mis-classify towards a certain class or tendency to consistently to over or under-estimate.

Somehow the term has been hijacked to mean 'discriminate on factors that are politically incorrect'. You can have a super racist model that's bias free, and most models blinded to protected factors are in fact statistically biased.

It's not constructive to conflate actual bias with political incorrectness.

Operational decision making, whether AI or human or statistical, faces an inherent trilemma: it's impossible to simultaneously treat everyone the same way, to have a useful model, and to have uniformly distributed 'bias'-free outcomes. At best a model can strive to achieve two of these factors.

See: https://www.youtube.com/watch?v=Zn7oWIhFffs

[1] https://www.reuters.com/article/us-amazon-com-jobs-automatio...

⬐ throwawayjava

Yes, mathematicians mean something different and specific when using the word bias. The average non-expert is not misusing the word. They are using the word to express a different -- and far more popular -- meaning.

Neither is wrong, but insisting that a naming clash carries any substantive significance on an underlying issue is just silly. Similarly, insisting that nonmathematicians should stop using a certain word unless they use it how mathematicians use it is a tad ridiculous.

Of anything, it's more reasonable for mathematicians to change their language. After all, their intended meaning is far less commonly understood.

⬐ TomMckenny

If an algorithm is clearly sorting on irrelevant criterion, especially a black box algorithm, we normally assume it's a bug. It's not reasonable to reverse that, assume the code is incapable of being mistaken and say that obviously irrelevant criterion are somehow correct in an unknown way.

Amazon's problem is a bug, they even describe it's nature. And given how flawed their recommendation algorithms are, it's especially unreasonable to assume this one is infallible.

So that linked Reuters does not show a failure to find bias, if anything it shows a design error.

⬐ missosoup

You say "obviously irrelevant criterion"

Data says criterion is an eigenvalue and no matter how hard amazon tried to blind the solution to that eigenvalue, the ML system kept finding ways to infer it because it was that strongly correlated with the fitness function.

This is the difference between political newspeak '''bias''' and actual bias. Amazon scrapped the model despite it performing just fine and being bias-free, because it kept finding ways to discriminate on a protected attribute which is a PR nightmare in the age of political outrage cancel culture. It's fine to explicitly decide that some attributes should not be discriminated upon, but this comes with a cost either in terms of model utility or in terms of discrimination against other demographics. There's no way around this. In designing operational decision making systems, one must explicitly choose a victim demographic or not to implement the system at all. There's no everyone-wins scenario.

The harm of the newspeak version of '''bias''' is that it misleads people into thinking that making system inputs uniform somehow makes it bias-free when the opposite is typically true. Worse, it creates the impression that some kind of magical bias-free system can exist where everyone is treated fairly, even though we've formally demonstrated that to be false.

No amount of white-boxing or model transparency will get around this trilemma. The sooner the industry comes to grips with it and learns to explicitly wield it when required, the better.

⬐ XuMiao

Supervised learning algorithms assume that the input data are iid of the future. This is not valid in most of the real applications. The observation that we see men more than women in programming does not necessarily generalize to the future. That's why online learning provides an exploitation vs exploration mechanism to minimize the bias in the hindsight. In many applications, people just forgot about this simple strategy and blame the bias caused by supervised learning to the black box model.

Of course, black box AI itself is not the right solution. As more and more cross domain multitask settings emerge, open box AI will gradually take off. It is about compositional capability like functor and monad in functional language. Explanable or not is just a communication problem which is parallel to the ultimate intelligence problem. It is very possible that human intelligence is bounded.

⬐ TomMckenny

>No amount of white-boxing or model transparency will get around this trilemma. The sooner the industry comes to grips with it and learns to explicitly wield it when required, the better.

Agreed. The optima of multiple criterion will essentially never intersect.

But for Amazon, there is no evidence the tool was accurately selecting the best candidates. They themselves never said it was. After all, altering word choices in a trivial way dramatically affects ranking. On the points you mention, why should we assume their data was relevant or their fitness function even doing what they thought? If they were naive enough, they could just be building something that predicts what the staff of an e-commerce monopoly in a parallel universe will look like.

The most likely story is that they failed at what they were doing. Part of that failure happened to be controversial and so got unwanted attention. I would guess there were quite a few incredible correlations the tool "discovered" that did not get to press.

At any rate, their recommendation engine is more important and has been worked on longer yet it is conspicuously flawed. When their recommendation tool inspires awe then maybe we could take their recruiting engine seriously enough to imagine it has found deep socio-psychological-genetic secrets.

⬐ cmendel

Hold up. Bias is being used in two different manners because it has two differed meanings. When you are using bias in industry you are talking about a minor mathematical factor added to a learning rate. When we talk about human bias, we aren't. The term was never hijacked, it just has multiple meanings.

⬐ derangedHorse

I’m not sure if he was using any of those known definitions of bias though. I’m not sure if I would define bias as the consistent act of mis-classification

This 'algorithm has bias' kind of headline needs to die.

Every system has '''bias''' in the colloquial sense so long as the inputs to that system are not uniformly distributed. The only thing the designer of the system can do is make a conscious choice about which way to '''bias''' the system and be able to justify that choice.

In practical terms it means that systems that deal with diverse populations are always going to be 'unfair' towards some group of people. The only control we have is who that group of people will be. This is inherent and unavoidable. We need to accept this fact and keep it in consideration when designing such systems (or choosing not to design them in the first place).

{procedural fairness, group fairness, utility}. Pick two.

https://www.youtube.com/watch?v=Zn7oWIhFffs

⬐ jakelazaroff

You're right, in that there is a tension between systems behaving equally and equitably, and that either one can be seen as "bias". (Although there can of course be systems that by design behave neither equally nor equitably, whether intentional or not.)

I think the "algorithm has bias" type of headlines proliferate because there's a common misconception that algorithms are somehow "objective". See for example Ryan Saavedra implying algorithms can't be racist because they're "driven by math": https://twitter.com/realsaavedra/status/1087627739861897216?...

⬐ solotronics

It comes down to the world frame some folks have. To them the world should have total fairness between all different people irregardless of predisposition, skill, situational luck, or effort put in. If it is not fair by their system they can fix it by pressing down on the scale. If any system or person has a fact that disrupts their worldview they are labeled things like "biased/racist/patriarchal/etc". Their desire for absolute equity requires rejecting facts that show people are different.

To people with this worldview an algorithm will always be "biased" because it simply reflects statistics instead of their view of they see as "right" which is with their benevolent finger on the scale.

I edited this slightly to try and remove any us vs them rhetoric. I am not passing any judgement just stating what I have observed.

⬐ cycrutchfield

That’s gonna be a yikes from me, dog.

Edited this (since you did as well) to say that simply pointing to statistics and saying that “facts don’t care about your feelings” is sophomoric and probably intentionally disingenuous. For instance, statistically in the US black people are more likely to commit crimes than white people. So should we implement systems that target suspects based on the color of their skin? Obviously not. There is a very important confounding factor, socioeconomic status, which happens to be very correlated with skin color in the US due to a long history of racial discrimination. So saying that “these systems should not be racially biased” is an important criteria.

⬐ 8f2ab37a-ed6c

We already discriminate as of today based on gender statistics in areas such as car insurance, where men pay more on average, because they're statistically more likely to cause accidents and drive recklessly. Why can't statistics be extended to other areas?

⬐ astura

Well, not everywhere allows discrimination based on sex, some US states don't, for example (California just joined that club last year). Plus the EU doesn't either.

People generally find a discrimination "fair" if it's based on something that you can change whereas an "unfair" discrimination is typically something that you are.

⬐ cycrutchfield

Do you really think that the melanin content of your skin has any causal relation to how likely you are to commit a crime, or any other outcome?

⬐ pnako

Literally no one is treating race as if it was about "melanin content of [one's] skin", so that's a strawman if I ever saw one.

It's a very visible trait, strongly associated with ancestry, and thus associated with many other, more important, traits.

⬐ cycrutchfield

Well, police don’t perform actions based on the identity of your parents or grandparents. Skin color is evidently a pretty important feature in terms of determining the actions that police may take.

But, let’s follow your logic for a second. It seems you are suggesting that it is acceptable for insurance companies to ask you for the identity of your parents or grandparents to decide what rates to charge you. Does that sound like an outcome that you think would be acceptable?

⬐ pnako

Not necessarily (I haven't thought about the issue too much).

My point is more that, if insurance companies discriminate using any non-genetic, non-biological criterion, the outcome would still not be a distribution strictly representative of the population. Because people's choices are influenced by their background. For example, it's quite probable that the distribution of brand and type of car is not uniform across racial lines (or between men and women). Would you consider the outcome resulting from this to be biased?

In fact I'm pretty sure it's how European insurance companies still manage to charge more for men, despite being forbidden to explicitly ask that information. That information still affects the pricing model through things like occupation, etc.

⬐ plurple

Do you really think the melanin content of your skin has any causal relation to how well you can play basketball? Or maybe melanin content isn’t the only difference.

⬐ chronic829

Also, men dominate the trucking and construction industry. Why are we not pushing for gender equality there?

Hint: the answer is money.

⬐ mantap

In the EU exactly that kind of gender discrimination is illegal. Insurers are not allowed to charge men more than women, just for being men.

The question is, do you want to live in a society where prices are set based on one's genes. I sure don't.

⬐ missosoup

In the age of ML this is almost impossible to police and very difficult to implement even if you want to.

You blind your model to gender but it learns to infer it from names. You blind it to names, it learns to infer it from other inputs. By the time you've blinded your model to anything that gives away gender, it no longer has utility. If you instead intentionally alter the model to still have utility but impact both genders equally, it's no longer procedurally fair and does in fact discriminate.

This is the trilemma mentioned above. You must choose which of those factors you care about and which one you'll sacrifice. If you're unable to make that choice and morally stand behind it, then probably you shouldn't be building said model/system in the first place.

This is one of the major topics of the ML ethics conversation.

⬐ ahbyb

>The question is, do you want to live in a society where prices are set based on one's genes. I sure don't.

Those genes do clearly make us behave in different ways, so why not?

I'm not a feminist, but it would be fairer for women to pay less for the same insurance since they are less likely to get in trouble with their cars. Why should they pay more?

⬐ solotronics

I think the argument reduces to since we can get hyper accurate data on an individual level how can we treat people fairly? If an insurance algorithm charges one person who is higher risk 100x more than someone else is it really a fair form of insurance?

⬐ briandear

So what the EU does is just charge more for everyone which means women pay higher rates for car insurance despite having lower accident rates. So in the goal of equality, women are being forced to pay more than they should based on their level of risk. That’s exactly the opposite of discrimination — women pay more despite lower risk, which makes that EU law, discriminatory.

⬐ 8f2ab37a-ed6c

I think that's an interesting point, and I can't say I have a well thought out position on it, but I'm happy to have stumbled upon this conversation.

One one hand, I feel ok paying extra for my health care to cover others who might have been left out previously because of individual pre-existing conditions. Us pooling all together to take care of every citizen makes sense.

However in a purely hypothetical scenario where say men are 1000x worse at something than women, let's say driving, say car insurance for women is $100 and it's $100k for men, should we really charge women 50k just to make it fair based on the "we must ignore gender" principle? I realize in the real world there might be no such stark contrast between genders or ethnicities, in which case giving everybody the same higher rate would probably be more sensible.

⬐ megablast

Do they commit more crimes, or do they get caught committing more crimes? And is that because they are targeted more for searches?

⬐ pnako

The problem is that people still tend to complain about the outcome if it's not an egalitarian distribution, regardless of the accuracy of the inputs.

Let's say I want to identify violent criminals based on things like type of jewelry or clothes they wear, or tattoos (symbol, location). Reasonable model: it tries to identify gang membership by visible gang membership symbols (i.e. it exploits something gangsters themselves signal). It's not perfect (false positives: hip hop artists who are not necessarily actual gangsters) but it's probably a decent model. I would not be surprised if the outcome did not match the racial distribution of the US population, precisely for the reason you mentioned. Would that model be racially biased?

⬐ missosoup

> So should we implement systems that target suspects based on the color of their skin? Obviously not.

Why not? Why is that obvious?

Utility: if the system isn't effective, there will be additional crime and additional victims of crime ('''bias''' against those victims)

Group fairness: the system should target the population in a uniform manner across all skin colours (actual statistical bias to favour skin colours over-represented in crime)

Procedural fairness: the system should follow the same process for all skin colours ('''bias''' against skin colours over-represented in crime)

In a world where skin colour isn't uniformly distributed across crime rates, you can't have all of those. You must explicitly sacrifice one of those factors to have a chance of satisfying the other two. Ignoring this trilemma doesn't make it go away. No matter what you choose, including inaction, there will be a harmed party.

This was an actual scenario faced by the COMPAS parole sentencing system. There are no easy answers.

P.S.

You are surrounded by systems that target you based on the colour of your skin every day, and the designers of most of them aren't even aware of it. Scandals like 'Woman slams ‘racist’ Boots for putting security tags on hair products for black customers but not those aimed at white people'[1] are driven by very simple models and very limited data like [sku, shrinkage rate] that you would think can't possibly give away skin colour. And yet they do. And if you want to make those models 'not racist' then you do actually have to statistically bias them on skin colour. Again, no easy answers.

[1] https://www.thesun.co.uk/news/9941383/boots-racism-security-...

See also Stucchio's impossibility theorem: it is impossible for a given algorithm to be procedurally fair, representationally fair, and utilitarian at the same time:

Video: https://www.youtube.com/watch?v=Zn7oWIhFffs

Slides: https://www.chrisstucchio.com/pubs/slides/crunchconf_2018/sl...

⬐ missosoup

Good presentation

⬐ AstralStorm

The general point is that you have to robustly compromise and satisfice all the goals. People tend to be rather good at it when taken as a group. (Any particular person may be bad at a given subset of all problems.)

It is a kind of optimality condition on all three goals.

The robustness additionally means that should conditions change, the algorithm usually will become better not worse and should a degradation still happen, it will be graceful and not catastrophical.

It's a hard and open problem in ML and especially ANN, design of robust solutions in the space. Most have really bad problems with it even when debiased.

⬐ sgt101

Hello - do you have a good reference to this area? I bang on about similar ideas whenever allowed, but haven't found good support in the literature.

Not that I mind that too much!

That's a quite interesting slide deck! For others interested, I've not watched it yet, but this appears to be the talk https://www.youtube.com/watch?v=Zn7oWIhFffs

I was interested in the slide deck and wanted to hear the author expand on some of his thoughts, found a video of his presentation if anyone else is interested.

https://www.youtube.com/watch?v=Zn7oWIhFffs

⬐ Balgair

Wow, this is really good. I hadn't thought of some of these things and he exposes some points that should have been obvious to me. For example: You aren't guaranteed to be able to maximize two functions at the same time.