A checklist for developing fair models

With the preceding information, we can create a short checklist that can be used when creating fair models. Each issue comes with several sub-issues.

What is the goal of the model developers?

• Is fairness an explicit goal?

• Is the model evaluation metric chosen to reflect the fairness of the model?

• How do model developers get promoted and rewarded?

• How does the model influence business results?

• Would the model discriminate against the developer's demographic?

• How diverse is the development team?

• Who is responsible when things go wrong?

Is the data biased?

• How was the data collected?

• Are there statistical misrepresentations in the sample?

• Are sample sizes for minorities adequate?

• Are sensitive variables included?

• Can sensitive variables be inferred from the data?

• Are there interactions between features that might only affect subgroups?

Are errors biased?

• What are the error rates for different subgroups?

• What is the error rate of a simple, rule-based alternative?

• How do the...