Question

In your opinion, what is explainability in the context of machine learning?

Comprehensibility above all

The most important property is how comprehensible the explanations are

• Can the audience actually understand the explanation?

The social context of explanations is especially important here

• You would all probably be comfortable with interpretation of linear or logistic regression models

How to achieve interpretability

Two methods:

• Use models that are intrinsically interpretable
• Use post-hoc methods for explainability

Weights

In linear models, we can interpret the weights

• All else equal

Lots of collective experience

• Both among social scientists, but also other fields
• Going to focus on some visuals

Weight plots

_{^{Source: Molnar, 2022}}

Weight feature plot

_{^{Source: Molnar, 2022}}

Individual explanations

_{^{Source: Molnar, 2022}}

Selective explanations

LASSO models create sparse models due to the \(L_2\) norm

Can be done to either

• Increase performance
• Reduce complexity

\(\alpha\) can be tuned such that a set amount of weights are non-zero

• Utilize same methods as before

Drawbacks

• No interactions unless you include them

• Linear models do not perform well with non-linear data

• All else equal interpretation

  • If features covary, this can create weird interpretations

• If features covary, model may arbitrarily select one

Decision trees as flowcharts

Decision trees can be plotted

• Easy to understand
• A collection of if-else statements and splits
• Easy to imagine counterfactuals
• They always go left in sklearn

Maximum amount of leafs is maximum depth to the power of two

A shallow flowchart

A deep flowchart

Question

Do you consider linear regression or decision trees to be most explainable?

• Does it depend on the amount of features or complexity of the model?

The social aspect of explainability

I wager that most people here would default to linear regression

You would have no problem interpreting coefficients

• But what about the receiver of the explanation?

I posit that decision trees are more easily understood for a layman

• Especially for a single given observation, e.g. the layman’s own observation
• Cutoffs readily available for counterfactuals

Mean decrease in impurity

Another method for trees, which calculates global feature importance

For all features:

• Go through all the splits for which the feature was used
• Measure how much it has reduced the MSE or Gini index compared to the parent node

Scale sum of importances to 100

• Interpretation: Importance as a fraction of total decrease in impurity

Titanic with two random features

_{^{Source: sklearn}}

Question

This method is generally biased and favors high cardinality features.

Why?

Drawbacks

The method favors high cardinality features because these features are more often split on

• E.g. once a decision tree has split on a binary feature, it won’t split on it again
• See this example from sklearn

Can only say something about the model in relation to the training data

As a result, not often used

• Permutation feature importance used instead

Shedding light on the black boxes

Some models are so complex that we cannot understand them or their components

Here we can use model agnostic explanation methods

• Translucency is gone

Permutation feature importance

Permute a feature

• This breaks the dependence between \(X\) and \(y\)
• Also breaks any interactions

How much does this loss of information change the score (e.g. MSE)

• Nice because this is the object of main interest!

Can be computed for both train and test

What data to use

Most commonly done on test data

• Realistic error estimates
• The object of interest

However, training data will show what features the model uses

• Can be used

Permutation plots

_{^{Source: sklearn}}

Average dependence

What happens when we set a feature \(X\) with \(x\)?

• In essence, hold all values fixed except one (or two)

Plot how the average prediction changes as a function of \(X\)

• This is called a partial dependence plot
• Due to the averaging, this is a global method

Partial dependence plot

_{^{Source: Molnar, 2022}}

Uncovering the heterogeneity

How to avoid hiding heterogeneous effects?

Plot a partial dependence for all observations!

• This is called an individual conditional expectation (ICE) plot

As we no longer average over all observations, it is local

ICE plots

_{^{Source: Molnar, 2022}}

Centered ICE plots

_{^{Source: Molnar, 2022}}

Pros and cons

Pros

• Easy to interpret
• Can display heterogeneity

Cons

• Now limited to just one feature
• Still all else equal dependencies
• Can be hard to visualize for large datasets

Roadmap

Shapley Additive Explanations allocate prediction outputs as if a game (Shapley values) using a local interpretable model (LIME)

• As such, we will (quickly) cover these

There’s a lot of math and pseudo-code in the papers and book regarding SHAP and it subcomponents

• This I have omitted

LIME

Local interpretable model-agnostic explanations introduced by Ribeiro et al. (2016)

• Only focus on local fidelity

Create a low-complexity (intrinsically interpretable) model for each observation

• E.g. LASSO

Available stand-alone in the package LIME (not covered in exercises)

LIME in a nutshell

Select observation to explain

• Perturb the observation and compute black box predictions
• Weight the new samples according to proximity (using some kernel)
• Train a weighted, interpretable model on the dataset with the variations
• Explain the prediction by interpreting the local model

Devil in the details

Big question: What kernel bandwidth to use?

• LIME implementation in Python has a fixed kernel with a fixed bandwidth

LIME visually

LIME fitting procedure

_{^{Source: Molnar, 2022}}

Different kernels

LIME with different kernels and true dependence

_{^{Source: Molnar, 2022}}

Shapley values

Allocate payout using Shapley values from cooperative game theory

• Features are players, prediction is payout

Theoretically grounded

Allocation of payouts

Groups of players (coalitions) can either participate in the game or not

• If a player enters the game with a coalition, this can change the payout

This change in payoff is distributed to the players in the coalition

Game is played for each instance

• A local method

An example

Marginal contributions in feature sets

_{^{Source: aidancooper.co.uk}}

One problem

Need to calculate models with the power set of all features

• Retraining a huge amount of models

This quickly becomes computationally expensive

SHAP

To reduce compute, we return to SHAP (Lundberg & Lee, 2017)

• SHAP and Shapley values are not the same

Essentially: Change kernel in LIME with a weighing scheme based on Shapley values!

• Based not on distance, but how many features are ‘present’

Why?

• Retains the nice guarantees from Shapley values!

Non-participation

To ‘simulate not participating’, we permute the features

• We break the dependency, just like in permutation feature importance

By doing this, we retain an (uninformative) input (which the model requires) and do not need to retrain!

• Much cheaper to compute

A simpler example

_{^{Source: nicolas-hug.com}}

Enough theory

Lets talk plots

We remember that SHAP values are all individual

• Thus based on a collection of observations, i.e. a single, train set, test set or a group of interest

Through clever plots and summary statistics, we can still obtain global insights

All easy to compute and plot through SHAP

Force plot

_{^{Source: Molnar, 2022}}

Beeswarm plots

_{^{Source: Molnar, 2022}}

Feature importance plots

_{^{Source: Molnar, 2022}}

Dependence plots

_{^{Source: Molnar, 2022}}

Dependence plots with interactions

_{^{Source: Molnar, 2022}}

Further information

The rest of the book by Molnar has more methods

• e.g. counterfactual explanations

Text and image also have methods developed for them

• Not covered due to a mainly tabular focus in this course