对于模型可解释性的材料较少，这里对立做个整顿。
如果想系统性学习，举荐浏览 Interpretable machine learning 这本书。

What

可解释性是人能够了解零碎决策的一个度量 (Interpretability is the degree to which a human can understand the cause of a decision.)

Why

随着大量科学家在机器学习畛域的继续投入，现在各种模型形形色色，且毋庸置疑这些模型在生产环境都产生的肯定的成果。

但计算机系统做出的这些预测或决策的起因并无奈被人类感知，因此人类无奈直观的感知这些预测的成果或起因。

模型都是对事实世界的模仿，然而都不是 100% 的还原，因而没有一个繁多 metric(如 auc) 能形容模型在生成环境中的实在体现。

此外机器学习零碎并不知道本人在预测什么内容。

这个问题在某些畛域可能十分重要（例如临床医学，反欺诈）。

How(实践)

• 应用有自解释性的模型 (WB)
• 基于模型不可知论的黑盒探测 (BB)
• 深度学习 / 神经网络模型的可解释性

自解释性的模型

• 线性回归模型
• 逻辑回归
• 决策树

白盒解释，须要晓得模型具体实现，以及训练数据，甚至可能须要改模型代码。

黑盒探测

• Partial Dependence Plot (PDP)
• Global Surrogate
• Local Surrogate Model-Agnostic Method(LIME)
• Anchors(If-Then-That)
• Shapley Values
• SHAP

PDP

A partial dependence plot can show whether the relationship between the target and a feature is linear, monotonic or more complex.

Advantages: intuitive
Disadvantages:The assumption of independence is the biggest issue with PD plots.

Global Surrogate

A global surrogate model is an interpretable model that is trained to approximate the predictions of a black box model. 

Advantages: flexible.Any interpretable models can be used. 
Disadvantages:You have to be aware that you draw conclusions about the model and not about the data, since the surrogate model never sees the real outcome.

Local Surrogate Model-Agnostic Method(LIME)

Local surrogate models are interpretable models that are used to explain individual predictions of black box machine learning models.

Advantages: They make human-friendly explanations.LIME is one of the few methods that works for tabular data, text and images.
Disadvantages: The correct definition of the neighborhood differs./instability.

Anchor

A rule anchors a prediction if changes in other feature values do not affect the prediction. 
Anchors utilizes reinforcement learning techniques in combination with a graph search algorithm to reduce the number of model calls (and hence the required runtime) to a minimum while still being able to recover from local optima. 

Advantages: Like Lime. 
    Anchors are subsettable(shown as example).
    Works for non-linear or complex in an instance’s neighborhood(reinforcement learning).
    Can be parallelized.
    
Disadvantages: highly configurable.
    many scenarios require discretization.
    many calls to the ML model.

Shapley Values

SHAP

explore timeline

PDP(2001) –> LIME(2016) –> Anchors(2018)
Shapley Values(2014) –> SHAP(2016)

深度学习 / 神经网络模型

深度学习 / 神经网络模型的解释能够应用黑盒探测来实现，除此之外还有一些专门针对神经网络的可解释性科研摸索。

• Feature Visualization
• Network dissection
• Pixel Attribution (Saliency Maps)

When to use

• Your model makes significant impact.
• When the problem is not well studied, or explore in a very new area.

In Action(实战)

python: sklearn、keras、alibi

R: iml

seldon 是一个模型生命周期治理的零碎，有点相似于咱们的 PAS 加一部分 DAG 的性能，他们基于上述 LIME 和 Anchor 等实现了一套模型解释和异样检测的框架并开源了，咱们能够借鉴、摸索下。

模型 outlier, adversarial and drift

https://github.com/SeldonIO/a…

模型 inspection and interpretation

https://github.com/SeldonIO/a…

具体参考 explain.ipynb