关于人工智能:使用PyOD进行异常值检测

6次阅读

共计 2998 个字符，预计需要花费 8 分钟才能阅读完成。

异样值检测各个领域的要害工作之一。PyOD 是 Python Outlier Detection 的缩写，能够简化多变量数据集中辨认异样值的过程。在本文中，咱们将介绍 PyOD 包，并通过理论给出具体的代码示例

PyOD 为异样值检测提供了宽泛的算法汇合，实用于有监督和无监督的场景。无论解决的是带标签的数据还是未带标签的数据，PyOD 都提供了一系列技术来满足特定需要。PyOD 的突出个性之一是其用户敌对的 API，使老手和有教训的从业者都能够轻松的拜访它。

咱们从一个简略的例子开始，利用 k 近邻 (kNN) 算法进行离群值检测。

首先从 PyOD 导入必要的模块

 from pyod.models.knn import KNN 
 from pyod.utils.data import generate_data
 from pyod.utils.data import evaluate_print

咱们生成具备预约义离群率的合成数据来模仿异样值。

 contamination = 0.1  # percentage of outliers
 n_train = 200  # number of training points
 n_test = 100  # number of testing points
 X_train, X_test, y_train, y_test = generate_data(n_train=n_train, n_test=n_test, contamination=contamination)

初始化 kNN 检测器，将其与训练数据拟合，并取得离群值预测。

 clf_name = 'KNN'
 clf = KNN()
 clf.fit(X_train)

应用 ROC 和 Precision @ Rank n 指标评估训练模型在训练和测试数据集上的性能。

 print("\nOn Training Data:")
 evaluate_print(clf_name, y_train, clf.decision_scores_)
 print("\nOn Test Data:")
 evaluate_print(clf_name, y_test, clf.decision_function(X_test))

最初能够应用内置的可视化性能可视化离群检测后果。

 from pyod.utils.data import visualize
 
 visualize(clf_name, X_train, y_train, X_test, y_test, clf.labels_,
           clf.predict(X_test), show_figure=True, save_figure=False)

这是一个简略的用法示例

异样值检测有时会受到模型不稳定性的影响，特地是在无监督的状况下。所以 PyOD 提供了模型组合技术来进步鲁棒性。

 import numpy as np
 from sklearn.model_selection import train_test_split
 from scipy.io import loadmat
 
 from pyod.models.knn import KNN
 from pyod.models.combination import aom, moa, average, maximization, median
 from pyod.utils.utility import standardizer
 from pyod.utils.data import generate_data
 from pyod.utils.data import evaluate_print
 
 X, y = generate_data(train_only=True)  # load data
 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4)
 
 # standardizing data for processing
 X_train_norm, X_test_norm = standardizer(X_train, X_test)
 
 n_clf = 20  # number of base detectors
 
 # Initialize 20 base detectors for combination
 k_list = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140,
             150, 160, 170, 180, 190, 200]
 
 train_scores = np.zeros([X_train.shape[0], n_clf])
 test_scores = np.zeros([X_test.shape[0], n_clf])
 
 print('Combining {n_clf} kNN detectors'.format(n_clf=n_clf))
 
 for i in range(n_clf):
     k = k_list[i]
 
     clf = KNN(n_neighbors=k, method='largest')
     clf.fit(X_train_norm)
 
     train_scores[:, i] = clf.decision_scores_
     test_scores[:, i] = clf.decision_function(X_test_norm)
 
 # Decision scores have to be normalized before combination
 train_scores_norm, test_scores_norm = standardizer(train_scores,
                                                     test_scores)
 # Combination by average
 y_by_average = average(test_scores_norm)
 evaluate_print('Combination by Average', y_test, y_by_average)
 
 # Combination by max
 y_by_maximization = maximization(test_scores_norm)
 evaluate_print('Combination by Maximization', y_test, y_by_maximization)
 
 # Combination by median
 y_by_median = median(test_scores_norm)
 evaluate_print('Combination by Median', y_test, y_by_median)
 
 # Combination by aom
 y_by_aom = aom(test_scores_norm, n_buckets=5)
 evaluate_print('Combination by AOM', y_test, y_by_aom)
 
 # Combination by moa
 y_by_moa = moa(test_scores_norm, n_buckets=5)
 evaluate_print('Combination by MOA', y_test, y_by_moa)

如果下面代码提醒谬误，须要装置 combo 包