Efficiency of different measures for defining the applicability domain of classification models

Klingspohn, Waldemar; Mathea, Miriam; ter Laak, Antonius; Heinrich, Nikolaus; Baumann, Knut

doi:10.1186/s13321-017-0230-2

Journal of Cheminformatics

Table 2 AUC ROC for all classification techniques and AD measures

From: Efficiency of different measures for defining the applicability domain of classification models

	MUSK2	QSAR	BBB	PGP	FXa	Liver	hERG	Cancer	Ames	CYP1A2	Mean rank	#Signif.^a
RF
\(\bar{p}_{RFC}\)	0.99 ^b	0.93	0.86	0.85	0.98	0.59	0.86	0.64	0.87	0.90	1.95	10
\(PROBSTD_{RFR}\)	0.99	0.93	0.85	0.85	0.98	0.59	0.86	0.64	0.86	0.90	2.25	10
\(\bar{p}_{RFR}\)	0.99	0.93	0.85	0.85	0.97	0.59	0.86	0.64	0.86	0.90	2.60	10
\(STD_{RFR}\)	0.99	0.93	0.84	0.84	0.98	0.58	0.85	0.63	0.85	0.90	3.45	8
\(\cos_{\alpha }\)	0.95	0.87	0.82	0.82	0.97	0.56	0.78	0.61	0.81	0.84	6.10	5
\(\gamma_{Euc}\)	0.95	0.86	0.82	0.81	0.97	0.55	0.79	0.61	0.79	0.85	6.50	4
\(\kappa_{Euc}\)	0.94	0.86	0.81	0.80	0.97	0.54	0.80	0.61	0.79	0.85	7.10	4
\(\delta_{Euc}\)	0.94	0.86	0.84	0.79	0.96	0.58	0.78	0.59	0.80	0.82	7.45	1
\(\delta_{\text{Tan}}\)	0.92	0.85	0.79	0.80	0.95	0.55	0.77	0.60	0.78	0.83	9.10	0
\(\gamma_{\text{Tan}}\)	0.91	0.85	0.78	0.80	0.94	0.56	0.78	0.58	0.78	0.81	9.70	0
\(\kappa_{\text{Tan}}\)	0.92	0.85	0.78	0.79	0.94	0.57	0.76	0.59	0.78	0.81	9.80	0
Range	0.08	0.08	0.08	0.06	0.04	0.05	0.10	0.06	0.09	0.09
NN
\(\bar{p}_{NNR}\)	1.00	0.92	0.83	0.84	0.98	0.57	0.81	0.62	0.84	0.89	2.00	10
\(\bar{p}_{NNC}\)	1.00	0.92	0.83	0.84	0.98	0.56	0.82	0.61	0.84	0.89	2.25	9
\(PROBSTD_{NNR}\)	1.00	0.92	0.82	0.83	0.98	0.57	0.81	0.62	0.84	0.89	2.30	10
\(STD_{NNR}\)	1.00	0.92	0.79	0.82	0.98	0.56	0.80	0.61	0.83	0.88	3.50	9
\(\gamma_{Euc}\)	0.99	0.86	0.76	0.82	0.96	0.52	0.77	0.59	0.77	0.85	6.65	4
\(\kappa_{Euc}\)	0.99	0.86	0.76	0.81	0.96	0.52	0.78	0.59	0.77	0.85	6.75	4
\(\cos_{\alpha }\)	0.99	0.88	0.76	0.76	0.96	0.53	0.73	0.59	0.79	0.83	7.15	4
\(\delta_{\text{Tan}}\)	0.97	0.85	0.75	0.81	0.95	0.53	0.76	0.58	0.75	0.82	8.70	2
\(\delta_{Euc}\)	0.98	0.86	0.78	0.73	0.95	0.54	0.74	0.58	0.77	0.81	8.05	0
\(\kappa_{\text{Tan}}\)	0.96	0.86	0.76	0.76	0.93	0.55	0.74	0.57	0.75	0.80	9.10	0
\(\gamma_{\text{Tan}}\)	0.96	0.85	0.75	0.77	0.93	0.55	0.74	0.57	0.75	0.80	9.55	0
Range	0.04	0.07	0.08	0.11	0.05	0.05	0.09	0.05	0.09	0.09
SVM
\(\hat{p}_{SVR}\)	1.00	0.90	0.86	0.84	0.97	0.57	0.83	0.61	0.84	0.87	1.50	9
\(\hat{p}_{SVC}\)	1.00	0.91	0.87	0.82	0.97	0.56	0.83	0.60	0.84	0.88	1.60	9
\(\cos_{\alpha }\)	0.99	0.87	0.80	0.79	0.95	0.56	0.77	0.59	0.79	0.82	4.30	6
\(\gamma_{Euc}\)	0.99	0.86	0.82	0.75	0.95	0.54	0.79	0.59	0.78	0.83	4.35	5
\(\kappa_{Euc}\)	0.99	0.86	0.82	0.74	0.95	0.53	0.79	0.59	0.78	0.83	4.75	5
\(\delta_{Euc}\)	0.98	0.85	0.83	0.74	0.93	0.56	0.75	0.58	0.77	0.80	5.80	0
\(\delta_{\text{Tan}}\)	0.98	0.85	0.81	0.74	0.92	0.53	0.76	0.58	0.76	0.80	6.80	0
\(\kappa_{\text{Tan}}\)	0.97	0.84	0.82	0.72	0.91	0.55	0.75	0.57	0.75	0.78	7.70	0
\(\gamma_{\text{Tan}}\)	0.97	0.84	0.81	0.72	0.90	0.54	0.75	0.57	0.75	0.78	8.20	0
Range	0.03	0.07	0.07	0.12	0.07	0.04	0.08	0.04	0.09	0.10
MB
\(\bar{f}_{MB}\)	0.93	0.90	0.80	0.84	0.97	0.59	0.83	0.58	0.76	0.85	1.20	10
\(\hat{v}_{MB}\)	0.93	0.89	0.79	0.83	0.97	0.58	0.83	0.58	0.75	0.82	1.80	9
\(\cos_{\alpha }\)	0.89	0.86	0.75	0.80	0.95	0.57	0.74	0.56	0.74	0.80	4.20	6
\(\kappa_{Euc}\)	0.90	0.84	0.73	0.73	0.95	0.55	0.77	0.56	0.72	0.80	4.85	4
\(\gamma_{Euc}\)	0.89	0.84	0.73	0.74	0.95	0.55	0.76	0.56	0.72	0.80	5.10	4
\(\delta_{Euc}\)	0.87	0.83	0.75	0.74	0.93	0.57	0.77	0.55	0.70	0.75	5.30	1
\(\delta_{\text{Tan}}\)	0.86	0.82	0.69	0.73	0.92	0.56	0.76	0.55	0.69	0.75	6.90	0
\(\gamma_{\text{Tan}}\)	0.81	0.79	0.69	0.72	0.89	0.56	0.77	0.54	0.66	0.72	7.80	0
\(\kappa_{\text{Tan}}\)	0.81	0.79	0.69	0.72	0.89	0.57	0.76	0.54	0.66	0.72	7.85	0
Range	0.12	0.11	0.11	0.12	0.08	0.04	0.09	0.04	0.10	0.13
k-NN
\(\hat{p}_{kNN}\)	0.97	0.90	0.83	0.80	0.97	0.57	0.80	0.61	0.82	0.87	1.10	10
\(\cos_{\alpha }\)	0.94	0.87	0.76	0.79	0.97	0.52	0.78	0.59	0.78	0.83	2.95	7
\(\gamma_{Euc}\)	0.94	0.85	0.77	0.71	0.97	0.51	0.77	0.60	0.77	0.83	3.40	8
\(\kappa_{Euc}\)	0.93	0.85	0.76	0.69	0.96	0.51	0.78	0.59	0.77	0.83	4.05	6
\(\delta_{Euc}\)	0.91	0.82	0.76	0.73	0.94	0.55	0.74	0.58	0.75	0.79	4.70	0
\(\delta_{\text{Tan}}\)	0.90	0.82	0.74	0.68	0.93	0.53	0.72	0.58	0.74	0.79	5.75	0
\(\kappa_{\text{Tan}}\)	0.88	0.80	0.73	0.66	0.91	0.56	0.70	0.57	0.72	0.77	6.90	1
\(\gamma_{\text{Tan}}\)	0.88	0.79	0.73	0.66	0.91	0.55	0.71	0.57	0.72	0.76	7.15	0
Range	0.09	0.11	0.10	0.14	0.06	0.06	0.10	0.04	0.10	0.11
LDA
\(\hat{p}_{LDA}\)	0.93	0.90	0.77	0.85	0.97	0.54	0.84	0.58	0.76	0.85	1.10	9
\(\cos_{\alpha }\)	0.86	0.87	0.75	0.83	0.97	0.52	0.76	0.57	0.71	0.81	3.55	6
\(\gamma_{Euc}\)	0.86	0.86	0.73	0.79	0.96	0.52	0.79	0.57	0.69	0.82	3.60	6
\(\kappa_{Euc}\)	0.87	0.85	0.72	0.78	0.96	0.52	0.79	0.56	0.69	0.82	3.90	4
\(\delta_{Euc}\)	0.84	0.84	0.75	0.78	0.94	0.54	0.77	0.55	0.69	0.78	4.55	0
\(\delta_{\text{Tan}}\)	0.86	0.83	0.69	0.77	0.93	0.53	0.78	0.55	0.67	0.78	5.55	1
\(\kappa_{\text{Tan}}\)	0.82	0.81	0.70	0.76	0.92	0.53	0.77	0.54	0.67	0.75	6.75	0
\(\gamma_{\text{Tan}}\)	0.82	0.81	0.69	0.76	0.92	0.53	0.77	0.54	0.67	0.74	7.00	0
Range	0.11	0.09	0.08	0.09	0.05	0.02	0.08	0.04	0.09	0.11

^aNumber of data sets where the AD measure performs significantly better than chance based on the 95th percentile (α = 0.05) of the permutation test (see Additional file 1 for a description of the permutation test and Additional file 2 for code of the permutation test)
^bUnderlined values indicate that the AD measure performs significantly better than chance based on the permutation test (for details of the permutation test see also footnote a)

Back to article page

ISSN: 1758-2946

Contact us

Submission enquiries: journalsubmissions@springernature.com