Skip to main content

Table 1 Comparing AUPRC to other virtual screening measures when improving a reference ranking (a) at different positions (b–e)

From: Filtered circular fingerprints improve either prediction or runtime performance while retaining interpretability

Ranking of active (1) and inactive (0) compounds due to predicted probability

(a)

11011111110111000000010000000000000000000000010000000000000000100

(b)

11011111110111000000010000000000000000000000010000000000001000000

(c)

11011111110111000000010000000000000000000100000000000000000000100

(d)

11011111110111000100000000000000000000000000010000000000000000100

(e)

11111101110111000000010000000000000000000000010000000000000000100

 

AUROC

\(\Delta \)

AUPRC

\(\Delta \)

EF-5%

\(\Delta \)

BEDROC-20

\(\Delta \)

BEDROC-100

\(\Delta \)

(a)

0.864

 

0.772

 

2.9

 

0.884

 

0.885

 

(b)

0.869

+0.005

0.773

+0.001

2.9

0

0.889

+0.005

0.889

+0.003

(c)

0.869

+0.005

0.774

+0.002

2.9

0

0.889

+0.005

0.89

+0.004

(d)

0.869

+0.005

0.78

+0.009

2.9

0

0.89

+0.006

0.892

+0.006

(e)

0.869

+0.005

0.822

+0.051

4.3

+1.4

0.89

+0.006

0.893

+0.008

  1. Each bit resembles an active (1) or inactive compound (0). Each bit-string (a–e) corresponds to the result of a classifier that ranks all test-set compounds according to their predicted probability of being active. A perfect prediction would list all “1s” before “0s”. The reference prediction (a) has an AUROC (the area under the ROC curve) score of 0.864 (Hence, the probability that a randomly drawn active compound is ranked higher than an inactive compound is 86.4%). We improve the reference prediction (a) by modifying the predicted probability of a single compound at different positions in the ranking: in (b–d), a single active compound is predicted with higher probability and shifted 4 positions upwards in the ranking. In (e), a single inactive compound is predicted with lower probability and is moved 4 positions downwards in the ranking. When performing virtual screening, the last change (e) is probably most important to us, as we are interested in the compounds that are most likely active. However, the change in AUROC is constant for (b–e). EF (enrichment factor) and BEDROC (Boltzmann-Enhanced Discrimination of ROC) have the disadvantage of relying on a user defined parameter. Moreover, EF-5% changes only if the number (not the ordering) of active compounds within the top \(\chi \) bits (here: 3) differs. On the contrary, AUPRC (the area under precision recall curve) has the desired property that it increases more when the affected ranking position is higher without relying on a parameter (Please find ROC and precision recall curves for (a–e) in the Additional file 1)
Back to article page

Journal of Cheminformatics

ISSN: 1758-2946

Contact us