Reply to the comment made by Šicho, Vorśilák and Svozil on ‘The Power metric: a new statistically robust enrichment-type metric for virtual screening applications with early recovery capability’

The authors of the comment [1] raised an interesting remark about the relation between the power metric (PM) [2] and the precision metric (PR), also known as the positive predictive value (PPV).

In fact, this relation was noted before by the authors of the article that introduced the power metric [2]. Actually, this relationship is shared by all enrichment-type metrics, like the enrichment factor (EF) and ROC enrichment (ROCE), as can be noted by these equations:

in which R _i and R _a being the proportion of active and inactive instances in the whole dataset with N instances:

with n _a and n _i the number of active and inactive instances in the dataset.

This relationship was one of the reasons to classify the power metric as another enrichment-type metric. In fact, all enrichment-type metrics can be expressed by the same representation:

in which the threshold χ will be the cutoff that defines the hitlist of selected compounds. It can be expressed differently for each particular metric:

1. (a)

   in EF, χ is the fraction of compounds selected (χ = N _s /N), related to the number of true and false positives (TP and FP):

   $$\chi = \frac{TP + FP}{N}$$

   (7)
2. (b)

   in ROCE, χ can be related to the fraction of inactive instances wrongly classified as positives:

   $$\chi = FPR = \frac{FP}{{n_{i} }}$$

   (8)
3. (c)

   in PM, χ can be related to the sum of the true and false positive rates:

   $$\chi = TPR + FPR = \frac{TP}{{n_{a} }} + \frac{FP}{{n_{i} }}$$

   (9)

Due to these characteristics all these metrics are interconvertible.

A second remark made by Šicho, Vorśilák and Svozil [1] is that the power metric ‘should be accompanied by a metric taking negative classification into account’. We do not entirely agree with this statement as one can estimate all other metrics from the 2-by-2 contingency (confusion) matrix using only the power metric value and the user-defined threshold χ. Combining Eqs. (6) and (9), we can redefine PM as a function of χ and FPR:

and derive:

In addition, using the number of actives and inactives, all values of TP, FP, TN (true negatives) and FN (false negatives) can be calculated, and from these values any metric can be derived.

The fact that all these metrics are functionally related to the precision metric do not invalidated them as being useful metrics (‘not suitable for performance assessment’, as stated by the authors of the comment). All these metrics have their scopes, strengths and weaknesses. Each one has its meaning and can be used by the user depending on the desired aims. For example, the precision or EF metrics might be more appropriate if the user is more concerned about false positives, while in applications with more emphasis on true positive rates the PM or ROCE metrics would be recommended instead.

In order to have a better understanding on the interpretation of the power metric, lets investigate the dependency of PM on threshold χ. In case of a ‘perfect’ screening method in which FPR approaches zero, the PM tends to approach one (Eq. 10) and TPR tends to become equal to χ (Eq. 9). Thus, in this case the maximum value of the TPR is limited by the user-defined threshold value χ:

and the PM could be expressed as:

This leads us to the interpretation of the PM as the fraction of active compounds that are correctly predicted in relation to the maximum fraction of active compounds that could be recovered at the chosen threshold χ, or, in other words, PM express the probability of an active compound to be correctly classified.