Full “Laplacianised” posterior naive Bayesian algorithm

Background In the last decade the standard Naive Bayes (SNB) algorithm has been widely employed in multi–class classification problems in cheminformatics. This popularity is mainly due to the fact that the algorithm is simple to implement and in many cases yields respectable classification results. Using clever heuristic arguments “anchored” by insightful cheminformatics knowledge, Xia et al. have simplified the SNB algorithm further and termed it the Laplacian Corrected Modified Naive Bayes (LCMNB) approach, which has been widely used in cheminformatics since its publication. In this note we mathematically illustrate the conditions under which Xia et al.’s simplification holds. It is our hope that this clarification could help Naive Bayes practitioners in deciding when it is appropriate to employ the LCMNB algorithm to classify large chemical datasets. Results A general formulation that subsumes the simplified Naive Bayes version is presented. Unlike the widely used NB method, the Standard Naive Bayes description presented in this work is discriminative (not generative) in nature, which may lead to possible further applications of the SNB method. Conclusions Starting from a standard Naive Bayes (SNB) algorithm, we have derived mathematically the relationship between Xia et al.’s ingenious, but heuristic algorithm, and the SNB approach. We have also demonstrated the conditions under which Xia et al.’s crucial assumptions hold. We therefore hope that the new insight and recommendations provided can be found useful by the cheminformatics community.


Background
Broadly speaking there are two conceptually different ways to solve statistical problems: the frequentist and the Bayesian approaches. On the pros and cons of each method there are numerous excellent review articles and text books, such as the recent book by Murphy [1]. Unlike the frequentist approach, in the Bayesian approach any a priori knowledge about the probability distribution function that one assumes might have generated the given data (in the first place) can be taken into account when estimating this distribution function from the data at hand. If the data are noise-free and "complete", the role of the a priori information in estimating the distribution function diminishes drastically. However, the a priori information can be http://www.jcheminf.com/content/5/1/37 the standard Naive Bayesian algorithm, albeit for binary classification problems; slight variants of this algorithm for multi-class classification can also be found in [3,4]. According to Rogers et al. [5], Rogers being a co-author of the work presented in [2], "the standard Naive Bayes was modified by considering only the effect of the presence of a feature and not its absence". There are also a few more noticeable aspects of this proposed simplification: (a) the authors cleverly estimate directly -albeit heuristically -the a posteriori class probability for the present feature; (b) these authors (rather ingeniously) incorporate a Laplacian-correction into the estimated posterior class probability; and (c) the authors deem absent features not discriminating enough and therefore discard their contributions to the estimation of the posterior class. More than anything else it is this omission of the absent features from the Standard Naive Bayes (SNB) algorithm that makes Xia et al.'s proposed Naive Bayes Algorithm, termed Laplacian Corrected Modified Naive Bayes (LCMNB), (and its variants by different groups) computationally fast.
It is these three points, (a), (b) and (c), that we expound on in a mathematical setting to demonstrate under which conditions they hold -not only in an abstract sense, but also in the practical sense for a NB practitioner to make an informed decision as to when it is appropriate to employ SNB or LCMNB, in the cheminformatics context.

Naive Bayes
From Bayes' theorem recall that [6]: where x = (x 1 , x 2 , ..., x L ) and ω i denote the feature vectors and class labels, respectively; x j and L being as described before, whereas i is just an index for the class labels. The terms p(ω i |x), p(x|ω i ), p(ω i ), and p(x) refer to the posterior probability for ω i given x, the descriptor vector distribution conditioned on class ω i , the a priori probability of class ω i occurring, and the descriptor vector density function, respectively -for more details, see ref. [3,4,6]. The left hand side of Eq. 1 can be expressed as follows [1,7] By virtue of Bayes' theorem p(ω i |x 1 , x 2 , ..., x L ) can be rewritten as which in turn allows us to rewrite Eq. 2 as Making use of the chain rule of probability [1,8], we can express p(x 1 , x 2 , ..., x L |ω i ) as Plugging the right hand side of the equation above into Eq. 4 results in In practice, it is extremely difficult to estimate p(ω i |x) or p(x|ω i ). This reality inevitably forces one to make concessions over the degree of accuracy the estimated p(ω i |x) or p(x|ω i ) can deliver. One widely employed scheme to obtain these probability distributions with compromised accuracy is to assume that individual descriptors x j , j = 1, 2, ..., L, are independent conditional on ω i . It is this naive assumption of independence among features to which the term "Naive" in "Naive Bayesian" refers.
Under this naive assumption, in Eq. 6, p( Multiplying top and bottom of Eq. 7 by Then making use of the fact that p( , we can rewrite Eq. 9 as or more compactly as p(x 1 ,x 2 ,....,x L ) is common to all classes and therefore plays no role in classification. Thus, in practice (in the Naive Bayes context with which this work is concerned) one is required to estimate p(ω i |x j ) and p(ω i ).
Since generative approaches can be informative and "simpler" than their discriminative counterparts [9], we make use of Bayes' theorem again, i.e., p( and then estimate p(ω i |x j ) through with C referring to the number of classes. p(ω i ) denotes the a priori class probability, which is relatively easy to estimate. Thus, in our Bayesian context, the estimation of p(ω j |x j ) boils down in practice to estimating p(x j |ω i ).
Estimation of p(x j |ω i ), with x j = 1 and 0 p(x j |ω i ) can be estimated using the given data and assuming a Beta distribution as an a priori distribution for p(x j |ω i ) [10]. (There are other possible prior distributions from which one can choose, but we select the Beta distribution for reasons that will transpire later). As described [11]: and of course where N ω i and N ij , respectively, denote the number compounds in class ω i , and number of compounds in this class with descriptor x j assuming value 1. β i and α i are Beta distribution hyper-parameters per class and the valid range of values that these hyper-parameter can assume are as defined in Appendix A. When α i and β i equal 1, α i and β i + α i in Eqs. 12-13 can be viewed as a "Laplacian correction".

Results and discussion
Estimation of p(ω i |x j = 1) and p(ω i |x j = 0) Estimation of p(ω i |x j = 1): In Our Approach Remark 1. Assume that we have N chemical compounds (and their activity labels) available for training, where N ω i of these compounds belong to class ω i .

Remark 2.
Assume that the class a priori distribution is taken as p(ω i ) = N ω i N , where N ω i >> α i + β i (which is a valid assumption as found in any realistic large chemical dataset).
By virtue of Remark 1 and Eq. 12, the estimate of p(ω i |x j = 1) becomes (recall that p( Because of Remark 2, Eq. 14 can be simplified to where N + j is the number of times x j assumes the value 1.

Estimation of p(ω i |x j = 1): In Xia et al.'s Formulation
In the approach of Xia et al., p(ω i |x j = 1) is estimated as where K is as defined in Xia et al. and in their paper A i is given as Eq. 16 constitutes what Xia et al. term "the Laplacian-Corrected Modified Naive Bayes (LCMNB)" estimator for p(ω i |x j = 1). If α i in Eq. 15 is set to A i , Eq. 15 is exactly equivalent to Xia et al.'s estimator for p(ω i |x j = 1) as can be seen in Eq. 16.
We note in passing that in Xia et al.'s case, C = 2 and p(ω 2 ) = 1 K , which in their nomenclature denoted by p(Active) -that is, Initially we employed the Beta a priori distribution for the class conditional distribution to ascertain the equivalence of Eqs. 15 and 16. Fortunately, however, we have ended up with the general equations (Eqs. 14 -15) that not only encapsulate the LCMNB scheme of Xia et al., but also subsume the other various variants of LCMNB, such as those discussed in Nidhi et al. and Nigsch et al.'s papers [3,4].
At any rate, let us proceed to the nub of this work: Identifying the conditions under which the LCMNB algorithm holds with respect to the SNB algorithm. But first we need to describe the estimation of p(ω i |x j = 0).

Estimation of p(ω i |x j = 0): In Our Approach
In regard to the case of x j = 0, we make use of Remark 1, Remark 2 and Eq. 13, which yield an estimator for http://www.jcheminf.com/content/5/1/37 p(ω i |x j = 0) as

Naive Bayes: scoring function from
For notational convenience let us denote in Eq. 17 by ξ ij and ν ij , respectively. Thus, p(ω i |x j = 1) and p(ω i |x j = 0) may be written more succinctly as p(ω i |x j ) = ξ , which allows us to express Eq. 11 more compactly as Now we come to the core of this work, under which conditions does the LCMNB algorithm hold with respect to the SNB algorithm? Before we answer this question, we deem it instructive and more insightful to map Eq. 18 monotonically to a discriminant function, a "scoring function" (so to speak).
To this end, taking the logarithm of Eq. 18 results in is common to all classes and therefore does not play any role in classifying a given new compound. In other words, for practical classification purposes we are only interested in class dependent terms, i.e.,

Conditions
In Xia et al.'s approach, the LCMNB algorithm, is none other than j x j ln ξ ij − ln p(ω i ) in Eq. 21. This means that in Xia et al.'s scheme the contributions from the terms depending on x j = 0 for a given class, i.e., are discarded. To the best of our knowledge, neither in Xia et al. nor in any other paper on the LCMNB approach has it been demonstrated that (i) the contribution of Eq. 22 is zero, i.e., equally, in these papers, it has not been shown that (ii) nor has it been established that (iii) Thus unless one (or more) of the above -(i), (ii) and (iii) -is (are) met, the assumption on which the Modified Naive Bayesian algorithm is based is questionable and therefore its practitioners should pay attention to this discrepancy; clearly it is not justifiable to discard from the onset the contribution of j (1 − x j ) ln ν ij − (L − 1) × ln p(ω i ) simply because features x j are absent, i.e. x j = 0.
For completeness, we consider also the case of the highly popular class prior distribution, p(ω i ) = 1 C , i.e. p(ω 1 ) = p(ω 2 ) = ... = p(ω C ). We hasten to add that this option was not included in the LCMNB scheme. At any rate, by simply repeating the arguments in the preceding sections, it is straightforward to show that one ends up with Eq. 21. In this scenario, though, L × ln p(ω i ) is common to all classes and therefore does not play a role in classifying a new compound, i.e., D ω i (x) reduces to  We state from the onset, in the following derivation we follow closely the descriptions given in ref. [10]. We also note, for clarity's sake, in the following analyses we abuse notation and use x jk for both the random variable and its realization.
In this work, x ∈ {0, 1} L , i.e. x j ∈ {0, 1} and suppose that x j are independent Bernoulli random variables (and this is in fact the assumption made in the Naive Bayesian approach). Thus, in the Naive Bayesian setting p(x|ω i ) can be given as where μ ij is an estimate for the conditional probability that feature j occurs in class ω i , and is what we are trying to estimate given a set of compounds assumed to belong to class ω i . (In our context, μ ij is an estimator for p(x j |ω i ), where p(x j |ω i ) is as defined in the text.) To estimate μ ij in a Bayesian framework, we first view μ ij as a random variable, then choose an "appropriate" prior and likelihood for the random variable μ ij .
Let us suppose that our a priori knowledge about the random variable μ ij indicates that μ ij is described by a Beta distribution, i.e., where B(α i , β i ) ensures that the Beta distribution is normalised Using the Bayes' theorem, then the posterior probability for μ ij on the training data can be given by where f (x j1 , x j2 , ..., x jN ω i |μ ij ) refers to the likelihood, and x j1 , x j2 ,... and x jN ω i denote the j th feature of the N ω i samples/compounds from class ω i . As the samples are assumed independent, then f (x j1 , x j2 , ..., i.e., π(μ ij |x j1 , x j2 , ..., Clearly, in Eq. 34, the posterior density for μ ij given the samples x j1 , x j2 , ..., x jN ω i has the same form as the prior for μ ij [11], i.e., π(μ ij |x j1 , x j2 , ..., x jN ω i ) which is none other than another Beta distribution. This means that the Bayes estimator of μ ij , which is the estimate we are interested in, is the mean of the posterior distribution obtained [11]: In other words, QED.
An accessible description of the derivation of Eq. 37 can be found in ref. [10].