Skip to main content

Table 2 Comparison of prediction and uncertainty quantification performance on benchmark datasets

From: Uncertainty-aware prediction of chemical reaction yields with graph neural networks

Dataset Training/test split Measure YieldBERT YieldBERT-DA Proposed
\(\lambda = 0\) \(\lambda = 1\) \(\lambda = 0.1\)
Buchwald-Hartwig 70/30 MAE (%p) 3.990 ± 0.153 3.090 ± 0.118 3.009 ± 0.045 2.953 ± 0.058 \(\mathbf{2.920} \pm \mathbf{0.056}\)
RMSE (%p) 6.014 ± 0.272 4.799 ± 0.261 4.509 ± 0.116 4.535 ± 0.136 \(\mathbf{4.433} \pm \mathbf{0.085}\)
R\(^2\) 0.951 ± 0.005 0.969 ± 0.004 0.973 ± 0.002 0.972 ± 0.002 \(\mathbf{0.974} \pm \mathbf{0.001}\)
Spearman \(\rho\) 0.439 ± 0.037 0.254 ± 0.027 \(\mathbf{0.445} \pm \mathbf{0.020}\) 0.421 ± 0.031
50/50 MAE (%p) 4.792 ± 0.124 3.744 ± 0.150 3.614 ± 0.095 \(\mathbf{3.482} \pm \mathbf{0.107}\) 3.497 ± 0.090
RMSE (%p) 7.288 ± 0.198 5.877 ± 0.348 5.484 ± 0.193 5.481 ± 0.355 \(\mathbf{5.387} \pm \mathbf{0.202}\)
R\(^2\) 0.928 ± 0.004 0.953 ± 0.006 0.959 ± 0.003 0.959 ± 0.005 \(\mathbf{0.961} \pm \mathbf{0.003}\)
Spearman \(\rho\) \(\mathbf{0.460} \pm \mathbf{0.021}\) 0.227 ± 0.021 0.419 ± 0.020 0.401 ± 0.014
30/70 MAE (%p) 6.075 ± 0.222 4.833 ± 0.167 4.677 ± 0.174 \(\mathbf{4.463} \pm \mathbf{0.150}\) 4.483 ± 0.165
RMSE (%p) 9.338 ± 0.424 7.822 ± 0.463 7.227 ± 0.407 7.053 ± 0.439 \(\mathbf{6.970} \pm \mathbf{0.403}\)
R\(^2\) 0.882 ± 0.011 0.917 ± 0.010 0.929 ± 0.008 0.933 ± 0.009 \(\mathbf{0.934} \pm \mathbf{0.008}\)
Spearman \(\rho\) \(\mathbf{0.464} \pm \mathbf{0.020}\) 0.229 ± 0.035 0.407 ± 0.022 0.385 ± 0.029
20/80 MAE (%p) 6.862 ± 0.212 5.781 ± 0.252 5.605 ± 0.236 5.319 ± 0.179 \(\mathbf{5.311} \pm \mathbf{0.154}\)
RMSE (%p) 10.306 ± 0.303 9.164 ± 0.668 8.567 ± 0.472 8.357 ± 0.400 \(\mathbf{8.204} \pm \mathbf{0.372}\)
R\(^2\) 0.857 ± 0.008 0.886 ± 0.017 0.901 ± 0.011 0.906 ± 0.009 \(\mathbf{0.909} \pm \mathbf{0.008}\)
Spearman \(\rho\) \(\mathbf{0.457} \pm \mathbf{0.017}\) 0.208 ± 0.044 0.373 ± 0.040 0.343 ± 0.029
10/90 MAE (%p) 8.607 ± 0.387 7.705 ± 0.236 7.605 ± 0.420 7.244 ± 0.229 \(\mathbf{7.196} \pm \mathbf{0.274}\)
RMSE (%p) 12.393 ± 0.499 11.633 ± 0.293 11.468 ± 0.699 11.002 ± 0.436 \(\mathbf{10.875} \pm \mathbf{0.448}\)
R\(^2\) 0.793 ± 0.016 0.818 ± 0.009 0.822 ± 0.022 0.837 ± 0.013 \(\mathbf{0.841} \pm \mathbf{0.013}\)
Spearman \(\rho\) \(\mathbf{0.432} \pm \mathbf{0.024}\) 0.148 ± 0.036 0.384 ± 0.040 0.345 ± 0.031
5/95 MAE (%p) 12.117 ± 0.789 \(\mathbf{9.651} \pm \mathbf{0.338}\) 10.056 ± 0.501 10.609 ± 1.610 9.677 ± 0.408
RMSE (%p) 16.740 ± 0.950 14.073 ± 0.687 14.636 ± 0.672 14.693 ± 1.467 \(\mathbf{14.041} \pm \mathbf{0.492}\)
R\(^2\) 0.622 ± 0.042 0.733 ± 0.027 0.711 ± 0.026 0.707 ± 0.063 \(\mathbf{0.734} \pm \mathbf{0.019}\)
Spearman \(\rho\) \(\mathbf{0.411} \pm \mathbf{0.024}\) 0.002 ± 0.058 0.398 ± 0.141 0.399 ± 0.058
2.5/97.5 MAE (%p) 15.979 ± 0.817 12.243 ± 0.631 12.409 ± 0.558 13.508 ± 2.745 \(\mathbf{11.747} \pm \mathbf{1.005}\)
RMSE (%p) 20.463 ± 0.623 17.151 ± 0.677 17.384 ± 0.775 17.992 ± 2.530 \(\mathbf{16.586} \pm \mathbf{1.364}\)
R\(^2\) 0.436 ± 0.034 0.604 ± 0.031 0.593 ± 0.037 0.556 ± 0.130 \(\mathbf{0.628} \pm \mathbf{0.062}\)
Spearman \(\rho\) \(\mathbf{0.381} \pm \mathbf{0.038}\) 0.016 ± 0.067 0.309 ± 0.176 0.300 ± 0.075
Suzuki-Miyaura 70/30 MAE (%p) 8.128 ± 0.344 6.598 ± 0.270 6.233 ± 0.207 6.118 ± 0.212 \(\mathbf{6.116} \pm \mathbf{0.223}\)
RMSE (%p) 12.073 ± 0.463 10.524 ± 0.482 9.522 ± 0.454 9.495 ± 0.430 \(\mathbf{9.467} \pm \mathbf{0.459}\)
R\(^2\) 0.815 ± 0.013 0.859 ± 0.012 0.885 ± 0.010 0.885 ± 0.009 \(\mathbf{0.886} \pm \mathbf{0.010}\)
Spearman \(\rho\) \(\mathbf{0.439} \pm \mathbf{0.018}\) 0.324 ± 0.026 0.432 ± 0.024 0.425 ± 0.026
50/50 MAE (%p) 8.922 ± 0.235 7.539 ± 0.153 6.872 ± 0.089 \(\mathbf{6.702} \pm \mathbf{0.082}\) 6.725 ± 0.089
RMSE (%p) 13.148 ± 0.270 11.797 ± 0.250 10.272 ± 0.138 \(\mathbf{10.225} \pm \mathbf{0.128}\) \(\mathbf{10.225} \pm \mathbf{0.135}\)
R\(^2\) 0.780±0.009 0.823 ± 0.007 0.866 ± 0.003 \(\mathbf{0.867} \pm \mathbf{0.003}\) \(\mathbf{0.867} \pm \mathbf{0.003}\)
Spearman \(\rho\) \(\mathbf{0.439} \pm \mathbf{0.019}\) 0.322 ± 0.021 0.432 ± 0.017 0.430 ± 0.012
30/70 MAE (%p) 10.094 ± 0.346 8.804 ± 0.249 8.021 ± 0.094 \(\mathbf{7.740} \pm \mathbf{0.109}\) 7.847 ± 0.094
RMSE (%p) 14.614 ± 0.381 13.337 ± 0.357 11.726 ± 0.152 \(\mathbf{11.526} \pm \mathbf{0.166}\) 11.593 ± 0.136
R\(^2\) 0.729 ± 0.014 0.774 ± 0.012 0.825 ± 0.004 \(\mathbf{0.831} \pm \mathbf{0.005}\) 0.829 ± 0.004
Spearman \(\rho\) \(\mathbf{0.432} \pm \mathbf{0.018}\) 0.292 ± 0.012 0.428 ± 0.013 0.417 ± 0.008
20/80 MAE (%p) 11.229 ± 0.247 10.017 ± 0.338 9.147 ± 0.185 \(\mathbf{8.726} \pm \mathbf{0.172}\) 8.793 ± 0.191
RMSE (%p) 15.966 ± 0.381 14.851 ± 0.576 13.115 ± 0.298 12.754 ± 0.316 \(\mathbf{12.734} \pm \mathbf{0.347}\)
R\(^2\) 0.676 ± 0.015 0.719 ± 0.022 0.781 ± 0.010 0.793 ± 0.010 \(\mathbf{0.794} \pm \mathbf{0.011}\)
Spearman \(\rho\) \(\mathbf{0.432} \pm \mathbf{0.014}\) 0.274 ± 0.020 0.429 ± 0.017 0.408 ± 0.018
10/90 MAE (%p) 13.528 ± 0.395 11.954 ± 0.443 11.439 ± 0.185 \(\mathbf{10.625} \pm \mathbf{0.249}\) 10.739 ± 0.211
RMSE (%p) 18.734 ± 0.530 17.129 ± 0.683 15.967 ± 0.326 \(\mathbf{15.097} \pm \mathbf{0.421}\) 15.164 ± 0.344
R\(^2\) 0.554 ± 0.025 0.627 ± 0.030 0.676 ± 0.013 \(\mathbf{0.711} \pm \mathbf{0.016}\) 0.708 ± 0.013
Spearman \(\rho\) 0.389 ± 0.022 0.221 ± 0.027 \(\mathbf{0.390} \pm \mathbf{0.019}\) 0.382 ± 0.019
5/95 MAE (%p) 15.695 ± 0.618 14.294 ± 0.507 14.214 ± 0.504 \(\mathbf{13.364} \pm \mathbf{0.223}\) 13.451 ± 0.353
RMSE (%p) 21.181 ± 0.724 20.016 ± 0.661 19.421 ± 0.588 \(\mathbf{18.463} \pm \mathbf{0.308}\) 18.511 ± 0.392
R\(^2\) 0.430 ± 0.040 0.491 ± 0.034 0.521 ± 0.029 \(\mathbf{0.567} \pm \mathbf{0.014}\) 0.565 ± 0.018
Spearman \(\rho\) 0.355 ± 0.026 0.144 ± 0.052 \(\mathbf{0.389} \pm \mathbf{0.045}\) 0.330 ± 0.034
2.5/97.5 MAE (%p) 17.666 ± 0.496 17.587 ± 0.690 18.061 ± 0.571 \(\mathbf{16.705} \pm \mathbf{1.090}\) 17.189 ± 0.813
RMSE (%p) 22.967 ± 0.804 23.780 ± 0.793 24.121 ± 0.655 \(\mathbf{22.156} \pm \mathbf{1.273}\) 22.943 ± 0.887
R\(^2\) 0.330 ± 0.047 0.282 ± 0.047 0.261 ± 0.039 \(\mathbf{0.375} \pm \mathbf{0.072}\) 0.331 ± 0.051
Spearman \(\rho\) \(\mathbf{0.291} \pm \mathbf{0.025}\) 0.028 ± 0.054 0.280 ± 0.074 0.223 ± 0.081