Table 2 Comparison of prediction and uncertainty quantification performance on benchmark datasets
From: Uncertainty-aware prediction of chemical reaction yields with graph neural networks
Dataset | Training/test split | Measure | YieldBERT | YieldBERT-DA | Proposed | ||
|---|---|---|---|---|---|---|---|
\(\lambda = 0\) | \(\lambda = 1\) | \(\lambda = 0.1\) | |||||
Buchwald-Hartwig | 70/30 | MAE (%p) | 3.990 ± 0.153 | 3.090 ± 0.118 | 3.009 ± 0.045 | 2.953 ± 0.058 | \(\mathbf{2.920} \pm \mathbf{0.056}\) |
RMSE (%p) | 6.014 ± 0.272 | 4.799 ± 0.261 | 4.509 ± 0.116 | 4.535 ± 0.136 | \(\mathbf{4.433} \pm \mathbf{0.085}\) | ||
R\(^2\) | 0.951 ± 0.005 | 0.969 ± 0.004 | 0.973 ± 0.002 | 0.972 ± 0.002 | \(\mathbf{0.974} \pm \mathbf{0.001}\) | ||
Spearman \(\rho\) | – | 0.439 ± 0.037 | 0.254 ± 0.027 | \(\mathbf{0.445} \pm \mathbf{0.020}\) | 0.421 ± 0.031 | ||
50/50 | MAE (%p) | 4.792 ± 0.124 | 3.744 ± 0.150 | 3.614 ± 0.095 | \(\mathbf{3.482} \pm \mathbf{0.107}\) | 3.497 ± 0.090 | |
RMSE (%p) | 7.288 ± 0.198 | 5.877 ± 0.348 | 5.484 ± 0.193 | 5.481 ± 0.355 | \(\mathbf{5.387} \pm \mathbf{0.202}\) | ||
R\(^2\) | 0.928 ± 0.004 | 0.953 ± 0.006 | 0.959 ± 0.003 | 0.959 ± 0.005 | \(\mathbf{0.961} \pm \mathbf{0.003}\) | ||
Spearman \(\rho\) | – | \(\mathbf{0.460} \pm \mathbf{0.021}\) | 0.227 ± 0.021 | 0.419 ± 0.020 | 0.401 ± 0.014 | ||
30/70 | MAE (%p) | 6.075 ± 0.222 | 4.833 ± 0.167 | 4.677 ± 0.174 | \(\mathbf{4.463} \pm \mathbf{0.150}\) | 4.483 ± 0.165 | |
RMSE (%p) | 9.338 ± 0.424 | 7.822 ± 0.463 | 7.227 ± 0.407 | 7.053 ± 0.439 | \(\mathbf{6.970} \pm \mathbf{0.403}\) | ||
R\(^2\) | 0.882 ± 0.011 | 0.917 ± 0.010 | 0.929 ± 0.008 | 0.933 ± 0.009 | \(\mathbf{0.934} \pm \mathbf{0.008}\) | ||
Spearman \(\rho\) | – | \(\mathbf{0.464} \pm \mathbf{0.020}\) | 0.229 ± 0.035 | 0.407 ± 0.022 | 0.385 ± 0.029 | ||
20/80 | MAE (%p) | 6.862 ± 0.212 | 5.781 ± 0.252 | 5.605 ± 0.236 | 5.319 ± 0.179 | \(\mathbf{5.311} \pm \mathbf{0.154}\) | |
RMSE (%p) | 10.306 ± 0.303 | 9.164 ± 0.668 | 8.567 ± 0.472 | 8.357 ± 0.400 | \(\mathbf{8.204} \pm \mathbf{0.372}\) | ||
R\(^2\) | 0.857 ± 0.008 | 0.886 ± 0.017 | 0.901 ± 0.011 | 0.906 ± 0.009 | \(\mathbf{0.909} \pm \mathbf{0.008}\) | ||
Spearman \(\rho\) | – | \(\mathbf{0.457} \pm \mathbf{0.017}\) | 0.208 ± 0.044 | 0.373 ± 0.040 | 0.343 ± 0.029 | ||
10/90 | MAE (%p) | 8.607 ± 0.387 | 7.705 ± 0.236 | 7.605 ± 0.420 | 7.244 ± 0.229 | \(\mathbf{7.196} \pm \mathbf{0.274}\) | |
RMSE (%p) | 12.393 ± 0.499 | 11.633 ± 0.293 | 11.468 ± 0.699 | 11.002 ± 0.436 | \(\mathbf{10.875} \pm \mathbf{0.448}\) | ||
R\(^2\) | 0.793 ± 0.016 | 0.818 ± 0.009 | 0.822 ± 0.022 | 0.837 ± 0.013 | \(\mathbf{0.841} \pm \mathbf{0.013}\) | ||
Spearman \(\rho\) | – | \(\mathbf{0.432} \pm \mathbf{0.024}\) | 0.148 ± 0.036 | 0.384 ± 0.040 | 0.345 ± 0.031 | ||
5/95 | MAE (%p) | 12.117 ± 0.789 | \(\mathbf{9.651} \pm \mathbf{0.338}\) | 10.056 ± 0.501 | 10.609 ± 1.610 | 9.677 ± 0.408 | |
RMSE (%p) | 16.740 ± 0.950 | 14.073 ± 0.687 | 14.636 ± 0.672 | 14.693 ± 1.467 | \(\mathbf{14.041} \pm \mathbf{0.492}\) | ||
R\(^2\) | 0.622 ± 0.042 | 0.733 ± 0.027 | 0.711 ± 0.026 | 0.707 ± 0.063 | \(\mathbf{0.734} \pm \mathbf{0.019}\) | ||
Spearman \(\rho\) | – | \(\mathbf{0.411} \pm \mathbf{0.024}\) | 0.002 ± 0.058 | 0.398 ± 0.141 | 0.399 ± 0.058 | ||
2.5/97.5 | MAE (%p) | 15.979 ± 0.817 | 12.243 ± 0.631 | 12.409 ± 0.558 | 13.508 ± 2.745 | \(\mathbf{11.747} \pm \mathbf{1.005}\) | |
RMSE (%p) | 20.463 ± 0.623 | 17.151 ± 0.677 | 17.384 ± 0.775 | 17.992 ± 2.530 | \(\mathbf{16.586} \pm \mathbf{1.364}\) | ||
R\(^2\) | 0.436 ± 0.034 | 0.604 ± 0.031 | 0.593 ± 0.037 | 0.556 ± 0.130 | \(\mathbf{0.628} \pm \mathbf{0.062}\) | ||
Spearman \(\rho\) | – | \(\mathbf{0.381} \pm \mathbf{0.038}\) | 0.016 ± 0.067 | 0.309 ± 0.176 | 0.300 ± 0.075 | ||
Suzuki-Miyaura | 70/30 | MAE (%p) | 8.128 ± 0.344 | 6.598 ± 0.270 | 6.233 ± 0.207 | 6.118 ± 0.212 | \(\mathbf{6.116} \pm \mathbf{0.223}\) |
RMSE (%p) | 12.073 ± 0.463 | 10.524 ± 0.482 | 9.522 ± 0.454 | 9.495 ± 0.430 | \(\mathbf{9.467} \pm \mathbf{0.459}\) | ||
R\(^2\) | 0.815 ± 0.013 | 0.859 ± 0.012 | 0.885 ± 0.010 | 0.885 ± 0.009 | \(\mathbf{0.886} \pm \mathbf{0.010}\) | ||
Spearman \(\rho\) | – | \(\mathbf{0.439} \pm \mathbf{0.018}\) | 0.324 ± 0.026 | 0.432 ± 0.024 | 0.425 ± 0.026 | ||
50/50 | MAE (%p) | 8.922 ± 0.235 | 7.539 ± 0.153 | 6.872 ± 0.089 | \(\mathbf{6.702} \pm \mathbf{0.082}\) | 6.725 ± 0.089 | |
RMSE (%p) | 13.148 ± 0.270 | 11.797 ± 0.250 | 10.272 ± 0.138 | \(\mathbf{10.225} \pm \mathbf{0.128}\) | \(\mathbf{10.225} \pm \mathbf{0.135}\) | ||
R\(^2\) | 0.780±0.009 | 0.823 ± 0.007 | 0.866 ± 0.003 | \(\mathbf{0.867} \pm \mathbf{0.003}\) | \(\mathbf{0.867} \pm \mathbf{0.003}\) | ||
Spearman \(\rho\) | – | \(\mathbf{0.439} \pm \mathbf{0.019}\) | 0.322 ± 0.021 | 0.432 ± 0.017 | 0.430 ± 0.012 | ||
30/70 | MAE (%p) | 10.094 ± 0.346 | 8.804 ± 0.249 | 8.021 ± 0.094 | \(\mathbf{7.740} \pm \mathbf{0.109}\) | 7.847 ± 0.094 | |
RMSE (%p) | 14.614 ± 0.381 | 13.337 ± 0.357 | 11.726 ± 0.152 | \(\mathbf{11.526} \pm \mathbf{0.166}\) | 11.593 ± 0.136 | ||
R\(^2\) | 0.729 ± 0.014 | 0.774 ± 0.012 | 0.825 ± 0.004 | \(\mathbf{0.831} \pm \mathbf{0.005}\) | 0.829 ± 0.004 | ||
Spearman \(\rho\) | – | \(\mathbf{0.432} \pm \mathbf{0.018}\) | 0.292 ± 0.012 | 0.428 ± 0.013 | 0.417 ± 0.008 | ||
20/80 | MAE (%p) | 11.229 ± 0.247 | 10.017 ± 0.338 | 9.147 ± 0.185 | \(\mathbf{8.726} \pm \mathbf{0.172}\) | 8.793 ± 0.191 | |
RMSE (%p) | 15.966 ± 0.381 | 14.851 ± 0.576 | 13.115 ± 0.298 | 12.754 ± 0.316 | \(\mathbf{12.734} \pm \mathbf{0.347}\) | ||
R\(^2\) | 0.676 ± 0.015 | 0.719 ± 0.022 | 0.781 ± 0.010 | 0.793 ± 0.010 | \(\mathbf{0.794} \pm \mathbf{0.011}\) | ||
Spearman \(\rho\) | – | \(\mathbf{0.432} \pm \mathbf{0.014}\) | 0.274 ± 0.020 | 0.429 ± 0.017 | 0.408 ± 0.018 | ||
10/90 | MAE (%p) | 13.528 ± 0.395 | 11.954 ± 0.443 | 11.439 ± 0.185 | \(\mathbf{10.625} \pm \mathbf{0.249}\) | 10.739 ± 0.211 | |
RMSE (%p) | 18.734 ± 0.530 | 17.129 ± 0.683 | 15.967 ± 0.326 | \(\mathbf{15.097} \pm \mathbf{0.421}\) | 15.164 ± 0.344 | ||
R\(^2\) | 0.554 ± 0.025 | 0.627 ± 0.030 | 0.676 ± 0.013 | \(\mathbf{0.711} \pm \mathbf{0.016}\) | 0.708 ± 0.013 | ||
Spearman \(\rho\) | – | 0.389 ± 0.022 | 0.221 ± 0.027 | \(\mathbf{0.390} \pm \mathbf{0.019}\) | 0.382 ± 0.019 | ||
5/95 | MAE (%p) | 15.695 ± 0.618 | 14.294 ± 0.507 | 14.214 ± 0.504 | \(\mathbf{13.364} \pm \mathbf{0.223}\) | 13.451 ± 0.353 | |
RMSE (%p) | 21.181 ± 0.724 | 20.016 ± 0.661 | 19.421 ± 0.588 | \(\mathbf{18.463} \pm \mathbf{0.308}\) | 18.511 ± 0.392 | ||
R\(^2\) | 0.430 ± 0.040 | 0.491 ± 0.034 | 0.521 ± 0.029 | \(\mathbf{0.567} \pm \mathbf{0.014}\) | 0.565 ± 0.018 | ||
Spearman \(\rho\) | – | 0.355 ± 0.026 | 0.144 ± 0.052 | \(\mathbf{0.389} \pm \mathbf{0.045}\) | 0.330 ± 0.034 | ||
2.5/97.5 | MAE (%p) | 17.666 ± 0.496 | 17.587 ± 0.690 | 18.061 ± 0.571 | \(\mathbf{16.705} \pm \mathbf{1.090}\) | 17.189 ± 0.813 | |
RMSE (%p) | 22.967 ± 0.804 | 23.780 ± 0.793 | 24.121 ± 0.655 | \(\mathbf{22.156} \pm \mathbf{1.273}\) | 22.943 ± 0.887 | ||
R\(^2\) | 0.330 ± 0.047 | 0.282 ± 0.047 | 0.261 ± 0.039 | \(\mathbf{0.375} \pm \mathbf{0.072}\) | 0.331 ± 0.051 | ||
Spearman \(\rho\) | – | \(\mathbf{0.291} \pm \mathbf{0.025}\) | 0.028 ± 0.054 | 0.280 ± 0.074 | 0.223 ± 0.081 | ||