Skip to main content

Table 2 Comparison of prediction and uncertainty quantification performance on benchmark datasets

From: Uncertainty-aware prediction of chemical reaction yields with graph neural networks

Dataset

Training/test split

Measure

YieldBERT

YieldBERT-DA

Proposed

\(\lambda = 0\)

\(\lambda = 1\)

\(\lambda = 0.1\)

Buchwald-Hartwig

70/30

MAE (%p)

3.990 ± 0.153

3.090 ± 0.118

3.009 ± 0.045

2.953 ± 0.058

\(\mathbf{2.920} \pm \mathbf{0.056}\)

RMSE (%p)

6.014 ± 0.272

4.799 ± 0.261

4.509 ± 0.116

4.535 ± 0.136

\(\mathbf{4.433} \pm \mathbf{0.085}\)

R\(^2\)

0.951 ± 0.005

0.969 ± 0.004

0.973 ± 0.002

0.972 ± 0.002

\(\mathbf{0.974} \pm \mathbf{0.001}\)

Spearman \(\rho\)

0.439 ± 0.037

0.254 ± 0.027

\(\mathbf{0.445} \pm \mathbf{0.020}\)

0.421 ± 0.031

50/50

MAE (%p)

4.792 ± 0.124

3.744 ± 0.150

3.614 ± 0.095

\(\mathbf{3.482} \pm \mathbf{0.107}\)

3.497 ± 0.090

RMSE (%p)

7.288 ± 0.198

5.877 ± 0.348

5.484 ± 0.193

5.481 ± 0.355

\(\mathbf{5.387} \pm \mathbf{0.202}\)

R\(^2\)

0.928 ± 0.004

0.953 ± 0.006

0.959 ± 0.003

0.959 ± 0.005

\(\mathbf{0.961} \pm \mathbf{0.003}\)

Spearman \(\rho\)

\(\mathbf{0.460} \pm \mathbf{0.021}\)

0.227 ± 0.021

0.419 ± 0.020

0.401 ± 0.014

30/70

MAE (%p)

6.075 ± 0.222

4.833 ± 0.167

4.677 ± 0.174

\(\mathbf{4.463} \pm \mathbf{0.150}\)

4.483 ± 0.165

RMSE (%p)

9.338 ± 0.424

7.822 ± 0.463

7.227 ± 0.407

7.053 ± 0.439

\(\mathbf{6.970} \pm \mathbf{0.403}\)

R\(^2\)

0.882 ± 0.011

0.917 ± 0.010

0.929 ± 0.008

0.933 ± 0.009

\(\mathbf{0.934} \pm \mathbf{0.008}\)

Spearman \(\rho\)

\(\mathbf{0.464} \pm \mathbf{0.020}\)

0.229 ± 0.035

0.407 ± 0.022

0.385 ± 0.029

20/80

MAE (%p)

6.862 ± 0.212

5.781 ± 0.252

5.605 ± 0.236

5.319 ± 0.179

\(\mathbf{5.311} \pm \mathbf{0.154}\)

RMSE (%p)

10.306 ± 0.303

9.164 ± 0.668

8.567 ± 0.472

8.357 ± 0.400

\(\mathbf{8.204} \pm \mathbf{0.372}\)

R\(^2\)

0.857 ± 0.008

0.886 ± 0.017

0.901 ± 0.011

0.906 ± 0.009

\(\mathbf{0.909} \pm \mathbf{0.008}\)

Spearman \(\rho\)

\(\mathbf{0.457} \pm \mathbf{0.017}\)

0.208 ± 0.044

0.373 ± 0.040

0.343 ± 0.029

10/90

MAE (%p)

8.607 ± 0.387

7.705 ± 0.236

7.605 ± 0.420

7.244 ± 0.229

\(\mathbf{7.196} \pm \mathbf{0.274}\)

RMSE (%p)

12.393 ± 0.499

11.633 ± 0.293

11.468 ± 0.699

11.002 ± 0.436

\(\mathbf{10.875} \pm \mathbf{0.448}\)

R\(^2\)

0.793 ± 0.016

0.818 ± 0.009

0.822 ± 0.022

0.837 ± 0.013

\(\mathbf{0.841} \pm \mathbf{0.013}\)

Spearman \(\rho\)

\(\mathbf{0.432} \pm \mathbf{0.024}\)

0.148 ± 0.036

0.384 ± 0.040

0.345 ± 0.031

5/95

MAE (%p)

12.117 ± 0.789

\(\mathbf{9.651} \pm \mathbf{0.338}\)

10.056 ± 0.501

10.609 ± 1.610

9.677 ± 0.408

RMSE (%p)

16.740 ± 0.950

14.073 ± 0.687

14.636 ± 0.672

14.693 ± 1.467

\(\mathbf{14.041} \pm \mathbf{0.492}\)

R\(^2\)

0.622 ± 0.042

0.733 ± 0.027

0.711 ± 0.026

0.707 ± 0.063

\(\mathbf{0.734} \pm \mathbf{0.019}\)

Spearman \(\rho\)

\(\mathbf{0.411} \pm \mathbf{0.024}\)

0.002 ± 0.058

0.398 ± 0.141

0.399 ± 0.058

2.5/97.5

MAE (%p)

15.979 ± 0.817

12.243 ± 0.631

12.409 ± 0.558

13.508 ± 2.745

\(\mathbf{11.747} \pm \mathbf{1.005}\)

RMSE (%p)

20.463 ± 0.623

17.151 ± 0.677

17.384 ± 0.775

17.992 ± 2.530

\(\mathbf{16.586} \pm \mathbf{1.364}\)

R\(^2\)

0.436 ± 0.034

0.604 ± 0.031

0.593 ± 0.037

0.556 ± 0.130

\(\mathbf{0.628} \pm \mathbf{0.062}\)

Spearman \(\rho\)

\(\mathbf{0.381} \pm \mathbf{0.038}\)

0.016 ± 0.067

0.309 ± 0.176

0.300 ± 0.075

Suzuki-Miyaura

70/30

MAE (%p)

8.128 ± 0.344

6.598 ± 0.270

6.233 ± 0.207

6.118 ± 0.212

\(\mathbf{6.116} \pm \mathbf{0.223}\)

RMSE (%p)

12.073 ± 0.463

10.524 ± 0.482

9.522 ± 0.454

9.495 ± 0.430

\(\mathbf{9.467} \pm \mathbf{0.459}\)

R\(^2\)

0.815 ± 0.013

0.859 ± 0.012

0.885 ± 0.010

0.885 ± 0.009

\(\mathbf{0.886} \pm \mathbf{0.010}\)

Spearman \(\rho\)

\(\mathbf{0.439} \pm \mathbf{0.018}\)

0.324 ± 0.026

0.432 ± 0.024

0.425 ± 0.026

50/50

MAE (%p)

8.922 ± 0.235

7.539 ± 0.153

6.872 ± 0.089

\(\mathbf{6.702} \pm \mathbf{0.082}\)

6.725 ± 0.089

RMSE (%p)

13.148 ± 0.270

11.797 ± 0.250

10.272 ± 0.138

\(\mathbf{10.225} \pm \mathbf{0.128}\)

\(\mathbf{10.225} \pm \mathbf{0.135}\)

R\(^2\)

0.780±0.009

0.823 ± 0.007

0.866 ± 0.003

\(\mathbf{0.867} \pm \mathbf{0.003}\)

\(\mathbf{0.867} \pm \mathbf{0.003}\)

Spearman \(\rho\)

\(\mathbf{0.439} \pm \mathbf{0.019}\)

0.322 ± 0.021

0.432 ± 0.017

0.430 ± 0.012

30/70

MAE (%p)

10.094 ± 0.346

8.804 ± 0.249

8.021 ± 0.094

\(\mathbf{7.740} \pm \mathbf{0.109}\)

7.847 ± 0.094

RMSE (%p)

14.614 ± 0.381

13.337 ± 0.357

11.726 ± 0.152

\(\mathbf{11.526} \pm \mathbf{0.166}\)

11.593 ± 0.136

R\(^2\)

0.729 ± 0.014

0.774 ± 0.012

0.825 ± 0.004

\(\mathbf{0.831} \pm \mathbf{0.005}\)

0.829 ± 0.004

Spearman \(\rho\)

\(\mathbf{0.432} \pm \mathbf{0.018}\)

0.292 ± 0.012

0.428 ± 0.013

0.417 ± 0.008

20/80

MAE (%p)

11.229 ± 0.247

10.017 ± 0.338

9.147 ± 0.185

\(\mathbf{8.726} \pm \mathbf{0.172}\)

8.793 ± 0.191

RMSE (%p)

15.966 ± 0.381

14.851 ± 0.576

13.115 ± 0.298

12.754 ± 0.316

\(\mathbf{12.734} \pm \mathbf{0.347}\)

R\(^2\)

0.676 ± 0.015

0.719 ± 0.022

0.781 ± 0.010

0.793 ± 0.010

\(\mathbf{0.794} \pm \mathbf{0.011}\)

Spearman \(\rho\)

\(\mathbf{0.432} \pm \mathbf{0.014}\)

0.274 ± 0.020

0.429 ± 0.017

0.408 ± 0.018

10/90

MAE (%p)

13.528 ± 0.395

11.954 ± 0.443

11.439 ± 0.185

\(\mathbf{10.625} \pm \mathbf{0.249}\)

10.739 ± 0.211

RMSE (%p)

18.734 ± 0.530

17.129 ± 0.683

15.967 ± 0.326

\(\mathbf{15.097} \pm \mathbf{0.421}\)

15.164 ± 0.344

R\(^2\)

0.554 ± 0.025

0.627 ± 0.030

0.676 ± 0.013

\(\mathbf{0.711} \pm \mathbf{0.016}\)

0.708 ± 0.013

Spearman \(\rho\)

0.389 ± 0.022

0.221 ± 0.027

\(\mathbf{0.390} \pm \mathbf{0.019}\)

0.382 ± 0.019

5/95

MAE (%p)

15.695 ± 0.618

14.294 ± 0.507

14.214 ± 0.504

\(\mathbf{13.364} \pm \mathbf{0.223}\)

13.451 ± 0.353

RMSE (%p)

21.181 ± 0.724

20.016 ± 0.661

19.421 ± 0.588

\(\mathbf{18.463} \pm \mathbf{0.308}\)

18.511 ± 0.392

R\(^2\)

0.430 ± 0.040

0.491 ± 0.034

0.521 ± 0.029

\(\mathbf{0.567} \pm \mathbf{0.014}\)

0.565 ± 0.018

Spearman \(\rho\)

0.355 ± 0.026

0.144 ± 0.052

\(\mathbf{0.389} \pm \mathbf{0.045}\)

0.330 ± 0.034

2.5/97.5

MAE (%p)

17.666 ± 0.496

17.587 ± 0.690

18.061 ± 0.571

\(\mathbf{16.705} \pm \mathbf{1.090}\)

17.189 ± 0.813

RMSE (%p)

22.967 ± 0.804

23.780 ± 0.793

24.121 ± 0.655

\(\mathbf{22.156} \pm \mathbf{1.273}\)

22.943 ± 0.887

R\(^2\)

0.330 ± 0.047

0.282 ± 0.047

0.261 ± 0.039

\(\mathbf{0.375} \pm \mathbf{0.072}\)

0.331 ± 0.051

Spearman \(\rho\)

\(\mathbf{0.291} \pm \mathbf{0.025}\)

0.028 ± 0.054

0.280 ± 0.074

0.223 ± 0.081