DEUE: Delta Ensemble Uncertainty Estimation for Robust EF Assessment

Reliable uncertainty estimation is critical for deploying deep learning models in high-stakes clinical tasks like Left Ventricular Ejection Fraction (LVEF) estimation from echocardiography. Our paper introduces DEUE (Delta Ensemble Uncertainty Estimation) — a sampling-free, architecture-agnostic method to estimate epistemic (model) uncertainty for regression tasks with just one forward pass (plus one lightweight backward pass) at inference.

The Problem with Current Approaches

Existing epistemic uncertainty estimators each come with significant trade-offs:

• Deep Ensembles: Well-calibrated but require storing and running multiple models per prediction
• MC Dropout: Needs many forward passes, increasing latency substantially
• Bayesian Methods: Often computationally expensive or inaccurate for modern large networks

For clinical deployment, we need:

1. Single-pass inference for real-time performance
2. Minimal extra memory requirements
3. Architecture independence to work with existing models
4. Calibrated uncertainty that correlates with prediction error and input quality

Our Solution: DEUE

DEUE combines two key ideas:

1. The Delta Method to propagate parameter uncertainty to output uncertainty
2. Lightweight ensemble usage to estimate per-parameter variances (diagonal covariance only)

Unlike traditional deep ensembles, DEUE doesn't average predictions or rely on output disagreements between models.

How It Works

Step 1: Train Multiple Models

• Train M identical architectures (we used ResNet18 (2+1)D, M=5) with different random initializations
• Use standard supervised training with no architectural changes

Step 2: Estimate Parameter Variances

• For each parameter w, compute empirical variance across the M trained models:

$${\hat{\Sigma }}_{jj}=\frac{1}{M-1}\sum _{i=1}^M(W_{i,j}-{\bar{W}}_j{)}^2$$

where ${\bar{W}}_j=\frac{1}{M}{\sum }_{i=1}^M{W}_{i,j}$ is the empirical mean.

• This gives us a diagonal covariance matrix $\hat{\Sigma }$ (inexpensive to store)

Step 3: Single-Model Inference

• Select the best performing model as reference
• For input $x$, compute gradient $\mathbf{g}={\nabla }_W{f}_W(x)$ via one backward pass
• Apply the Delta Method (display equation below).

$$U(x)\approx {\mathbf{g}}^{\top }\hat{\Sigma }\mathbf{g}$$

This scalar $U(x)$ is the epistemic uncertainty estimate.

Why This Is Efficient

Method	Storage	Runtime	Applicability
Deep Ensembles	Multiple full models	K forward passes	General
MC Dropout	Single model	~30 forward passes	Architecture-dependent
DEUE	Reference model + diagonal variances	1 forward + 1 backward	Architecture-agnostic

Experimental Results

We evaluated DEUE on the EchoNet-Dynamic dataset (10,031 AP4 echo videos) for ejection fraction regression:

• Training set: 7,465 videos
• Validation set: 1,288 videos
• Test set: 1,277 videos
• Architecture: ResNet18 (2+1)D
• Task: Single-value EF regression (0-100%)

Uncertainty Calibration

We compared DEUE against established baselines:

• Deep Ensembles (DE): 5 models with multi-forward inference
• Deep Bayesian Neural Network (DBNN): Variational approach
• Single Deterministic Regression (DDR): No epistemic uncertainty

Key Finding: DEUE's uncertainty calibration closely matched Deep Ensembles and DBNN. When we ranked test samples by uncertainty and progressively rejected the most uncertain predictions, all three methods showed similar RMSE reduction curves.

Qualitative Results

DEUE successfully identified challenging cases:

• High uncertainty: Off-axis views, incorrect color maps, heavy zoom
• Low uncertainty: Clean AP4 views with good image quality

This aligns perfectly with epistemic uncertainty semantics — the model is uncertain about cases that differ from its training distribution.

Clinical Impact

DEUE enables several clinical applications:

Quality Control

• Flag uncertain predictions for expert review or repeat acquisition
• Prioritize manual review of cases with high uncertainty
• Improve acquisition protocols by identifying common failure modes

Resource Optimization

• Triage critical cases requiring immediate attention
• Focus labeling efforts on persistently uncertain examples
• Optimize workflow by automating confident predictions

Risk Stratification

• Combine with clinical scores for comprehensive assessment
• Identify edge cases that may need additional testing
• Support clinical decision-making with confidence intervals

Future Directions

Enhanced Uncertainty Modeling

• Combine epistemic (DEUE) with aleatoric uncertainty for complete predictive intervals
• Explore gradient compression techniques for even lower latency
• Extend to multi-output regression tasks (e.g., chamber volumes)

Advanced Applications

• Apply to structured prediction tasks like segmentation
• Develop uncertainty-aware active learning strategies
• Create uncertainty calibration metrics for continuous clinical indices

Clinical Validation

• Conduct prospective studies with real clinical workflows
• Validate uncertainty thresholds for different risk categories
• Assess impact on diagnostic accuracy and efficiency

Technical Details

Mathematical Foundation

Problem Definition

For a deep learning algorithm $\mathcal{L}$ that maps dataset $D$ to parametric function $f_W$:

$$f_W=\mathcal{L}(D_{X,Y})\text{\hspace{1em}(1)}$$

We define uncertainty at point $x$ as expected squared prediction error:

$$U_{\mathcal{L}}(x):=\mathbb{E}[(f_W(x)-y{)}^2]=\int (f_W(x)-y{)}^{2}d{\mathcal{P}}_{(Y|x)}\text{\hspace{1em}(2)}$$

Delta Method Approximation

Using first-order Taylor expansion around optimal parameters $W^{\ast }$:

$$f_W(x)\simeq f_{W^{\ast }}(x)+g_{W^{\ast },x}^{\top }(W-W^{\ast })\text{\hspace{1em}(3)}$$

where $g_{W^{\ast },x}$ is the gradient ${\nabla }_W{f}_W(x)$ evaluated at $W^{\ast }$.

This leads to our uncertainty approximation:

$$U_{\mathcal{L}}(x)\simeq g_{W^{\ast },x}^{\top }\Sigma g_{W^{\ast },x}\text{\hspace{1em}(4)}$$

where $\Sigma$ is the parameter covariance matrix.

DEUE Implementation

Since computing full $\Sigma$ is intractable for deep networks, DEUE approximates it using ensemble statistics:

$$U_{\mathcal{L}}(x)\simeq g_{W,x}^{\top }{\hat{\Sigma }}_M{g}_{W,x}\text{\hspace{1em}(5)}$$

Where:

• $g_{W,x}$ is the gradient computed via backpropagation
• ${\hat{\Sigma }}_M$ is diagonal covariance from $M$ ensemble models
• $W$ represents the best-performing model parameters

Practical Algorithm

The algorithm implementation follows these concrete steps:

1. Select the best performing model as reference
2. For input x, compute gradient $\mathbf{g}={\nabla }_W{f}_W(x)=\partial f_W(x)/\partial W$ via one backward pass
3. Apply Delta Method approximation:

$$U(x)\approx {\mathbf{g}}^{\top }\hat{\Sigma }\mathbf{g}\text{\hspace{1em}(6)}$$

where $\mathbf{g}$ is the gradient vector and $\hat{\Sigma }$ is our pre-computed diagonal covariance matrix.

Implementation Considerations

• Memory: Store only diagonal covariance elements (one float per parameter)
• Computation: Single forward pass + backward pass for gradients
• Scalability: Linear in number of parameters, constant in ensemble size at inference

Conclusion

DEUE provides a practical solution for epistemic uncertainty estimation in clinical deep learning. By leveraging gradient-based uncertainty propagation with lightweight ensemble statistics, we achieve uncertainty quality comparable to expensive methods while maintaining single-model efficiency.

Key advantages:

• ✅ Single forward pass + one backward pass at inference
• ✅ Minimal additional memory requirements
• ✅ Architecture-agnostic approach
• ✅ Well-calibrated uncertainty estimates
• ✅ Clinically meaningful uncertainty patterns

This work represents an important step toward reliable uncertainty quantification in medical AI, enabling safer deployment of deep learning models in clinical practice.

---

This research was conducted at the University of British Columbia (UBC) in collaboration with Vancouver General Hospital.

Citation: Kazemi Esfeh, M.M., Gholami, Z., Luong, C., Tsang, T., Abolmaesumi, P. "DEUE: Delta Ensemble Uncertainty Estimation for a More Robust Estimation of Ejection Fraction."

Questions or collaboration interests? See the About page for contact details.