Designing Imaging Systems with Mutual Information: A Practical Tutorial

Overview

Modern imaging systems—from smartphone cameras to medical MRI scanners—often produce measurements that no human ever sees directly. Instead, algorithms convert raw sensor data into interpretable outputs: your phone's image pipeline, an MRI reconstruction, or a self-driving car's perception stack. What truly matters is not how the measurements look, but how much useful information they contain. This tutorial explains a framework that directly evaluates and optimizes imaging hardware based on mutual information—a single number that captures the system's ability to distinguish objects, regardless of how the measurements appear.

Designing Imaging Systems with Mutual Information: A Practical Tutorial — Source: bair.berkeley.edu

Traditional metrics like resolution, SNR, and sampling rate treat quality factors separately, making it hard to trade off between them. Training an end-to-end neural network for reconstruction lumps hardware and algorithm quality together. Our method, presented at NeurIPS 2025, estimates mutual information from noisy measurements alone plus a noise model. It predicts performance across four imaging domains and produces designs that match state-of-the-art end-to-end methods while requiring less memory, less compute, and no task-specific decoder.

Prerequisites

Before diving into the tutorial, ensure you have:

Basic knowledge of imaging physics (lenses, sensors, noise sources).
Familiarity with information theory (mutual information, entropy).
Programming skills in Python (NumPy, PyTorch/TensorFlow).
Optional but helpful: experience with neural network-based mutual information estimators (e.g., MINE, InfoNCE).

Step-by-Step Guide

1. Define the Imaging System Model

An imaging system consists of an encoder (optical system) that maps objects to noiseless images, and a noise source that corrupts those images into measurements. Formally, let X be the object (high-dimensional variable), Y the noiseless image, and Z the noisy measurement. We assume a known noise model p(Z|Y) (e.g., Poisson for photon counting, Gaussian for thermal noise). The goal is to compute I(X; Z)—the mutual information between objects and measurements.

Example code snippet for a simple camera model (using PyTorch):

import torch
import torch.nn as nn

class Encoder(nn.Module):
    def __init__(self):
        super().__init__()
        # Placeholder: real complexity is in optical design
        self.conv = nn.Conv2d(1, 1, kernel_size=5, padding=2, bias=False)
    def forward(self, x):
        return self.conv(x)  # noiseless image

class NoiseModel:
    def __init__(self, std=0.1):
        self.std = std
    def corrupt(self, y):
        return y + torch.randn_like(y) * self.std

2. Choose a Noise Model

Mutual information estimation relies on knowing p(Z|Y). In practice, you must calibrate your sensor to derive an accurate noise model. Common choices:

Gaussian noise: additive, independent with known variance.
Poisson noise: shot noise from photon counting, variance proportional to signal.
Mixture models: e.g., Poisson+Gaussian for scientific cameras.

Incorrect noise models lead to severely over- or under-estimated information.

3. Estimate Mutual Information from Measurements

Directly estimating I(X;Z) for high-dimensional X is notoriously difficult. Our framework uses a neural mutual information estimator that only requires noisy measurements and a noise model—no explicit object distribution needed. The estimator learns to discriminate between pairs (X,Z) that come from the joint distribution vs. product of marginals.

One robust approach is the InfoNCE (NT-Xent) loss, commonly used in contrastive learning:

def info_nce_loss(z_joint, z_marginal, temperature=0.1):
    # z_joint: batch of measurements from same object
    # z_marginal: batch of measurements from different objects
    logits = torch.cat([z_joint @ z_joint.T, z_joint @ z_marginal.T], dim=1)
    labels = torch.zeros(z_joint.size(0), dtype=torch.long)
    return nn.CrossEntropyLoss()(logits / temperature, labels)

Important: This estimator requires samples from both the joint distribution (measurement and its originating object) and the marginal (random measurement with random object). For real systems, you can generate these by simulating objects and passing them through the encoder + noise model.

4. Optimize the Imaging System

With an estimator in place, you can treat the encoder parameters (e.g., lens curvature, aperture size, sensor gain) as differentiable and maximize the estimated mutual information via gradient ascent. Because the estimator itself is a neural network, you can backpropagate through it:

def optimize_encoder(encoder, noise_model, num_steps=1000):
    opt = torch.optim.Adam(encoder.parameters(), lr=1e-3)
    for step in range(num_steps):
        objects = sample_objects()  # e.g., from a dataset
        y = encoder(objects)
        z = noise_model.corrupt(y)
        mi_estimate = info_nce_loss(z, shuffle(z))
        loss = -mi_estimate  # maximize MI
        loss.backward()
        opt.step()
    return encoder

This process directly maximizes information content without needing a separate decoder or reconstruction task.

5. Validate with Downstream Tasks

After optimization, verify that the improved mutual information translates to real-world performance. For example, train a simple classifier on the measurements (or reconstructed images) and compare accuracy. The original paper shows consistent gains across fluorescence microscopy, celestial imaging, face recognition, and MRI under different noise conditions.

Common Mistakes

Using mismatched noise models: Overly simple noise models (e.g., pure Gaussian when real detector has Poisson) inflate MI estimates and lead to poor hardware designs. Always validate noise parameters with calibration data.
Ignoring physical constraints: Mutual information can be artificially high if you allow unrealistically large apertures or zero read noise. In practice, lens size, sensor saturation, and power budget must bound the design space.
Assuming independent objects: The estimator assumes objects are drawn from some distribution. If object distribution is too peaked (e.g., all similar), MI will be small; if too broad, the task may be misaligned with your application. Choose a representative object prior.
Neglecting computational cost: Neural MI estimators can be biased if the critic network is too small or training insufficient. Use cross-validation and multiple random seeds to check stability.

Summary

This tutorial demonstrated how to replace traditional imaging quality metrics with mutual information—a single, task-agnostic metric that accounts for resolution, noise, and sampling in a unified way. By estimating I(X;Z) from noisy measurements and a known noise model, engineers can evaluate and optimize hardware designs without task-specific algorithms. The method is implemented via neural estimators (e.g., InfoNCE), requires no human-interpretable outputs, and has been validated across multiple domains. If you are designing imaging systems for AI-driven applications, this information-driven approach will give you a direct handle on hardware quality that traditional metrics miss.