torchuq.transform Subpackage¶

This section contains the Python API reference for the torchuq.transform subpackage, containing code for uncertainty transformations.

torchuq.transform.basic Module¶

class torchuq.transform.basic.Calibrator(input_type='auto')¶

The abstract base class for all calibrator classes.

Parameters: input_type (str) – the input prediction type. If input_type is ‘auto’ then it is automatically induced when Calibrator.train() or update() is called, it cannot be changed after the first call to train() or update(). Not all sub-classes support ‘auto’ input_type, so it is strongly recommended to explicitly specify the prediction type.

__init__(input_type='auto')¶

check_type(predictions)¶

Checks that the prediction has the correct shape specified by input_type.

Parameters: predictions (prediction object) – a batched prediction object, must match the input_type argument when calling __init__.

to(device)¶

Move this class and all the tensors it owns to a specified device.

Parameters: device (torch.device) – the device to move this class to.

train(predictions, labels, *args, **kwargs)¶

The train abstract class. Learn the recalibration map based on labeled data.

This function uses the training data to learn any parameters that is necessary to transform a low quality (e.g. uncalibrated) prediction into a higher quality (e.g. calibrated) prediction. It takes as input a set of predictions and the corresponding labels. In addition, a few recalibration algorithms — such as group calibration or multicalibration — can take as input additional side features, and the transformation depends on the side feature.

Parameters

predictions (object) – a batched prediction object, must match the input_type argument when calling __init__.
labels (tensor) – the labels with shape [batch_size]
side_feature (tensor) – some calibrator instantiations can use additional side feature, when used it should be a tensor of shape [batch_size, n_features]

Returns

an optional log object that contains information about training history.

Return type

object

update(predictions, labels, *args, **kwargs)¶

Same as Calibrator.train, but updates the calibrator online with the new data (while train erases any existing data in the calibrator and learns it from scratch)

Parameters

predictions (object) – a batched prediction object, must match the input_type argument when calling __init__.
labels (tensor) – the labels with shape [batch_size]
side_feature (tensor) – some calibrator instantiations can use additional side feature, when used it should be a tensor of shape [batch_size, n_features]

Returns

an optional log object that contains information about training history.

Return type

object

class torchuq.transform.basic.ConcatDistribution(distributions, dim=0)¶

Class that concat multiple distribution instances.

torch.distributions.Distribution does not yet have a concatenation function (as of 1.1), making it difficult to concate two distribution instances similar to concating two torch tensors. This class fills this gap by concating multiple distribution instances into a single class that behaves like torch.distributions.Distribution.

This class supports a subset of functions for torch.distributions.Distribution, including cdf, icdf, log_prob, sample, rsample, sample_n.

Parameters

distributions (list) – a list of torch Distribution instances.
dim – dimension to concat the distributions, any dimension other than the concat dimension must have equal size

__init__(distributions, dim=0)¶

cdf(value)¶

Returns the cumulative density (CDF) evaluated at value.

Parameters: value (tensor) – the values to evaluate the CDF.
Returns: the evaluated CDF.
Return type: tensor

icdf(value)¶

Returns the inverse cumulative density (ICDF) evaluated at value.

Parameters: value (tensor) – the values to evaluate the ICDF.
Returns: the evaluated ICDF.
Return type: tensor

log_prob(value)¶

Returns the log of the probability density evaluated at value.

Parameters: value (tensor) – the values to evaluate the ICDF.
Returns: the evaluated log_prob.
Return type: tensor

rsample(sample_shape=torch.Size([]))¶

Generates a sample_shape shaped (batch of) sample.

Parameters: sample_shape (torch.Size) – the shape of the samples.
Returns: the drawn samples.
Return type: tensor

sample_n(n)¶

Generates n batches of samples.

Parameters: n (int) – the number of batches of samples.
Returns: the drawn samples.
Return type: tensor

to(device)¶

Move this class and all the tensors it owns to a specified device.

Parameters: device (torch.device) – the device to move this class to.

class torchuq.transform.basic.DistributionBase¶

Abstract baseclass for a distribution that arises from conformal calibration.

This class behaves like torch.distribution.Distribution, and supports the cdf, icdf and rsample functions.

__init__()¶

log_prob(value)¶: Compute the log probability. This default implementation is not great as it is numerically unstable and require tricks to not throw faults.

rsample(sample_shape=torch.Size([]))¶: Draw a set of samples from the distribution

shape_inference(value)¶: Handle all unusual shapes

torchuq.transform.calibrate Module¶

class torchuq.transform.calibrate.DirichletCalibrator(verbose=False)¶

The class to recalibrate a categorical prediction with dirichlet calibration

This implements the dirichlet calibration algorithm with ODIR regularization. Dirichlet calibration is a method proposed to achieve classwise calibration. For details see https://arxiv.org/abs/1910.12656.

Parameters: verbose (bool) – if verbose=True print non-error messsages

__init__(verbose=False)¶

to(device)¶

Move all assets of this class to a torch device.

Parameters: device (device) – the torch device (such as torch.device(‘cpu’))

train(predictions, labels, param_lambda=0.001, param_mu=0.1, max_epochs=1000, *args, **kwargs)¶

Train the Dirichlet recalibration map

Parameters

predictions (tensor) – a batch of categorical predictions with shape [batch_size, num_classes]
labels (tensor) – a batch of labels with shape [batch_size]
param_lambda (float) – the regularization hyper-parameter. According to E.3 of https://arxiv.org/pdf/1910.12656.pdf, the best hyper-parameter is usually 1e-3
param_mu (float) – the regularization hyper-parameter.
max_epochs (int) – the maximum number of epochs to train the model. This function might terminate when training makes no additional progress before max_epochs is reached

Returns

a PerformanceRecord instance with detailed training log

Return type

PerformanceRecord

class torchuq.transform.calibrate.HistogramBinning(adaptive=True, bin_count='auto', verbose=False)¶

The class to recalibrate a categorical prediction with histogram binning.

The histogram binning algorithm partitions the samples into bins, computes the average confidence and the average accuracy in each bin, and increases / decreases the confidence of the samples to match the average accuracy.

Parameters

adaptive (bool) – if adaptive is true, use the same number of samples per bin, if adaptive if false, use equal width bins
bin_count (int or str) – the number of bins, if ‘auto’ set the number of bins automatically as sqrt(number of samples) / 5
verbose (bool) – if verbose=True print non-error messsages

__init__(adaptive=True, bin_count='auto', verbose=False)¶

to(device)¶

Move all assets of this class to a torch device.

Parameters: device (device) – the torch device (such as torch.device(‘cpu’))

train(predictions, labels, *args, **kwargs)¶

Train the recalibration map

Parameters

predictions (tensor) – a batch of categorical predictions with shape [batch_size, num_classes]
labels (tensor) – a batch of labels with shape [batch_size]

class torchuq.transform.calibrate.TemperatureScaling(verbose=False)¶

The class to recalibrate a categorical prediction with temperature scaling

Temeprature scaling is often the algorithm of choice when calibrating predictions from deep neural networks. It learns the following map from the original predictions to the new prediction: $p mapsto mathrm{softmax}(log p / T)$. The only learnable parameter — the temperature parameter $T$ — is tuned to maximize the log-likelihood of the labels. Temperature scaling requires very few samples to train because it only learns a single parameter $T$, yet despite the simplcity, empirical results show that temperature scaling achieves low calibration error when applied to deep network predictions.

Parameters: verbose (bool) – if verbose=True print detailed messsages

__init__(verbose=False)¶

to(device)¶

Move all assets of this class to a torch device.

Parameters: device (device) – the torch device (such as torch.device(‘cpu’))

train(predictions, labels, *args, **kwargs)¶

Find the optimal temperature with gradient descent.

Parameters

predictions (tensor) – a batch of categorical predictions with shape [batch_size, num_classes]
labels (tensor) – a batch of labels with shape [batch_size]

torchuq.transform.conformal Module¶

class torchuq.transform.conformal.ConformalBase(input_type='interval', score_func=0, verbose=False)¶

The base class for both ConformalCalibrator and ConformalIntervalPredictor

Parameters

input_type (str) – the input prediction type, currently supports point, interval, particle, quantile, distribution or ensemble.
score_func (int) – for certain prediction types there are multiple conformal score options. In this case they will be numbered 0, 1, …
verbose (bool) – if verbose=True then print extra messages.

__init__(input_type='interval', score_func=0, verbose=False)¶

to(device)¶

Move every torch tensor owned by this class to a new device

Parameters: device – a torch.device instance, alternatively it could be a torch.Tensor or a prediction object

train(predictions, labels)¶

Train the conformal calibration from scratch

Parameters

predictions (object) – a batch of original predictions. Must have the correct type that matches the input_type argument
labels (tensor) – a batch of labels, must be on the same device as predictions

update(predictions, labels)¶

Update the conformal calibrator online with new labels

Parameters

predictions (object) – a batch of original predictions. Must have the correct type that matches the input_type argument
labels (tensor) – a batch of labels, must be on the same device as predictions

class torchuq.transform.conformal.ConformalCalibrator(input_type='interval', interpolation='linear', score_func=0, verbose=False)¶

Transforms any prediction type into calibrated probabilities.

Parameters

input_type (str) – the input prediction type, currently supports point, interval, particle, quantile, distribution or ensemble.
interpolation (str) – ‘linear’, ‘random’ or ‘naf’, the interpolation used when computing the CDF/ICDF functions. NAF is slow but produces smoother CDFs. Random has better calibration error (it should have perfect calibration) but has non-continuous CDF. Linear achieves good trade-off between speed, smoothness of CDF and calibration error.
score_func (int) – for certain prediction types there are multiple conformal score options. In this case they will be numbered 0, 1, …
verbose (bool) – if verbose=True then print extra messages.

__init__(input_type='interval', interpolation='linear', score_func=0, verbose=False)¶

class torchuq.transform.conformal.ConformalIntervalPredictor(input_type='interval', coverage='exact', score_func=0, confidence=0.95, verbose=False)¶

Use conformal prediction to transform any prediction into intervals with correct coverage.

Parameters

input_type (str) – the input prediction type, currently supports point, interval, particle, quantile, distribution or ensemble.
coverage (str) – the coverage can be ‘exact’ or ‘1/n’. If the coverage is exact, then the algorithm can output [-inf, +inf] intervals
score_func (int) – for certain prediction types there are multiple conformal score options. In this case they will be numbered 0, 1, …
confidence (float) – the desired coverage. For example, when confidence=0.95 the label should belong to the predicted interval 95% of the times.
verbose (bool) – if verbose=True then print extra messages.

__init__(input_type='interval', coverage='exact', score_func=0, confidence=0.95, verbose=False)¶

class torchuq.transform.conformal.DistributionConformal(val_predictions, val_labels, test_predictions, score_func, iscore_func)¶

Abstract base class for a distribution that arises from conformal calibration. This class behaves like torch.distribution.Distribution, and supports the cdf, icdf and rsample functions.

Parameters

val_predictions – a set of validation predictions, the type must be compatible with score_func
val_labels – array [validation_batch_shape], a batch of labels
test_predictions – a set of test predictions, the type must be compatible with score func
score_func – non-conformity score function. A function that take as input a batched predictions q, and an array v of values with shape [n_evaluations, batch_shape] returns an array s of shape [n_evaluations, batch_shape] where s_{ij} is the score of v_{ij} under prediction q_j
iscore_func – inverse non-conformity score function: a function that take as input a batched prediction q, and an array s os scores with shape [n_evaluations, batch_shape] returns an array v of shape [n_evaluations, batch_shape] which is the inverse of score_func (i.e. iscore_func(q, score_func(q, v)) = v)

__init__(val_predictions, val_labels, test_predictions, score_func, iscore_func)¶

log_prob(value)¶

Returns the log of the probability density evaluated at value.

Parameters: value (tensor) – the values to evaluate the ICDF.
Returns: the evaluated log_prob.
Return type: tensor

rsample(sample_shape=torch.Size([]))¶

Generates a sample_shape shaped (batch of) sample.

Parameters: sample_shape (torch.Size) – the shape of the samples.
Returns: the drawn samples.
Return type: tensor

sample(sample_shape=torch.Size([]))¶

Generates a sample_shape shaped (batch of) sample.

Parameters: sample_shape (torch.Size) – the shape of the samples.
Returns: the drawn samples.
Return type: tensor

sample_n(n)¶

Generates n batches of samples.

Parameters: n (int) – the number of batches of samples.
Returns: the drawn samples.
Return type: tensor

shape_inference(value)¶: Change the shape of the input into a canonical shape

to(device)¶

Move this class and all the tensors it owns to a specified device.

Parameters: device (torch.device) – the device to move this class to.

class torchuq.transform.conformal.DistributionConformalLinear(val_predictions, val_labels, test_predictions, score_func, iscore_func, verbose=False)¶

Use linear interpolation for conformal calibration.

__init__(val_predictions, val_labels, test_predictions, score_func, iscore_func, verbose=False)¶

cdf(value)¶

The CDF at value. This function is differentiable

Parameters: value (tensor) – an array of shape [n_evaluations, batch_shape] or shape [batch_size].
Returns: the value of CDF at the queried values.
Return type: tensor

icdf(value)¶

Get the inverse CDF. This function is differentiable.

Parameters: value (tensor) – an array of shape [n_evaluations, batch_shape] or shape [batch_shape], each entry should take values in [0, 1] Supports automatic shape induction, e.g. if cdf has shape [n_evaluations, 1] it will automatically be converted to shape [n_evaluations, batch_shape]
Returns: the value of inverse CDF function at the queried cdf values
Return type: tensor

class torchuq.transform.conformal.DistributionConformalNAF(val_predictions, val_labels, test_predictions, score_func, iscore_func, verbose=True)¶

Using NAF interpolation for conformal calibration. This function behaves like torch Distribution.

__init__(val_predictions, val_labels, test_predictions, score_func, iscore_func, verbose=True)¶

cdf(value)¶

The CDF at value. This function is differentiable

Parameters: value (tensor) – an array of shape [n_evaluations, batch_shape] or shape [batch_size].
Returns: the value of CDF at the queried values.
Return type: tensor

icdf(value)¶

Get the inverse CDF. This function is differentiable.

Parameters: value (tensor) – an array of shape [n_evaluations, batch_shape] or shape [batch_shape], each entry should take values in [0, 1] Supports automatic shape induction, e.g. if cdf has shape [n_evaluations, 1] it will automatically be converted to shape [n_evaluations, batch_shape]
Returns: the value of inverse CDF function at the queried cdf values
Return type: tensor

class torchuq.transform.conformal.DistributionConformalRandom(val_predictions, val_labels, test_predictions, score_func, iscore_func, verbose=False)¶

Use randomization for conformal calibration.

This distribution does not have a differentiable CDF (i.e. it does not have a density), so the behavior of functions such as log_prob and plot_density are undefined.

__init__(val_predictions, val_labels, test_predictions, score_func, iscore_func, verbose=False)¶

cdf(value)¶

The CDF at value. This function is NOT differentiable

Parameters: value (tensor) – an array of shape [n_evaluations, batch_shape] or shape [batch_size].
Returns: the value of CDF at the queried values.
Return type: tensor

icdf(value)¶

Get the inverse CDF. This function is NOT differentiable

Parameters: value (tensor) – an array of shape [n_evaluations, batch_shape] or shape [batch_shape], each entry should take values in [0, 1] Supports automatic shape induction, e.g. if cdf has shape [n_evaluations, 1] it will automatically be converted to shape [n_evaluations, batch_shape]
Returns: the value of inverse CDF function at the queried cdf values
Return type: tensor

torchuq.transform.direct Module¶

class torchuq.transform.direct.DistributionKDE(loc, scale, weight=None)¶

A mixture of Gaussian distribution

Even though pytorch has native support for mixture distribution, it does not have an icdf method which is crucial and difficult to implement This implementation includes an icdf function

Parameters

loc (tensor) – array [batch_size, n_components], the set of centers of the Gaussian distributions
scale (tensor) – array [batch_size, n_components], the set of stddev of the Gaussian distributions
weight (tensor) – array [batch_size, n_components], the weight of each mixture component. If set to None then all mixture components have the same weight

__init__(loc, scale, weight=None)¶

cdf(value)¶

Computes the cumulative density evaluated at value

Parameters: value (tensor) – an array of shape [batch_size], [1] or [num_cdfs, batch_size]
Returns: the evaluated CDF.
Return type: tensor

icdf(value)¶

Computes the inverse cumulative density evaluated at value

Parameters: value (tensor) – an array of shape [batch_size], [1] or [num_cdfs, batch_size]
Returns: the evaluated inverse CDF.
Return type: tensor

log_prob(value)¶

Computes the log likelihood evaluated at value

Parameters: value (tensor) – an array of shape [batch_size], [1] or [num_cdfs, batch_size]
Returns: the evaluated log probability.
Return type: tensor

torchuq.transform.direct.categorical_to_topk(predictions, k=1)¶

Convert a categorical prediction to a a top-k prediction by taking the k indices with largest probability

This function is not differentiable.

Parameters

predictions (tensor) – a batch of categorical predictions.
k (int) – the number of predicted labels.

Returns

the topk prediction.

Return type

tensor

torchuq.transform.direct.categorical_to_uset(predictions, threshold=0.95)¶

Convert a categorical prediction to a set prediction by taking the labels with highest probability until their total probability exceeds threshold

This function is not differentiable.

Parameters

predictions (tensor) – a batch of categorical predictions.
threshold (float) – the minimum probability of the confidence set.

Returns

the set prediction.

Return type

tensor

torchuq.transform.direct.distribution_to_interval(predictions, confidence=0.95)¶

Convert a distribution prediction to an interval prediction by finding the smallest interval

Parameters

predictions (Distribution) – a batch of distribution predictions
confidence (float) – the probability of the credible interval.

Returns

a batch of interval predictions

Return type

tensor

torchuq.transform.direct.distribution_to_particle(predictions, n_particles=50)¶

Convert a distribution prediction to a particle prediction

Parameters

predictions (Distribution) – a batch of distribution predictions.
n_particles (int) – the number of particles to sample.

torchuq.transform.direct.distribution_to_point(predictions, functional='mean')¶

Convert a distribution prediction to a point prediction by taking the mean or the median

Parameters

predictions (Distribution) – a batch of distribution predictions.
functional (str) – can be ‘mean’ or ‘median’.

Returns

a batch of point predictions.

Return type

tensor

torchuq.transform.direct.distribution_to_quantile(predictions, quantiles=None, n_quantiles=None)¶

Convert a distribution prediction to a quantile prediction

Parameters

predictions (Distribution) – a batch of distribution predictions
quantiles (tensor) – a set of quantiles. One and only one of quantiles or n_quantiles can be specified (the other should be None).
n_quantiles (int) – the number of quantiles.

Returns

a batch of quantile predictions.

Return type

tensor

torchuq.transform.direct.ensemble_to_distribution(predictions)¶

Convert an ensemble prediction to a distribution prediction.

This is based on the conversion scheme described in https://papers.nips.cc/paper/2017/file/9ef2ed4b7fd2c810847ffa5fa85bce38-Paper.pdf

Parameters: predictions – an ensemble prediction.
Returns: a batch of distribution predictions.
Return type: Distribution

torchuq.transform.direct.interval_to_point(predictions)¶

Converts an interval prediction to a point prediction by taking the middle of the interval.

Parameters: predictions (tensor) – a batch of interval predictions.
Returns: a batch of point predictions
Return type: tensor

torchuq.transform.direct.particle_to_distribution(predictions, bandwidth_ratio=2.5)¶

Convert a particle prediction to a distribution prediction by kernel density estimation (KDE)

Parameters

predictions (tensor) – a batch of quantile predictions, should be an array of shape [batch_size, n_quantiles] or [batch_size, n_quantiles, 2].
bandwidth_ratio (float) – the bandwidth of the kernel density estimator (relative to the average distance between quantiles).

Returns

a batch of distribution predictions

Return type

Distribution

torchuq.transform.direct.particle_to_quantile(predictions)¶

Converts a particle prediction to a quantile prediction.

Not fully differentiable because of the sorting operation involved. Should be thought of as a permutation function

Parameters: predictions (tensor) – array [batch_size, n_particles], a batch of particle predictions
Returns: a batch of quantile predictions
Return type: tensor

torchuq.transform.direct.quantile_to_distribution(predictions, bandwidth_ratio=2.5)¶

Convert a quantile prediction to a distribution prediction by kernel density estimation (KDE).

Parameters

predictions (tensor) – a batch of quantile predictions, should be an array of shape [batch_size, n_quantiles] or [batch_size, n_quantiles, 2].
bandwidth_ratio (float) – the bandwidth of the kernel density estimator (relative to the average distance between quantiles).

Returns

a batch of distribution predictions

Return type

Distribution

torchuq.transform.direct.topk_to_uset(predictions, num_classes=- 1)¶

Convert a topk prediction to a uncertainty set prediction

Parameters

predictions (tensor) – a batch of topk predictions
num_classes (int) – total number of classes. If set to -1, the number of classes will be inferred as one greater than the largest class value in the input tensor.

Returns

the uset prediction

Return type

tensor

torchuq.transform.decision Module¶

class torchuq.transform.decision.CriticDecision(num_classes=1000, num_action=2)¶

__init__(num_classes=1000, num_action=2)¶: Initializes internal Module state, shared by both nn.Module and ScriptModule.

forward(predictions)¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class torchuq.transform.decision.DecisionCalibrator(verbose=True, save_path=None)¶

Recalibrate a categorical prediction to achieve decision calibration.

Parameters

verbose (bool) – if set to True than print additional performance information during training
save_path (float) – the path to checkpoint any training weights. If set to None the weights are not saved.

__init__(verbose=True, save_path=None)¶

to(device)¶

Move every torch tensor owned by this class to a new device

Parameters: device – a torch.device instance, alternatively it could be a torch.Tensor or a prediction object

train(predictions, labels, calib_steps=100, num_action=2, num_critic_epoch=500, test_predictions=None, test_labels=None, seed=0, *args, **kwargs)¶

Train the decision calibrator for calib_steps. If you call this function multiple times, this function does not erase previously trained calibration maps, and only appends additional recalibration steps

Parameters

predictions (tensor) – a categorical prediction with shape [batch_size, n_classes]
labels (tensor) – an array of int valued labels
calib_steps (int) – number of calibration iterations (this is the number of iteration steps in Algorithm 2 of the paper)
num_critic_epoch (int) – number of gradient descent steps when optimizing the worst case b in Algorithm 2 of the paper
test_predictions (tensor) – a categorical prediction for measuring test performance, can be set to None if measuring test performance is not needed
test_labels (tensor) – an array of int valued labels for measuring test performance, can be set to None if measuring test performance is not needed
seed – float, the random seed for reproducibility.

Returns

a PerformanceRecord object, the measured performance

Return type

recorder