Model Validation¶

Use confirmation runs to check whether a fitted OLS model predicts correctly for new, independent experimental results at selected factor settings. Replicates at the same setting are grouped, averaged, and compared with a prediction interval for their future mean (PIMean).

Quick Navigation:

Load Confirmation Runs - Import and validate independent experiments
Evaluate Confirmation Results - Compare observations with PIMean intervals
Confirmation Plots - Visualize predictions and residuals
Access Confirmation Data - Retrieve imported settings and responses
Clear Confirmation Runs - Remove confirmation data

Note

Fit the model with set_model_terms(), import_responses(), and compute_mlr_model() before loading confirmation runs. Confirmation data are kept separate from the original design and are not used to refit the model.

Load Confirmation Runs¶

Load independent confirmation experiments from an Excel or CSV file.

ConfirmationRunsMixin.load_confirmation_runs(source, coded=False)[source]¶

Load and validate external confirmation runs for a fitted model.

Parameters:

source (str | Path | DataFrame) – DataFrame or Excel/CSV file containing every factor and fitted-response column.
coded (bool) – Whether process and categorical factor values are coded. Mixture components are unaffected. Defaults to False.

Raises:

FileNotFoundError – If a supplied path does not exist.
TypeError – If coded is not Boolean.
ValueError – If no MLR model has been fitted or the file contains missing, duplicate, nonnumeric, or out-of-domain data.

Return type:

None

Notes

Each row is one confirmation run. Identical factor settings are treated as replicates.

The file must contain one column for every factor and every response in the fitted model. Each row represents one confirmation run; repeated factor settings are treated as replicates. Exp. Order, Exp. Idx, and other laboratory metadata are optional, and additional columns are ignored.

Set coded=True when process and categorical factors are stored using their coded values. Mixture components remain proportions regardless of this option.

Example:

design.load_confirmation_runs(
    source="confirmation_runs.xlsx",
    coded=False,
)

The same method accepts an in-memory DataFrame and copies it defensively:

design.load_confirmation_runs(source=confirmation_runs)

The loader checks factor levels, numeric responses, continuous and mixture bounds, mixture sums, and configured domain filters. If validation fails, previously loaded confirmation data remain unchanged.

Evaluate Confirmation Results¶

PIMean interval¶

For \(c\) future confirmation runs at a setting represented by model vector \(x_0\), doetools calculates

\[\hat y_0 \;\pm\; t_{1-\alpha/2,df_{res}} \sqrt{MS_{res}\left(\frac{1}{c}+ x_0^{\mathsf T}(X^{\mathsf T}X)^{+}x_0\right)}.\]

This is the PIMean interval described for confirmation experiments by A. Jensen [AJENSEN2016]. The \(1/c\) term represents the variability of the mean of \(c\) future observations, while the other term represents the variability of the prediction for a new observation. For \(c=1\), the formula reduces to the usual prediction interval for one future observation.

The superscript \(+\) denotes the Moore-Penrose pseudoinverse used by the implementation. A setting passes when its observed confirmation mean lies within the interval. This is a pointwise check for each setting; intervals are not adjusted for simultaneous inference across multiple settings.

ConfirmationRunsMixin.get_confirmation_results(response, alpha=0.05)[source]¶

Return grouped PIMean confirmation results for one response.

Parameters:

response (str) – Fitted response to evaluate.
alpha (float) – Pointwise significance level for the two-sided PIMean interval. Must lie strictly between 0 and 1. Defaults to 0.05.

Returns:

One row per distinct confirmation setting, including: factor values, replicate count, observed mean, prediction, residual, interval limits, and the Within PI result.

Return type:

DataFrame

Raises:

TypeError – If alpha is not numeric.
ValueError – If the response or confirmation data are unavailable, alpha is invalid, or the fitted model cannot provide a PIMean interval.

Notes

Replicates are grouped by identical factor settings. Residuals use Observed - Predicted. Intervals are pointwise and are not adjusted across multiple confirmation settings.

Example:

results = design.get_confirmation_results(
    response="Yield",
    alpha=0.05,
)
print(results)

The returned table contains the setting identifier, factor values, replicate count n, observed mean, prediction, residual, interval limits, and Within PI. Residuals follow Observed - Predicted and values retain full precision.

Confirmation Plots¶

Plot grouped observed means against predictions or inspect confirmation residuals.

GraphsMixin.plot_confirmation(response, *, view)[source]¶

Plot one confirmation diagnostic for a fitted response.

Return type:: Figure

Example:

prediction_figure = design.plot_confirmation(
    response="Yield", view="observed_vs_predicted"
)
residual_figure = design.plot_confirmation(
    response="Yield", view="residuals"
)

prediction_figure.show()
residual_figure.show()

Access Confirmation Data¶

Retrieve confirmation factor settings and measured responses without exposing internal DataFrames.

ConfirmationRunsMixin.get_confirmation_points(coded=False)[source]¶

Return confirmation factor settings, including replicated rows.

Parameters:

coded (bool) – Whether to return coded factor settings. Defaults to False.

Returns:

A defensive copy of the confirmation factor settings.

Return type:

DataFrame

Raises:

TypeError – If coded is not Boolean.
ValueError – If confirmation runs have not been loaded.

ConfirmationRunsMixin.get_confirmation_responses(response=None)[source]¶

Return all confirmation responses or one selected response.

Parameters:: response (str | None) – Response to select. If None, return every confirmation response. Defaults to None.
Returns:: A defensive copy of the selected response data.
Return type:: DataFrame
Raises:: ValueError – If confirmation runs have not been loaded or response is unavailable.

Example:

actual_points = design.get_confirmation_points(coded=False)
coded_points = design.get_confirmation_points(coded=True)
all_responses = design.get_confirmation_responses()
yield_response = design.get_confirmation_responses(response="Yield")

All getters return defensive copies. Passing a response name returns a one-column DataFrame.

Clear Confirmation Runs¶

Remove loaded confirmation data from the design.

ConfirmationRunsMixin.clear_confirmation_runs()[source]¶

Remove confirmation data without changing the fitted OLS model.

Notes

This clears confirmation points, responses, metadata, and setting IDs. It does not change the original design or fitted model.

Return type:: None

Clearing confirmation runs does not modify the original experiments, model terms, or fitted OLS model.

References¶

[AJENSEN2016]

Jensen, A. (2016). Confirmation Runs in Design of Experiments. Journal of Quality Technology, 48(2), 162-177, DOI: 10.1080/00224065.2016.11918157