Run

This module contains top level functionalities for initialising the Bayesian optimization loop, running it and getting MC samples from it. Furthermore some methods for plotting and diagnostics are provided.

Module that defines the Runner class, which handles input, the active learning loop, and the processing of results.

class run.Runner(loglike=None, bounds=None, ref_bounds=None, params=None, gpr='RBF', gp_acquisition='LogExp', initial_proposer='reference', convergence_criterion=None, mc=None, callback=None, callback_is_MPI_aware=False, options=None, checkpoint=None, load_checkpoint=None, seed=None, plots=False, verbose=3)

Bases: object

Class that takes care of constructing the Bayesian quadrature/likelihood characterization loop. After initialisation, the algorithm can be launched with Runner.run(), and, optionally after that, an MC process can be launched on the surrogate model with Runner.generate_mc_sample().

Parameters:

• loglike (callable or Cobaya model object) – Likelihood function (returning log-likelihood; requires additional argument bounds) or Cobaya Model instance (which contains all information about the parameters in the likelihood and their priors as well as the likelihood itself). It must not be specified if ‘resuming’ from a checkpoint (see load_checkpoint below).

• bounds (List of [min, max], or Dict {name: [min, max],...}) – List or dictionary of parameter bounds. If it is a dictionary, the keys need to correspond to the argument names of the likelihood function, and the values can be either bounds specified as [min, max], or bounds and labels, as {"prior": [min, max], "latex": [label]}. It does not need to be defined (will be ignored) if a Cobaya Model instance is passed as loglike.

• gpr (GaussianProcessRegressor, str, dict, optional (default="RBF")) – The GP used for interpolating the posterior. If None or “RBF” is given a GP with a constant kernel multiplied with an anisotropic RBF kernel and dynamic bounds is generated. The same kernel with a Matern 3/2 kernel instead of a RBF is generated if “Matern” is passed. This might be useful if the posterior is not very smooth. Otherwise a custom GP regressor can be defined as a dict containing the arguments of GaussianProcessRegressor, or passing an already initialized instance.

• gp_acquisition (GenericGPAcquisition, optional (default="LogExp")) – The acquisition object. If None is given the BatchOptimizer with a LogExp acquisition function is used (with the \(\zeta\) value chosen automatically depending on the dimensionality of the prior) and the GP’s X-values are preprocessed to be in the uniform hypercube before optimizing the acquistion function. It can also be passed an initialized instance, or a dict with arguments with which to initialize one.

• initial_proposer (InitialPointProposer, str, dict, optional (default="reference")) – Proposer used for drawing the initial training samples before running the Bayesian optimisation loop. As standard the samples are drawn from the model reference (prior if no reference is specified). Alternative options which can be passed as strings are "prior", "uniform". The "reference" proposer defaults to the prior if no reference distribution is provided. If defined as a dict with the proposer name as single key, the values will be passed as kwargs to the proposer.

• convergence_criterion (ConvergenceCriterion, str, dict, False, optional (default=None)) – The convergence criterion. If None is given the default criterion is used: CorrectCounter for BatchOptimizer with adaptive thresholds, and a combination of a less stringent CorrectCounter and a GaussianKL for NORA. Can be specified as a dict to initialize one or more ConvergenceCriterion classes with some arguments, or directly as an instance or class name of some ConvergenceCriterion. If False, no convergence criterion is used, and the process runs until the budget is exhausted.

• mc (dict, str, optional (default=None)) – Sampler and it options for the diagnosis and final MC samples as {<sampler_name >: {'option1': value1, ...}}, or simply a struing specifying the sampler name. These settings can be overriden when calling run.Runner.generate_mc_sample() using its sampler argument. By default, the best available nested sampler is used, with standard precision settings.

• options (dict, optional (default=None)) –

  A dict containing all options regarding the bayesian optimization loop. The available options are:

  • n_initial : Number of finite initial truth evaluations before starting the BO loop (default: 3 * number of dimensions)

  • max_initial : Maximum number of truth evaluations at initialization. If it is reached before n_initial finite points have been found, the run will fail. To avoid that, try decreasing the volume of your prior (default: 30 * (number of dimensions)**1.5).

  • max_total : Maximum number of attempted sampling points before the run stops. This is useful if you e.g. want to restrict the maximum computation resources (default: 70 * (number of dimensions)**1.5 or max_initial, whichever is largest).

  • max_finite : Maximum number of sampling points accepted into the GP training set before the run stops. This might be useful if you use the DontConverge convergence criterion, specifying exactly how many points you want to have in your GP. If you set this limit by hand and find that it is easily saturated, try decreasing the volume of your prior (default: max_total).

  • n_points_per_acq : Number of points which are aquired with Kriging believer for every acquisition step. It will be adjusted within a 20% tolerance to match a multiple of the number of parallel processes (default: number of dimensions).

  • fit_full_every : Number of iterations between full GP hyperparameters fits, including several restarts of the optimiser. Pass ‘np.inf’ or a large number to never refit with restarts (default : 2 * sqrt(number of dimensions)).

  • fit_simple_every : Similar to fit_full_every, but with a single optimiser run from the last optimum hyperparameters. Overridden by fit_full_every where it matches its periodicity. Pass np.inf or a large number to never refit from last optimum (default : 1, i.e. every iteration).

• callback (callable, optional (default=None)) – Function run each iteration after adapting the recently acquired points and the computation of the convergence criterion. This function should take the runner as argument: callback(runner_instance). When running in parallel, the function is run by the main process only, unless callback_is_MPI_aware=True.

• callback_is_MPI_aware (bool (default: False)) – If True, the callback function is called for every process simultaneously, and it is expected to handle parallelisation internally. If false, only the main process calls it.

• checkpoint (str, optional (default=None)) – Path for storing checkpointing information from which to resume in case the algorithm crashes. If None is given no checkpoint is saved.

• load_checkpoint ("resume" or "overwrite", must be specified if path is not None.) – Whether to resume from the checkpoint files if existing ones are found at the location specified by checkpoint.

• seed (int or numpy.random.Generator, optional) – Seed for the random number generator, or an already existing generator, for reproducible runs. For MPI runs, if a seed is passed, numpy.random.SeedSequence will be used to generate seeds for every ranks.

• plots (bool, dict (default: True)) – If True, produces some progress plots. One can also pass the arguments of Runner.plot_progress as a dict for finer control, e.g. {"timing": True, "convergence": True, "trace": False, "slices": False, "ext": "svg"}.

• verbose (1, 2, 3, optional (default: 3)) – Level of verbosity. 3 prints Infos, Warnings and Errors, 2 Warnings and Errors, and 1 only Errors. Should be set to 2 or 3 if problems arise. Is passed to the GP, Acquisition and Convergence criterion if they are built automatically.

Attributes:

• model (Cobaya model) – The model that was used to run the GP on (if running in parallel, needs to be passed for all processes).

• gpr (GaussianProcessRegressor) – This can be used to call an MCMC sampler for getting marginalized properties. This is the most crucial component.

• gp_acquisition (GenericGPAcquisition) – The acquisition object that was used for the active sampling procedure.

• convergence_criterion (Convergence_criterion) – The convergence criterion used for determining convergence. Depending on the criterion used this also contains the approximate covariance matrix of the posterior distribution which can be used by the MCMC sampler.

• options (dict) – The options dict used for the active sampling loop.

• progress (Progress) – Object containing per-iteration progress information: number of finite training points, number of GP evaluations, timing of different parts of the algorithm, and value of the convergence criterion.

_construct_gpr(gpr)

Constructs or passes the GPR.

_construct_gp_acquisition(gp_acquisition)

Constructs or passes the GPAcquisition instance.

_construct_initial_proposer(initial_proposer)

Constructs or passes the initial proposer.

_construct_convergence_criterion(convergence_criterion, acq_has_mc=False)

Constructs or passes the convergence criterion.

_construct_mc_options(mc_options)

Parses the choice and options of the default MC sampler.

property d

Dimensionality of the problem.

property params

Returns the list of parameter names.

property labels

Returns the list of labels.

logp(X)

Wrapper for the surrogate posterior. Call with a point or a list of them.

This is the full posterior. If the prior is uniform the likelihood function can be recovered by summing self.log_prior_volume to this function.

Always returns an array.

logL(X)

Wrapper for the surrogate likelihood. Call with a point or a list of them.

Always returns an array.

logp_truth(X)

Wrapper for the true log-posterior. Call with a single point.

This is the full posterior. If the prior is uniform the likelihood function can be recovered by summing self.log_prior_volume to this function.

Always returns a scalar.

logL_truth(X)

Wrapper for the true log-likelihood. Call with a single point.

Always returns a scalar.

logpost_eval_and_report(X, level=None)

Simple wrapper to evaluate and return the true log-posterior at X, and log it with the given level.

logprior(X)

Returns the log-prior.

log(msg, level=None)

Print a message if its verbosity level is equal or lower than the given one (or always if level=None.

ensure_paths(plots=False)

Creates paths for checkpoint and plots.

property n_total_left

Number of truth evaluations before stopping.

property n_finite_left

Number of truth evaluations with finite return value before stopping.

banner(text, max_line_length=79, prefix='| ', suffix=' |', header='=', footer='=', level=3)

Creates an iteration banner.

read_checkpoint(truth=None)

Loads checkpoint files to be able to resume a run or save the results for further processing.

Parameters:

truth (gpry.truth.Truth, optional) – If passed, it will be used instead of the loaded one.

save_checkpoint(update_truth=False)

Saves checkpoint files to be able to resume a run or save the results for further processing.

_share_gpr(root=0)

Shares the GPR of the main process, restoring each process’ RNG.

_share_convergence_from_main()

Shares the convergence criterion from the main process, aware of whether any of the criteria handles MPI by itself.

run()

Runs the acquisition-training-convergence loop until either convergence or a stopping condition is reached.

do_initial_training()

Draws an initial sample for the gpr GP model until it has a training set of size n_initial, counting only finite-target points (“finite” here meaning over the threshold of the SVM classifier, if present).

This function is MPI-aware and broadcasts the initialized GPR to all processes.

_eval_truth_parallel(new_X)

Performs the evaluation of the true log-posterior in parallel.

Returns all y’s at rank 0 (None otherwise), and a short report msg.

_check_convergence_parallel(new_X, new_y, y_pred)

Checks algorithmic convergence.

It needs to be called from all MPI processes.

update_mean_cov(use_mc_sample=None)

Updates and shares mean and cov if available, preferring the MC-sample one if indicated, and otherwise checking GPAcquisition first, and Convergence second if not present in GPAcquisition.

set_fiducial_point(X, logpost=None, loglike=None)

Set a single fiducial point in parameter space, to be included in the plots and in some tests.

Recover with self.fiducial_[X|logpost|loglike|weight].

Parameters:

• X (array-like, shape = (n_dim)) – Fiducial point coordinates.

• logpost (float, optional) – Fiducial point log-posterior density

• loglike (float, optional) – Fiducial point log-likelihood density (use if the prior density not known).

:raises TypeError : if a mismatch found between the different inputs.:

set_fiducial_MC(X, logpost=None, loglike=None, weights=None)

Set a reference Monte Carlo sample of the true posterior, to be included in the plots and in some tests.

Recover with self.fiducial_MC_[X|logpost|loglike|weight].

Parameters:

• X (array-like, shape = (n_samples, n_dim)) – Fiducial 2d array of MC samples.

• logpost (array-like, shape = (n_samples), optional) – Log-posterior density for the points in the MC sample.

• loglike (array-like, shape = (n_samples), optional) – Log-likelihood density for the points in the MC sample (use if the prior density is not known).

• weights (array-like, shape = (n_samples), optional) – Weights of points in the MC sample (assumed equal weights if not specified).

:raises TypeError : if a mismatch found between the different inputs.:

plot_progress(ext='png', timing=True, convergence=True, trace=True, slices=False, corner=False)

Creates some progress plots and saves them at path (assumes path exists).

Parameters:

• ext (str (default "png")) – Format for the plots, among the available ones in matplotlib.

• timing (bool (default: True)) – Plot histogram of timing per iteration (totals in legend).

• convergence (bool (default: True)) – Plot the evolution of the convergence criterion (included in trace plot).

• trace (bool (default: True)) – Plot the evolution of the run: convergence criterion, surrogate log(p) and parameters.

• slices (bool (default: False)) – Plots slices per training samples. Slow – use for diagnosis only.

• corner (bool (default: False)) – Creates a corner plot (contours for current GP shown only if using NORA). Slow – use for diagnosis only.

generate_mc_sample(sampler=None, add_options=None, output=None, resume=False)

Runs an MC process using a nested sampler, or Cobaya.

The result can be retrieved using the run.Runner.last_mc_samples() method.

Parameters:

• sampler (str, dict, optional) – Sampler to be initialised. If a string, it must be nested or the name of a sampler recognised by Cobaya (e.g. mcmc). If nested, the best available nested sampler is used: PolyChord if available, otherwise UltraNest. If defined as a dict, the options specified as keys and values will be passed to the sampler, in the nested case nlive, num_repeats (length of slice chains for PolyChord), precision_criterion (fraction of the evidence contained in the set of live points), nprior (number of initial samples of the prior). If using a Cobaya sampler, the options mentioned in the Cobaya documentation of the particular sampler can be passed. If undefined, uses the sampler specified by the mc argument at initialzation (‘nested’, if unspecified).

• add_options (dict, optional) – DEPRECATED: pass options by specifying the sampler argument as a dict.

• output (path, optional (default: checkpoint/chains, if checkpoint != None)) – The path where the resulting Monte Carlo sample shall be stored. If passed explicitly False, produces no output.

• resume (bool, optional (default=False)) – Whether to resume from existing output files (True) or force overwrite (False)

last_mc_samples(copy=True, as_pandas=False, as_getdist=False)

Returns the last MC sample from the surrogate model as a dict with keys w (weights), X, logpost, logprior and loglike.

If None is stored as weights, all samples should be assumed to have equal weight.

The MC samples can optionally be returned as a Pandas DataFrame or a GetDist MCSamples.

diagnose_last_mc_sample()

Diagnoses the last MC sample to check consistency.

This is an MPI-aware method that should be run simultaneously from all processes if running with MPI.

Return type:

True if the test succeeded, False otherwise.

plot_mc(samples_or_samples_folder=None, add_training=True, add_samples=None, output=None, output_dpi=200, ext='png')

Creates a triangle plot of an MC sample of the surrogate model, and optionally shows some evaluation locations.

Parameters:

• samples_or_samples_folder (cobaya.SampleCollection, getdist.MCSamples, str) – MC samples returned by a call to the Runner.generate_mc_sample() method, the output path where they were written, or a getdist.MCSamples instance. If not specified (default) it will try to use the last set of samples generated.

• add_training (bool, optional (default=True)) – Whether the training locations are plotted on top of the contours.

• add_samples (dict(label, (cobaya.SampleCollection, getdist.MCSamples, str))) – Extra MC samples to be added to the plot, specified as dict with labels as keys, and the same type as samples_or_samples_folder as values. Default: None.

• output (str or os.path, optional (default=None)) – The location to save the generated plot in. If None it will be saved in checkpoint_path/images/Surrogate_triangle.pdf or ./images/Surrogate_triangle.png if checkpoint_path is None

• output_dpi (int (default: 200)) – The resolution of the generated plot in DPI.

• ext (str (default: “png” if output not defined; else ignore)) – Format for the plot.

plot_distance_distribution(samples_or_samples_folder=None, show_added=True, output=None, output_dpi=200, ext='png')

Plots the distance distribution of the training points with respect to the confidence ellipsoids (in a Gaussian approx) derived from an MC sample of the surrogate model.

Parameters:

• samples_or_samples_folder (cobaya.SampleCollection, getdist.MCSamples, str) – MC samples returned by a call to the Runner.generate_mc_sample() method, the output path where they were written, or a getdist.MCSamples instance. If not specified (default) it will try to use the last set of samples generated.

• show_added (bool (default True)) – Colours the stacks depending on how early or late the corresponding points were added (bluer stacks represent newer points).

• output (str or os.path, optional (default=None)) – The location to save the generated plot in. If None it will be saved in .png format at checkpoint_path/images/, or ./images/ if checkpoint_path was None.

• output_dpi (int (default: 200)) – The resolution of the generated plot in DPI.

• ext (str (default: “png” if output not defined; else ignore)) – Format for the plot