infinities_classifier

This module handles classifiers for training points that categorize them as either “finite” (interesting) or “infinite” (non-interesting). This classification defines regions which are “safe” to explore in contrast to regions which are “unsafe” to explore since they are infinite. This is done in an attempt to hinder the exploration of parts of the parameter space which have a \(-\infty\) log-posterior value. These values need to be filtered out since feeding them to the GP Regressor will break it. Nevertheless this is important information that we shouldn’t throw away.

We also make use of these classifiers when doing the MC runs of the surrogate log-p to tell the model which regions it shouldn’t visit. In essence our process shrinks the prior to a region where the model thinks that all values of the log-posterior distribution are finite.

At the moment, it impements a Support vector machine (SVM) with an RBF kernel and a hyper-rectangular constraint fitted to a fraction of the highest log-p samples.

class gpry.infinities_classifier.InfinitiesClassifiers(bounds, nstd_calculator, keep_min=None, random_state=None, **classifiers)

Bases: object

property trust_bounds

The bounds of a smaller trust region possibly defined by a classifier (in particular if infinities_classifier.TrustRegion is being used. Otherwise returns the original prior bounds.

update_threshold_definition(nstd_calculator)

Updates the classifier’s thresholds after a change in the transformation defining the space of the y’s.

Parameters:

nstd_calculator (callable) – Function able to translate threshold values from sigma units to the space in which the classifiers are defined (including preprocessing, if present.

fit(X, y, nstd_calculator=None, keep_min=None, i_sorted=None, validate=True)

Fits the classifiers.

If the representation space of the y’s has changed, it is necessary to pass an updated nstd_calculator to update the thresholds.

Parameters:

• X (array-like, shape = (n_samples, dimensionality)) – Training data.

• y (array-like, shape = (n_samples, [n_output_dims])) – Target values. They can have any value smaller than positive infinity, including negative infinity.

• nstd_calculator (callable) – Function able to translate threshold values from sigma units to the space in which the classifiers are defined (including preprocessing, if present.

• keep_min (int, optional) – If passed, and there are fewer than keep_min points above the threshold, the threshold is ignored and the keep_min largest points are returned. Exception: fewer than keep_min points in y that are not -np.inf. If none passed, uses by default max(2, dimensionality).

• i_sorted (bool or array-like, int, 1-dimensional, optional) – Sorting indices for y. If True passed, y is assumed sorted. Passing sorting indices (or True) greatly improves the efficiency of this function, saving a sort of y.

• validate (bool (default: True)) – If passed, check consistency of input arrays.

Returns:

i_finite – Indices of elements in y classified as “finite” according to the threshold.

Return type:

array-like int

get_classifier_min_threshold(ignore=None)

Returns the classifier with the minimum threshold, or None of none present or all ignored.

Parameters:

ignore (list(str), optional) – List of classifiers to ignore in this call.

Returns:

name_or_strictest_classifier

Return type:

str

get_highest_current_threshold(ignore=None)

Returns the minimum differential threshold among all classifiers.

Parameters:

ignore (list(str), optional) – List of classifiers to ignore in this call.

Returns:

threshold

Return type:

float (positive)

i_finite_y(y, i_sorted=None, ignore=None, validate=True)

Returns the indices of the finite points, depending on the current fit of the classifiers.

Parameters:

• y (array-like, shape = (n_samples, [n_output_dims])) – Target values. They can have any value smaller than positive infinity, including negative infinity.

• i_sorted (bool or array-like, int, 1-dimensional, optional) – Sorting indices for y. If True passed, y is assumed sorted. Passing sorting indices (or True) greatly improves the efficiency of this function, saving a sort of y.

• ignore (list(str), optional) – List of classifiers to ignore in this call.

• validate (bool (default: True)) – If passed, check consistency of input arrays.

Returns:

i_finite – Indices of elements in y classified as “finite” according to the threshold, in sorting order.

Return type:

array-like int

is_finite_y(y, i_sorted=None, ignore=None, validate=True)

Returns True for the finite indices of the input array, depending on the current threshold and maximum y value in the training set (not the input).

Parameters:

• y (array-like, shape = (n_samples, [n_output_dims])) – Target values. They can have any value smaller than positive infinity, including negative infinity.

• i_sorted (bool or array-like, int, 1-dimensional, optional) – Sorting indices for y. If True passed, y is assumed sorted. Passing sorting indices (or True) greatly improves the efficiency of this function, saving a sort of y.

• ignore (list(str), optional) – List of classifiers to ignore in this call.

• validate (bool (default: True)) – If passed, check consistency of input arrays.

Returns:

y_finite – Classification of given points according to the threshold.

Return type:

array-like bool

is_finite_X(X, ignore=None, validate=True)

Returns True for locations predicted to have a finite posterior probability density, based on the output of the classifiers’ predict methods.

Parameters:

• X (array-like, shape = (n_samples, dimensionality)) – Training data.

• ignore (list(str), optional) – List of classifiers to ignore in this call.

• validate (bool (default: True)) – If passed, check consistency of input arrays.

Returns:

X_finite – Whether each of the points are located inside the trust region.

Return type:

array-like bool

set_random_state(random_state)

class gpry.infinities_classifier.ThresholdClassifier(threshold, nstd_calculator)

Bases: object

Parent class for classifiers that separate the target values according to some given threshold.

Parameters:

• threshold (numeric) – Differential threshold below the maximum element in y over which points are selected as “finite”. Must be positive, can be np.inf.

• nstd_calculator (callable) – Function able to translate threshold values from sigma units to the space in which the classifiers are defined (including preprocessing, if present).

update_threshold_definition(nstd_calculator)

Converts the units of sigma in the theshold definition and transforms it into the right y space.

Call every time the representation space of the y’s changes, or alternatively pass the nstd_calculator in the corresponding call of the fit method.

Parameters:

nstd_calculator (callable) – Function able to translate threshold values from sigma units to the space in which the classifiers are defined (including preprocessing, if present).

property fitted

i_finite_y(y, i_sorted=None, validate=True)

Returns the indices of the finite points, depending on the current threshold and maximum y value in the training set (not the input).

Parameters:

• y (array-like, shape = (n_samples, [n_output_dims])) – Target values. They can have any value smaller than positive infinity, including negative infinity.

• i_sorted (bool or array-like, int, 1-dimensional, optional) – Sorting indices for y. If True passed, y is assumed sorted. Passing sorting indices (or True) greatly improves the efficiency of this function, saving a sort of y.

• validate (bool (default: True)) – If passed, check consistency of input arrays.

Returns:

i_finite – Indices of elements in y classified as “finite” according to the threshold, in sorting order.

Return type:

array-like int

is_finite_y(y, i_sorted=None, validate=True)

Returns True for the finite indices of the input array, depending on the current threshold and maximum y value in the training set (not the input).

Parameters:

• y (array-like, shape = (n_samples, [n_output_dims])) – Target values. They can have any value smaller than positive infinity, including negative infinity.

• i_sorted (bool or array-like, int, 1-dimensional, optional) – Sorting indices for y. If True passed, y is assumed sorted. Passing sorting indices (or True) greatly improves the efficiency of this function, saving a sort of y.

• validate (bool (default: True)) – If passed, check consistency of input arrays.

Returns:

y_finite – Classification of given points according to the threshold.

Return type:

array-like bool

is_finite_X(X, validate=True)

Returns True for locations predicted to have a finite posterior probability density, based on the output of the classifier’s predict method.

Parameters:

• X (array-like, shape = (n_samples, dimensionality)) – Training data.

• validate (bool (default: True)) – If passed, check consistency of input arrays.

Returns:

X_finite – Whether each of the points are located inside the trust region.

Return type:

array-like bool

fit(X, y, nstd_calculator=None, keep_min=None, i_sorted=None, validate=True)

Fits the classifier to some X, y, given a threshold (differential, see below).

If the representation space of the y’s has changed, it is necessary to pass an updated nstd_calculator to update the thresholds.

Parameters:

• X (array-like, shape = (n_samples, dimensionality)) – Training data.

• y (array-like, shape = (n_samples, [n_output_dims])) – Target values. They can have any value smaller than positive infinity, including negative infinity.

• nstd_calculator (callable) – Function able to translate threshold values from sigma units to the space in which the classifiers are defined (including preprocessing, if present).

• keep_min (int, optional) – If passed, and there are fewer than keep_min points above the threshold, the threshold is ignored and the keep_min largest points are returned. Exception: fewer than keep_min points in y that are not -np.inf. If none passed, uses by default max(2, dimensionality).

• i_sorted (bool or array-like, int, 1-dimensional, optional) – Sorting indices for y. If True passed, y is assumed sorted. Passing sorting indices (or True) greatly improves the efficiency of this function, saving a sort of y.

• validate (bool (default: True)) – If passed, check consistency of input arrays.

Returns:

i_finite – Indices of elements in y classified as “finite” according to the threshold, in sorting order.

Return type:

array-like int

static i_finite_threshold(y, threshold, is_abs=False, keep_min=None, i_sorted=None, validate=True)

Selects indices of finite y’s. More efficient if sorting indiced for y are passed.

Parameters:

• y (array-like, 1-dimensional) – Array of input values. They can have any value smaller than positive infinity, including negative infinity.

• threshold (numeric) – Differential threshold below the maximum element in y over which points are selected as “finite”. Must be positive, can be np.inf.

• is_abs (bool (default: False)) – If True, the threshold is understood as absolute, instead of a delta with respect to the max value in y.

• keep_min (int, optional) –

  If passed, and there are fewer than keep_min points above the threshold,

  the threshold is ignored and the keep_min largest points are returned. Exception: fewer than keep_min points in y that are not -np.inf.

• i_sorted (bool or array-like, int, 1-dimensional, optional) – Sorting indices for y. If True passed, y is assumed sorted. Passing sorting indices (or True) greatly improves the efficiency of this function, saving a sort of y.

• validate (bool (default: True)) – If passed, check consistency of input arrays.

Returns:

Indices of elements in y classified as “finite” according to the threshold, in sorting order, and actual threshold used, which is different from the given one only if keep_min needed to be used to enlarge the set of finite points.

Return type:

(i_sorted_finite, threshold)

class gpry.infinities_classifier.SVM(threshold, nstd_calculator, C=10000000.0, kernel='rbf', degree=3, gamma='scale', coef0=0.0, shrinking=True, probability=False, tol=0.001, cache_size=200, class_weight=None, verbose=False, max_iter=-1, decision_function_shape='ovr', break_ties=False, random_state=None)

Bases: SVC, ThresholdClassifier

Wrapper for the sklearn RBF kernel SVM.

Classifies points as finite of non-finite, in order to exclude the latter from the training set of a parent GPR. It keeps track of the full training set, including classified-infinite points.

The classification is performed using a threshold understood as a positive difference against the current maximum y in the training set. The threshold is passed at fitting time and not at initialisation, in case the classifier is defined in a transformed coordinate space, with a transformation that changes through the training of the parent GPR. (NB: passing the threshold every time is a compromise that allows to keep the full training set stored in this object with non-reduced y values while avoiding preprocessing overhead.)

Also in case there is a coordinate transformation, the training set of this object should not be obtained directly, but via properties of the parent GP instead that will undo the transformation. The same applying to calling any method directly.

Parameters:

• threshold (numeric) – Differential threshold below the maximum element in y over which points are selected as “finite”. Must be positive, can be np.inf.

• nstd_calculator (callable) – Function able to translate threshold values from sigma units to the space in which the classifiers are defined (including preprocessing, if present).

Cfloat, default=1e7

Regularization parameter. The strength of the regularization is inversely proportional to C. Must be strictly positive. The penalty is a squared l2 penalty.

kernel{‘linear’, ‘poly’, ‘rbf’, ‘sigmoid’, ‘precomputed’}, default=’rbf’

Specifies the kernel type to be used in the algorithm. It must be one of ‘linear’, ‘poly’, ‘rbf’, ‘sigmoid’, ‘precomputed’ or a callable. If none is given, ‘rbf’ will be used. If a callable is given it is used to pre-compute the kernel matrix from data matrices; that matrix should be an array of shape (n_samples, n_samples).

degreeint, default=3

Degree of the polynomial kernel function (‘poly’). Ignored by all other kernels.

gamma{‘scale’, ‘auto’} or float, default=’scale’

Kernel coefficient for ‘rbf’, ‘poly’ and ‘sigmoid’.

• if gamma='scale' (default) is passed then it uses 1 / (n_features * X.var()) as value of gamma,

• if ‘auto’, uses 1 / n_features.

coef0float, default=0.0

Independent term in kernel function. It is only significant in ‘poly’ and ‘sigmoid’.

shrinkingbool, default=True

Whether to use the shrinking heuristic.

probabilitybool, default=False

Whether to enable probability estimates. This must be enabled prior to calling fit, will slow down that method as it internally uses 5-fold cross-validation, and predict_proba may be inconsistent with predict.

tolfloat, default=1e-3

Tolerance for stopping criterion.

cache_sizefloat, default=200

Specify the size of the kernel cache (in MB).

class_weightdict or ‘balanced’, default=None

Set the parameter C of class i to class_weight[i]*C for SVC. If not given, all classes are supposed to have weight one. The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y)).

verbosebool, default=False

Enable verbose output. Note that this setting takes advantage of a per-process runtime setting in libsvm that, if enabled, may not work properly in a multithreaded context.

max_iterint, default=-1

Hard limit on iterations within solver, or -1 for no limit.

decision_function_shape{‘ovo’, ‘ovr’}, default=’ovr’

Whether to return a one-vs-rest (‘ovr’) decision function of shape (n_samples, n_classes) as all other classifiers, or the original one-vs-one (‘ovo’) decision function of libsvm which has shape (n_samples, n_classes * (n_classes - 1) / 2). However, one-vs-one (‘ovo’) is always used as multi-class strategy. The parameter is ignored for binary classification.

break_tiesbool, default=False

If true, decision_function_shape='ovr', and number of classes > 2, predict will break ties according to the confidence values of decision_function; otherwise the first class among the tied classes is returned. Please note that breaking ties comes at a relatively high computational cost compared to a simple predict.

random_stateint, RandomState instance or None, default=None

Controls the pseudo random number generation for shuffling the data for probability estimates. Ignored when probability is False. Pass an int for reproducible output across multiple function calls.

Attributes:

• all_finite (bool) – Is true when all posterior values which have been sampled are finite which removes the need for fitting the SVM.

• class_weight_ (ndarray of shape (n_classes,)) – Multipliers of parameter C for each class. Computed based on the class_weight parameter.

• classes_ (ndarray of shape (n_classes,)) – The classes labels.

• coef_ (ndarray of shape (n_classes * (n_classes - 1) / 2, n_features)) – Weights assigned to the features (coefficients in the primal problem). This is only available in the case of a linear kernel. coef_ is a readonly property derived from dual_coef_ and support_vectors_.

• dual_coef_ (ndarray of shape (n_classes -1, n_SV)) – Dual coefficients of the support vector in the decision function, multiplied by their targets. For multiclass, coefficient for all 1-vs-1 classifiers. The layout of the coefficients in the multiclass case is somewhat non-trivial.

• fit_status_ (int) – 0 if correctly fitted, 1 otherwise (will raise warning)

• intercept_ (ndarray of shape (n_classes * (n_classes - 1) / 2,)) – Constants in decision function.

• n_features_in_ (int) – Number of features seen during fit.

• feature_names_in_ (ndarray of shape (n_features_in_,)) – Names of features seen during fit. Defined only when X has feature names that are all strings.

• support_ (ndarray of shape (n_SV)) – Indices of support vectors.

• support_vectors_ (ndarray of shape (n_SV, n_features)) – Support vectors.

• n_support_ (ndarray of shape (n_classes,), dtype=int32) – Number of support vectors for each class.

• probA_ (ndarray of shape (n_classes * (n_classes - 1) / 2))

• probB_ (ndarray of shape (n_classes * (n_classes - 1) / 2)) – If probability=True, it corresponds to the parameters learned in Platt scaling to produce probability estimates from decision values. If probability=False, it’s an empty array. Platt scaling uses the logistic function 1 / (1 + exp(decision_value * probA_ + probB_)) where probA_ and probB_ are learned from the dataset..

• shape_fit_ (tuple of int of shape (n_dimensions_of_X,)) – Array dimensions of training vector X.

set_random_state(random_state)

fit(X, y, nstd_calculator=None, keep_min=None, i_sorted=None, validate=True)

Fits the SVM with two categorial classes:

• \(\tilde{y}=True\) Finite points

• \(\tilde{y}=False\) Infinite points

where \(\tilde{y}\) is produced after checking the input y’s against an internal threshold value, which may also be adjusted at this step.

If the representation space of the y’s has changed, it is necessary to pass an updated nstd_calculator to update the thresholds.

Parameters:

• X (array-like, shape = (n_samples, dimensionality)) – Training data.

• y (array-like, shape = (n_samples, [n_output_dims])) – Target values. They can have any value smaller than positive infinity, including negative infinity.

• nstd_calculator (callable) – Function able to translate threshold values from sigma units to the space in which the classifiers are defined (including preprocessing, if present).

• keep_min (int, optional) – If passed, and there are fewer than keep_min points above the threshold, the threshold is ignored and the keep_min largest points are returned. Exception: fewer than keep_min points in y that are not -np.inf. If none passed, uses by default max(2, dimensionality).

• i_sorted (bool or array-like, int, 1-dimensional, optional) – Sorting indices for y. If True passed, y is assumed sorted. Passing sorting indices (or True) greatly improves the efficiency of this function, saving a sort of y.

• validate (bool (default: True)) – If passed, check consistency of input arrays.

Returns:

i_finite – Indices of elements in y classified as “finite” according to the threshold, in sorting order.

Return type:

array-like int

predict(X, validate=True)

Wrapper for the predict method of the SVM. Returns a boolean array which is true at locations where the SVM predicts a finite posterior distribution and False where it predicts infinite values.

Parameters:

• X (array-like, shape = (n_samples, dimensionality)) – Training data.

• validate (bool (default: True)) – If passed, check consistency of input arrays.

Returns:

X_finite – A boolean array which is True at locations predicted finite posterior and False at locations with predicted infinite posterior.

Return type:

array-like bool

Raises:

ValueError – “ndarray is not C-contiguous”: May be raised if validate is False. Call numpy.ascontiguousarray() on the input before the call.

class gpry.infinities_classifier.TrustRegion(threshold, nstd_calculator, bounds, factor=None)

Bases: ThresholdClassifier

Class managing the region within we trust the model to be finite, as a hyper-rectangle defined by the points with log-posterior over some value, expressed as a different with the current maximum value.

Parameters:

• threshold (numeric) – Differential threshold below the maximum element in y over which points are selected as “finite”. Must be positive, can be np.inf.

• nstd_calculator (callable) – Function able to translate threshold values from sigma units to the space in which the classifiers are defined (including preprocessing, if present).

• bounds (array-like, shape = (dimensionality, 2)) – Bounds of the problem

• factor (float (positive), optional) – If defined as a positive float, enlarges the size of the minimal hyper-rectangle by the given factor. If undefined, it keeps the boundaries defined by the threshold.

property d

Dimension of the space.

property trust_bounds

Returns a copy of the trust bounds (the original ones if never fit or factor=None).

fit(X, y, nstd_calculator=None, keep_min=None, i_sorted=None, validate=True)

Adjusts the boundaries of the trust region so that it contains the points over the theshold. More efficient if sorting indiced for y are passed. Indices are returned as sorting indices.

If the representation space of the y’s has changed, it is necessary to pass an updated nstd_calculator to update the thresholds.

Parameters:

• X (array-like, shape = (n_samples, dimensionality)) – Training data.

• y (array-like, shape = (n_samples, [n_output_dims])) – Target values. They can have any value smaller than positive infinity, including negative infinity.

• nstd_calculator (callable) – Function able to translate threshold values from sigma units to the space in which the classifiers are defined (including preprocessing, if present).

• keep_min (int, optional) – If passed, and there are fewer than keep_min points above the threshold, the threshold is ignored and the keep_min largest points are returned. Exception: fewer than keep_min points in y that are not -np.inf. If none passed, uses by default max(2, dimensionality).

• i_sorted (bool or array-like, int, 1-dimensional, optional) – Sorting indices for y. If True passed, y is assumed sorted. Passing sorting indices (or True) greatly improves the efficiency of this function, saving a sort of y.

• validate (bool (default: True)) – If passed, check consistency of input arrays.

Returns:

i_finite – Indices of elements in y classified as “finite” according to the threshold, in sorting order.

Return type:

array-like int

predict(X, validate=True)

Returns True for locations inside the trust region, and False otherwise.

Parameters:

• X (array-like, shape = (n_samples, dimensionality)) – Training data.

• validate (bool (default: True)) – If passed, check consistency of input arrays.

Returns:

X_finite – Whether each of the points are located inside the trust region.

Return type:

array-like bool