Hyper-parameters
Hyper-parameters can be can be used to automatically pick the model/plugin results with the highest score, as defined in the plugin results JSON by a key called score and setting a numerical value. When you don't output a score key at all, or use a null value, that plugin result will be discarded and not used when returning the maximum score plugin result.
Hyper-parameters are only available for intelligence plugins.
When you define a plugin input variable, you can enter your hyper-parameter format in the Auto-optimized value field.
The format starts with the type of the range, which can be integer, float, category or boolean. After that specify the range values, where , supplies two separate values basically like an array, while : specifies an inclusive range of values.
A few examples:
- integer 1:100 — a range of integers from 1 up to and including 100.
- integer 1,4,10 — three integer values, 1, 4 and 10.
- float 0.0:1.0 — a range of floats from 0.0 up to and including 1.0.
- float 1.22,2.33 — two float values, 1.22 and 2.33.
- category blue,green — two text values,
blueandgreen. - boolean true,false — two boolean values,
trueandfalse.
Each value of the hyper-parameter will be varied on every plugin run, and its current value ends up under inputParams. See JSON Manifest section again on how input params are supplied to the plugin via a JSON manifest.
The end-users running the plugin isn't asked to supply a value for this plugin variable at all, because hyper-parameter tuning will detect the optimal value for them. Same when using the Insight API to run the plugin; this plugin hyper-parameter doesn't have to be supplied. If you still want to give to end-user the option to override and supply their own fixed value, check [✅] Allow user to override this auto-optimized variable with own value..
Early stopping
Early stopping of experimentation can be forced — for that specific stage — by outputting {..., stopEarly: true} in the root of the Plugin Results JSON object.
Multiple Stages
A plugin can have multiple stages, defined by the process object after the initial plugin run (see Plugin Results Format).
Because these additional stages run in parallel and are independent of each other — they depend only on the initial stage and its parameters — the single optimal plugin result is returned from the run that had higest average score among all stages for that run.
If you have multiple stages, you often don't needs all hyper-parameters and variations for all stages. For example if you have two parameters, maybe param_a should only be used in the initial stage, while param_b should be used for any additional stages. Or maybe you don't want to use any hyper-parameters for the initial or additional stages.
In that case you can specify in the plugin JSON output for the initial stage, the following keys on the root level object:
{..., hyperParamsForInitial: ["param_a"]}. This indicates which hyper-parameters are to be used for the initial stage. Use empty array[]to indicate no hyper-params are to be used in the initial stage.{..., hyperParamsForProcess: ["param_b"]}. This indicates which hyper-parameters are to be used for the additional stages. Use empty array[]to indicate no hyper-params are to be used in the additional stages.
If no hyperParamsForInitial and hyperParamsForProcess are specified, it will only use hyper-params for the initial stage.
By returning null for the additional stages' results key, for example when the param value is not the default one, it effectively skips all variations except for the default value. Note that for the initial stage we always need to return a score for it to be a valid experiment. If you don't want to score the initial stage, just always return 1.0.
If you do want a stage to returns its results, but not take it's score into consideration when looking for the highest average run, just return null for score, or don't return score at all in the JSON manifest.
Generally, only results with a status code of success returned are made available, or used in returning the highest average variation (see Plugin Results Format).
Note that the process object to specify additional stages is always taken from the initial run with the default parameters, never for any hyper-param variation.
Metrics
The plugin result JSON can have an optional custom metrics object, so that for each experiment variations you can debug the results if needed:
When you run a plugin with hyper-parameters, the metrics and scores for different parameter combinations can be found in the browser console.
Experimenting with different datasets
Hyper-param tuning can be used to experiment not only with model input parameters, but also different datasets for example.
First we define a plugin input variable called dataset_param, and its hyper-parameter value could be for example category 50,60,70,80,85,90,95.
Each value could represent the Moment where X% of users didn't convert yet. In the initial stage, we can then specify a process object that contains all these dataset moment variations, for the additional stage (see also the Result Format section):
What happens is that after the initial stage, it will run the buildModelStage stage that has access to the seven datasets. Then based on the variation of the dataset_param value, we pick the corresponding dataset by the same key. If we then output the score using the analysis done for that dataset, we automatically end up with the best scoring result, based on one of the seven datasets.
Don't forget to add {..., hyperParamsForProcess: ["dataset_param"]} to your initial plugin run results JSON, otherwise it will not use that hyper-param for additional stages.
Deployable and batch plugins
In case a plugin is marked as Deployable Plugin or Batch Plugin in the initial plugin results JSON, the plugin JSON manifest specified for those two, contain the storage dataSets and options from the run that has the highest average score.
In general adding hyper-parameters doesn't differ in any way the way you work with the manifest — the format is exactly the same as if no hyper-parameters would be used — the platform just generates a new manifest for each experimental run.