Merge pull request #695 from Epistimio/release-v0.2.0rc1

Release v0.2.0rc1

LICENSE

@@ -1,6 +1,6 @@
 Software License Agreement (BSD License)
 
- Copyright (c) 2017-2020, Epistímio.
+ Copyright (c) 2017-2021, Epistímio.
  All rights reserved.
 
  Redistribution and use in source and binary forms, with or without

README.rst

@@ -118,7 +118,7 @@ If you use Oríon for published work, please cite our work using the following b
 
 .. code-block:: bibtex
 
- @software{xavier_bouthillier_2021_0_1_15,
+ @software{xavier_bouthillier_2021_0_2_0,
  author = {Xavier Bouthillier and
  Christos Tsirigotis and
  François Corneau-Tremblay and
@@ -142,10 +142,10 @@ If you use Oríon for published work, please cite our work using the following b
  Pascal Lamblin and
  Christopher Beckham},
  title = {{Epistimio/orion: Asynchronous Distributed Hyperparameter Optimization}},
- month = may,
+ month = nov,
  year = 2021,
  publisher = {Zenodo},
- version = {v0.1.17},
+ version = {v0.2.0},
  doi = {10.5281/zenodo.3478592},
  url = {https://doi.org/10.5281/zenodo.3478592}
  }

ROADMAP.md

@@ -1,25 +1,35 @@
 # Roadmap
-Last update Sep 14th, 2021
+Last update Nov 23rd, 2021
 
 ## Next releases - Short-Term
 
-### v0.2
+### v0.2.1
 
-#### Generic `Optimizer` interface supporting various types of algorithms
+- New master process to enhance parallelisation efficiency.
+- [PBT](https://arxiv.org/abs/1711.09846)
 
-Change interface to support trial object instead of curated lists. This is necessary to support algorithms such as PBT.
+### v0.2.2
+
+- Use shared algo serialization instead of replications to enhance parallelisation efficiency.
+- [DEBH](https://arxiv.org/abs/2105.09821)
+
+### v0.2.3
+
+- [HEBO](https://github.com/huawei-noah/HEBO/tree/master/HEBO/archived_submissions/hebo)
+
+### v0.2.4
 
-#### More Optimizers
-- [PBT](https://arxiv.org/abs/1711.09846)
 - [BOHB](https://ml.informatik.uni-freiburg.de/papers/18-ICML-BOHB.pdf)
 
+## Next releases - Mid-Term
+
 #### Simple dashboard specific to monitoring and benchmarking of Black-Box optimization
 - Specific to hyper parameter optimizations
 - Provide status of experiments
 
 #### Leveraging previous experiences
 Leveraging the knowledge base contained in the EVC of previous trials to optimize and drive new
- trials.
+trials.
 
 ## Next releases - Long-Term
 

docs/src/code/algo/asha.rst

@@ -3,6 +3,6 @@ Asynchronous Successive Halving Algorithm
 
 Can't build documentation because of import order.
 Sphinx is loading ``orion.algo.asha`` before ``orion.algo`` and therefore
-there is a cycle between the definition of ``OptimizationAlgorithm`` and
+there is a cycle between the definition of ``BaseAlgorithm`` and
 ``ASHA`` as the meta-class ``Factory`` is trying to import ``ASHA``.
 `PR #135 <https://github.com/Epistimio/orion/pull/135/files>`_ should get rid of this problem.

docs/src/code/core/io/database.rst

@@ -12,5 +12,3 @@ Databases
 .. automodule:: orion.core.io.database
  :members:
  :show-inheritance:
-
-

docs/src/code/core/utils.rst

@@ -9,7 +9,6 @@ Utilities
  utils/format_trials
  utils/format_terminal
  utils/singleton
- utils/points
 
 .. automodule:: orion.core.utils
  :members:

docs/src/install/gettingstarted.rst

@@ -55,7 +55,7 @@ For the previous example, we would run
 
 .. code-block:: console
 
- $ orion hunt -n <experiment name> script.py --lr~'loguniform(1e-5, 1.0)'
+ $ orion hunt -n <experiment name> --max-trials 10 python script.py --lr~'loguniform(1e-5, 1.0)'
 
 This is going to start the optimization process using the default optimization algorithm and sample
 the values for the ``lr`` hyper-parameter in a log uniform distribution between 0.00001 et 1.0. Each

docs/src/user/config.rst

@@ -110,7 +110,7 @@ Full Example of Global Configuration
  heartbeat: 120
  interrupt_signal_code: 130
  max_broken: 10
- max_idle_time: 60
+ reservation_timeout: 60
  max_trials: 1000000000
  user_script_config: config
 
@@ -365,7 +365,7 @@ Worker
  heartbeat: 120
  interrupt_signal_code: 130
  max_broken: 10
- max_idle_time: 60
+ reservation_timeout: 60
  max_trials: 1000000000
  user_script_config: config
 
@@ -464,21 +464,37 @@ max_broken
  Maximum number of broken trials before worker stops.
 
 
+.. _config_worker_reservation_timeout:
+
+reservation_timeout
+~~~~~~~~~~~~~~~~~~~
+
+:Type: int
+:Default: 60
+:Env var: ORION_RESERVATION_TIMEOUT
+:Description:
+ Maximum time the experiment can spend trying to reserve a new suggestion. Such timeout are
+ generally caused by slow database, large number of concurrent workers leading to many race
+ conditions or small search spaces with integer/categorical dimensions that may be fully
+ explored.
+
 
 .. _config_worker_max_idle_time:
 
 max_idle_time
 ~~~~~~~~~~~~~
 
+.. warning::
+
+ **DEPRECATED.** This argument will be removed in v0.3.
+ Use :ref:`config_worker_reservation_timeout` instead.
+
 :Type: int
 :Default: 60
 :Env var: ORION_MAX_IDLE_TIME
 :Description:
- Maximum time the producer can spend trying to generate a new suggestion.Such timeout are
- generally caused by slow database, large number of concurrent workers leading to many race
- conditions or small search spaces with integer/categorical dimensions that may be fully
- explored.
-
+ (DEPRECATED) This argument will be removed in v0.3. Use :ref:`config_worker_reservation_timeout`
+ instead.
 
 
 .. _config_worker_interrupt_signal_code:

docs/src/user/parallel.rst

@@ -42,7 +42,7 @@ Executor backends
 It is also possible to execute multiple workers using the argument ``--n-workers`` in commandline
 or ``experiment.workon(n_workers)`` using the python API. The workers will work together
 using the same mechanisms explained above, but an
-:class:`orion.executor.base.Executor` backend will be used in addition
+:class:`orion.executor.base.BaseExecutor` backend will be used in addition
 to spawn the workers and maintain them alive. The default backend is :ref:`executor-joblib`.
 
 You can configure it

docs/src/user/storage.rst

@@ -477,18 +477,22 @@ Here's an example on how you could remove an experiment
 --------------
 
 .. automethod:: orion.core.io.database.Database.read
+ :noindex:
 
 :hidden:`write`
 ---------------
 
 .. automethod:: orion.core.io.database.Database.write
+ :noindex:
 
 :hidden:`remove`
 ----------------
 
 .. automethod:: orion.core.io.database.Database.remove
+ :noindex:
 
 :hidden:`read_and_write`
 ------------------------
 
 .. automethod:: orion.core.io.database.Database.read_and_write
+ :noindex:

setup.py

@@ -46,19 +46,24 @@
  "console_scripts": [
  "orion = orion.core.cli:main",
  ],
- "OptimizationAlgorithm": [
+ "BaseAlgorithm": [
  "random = orion.algo.random:Random",
  "gridsearch = orion.algo.gridsearch:GridSearch",
  "asha = orion.algo.asha:ASHA",
  "hyperband = orion.algo.hyperband:Hyperband",
  "tpe = orion.algo.tpe:TPE",
  "EvolutionES = orion.algo.evolution_es:EvolutionES",
  ],
- "Storage": [
+ "Database": [
+ "ephemeraldb = orion.core.io.database.ephemeraldb:EphemeralDB",
+ "pickleddb = orion.core.io.database.pickleddb:PickledDB",
+ "mongodb = orion.core.io.database.mongodb:MongoDB",
+ ],
+ "BaseStorageProtocol": [
  "track = orion.storage.track:Track",
  "legacy = orion.storage.legacy:Legacy",
  ],
- "Executor": [
+ "BaseExecutor": [
  "singleexecutor = orion.executor.single_backend:SingleExecutor",
  "joblib = orion.executor.joblib_backend:Joblib",
  "dask = orion.executor.dask_backend:Dask",

src/orion/algo/asha.py

@@ -155,6 +155,8 @@ def __init__(
 
  self.brackets = self.create_brackets()
 
+ self.seed_rng(seed)
+
  def compute_bracket_idx(self, num):
  def assign_resources(n, remainings, totals):
  if n == 0 or remainings.sum() == 0:
@@ -192,7 +194,7 @@ def sample(self, num):
  return samples
 
  def suggest(self, num):
- return super(ASHA, self).suggest(1)
+ return super(ASHA, self).suggest(num)
 
  def create_bracket(self, i, budgets, iteration):
  return ASHABracket(self, budgets, iteration)
@@ -217,32 +219,39 @@ def sample(self, num):
  should_have_n_trials = self.rungs[0]["n_trials"]
  return self.hyperband.sample_for_bracket(num, self)
 
- def get_candidate(self, rung_id):
- """Get a candidate for promotion"""
+ def get_candidates(self, rung_id):
+ """Get a candidate for promotion
+
+ Raises
+ ------
+ TypeError
+ If get_candidates is called before the entire rung is completed.
+ """
  rung = self.rungs[rung_id]["results"]
  next_rung = self.rungs[rung_id + 1]["results"]
 
  rung = list(
  sorted(
- (objective, point)
- for objective, point in rung.values()
+ (objective, trial)
+ for objective, trial in rung.values()
  if objective is not None
  )
  )
  k = len(rung) // self.hyperband.reduction_factor
  k = min(k, len(rung))
 
+ candidates = []
  for i in range(k):
- point = rung[i][1]
- _id = self.hyperband.get_id(point, ignore_fidelity=True)
+ trial = rung[i][1]
+ _id = self.hyperband.get_id(trial, ignore_fidelity=True)
  if _id not in next_rung:
- return point
+ candidates.append(trial)
 
- return None
+ return candidates
 
  @property
  def is_filled(self):
- """ASHA's first rung can always sample new points"""
+ """ASHA's first rung can always sample new trials"""
  return False
 
  def is_ready(self, rung_id=None):
@@ -254,7 +263,7 @@ def promote(self, num):
 
  The rungs are iterated over in reversed order, so that high rungs
  are prioritised for promotions. When a candidate is promoted, the loop is broken and
- the method returns the promoted point.
+ the method returns the promoted trial.
 
  .. note ::
 
@@ -266,27 +275,34 @@ def promote(self, num):
  if num < 1 or self.is_done:
  return []
 
+ candidates = []
  for rung_id in range(len(self.rungs) - 2, -1, -1):
- candidate = self.get_candidate(rung_id)
- if candidate:
-
+ for candidate in self.get_candidates(rung_id):
  # pylint: disable=logging-format-interpolation
  logger.debug(
- "Promoting {point} from rung {past_rung} with fidelity {past_fidelity} to "
+ "Promoting {trial} from rung {past_rung} with fidelity {past_fidelity} to "
  "rung {new_rung} with fidelity {new_fidelity}".format(
- point=candidate,
+ trial=candidate,
  past_rung=rung_id,
- past_fidelity=candidate[self.hyperband.fidelity_index],
+ past_fidelity=candidate.params[self.hyperband.fidelity_index],
  new_rung=rung_id + 1,
  new_fidelity=self.rungs[rung_id + 1]["resources"],
  )
  )
 
- candidate = list(copy.deepcopy(candidate))
- candidate[self.hyperband.fidelity_index] = self.rungs[rung_id + 1][
- "resources"
- ]
+ candidate = candidate.branch(
+ status="new",
+ params={
+ self.hyperband.fidelity_index: self.rungs[rung_id + 1][
+ "resources"
+ ]
+ },
+ )
+
+ if not self.hyperband.has_suggested(candidate):
+ candidates.append(candidate)
 
- return [tuple(candidate)]
+ if len(candidates) >= num:
+ return candidates
 
- return []
+ return candidates