update docs, ray support, enable .xyz.plot

jcmgray · Jan 13, 2024 · ea5421b · ea5421b
1 parent da969da
commit ea5421b
Show file tree

Hide file tree

Showing 20 changed files with 367,203 additions and 174,018 deletions.
diff --git a/docs/conf.py b/docs/conf.py
@@ -148,4 +148,5 @@ def linkcode_resolve(domain, info):
  "numpy": ("https://numpy.org/doc/stable/", None),
  "xarray": ("http://xarray.pydata.org/en/stable/", None),
  "pandas": ("https://pandas.pydata.org/docs/", None),
+ "matplotlib": ("https://matplotlib.org/stable/", None),
 }
diff --git a/docs/examples/basic output example.ipynb b/docs/examples/basic output example.ipynb
diff --git a/docs/examples/complex output example.ipynb b/docs/examples/complex output example.ipynb
diff --git a/docs/examples/crop example.ipynb b/docs/examples/crop example.ipynb
diff --git a/docs/examples/dask distributed example.ipynb b/docs/examples/dask distributed example.ipynb
diff --git a/docs/examples/farming example.ipynb b/docs/examples/farming example.ipynb
diff --git a/docs/examples/visualize linear algebra.ipynb b/docs/examples/visualize linear algebra.ipynb
diff --git a/docs/examples/visualize matrix.ipynb b/docs/examples/visualize matrix.ipynb
diff --git a/docs/gen_parallel.ipynb b/docs/gen_parallel.ipynb
@@ -37,7 +37,7 @@
  "name": "stderr",
  "output_type": "stream",
  "text": [
- "100%|##########| 8/8 [00:02<00:00, 3.44it/s]\n"
+ "100%|##########| 8/8 [00:01<00:00, 6.56it/s]\n"
  ]
  },
  {
@@ -177,42 +177,95 @@
  "\n",
  "1. Creation with:\n",
  "\n",
- " * a unique ``name`` to call this set of runs\n",
+ " * a unique ``name`` to call this set of runs, defaulting the function name\n",
  " * a ``fn`` if not creating from an ``Harvester`` or ``Runner``\n",
  " * other optional settings such as ``batchsize`` controlling how many runs to group into one.\n",
  "\n",
+ ":::{hint}\n",
+ "You can automatically load all crops in the current directory (or a specific one) to a dictionary by calling the function {func}`xyzpy.load_crops`.\n",
+ ":::\n",
+ "\n",
  "2. **'Sow'**. Use {meth}`xyzpy.Crop.sow_combos` to write ``combos`` into batches on disk.\n",
  "\n",
  "3. **'Grow'**. Grow each batch. This can be done a number of ways:\n",
  "\n",
- " * In another different process, navigate to the same directory and run for example ``python -c \"import xyzpy; c = xyzpy.Crop(name=...); xyzpy.grow(i, crop=c)\"`` to grow the ith batch of crop with specified name. See {func}`~xyzpy.grow` for other options. This could manually be put in a script to run on a batch system.\n",
+ "* The script `xyzpy-grow` can be called from the command line to process all batches. It has signature:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "488e6578",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "usage: xyzpy-grow [-h] [--parent-dir PARENT_DIR] [--debug]\n",
+ " [--num-threads NUM_THREADS] [--num-workers NUM_WORKERS]\n",
+ " [--ray] [--gpus-per-task GPUS_PER_TASK]\n",
+ " [--verbosity VERBOSITY]\n",
+ " crop_name\n",
+ "\n",
+ "Grow crops using xyzpy-gen-cropping.\n",
+ "\n",
+ "positional arguments:\n",
+ " crop_name The name of the crop to grow.\n",
+ "\n",
+ "options:\n",
+ " -h, --help show this help message and exit\n",
+ " --parent-dir PARENT_DIR\n",
+ " The parent directory of the crop.\n",
+ " --debug Enable debugging mode.\n",
+ " --num-threads NUM_THREADS\n",
+ " The number of threads per worker (OMP_NUM_THREADS\n",
+ " etc.)\n",
+ " --num-workers NUM_WORKERS\n",
+ " The number of worker processes to use.\n",
+ " --ray Use a ray executor, either connecting to an existing\n",
+ " cluster, or starting a new one with num_workers\n",
+ " --gpus-per-task GPUS_PER_TASK\n",
+ " The number of gpus to request per task, if using a ray\n",
+ " executor. The overall GPUs available is set by\n",
+ " CUDA_VISIBLE_DEVICES, which ray follows.\n",
+ " --verbosity VERBOSITY\n",
+ " The verbosity level.\n"
+ ]
+ }
+ ],
+ "source": [
+ "!xyzpy-grow --help"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "27714fd0",
+ "metadata": {},
+ "source": [
  "\n",
- " * Use {meth}`xyzpy.Crop.grow_cluster` - experimental! This automatically generates and submits a script using SGE, PBS or SLURM. See its options and {func}`xyzpy.Crop.gen_cluster_script` for the template scripts.\n",
  "\n",
- " * Use {meth}`xyzpy.Crop.grow` or {meth}`xyzpy.Crop.grow_missing` to complete some or all of the batches locally. This can be useful to a) finish up a few missing/errored runs b) run all the combos with persistent progress, so that one can restart the runs at a completely different time/ with updated functions etc.\n",
+ "* In another python process, navigate to the same directory and run for example ``python -c \"import xyzpy; c = xyzpy.Crop(name=...); xyzpy.grow(i, crop=c)\"`` to grow the ith batch of crop with specified name. See {func}`~xyzpy.grow` for other options. This could manually be put in a script to run on a batch system.\n",
  "\n",
- "4. Watch the progress. ``Crop.__repr__`` will show how many batches have been completed of the total sown.\n",
+ "* Use [`Crop.grow_cluster`](xyzpy.gen.cropping.grow_cluster) - experimental! This automatically generates and submits a script using SGE, PBS or SLURM. See its options and [`Crop.gen_cluster_script`](xyzpy.gen.cropping.gen_cluster_script) for the template scripts.\n",
  "\n",
- "5. **'Reap'**. Once all the batches have completed, run {meth}`xyzpy.Crop.reap` to collect the results and remove the batches' temporary directory. If the crop originated from a ``Runner`` or ``Harvester``, the data will be labelled, merged and saved accordingly.\n",
+ "* Use {meth}`xyzpy.Crop.grow` or {meth}`xyzpy.Crop.grow_missing` to complete some or all of the batches in the local process. This can be useful to A. finish up a few missing/errored runs B. run all the combos with persistent progress, so that one can restart the runs at a completely different time/ with updated functions etc.\n",
  "\n",
- "See the full demonstrations in :ref:`Examples`.\n",
+ "4. Watch the progress. ``Crop.__str__`` will show how many batches have been completed of the total sown.\n",
  "\n",
+ "5. **'Reap'**. Once all the batches have completed, run {meth}`xyzpy.Crop.reap` to collect the results and remove the batches' temporary directory. If the crop originated from a ``Runner`` or ``Harvester``, the data will be labelled, merged and saved accordingly.\n",
  "\n",
  ":::{note}\n",
- "You can reap an unfinished ``Crop`` as long as there is at least one result by passing the ``allow_incomplete=True`` option to {meth}`~xyzpy.Crop.reap`.\n",
- "Note that missing results will be represented by ``numpy.nan`` which might effect the eventual ``dtype`` of harvested results.\n",
- "To avoid this, consider also setting ``sync=False`` to avoid writing anything to disk until the full ``Crop`` is finished.\n",
+ "You can reap an unfinished ``Crop`` as long as there is at least one result by passing the ``allow_incomplete=True`` option to {meth}`~xyzpy.Crop.reap`. Note that missing results will be represented by ``numpy.nan`` which might effect the eventual ``dtype`` of harvested results. To avoid this, consider also setting ``sync=False`` to avoid writing anything to disk until the full ``Crop`` is finished.\n",
  ":::\n",
  "\n",
- ":::{hint}\n",
- "You can automatically load all crops in the current directory (or a specific one) to a dictionary by calling the function {func}`xyzpy.load_crops`.\n",
- ":::\n"
+ "See the full demonstrations in {ref}`Examples`."
  ]
  }
  ],
  "metadata": {
  "kernelspec": {
- "display_name": "Python 3",
+ "display_name": "Python 3 (ipykernel)",
  "language": "python",
  "name": "python3"
  },
@@ -226,7 +279,7 @@
  "name": "python",
  "nbconvert_exporter": "python",
  "pygments_lexer": "ipython3",
- "version": "3.9.6"
+ "version": "3.11.3"
  }
  },
  "nbformat": 4,