Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[hotfix] change default requirements to production requirements #80

Merged
merged 8 commits into from
Sep 23, 2022
Merged

[hotfix] change default requirements to production requirements #80

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
cf6373f
[hotfix] change default requirements to production requirements
Croydon-Brixton Jul 13, 2022
5dcdd40
[feature] add optional mamba support in makefile
Croydon-Brixton Sep 23, 2022
adb232d
[fix] bump binder environment and fix dependencies to fix binder bugs
Croydon-Brixton Sep 23, 2022
343e1ff
[fix] fix problems with default requirements
Croydon-Brixton Sep 23, 2022
d8b1df8
[temp] remove rasterio pin to 1.1.8
Croydon-Brixton Sep 23, 2022
efbf476
[fix] loose fix requirements for main version
Croydon-Brixton Sep 23, 2022
2518dff
[clean] clean up requirements files
Croydon-Brixton Sep 23, 2022
b40b61a
[merge] merge staging branch into current
Croydon-Brixton Sep 23, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
repos:
- repo: https://github.com/psf/black
rev: 20.8b1
rev: 22.8.0
hooks:
- id: black
language_version: python3.8
language_version: python3.10
- repo: https://github.com/pycqa/pylint
rev: pylint-2.6.0
hooks:
Expand Down
25 changes: 17 additions & 8 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,12 @@ else
HAS_CONDA=True
endif

ifeq (,$(shell which mamba))
HAS_MAMBA=False
else
HAS_MAMBA=True
endif

#################################################################################
# COMMANDS #
#################################################################################
Expand All @@ -30,6 +36,7 @@ lint:

## Format geograph directory using black
format:
isort geograph
black geograph

## Set up pre-commit hooks
Expand All @@ -39,20 +46,22 @@ precommit:

## Set up python interpreter environment and install basic dependencies
env:
ifeq (True,$(HAS_CONDA))
@echo ">>> Detected conda, creating conda environment."
ifeq (True, $(HAS_MAMBA))
@echo ">>> Detected mamba, creating conda environment via mamba."

# Create the conda environment
mamba env create --prefix=./env -f requirements/environment.yml

@echo ">>> New mamba env created. Activate from project directory with:\nconda activate ./env"
else ifeq (True,$(HAS_CONDA))
@echo ">>> Detected conda, creating conda environment."

# Create the conda environment
conda env create --prefix=./env -f requirements/environment.yml

@echo ">>> New conda env created. Activate from project directory with:\nconda activate ./env"
else
@echo ">>> No conda detected. Falling back to virtualenv instead. The python verison will be that of your python3 interpreter."
$(PYTHON_INTERPRETER) -m pip install -q virtualenv virtualenvwrapper
@echo ">>> Installing virtualenvwrapper if not already installed.\nMake sure the following lines are in shell startup file\n\
export WORKON_HOME=$$HOME/.virtualenvs\nexport PROJECT_HOME=$$HOME/Devel\nsource /usr/local/bin/virtualenvwrapper.sh\n"
@bash -c "source `which virtualenvwrapper.sh`;mkvirtualenv $(PROJECT_NAME) --python=$(PYTHON_INTERPRETER)"
@echo ">>> New virtualenv created. Activate with:\nworkon $(PROJECT_NAME)"
@echo ">>> No conda detected. Please install conda or manually install requirements in your preferred python version."
endif

#################################################################################
Expand Down
36 changes: 33 additions & 3 deletions binder/environment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,38 @@
prefix: ./env
channels:
- conda-forge
- defaults
dependencies:
- python=3.8
- pip
- python=3.10
- pip==21.2.4
# linear algebra and general data analysis
- numpy ==1.23.3 # arrays, linear algebra
- pandas ==1.5.0 # tabular data analysis

# plotting
- folium ==0.12.1.post1 # plotting maps
- ipyleaflet ==0.17.1 # plotting ipywidget maps

# interactive computing
- tqdm ==4.64.1 # progress bars

# geospatial analysis requirements
# vector data
- geopandas ==0.11.1 # manipulating geospatial vector data
- shapely ==1.8.4 # working with vector shapes
- rtree ==1.0.0 # efficiently querying polygon data
# raster data
- rasterio ==1.3.2 # opening and loading raster data
- xarray ==2022.6.0 # useful data structures

# graph requirements
- networkx ==2.8.6 # manipulating graph data

# For binder
- rioxarray ==0.12.2
- seaborn ==0.12.0
- netcdf4 ==1.6.1

- pip:
- -r requirements.txt
- pylandstats ==2.4.2
- -e ..
38 changes: 0 additions & 38 deletions binder/requirements.txt
View file
Open in desktop

This file was deleted.

70 changes: 3 additions & 67 deletions requirements/dev-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,75 +1,11 @@
# NOTE: These requirements are used for developing code on the repo.
# As a standard they include certain formatters and linters.

# local package
-e ../.

# external requirements (mostly linters and formatters)
pylint # pylint linter
pylint==2.6.0 # pylint linter
mypy # python type checker
black # automatic formatting provider
black==22.8.0 # automatic formatting provider
pre-commit # for git precommit hooks
isort # automatic import sorter
python-dotenv # environment variable manager
pydocstyle # set pydocstyle

# linear algebra and general data analysis
numpy # arrays, linear algebra
scipy # linear algebra and numerical mathematics
numba # speeding up array operations
pandas # tabular data analysis

# plotting
matplotlib # general python plotting
seaborn # fancier plotting styles
descartes # geospatial plotting of shapefiles
folium # plotting maps
ipyleaflet # plotting ipywidget maps

# interactive computing
jupyterlab # jupyter notebooks
tqdm # progress bars

# geospatial analysis requirements
rasterio # opening and loading raster data
fiona # manipulating geospatial vector data
geopandas # manipulating geospatial vector data
shapely # working with vector shapes
pycrs # working with coordinate reference systems
geopy # convenient API requests to geocoders
xarray # useful data structures
rioxarray # adaptation of xarray for raterio.
dask[array] # allows to composite multiple satellite images stored in different shards
dask[dataframe] # allows more lazy operation for xarray.
dask[dataframe] # allows more lazy operation for xarray.
dask[distributed] # allows distributed computing
netCDF4 # makes sure that the default driver is netCDF4.
bottleneck # needed for fill forward (xarray.DataArray.ffil)

# additional
networkx # manipulating graph data
rtree # rtree library

# gdrive functionality
google-api-python-client
google-auth-httplib2
google-auth-oauthlib

# make videos for animating timeseries etc.
imageio
imageio-ffmpeg

# xgboost
xgboost # gradient boosted regression
sklearn # sklearn
graphviz # can plot the decission tree

#
pillow
torch==1.8
torchvision
wandb
pytorch-lightning
git+https://github.com/qubvel/segmentation_models.pytorch
hydra-core
twine # for publishing to PyPI
pydocstyle # set pydocstyle
76 changes: 2 additions & 74 deletions requirements/doc-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,77 +1,5 @@
# NOTE: These requirements are used for developing code on the repo.
# As a standard they include certain formatters and linters.

# local package
# -e ../.

# external requirements (mostly linters and formatters)
pylint # pylint linter
mypy # python type checker
black # automatic formatting provider
pre-commit # for git precommit hooks
isort # automatic import sorter
python-dotenv # environment variable manager
pydocstyle # set pydocstyle

# linear algebra and general data analysis
numpy # arrays, linear algebra
scipy # linear algebra and numerical mathematics
numba # speeding up array operations
pandas # tabular data analysis

# plotting
matplotlib # general python plotting
seaborn # fancier plotting styles
descartes # geospatial plotting of shapefiles
folium # plotting maps
ipyleaflet # plotting ipywidget maps

# interactive computing
jupyterlab # jupyter notebooks
tqdm # progress bars

# geospatial analysis requirements
rasterio # opening and loading raster data
fiona # manipulating geospatial vector data
geopandas # manipulating geospatial vector data
shapely # working with vector shapes
pycrs # working with coordinate reference systems
geopy # convenient API requests to geocoders
xarray # useful data structures
rioxarray # adaptation of xarray for raterio.
dask[array] # allows to composite multiple satellite images stored in different shards
dask[dataframe] # allows more lazy operation for xarray.
dask[dataframe] # allows more lazy operation for xarray.
dask[distributed] # allows distributed computing
netCDF4 # makes sure that the default driver is netCDF4.
bottleneck # needed for fill forward (xarray.DataArray.ffil)

# additional
networkx # manipulating graph data
rtree # rtree library

# gdrive functionality
google-api-python-client
google-auth-httplib2
google-auth-oauthlib

# make videos for animating timeseries etc.
imageio
imageio-ffmpeg

# xgboost
xgboost # gradient boosted regression
sklearn # sklearn
graphviz # can plot the decission tree

#
pillow
torch==1.8
torchvision
wandb
pytorch-lightning
git+https://github.com/qubvel/segmentation_models.pytorch
hydra-core
# NOTE: These requirements are used by readthedocs.io to autogenerate
# the documentation for this repo.

#docs
sphinx
Expand Down
27 changes: 25 additions & 2 deletions requirements/environment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,30 @@ channels:
- conda-forge
- defaults
dependencies:
- python=3.8
- python=3.10
- pip
# linear algebra and general data analysis
- numpy 1.* # arrays, linear algebra
- pandas 1.5.* # tabular data analysis

# plotting
- folium 0.12.* # plotting maps
- ipyleaflet 0.* # plotting ipywidget maps

# interactive computing
- tqdm 4.* # progress bars

# geospatial analysis requirements
# vector data
- geopandas 0.11.* # manipulating geospatial vector data
- shapely 1.* # working with vector shapes
- rtree 1.* # efficiently querying polygon data
# raster data
- rasterio 1.3.* # opening and loading raster data
- xarray 2022.* # useful data structures

# graph requirements
- networkx 2.8.* # manipulating graph data

- pip:
- -r dev-requirements.txt
- -e ..
27 changes: 0 additions & 27 deletions requirements/requirements.txt
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
"geopandas",
"shapely",
"rtree",
"rasterio==1.1.8",
"rasterio",
"xarray",
"networkx",
]
Expand Down