favorite_tools

Tools, packages and so on that I've used in my work and want to remember.

..this started out as a list of libraries that I just didn't want to forget, but now I'm having fun remembering all the libraries that I've used and loved in the past :)

Editors

Atom

• Vim Mode Plus
• Atom Reindent
• Terraform syntax highlighting
• Terraform linting
• Hydrogen - Run code blocks like in Jupyter notebooks
• Pretty JSON

Vim

• Syntastic

Git

• gitup - GUI for git, also visualizes branches nicely
• redate - useful if system time was wrong when comitting
• BFG Repo Cleaner - for finding and removing large files and passwords comitted to git

Shell

• bat - A cat(1) clone with wings.
• imgcat for iTerm2 - It's like cat but for images.

Fish

• fish
• Oh My Fish package manager
• bobthefish theme
• plugin-git
• chatgpt-prompt-wrapper
• zsh rocketship

zsh

• oh my zsh
• zsh-autosuggestions
• history

AWS

• aws-shell
• awslogs Much easier than having to use aws-shell or the AWS GUI
• saw Much like awslogs, but has a handy watch feature.

Hosted Services

AWS

• EC2
• Lambda
• S3
• RDS (Aurora)
• DynamoDB
• ElastiCache (Redis and memcached)
• DocumentDB
• Redshift
• VPC
• CloudFront
• Route 53
• API Gateway
• CodeBuild
• CodeDeploy
• CodePipeline
• CloudWatch
• CloudFormation
• CloudTrail
• IAM
• Cognito
• Secrets Manager
• Key Management Service
• AWS Amplify
• AppSync
• Simple Queue Service
• Elastic Container Registry
• aws batch

GCE

• bigquery

Cloud Infrastructure

• Terraform
• AWS Amplify - great for quick prototyping, although gets tricky if you try to do stuff not in the docs.
• brainboard

Frontend

• Material UI

Python

• prebaked lambda layers

Environment Management

• pyenv
  • pyenv virtualenv
• pipenv R.I.P 🧟🧟🧟
• pipdeptree - I use this when resolving dependency issues in pipenv
• virtualenvwrapper - makes it super easy to switch between virtualenvs
• autoenv and autoenv_fish
• direnv

Caching

• cachetools - Has a lot of useful caching decorators
• async-lru - Simple LRU caching for coroutines
• asyncache - Like cachetools, but supports coroutines

Messaging

• nameko - Python framework for building microservices
• pyzmq - Probably my most favorite messagig library of all times.

Job processing

• python-rq

Data Structures

• frozendict - Immutable dictionary
• bson, in particular bson.json_util for easy serialization of datetime and ObjectId types
• Python Sorted Containers

Data Science

• numpy
• pandas
• dask
• dask-kubernetes - I used this to spin up clustres for in memory analysis of TB sized data, then shutting down the VMs after completion.
• jupiter
• jupytext
• awswrangler
• papermill
• pandas-profiling

Machine Learning

• https://github.com/benfred/implicit/
• Annoy

Geospatial

• geopandas
• osmnx

Plotting

• bokeh - Great library for making streaming graphs and dashboards redered to javascript in the browser
• brewer2mpl - useful little package with nice matplitlib compatible color maps
• folium
• matplotlib - lately there are so many high-level python plotting libs available, so I don't Matplotlib use so much. But it's the old grandfather, and I used to swear by it, so on the list it goes :)level details.
• seaborn

DB

• pymongo
• sqlalchemy
• ponyORM
• motor
• aioredis

Web development

• flask
• aiohttp
• aiohttp-sentry
• fastapi
• sqlmodel

Testing

flake8 - My favorite linter. I find it easy to configure. pytest-cov - easy test coverage pytest-xdist - Run tests in parallel

Linting and Type Checking

• black - Keeps the python code syntax clean and readable.
• black for jupyter notebooks
• isort - Manages module import order
• mypy - static type checker. Remember to install mypy language server
• pycln - auto remove unused imports

Other

• tenacity - Super useful library for retrying. I use it often with HTTP calls on unstable connections.
• scoop - I used to use this to distribute workloads over clusters of machines in my lab. Probably outdated by now.
• faker I use this to create mock data for tests
• hashids
• tqdm - progress bars in scripts printed to terminal
• progressbar2 - another progressbar library. I use it similarly to tqdm.
• wget
• retry

Third Party Services

• Datadog - I use use datadog to monitor most of my deployed containers
• Sentry - Use it for services that contain a lot of state that's useful when debugging
• Apiary - Online parser and formatter of API Blueprint markup

Other tools

Airflow Docker Kubernetes

Firefox

• simple tab groups
• yomichan
• https everywhere
• Firefox multi-account containers
• uBlock origin
• facebook container, twitter container, amazon container, google container

Reading and Documentation

• Python 3 Patterns, Recipes and Idioms

Data engineering

• Panderas - Python dataframe schema
• Great Expectations - Data validation, e.g. for ETL
• dbt
• dolt

CICD

• Github action linter