Test matrix is slowing down the contribution process

[amureki]

Describe the issue

tox test matrix became convoluted because of the multiple edge cases, we need to think on how we can improve it.
I know, that changing the tool is not supposed to solve this problem, but maybe if we are going to touch the code, it'd be easier to switch to GitHub Actions matrix.

To Reproduce
Coming from #504

Expected behavior
A simpler process of updating Python and Django versions