Add RQPool management command

[chromium7]

Ref #621

---

Manual testing done:

1. Generate x number of jobs
2. Run rqworker and rqpool commands in burst mode with verbosity level of 0
3. Memory usage and time are measured using the following utility function: https://gist.github.com/chromium7/802741a5767cec234aef94e7b6149c92

Result Summary

	rqworker	rqpool	rqpool	rqpool
Number of Worker	4	4	8	24
Number of Jobs	1000	1000	1000	1000
Time Elapsed	10.336005s (per worker)	11.275090s	7.396444s	7.317025s
Peak Memory Usage	0.51407MB (per worker)	0.204661MB	0.214324MB	0.252267MB

Result Details

1000 jobs, rqworker * 4

$ python manage.py rqworker --burst --verbosity 0 & python manage.py rqworker --burst --verbosity 0 & python manage.py rqworker --burst --verbosity 0 & python manage.py rqworker --burst --verbosity 0


Function Name       : handle
Current memory usage: 0.380065MB
Peak                : 0.514592MB
Duration            : 10.309361s

Function Name       : handle
Current memory usage: 0.380201MB
Peak                : 0.51407MB
Duration            : 10.336005s

Function Name       : handle
Current memory usage: 0.380047MB
Peak                : 0.514037MB
Duration            : 10.310725s

Function Name       : handle
Current memory usage: 0.380201MB
Peak                : 0.514714MB
Duration            : 10.352149s

1000 jobs, rqpool with 4 workers

$ python manage.py rqpool --num-workers 4 --verbosity 0 --burst
Function Name       : handle
Current memory usage: 0.181683MB
Peak                : 0.204661MB
Duration            : 11.275090s

1000 jobs, rqpool with 8 workers

$ python manage.py rqpool --num-workers 8 --verbosity 0 --burst
Function Name       : handle
Current memory usage: 0.185166MB
Peak                : 0.214324MB
Duration            : 7.396444s

1000 jobs, rqpool with 24 workers

$ python manage.py rqpool --num-workers 24 --verbosity 0 --burst
Function Name       : handle
Current memory usage: 0.195509MB
Peak                : 0.252267MB
Duration            : 7.317025s

[chromium7]

some more manual testing with more jobs and multiple queues

Setup

Summary

	rqworker	rqpool	rqpool
Number of Worker	4	4	16
Number of Jobs	5500	5500	5500
Time Elapsed	51.016926s(per worker)	49.156525s	30.708005s
Peak Memory Usage	0.574323MB (per worker)	0.23773MB	0.264708MB

rqworker 4 separate workers

$ python manage.py rqworker default low medium --burst --verbosity 0 & \
python manage.py rqworker default low medium --burst --verbosity 0 & \
python manage.py rqworker default low medium --burst --verbosity 0 & \
python manage.py rqworker default low medium --burst --verbosity 0

Function Name       : handle
Current memory usage: 0.4393MB
Peak                : 0.574323MB
Duration            : 51.016926s

Function Name       : handle
Current memory usage: 0.439487MB
Peak                : 0.57451MB
Duration            : 51.103177s

Function Name       : handle
Current memory usage: 0.439599MB
Peak                : 0.574622MB
Duration            : 51.239900s

Function Name       : handle
Current memory usage: 0.43938MB
Peak                : 0.574456MB
Duration            : 51.193902s

rqworker-pool 4 workers

$ python manage.py rqworker-pool default low medium --num-workers 4 --verbosity 0 --burst
Function Name       : handle
Current memory usage: 0.185496MB
Peak                : 0.23773MB
Duration            : 49.156525s

rqworker-pool 16 workers (my laptop has 16 cores)

$ python manage.py rqworker-pool default low medium --num-workers 16 --verbosity 0 --burst
Function Name       : handle
Current memory usage: 0.192702MB
Peak                : 0.264708MB
Duration            : 30.708005s