-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Foot traffic data import and aggregations #479
base: master
Are you sure you want to change the base?
Foot traffic data import and aggregations #479
Conversation
antoniocarlon
commented
Apr 13, 2018
•
edited
Loading
edited
- Import foot traffic data: Import foot traffic data from file to table #474
- Aggregate foot traffic data: Create an aggregations table from the foot traffic data table #475
- Create higher (<22) zoom aggregations for foot traffic Create higher (<22) zoom aggregations for foot traffic #484
- Create metadata for aggregated tables Create metadata for aggregated tables #485
tasks/foot_traffic.py
Outdated
unnesting='+'.join([sum_unnest.format(date=d, size=HOURS_IN_A_DAY) for d in sundays]), | ||
numdates=len(sundays)), | ||
hour_aggregations=','.join([hour_aggregation.format(hour=i, size=7, | ||
hour_name='{num:02d}'.format(num=(i - 1))) for i in range(1, 25)]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (138 > 120 characters)
tasks/foot_traffic.py
Outdated
FROM ( | ||
SELECT quadkey, | ||
{oneday} oneday, {workday} workday, {weekend} weekend, | ||
{monday} monday, {tuesday} tuesday, {wednesday} wednesday, {thursday} thursday, {friday} friday, {saturday} saturday, {sunday} sunday, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (161 > 120 characters)
tasks/foot_traffic.py
Outdated
INSERT INTO "{output_schema}".{output_table} | ||
(quadkey, | ||
oneday, workday, weekend, monday, tuesday, wednesday, thursday, friday, saturday, sunday, | ||
h00, h01, h02, h03, h04, h05, h06, h07, h08, h09, h10, h11, h12, h13, h14, h15, h16, h17, h18, h19, h20, h21, h22, h23, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (137 > 120 characters)
tasks/foot_traffic.py
Outdated
|
||
aggregations = 'CEIL(({unnesting})::FLOAT / {numdates})' | ||
sum_unnest = 'UNNEST(COALESCE("{date}", array_fill(0, ARRAY[{size}])))' | ||
hour_aggregation = 'NULLIF(ARRAY[(ARRAY_AGG(monday))[{hour}]::integer, (ARRAY_AGG(tuesday))[{hour}]::integer, (ARRAY_AGG(wednesday))[{hour}]::integer, (ARRAY_AGG(thursday))[{hour}]::integer, (ARRAY_AGG(friday))[{hour}]::integer, (ARRAY_AGG(saturday))[{hour}]::integer, (ARRAY_AGG(sunday))[{hour}]::integer]::integer[], array_fill(0, ARRAY[{size}])) h{hour_name}' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (370 > 120 characters)
tasks/foot_traffic.py
Outdated
|
||
def _find_day_of_week(self, columns, day): | ||
return [d for d in columns | ||
if datetime.strptime(d, '{column_prefix}%Y%m%d'.format(column_prefix=DATA_COLUMN_PREFIX)).weekday() == day] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (123 > 120 characters)
tasks/foot_traffic.py
Outdated
|
||
def latlng2tile(lng, lat, z): | ||
x = math.floor((lng + 180) / 360 * math.pow(2, z)) | ||
y = math.floor((1 - math.log(math.tan(lat * math.pi / 180) + 1 / math.cos(lat * math.pi / 180)) / math.pi) / 2 * math.pow(2, z)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (132 > 120 characters)
tasks/foot_traffic.py
Outdated
SATURDAY = 5 | ||
SUNDAY = 6 | ||
|
||
def quadkey2tile(quadkey): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
expected 2 blank lines, found 1
tasks/foot_traffic.py
Outdated
import urllib.request | ||
import math | ||
from datetime import datetime | ||
from luigi import Task, IntParameter, LocalTarget |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
'luigi.IntParameter' imported but unused
tasks/foot_traffic.py
Outdated
sum_unnest = 'UNNEST(COALESCE("{date}", array_fill(0, ARRAY[{size}])))' | ||
hour_aggregation = 'NULLIF(ARRAY[(ARRAY_AGG(monday))[{hour}]::integer, (ARRAY_AGG(tuesday))[{hour}]::integer, (ARRAY_AGG(wednesday))[{hour}]::integer, (ARRAY_AGG(thursday))[{hour}]::integer, (ARRAY_AGG(friday))[{hour}]::integer, (ARRAY_AGG(saturday))[{hour}]::integer, (ARRAY_AGG(sunday))[{hour}]::integer]::integer[], array_fill(0, ARRAY[{size}])) h{hour_name}' | ||
|
||
view = ''' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
local variable 'view' is assigned to but never used
tasks/foot_traffic.py
Outdated
|
||
def _find_day_of_week(self, columns, day): | ||
return [d for d in columns | ||
if datetime.strptime(d, '{column_prefix}%Y%m%d'.format(column_prefix=DATA_COLUMN_PREFIX)).weekday() == day] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (123 > 120 characters)
tasks/foot_traffic.py
Outdated
fields=','.join([field_column.format(sumfield=','.join([sumfield.format(field=c, | ||
size=HOURS_IN_A_DAY, | ||
i=i) for i in range(1, HOURS_IN_A_DAY + 1)]), size=HOURS_IN_A_DAY, | ||
field=c) for c in data_columns]), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (171 > 120 characters)
continuation line over-indented for visual indent
tasks/foot_traffic.py
Outdated
'''.format( | ||
fields=','.join([field_column.format(sumfield=','.join([sumfield.format(field=c, | ||
size=HOURS_IN_A_DAY, | ||
i=i) for i in range(1, HOURS_IN_A_DAY + 1)]), size=HOURS_IN_A_DAY, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (158 > 120 characters)
tasks/foot_traffic.py
Outdated
|
||
def run(self): | ||
if self.zoom > MAX_ZOOM_LEVEL: | ||
raise ValueError('Zoom level {zoom} cannot be greater than the maximum available zoom level ({maxzoom})'.format(zoom=self.zoom, maxzoom=MAX_ZOOM_LEVEL)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (164 > 120 characters)
tasks/foot_traffic.py
Outdated
FROM ( | ||
SELECT {quadkey_field}, | ||
{oneday} oneday, {workday} workday, {weekend} weekend, | ||
{monday} monday, {tuesday} tuesday, {wednesday} wednesday, {thursday} thursday, {friday} friday, {saturday} saturday, {sunday} sunday, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (161 > 120 characters)
tasks/foot_traffic.py
Outdated
INSERT INTO "{output_schema}".{output_table} | ||
({quadkey_field}, | ||
oneday, workday, weekend, monday, tuesday, wednesday, thursday, friday, saturday, sunday, | ||
h00, h01, h02, h03, h04, h05, h06, h07, h08, h09, h10, h11, h12, h13, h14, h15, h16, h17, h18, h19, h20, h21, h22, h23, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (137 > 120 characters)
tasks/foot_traffic.py
Outdated
COMMENT ON COLUMN "{schema}".{table}.sunday IS 'Aggregated 0 to 24 hour data for a typical sunday'; | ||
COMMENT ON COLUMN "{schema}".{table}.oneday IS 'Aggregated 0 to 24 hour data for a typical day'; | ||
COMMENT ON COLUMN "{schema}".{table}.workday IS 'Aggregated 0 to 24 hour data for a typical workday'; | ||
COMMENT ON COLUMN "{schema}".{table}.weekend IS 'Aggregated 0 to 24 hour data for a typical weekend day'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (121 > 120 characters)
tasks/foot_traffic.py
Outdated
); | ||
COMMENT ON COLUMN "{schema}".{table}.monday IS 'Aggregated 0 to 24 hour data for a typical monday'; | ||
COMMENT ON COLUMN "{schema}".{table}.tuesday IS 'Aggregated 0 to 24 hour data for a typical tuesday'; | ||
COMMENT ON COLUMN "{schema}".{table}.wednesday IS 'Aggregated 0 to 24 hour data for a typical wednesday'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (121 > 120 characters)
tasks/foot_traffic.py
Outdated
'''.format( | ||
fields=','.join([field_column.format(sumfield=','.join([sumfield.format(field=c, | ||
size=HOURS_IN_A_DAY, | ||
i=i) for i in range(1, HOURS_IN_A_DAY + 1)]), size=HOURS_IN_A_DAY, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (158 > 120 characters)
tasks/foot_traffic.py
Outdated
sumfield = 'SUM((COALESCE({field}, array_fill(0, ARRAY[{size}])))[{i}])' | ||
query = ''' | ||
INSERT INTO "{schema}".{table} | ||
SELECT SUBSTRING({quadkey_field} FROM 1 FOR {zoom}) {quadkey_field}, ST_Centroid(ST_Union({geom_field})) {geom_field}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (134 > 120 characters)
|
||
def latlng2tile(lng, lat, z): | ||
x = math.floor((lng + 180) / 360 * math.pow(2, z)) | ||
y = math.floor((1 - math.log(math.tan(lat * math.pi / 180) + 1 / math.cos(lat * math.pi / 180)) / math.pi) / 2 * math.pow(2, z)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
undefined name 'math'
line too long (132 > 120 characters)
|
||
|
||
def latlng2tile(lng, lat, z): | ||
x = math.floor((lng + 180) / 360 * math.pow(2, z)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
undefined name 'math'
|
||
lon = x / n * 360.0 - 180.0 | ||
lat_rad = math.atan(math.sinh(math.pi * (1 - 2 * y / n))) | ||
lat = - math.degrees(lat_rad) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
undefined name 'math'
n = 2.0 ** z | ||
|
||
lon = x / n * 360.0 - 180.0 | ||
lat_rad = math.atan(math.sinh(math.pi * (1 - 2 * y / n))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
undefined name 'math'
unnesting='+'.join([sum_unnest.format(date=d, size=HOURS_IN_A_DAY) for d in sundays]), | ||
numdates=len(sundays)), | ||
hour_aggregations=','.join([hour_aggregation.format(hour=i, size=7, | ||
hour_name='{num:02d}'.format(num=(i - 1))) for i in range(1, 25)]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (138 > 120 characters)
); | ||
COMMENT ON COLUMN "{schema}".{table}.monday IS 'Aggregated 0 to 24 hour data for a typical monday'; | ||
COMMENT ON COLUMN "{schema}".{table}.tuesday IS 'Aggregated 0 to 24 hour data for a typical tuesday'; | ||
COMMENT ON COLUMN "{schema}".{table}.wednesday IS 'Aggregated 0 to 24 hour data for a typical wednesday'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (121 > 120 characters)
fields=','.join([field_column.format(sumfield=','.join([sumfield.format(field=c, | ||
size=HOURS_IN_A_DAY, | ||
i=i) for i in range(1, HOURS_IN_A_DAY + 1)]), size=HOURS_IN_A_DAY, | ||
field=c) for c in data_columns]), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (171 > 120 characters)
continuation line over-indented for visual indent
'''.format( | ||
fields=','.join([field_column.format(sumfield=','.join([sumfield.format(field=c, | ||
size=HOURS_IN_A_DAY, | ||
i=i) for i in range(1, HOURS_IN_A_DAY + 1)]), size=HOURS_IN_A_DAY, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (158 > 120 characters)
sumfield = 'SUM((COALESCE({field}, array_fill(0, ARRAY[{size}])))[{i}])' | ||
query = ''' | ||
INSERT INTO "{schema}".{table} | ||
SELECT SUBSTRING({quadkey_field} FROM 1 FOR {zoom}) {quadkey_field}, ST_Centroid(ST_Union({geom_field})) {geom_field}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (134 > 120 characters)
|
||
def run(self): | ||
if self.zoom > MAX_ZOOM_LEVEL: | ||
raise ValueError('Zoom level {zoom} cannot be greater than the maximum available zoom level ({maxzoom})'.format(zoom=self.zoom, maxzoom=MAX_ZOOM_LEVEL)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line too long (164 > 120 characters)
@@ -0,0 +1,573 @@ | |||
from datetime import datetime | |||
from luigi import Task, IntParameter, LocalTarget |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
'luigi.LocalTarget' imported but unused
@@ -0,0 +1,81 @@ | |||
import requests |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
'requests' imported but unused
What is the state of this PR? |
It's on hold (like the rest of the foot traffic project) |