Need some way to control digits used by jsonlite

[philbrierley]

I am finding that when uploading a data.table to big query, all values seem to end up in big query but rounded to 5 decimal places. Below is the call I'm making. The same data uploaded by other libraries has no such issue. Any pointers appreciated.

bq_table_upload(x=myBQTable, values=BQ_data_table, create_disposition='CREATE_NEVER', write_disposition='WRITE_APPEND')

[byapparov]

@philbrierley, do you posing repex with data to make it easier to investigate this.

Also, an you clarify

The same data uploaded by other libraries has no such issue. Any pointers appreciated.

Does it mean that uploading data.frame through bigrquery does not have rounding, or uploading data.table to bigquery through other R package does not have rounding?

[philbrierley]

hi @byapparov, thanks for the response.

1. it's hard to give a reproducible example as you wouldn't have access to my bigquery account. just try uploading a data.table to bigquery using

bq_table_upload(x=myBQTable, values=BQ_data_table, create_disposition='CREATE_NEVER', write_disposition='WRITE_APPEND')

where BQ_data_table is the data table, making sure some of the values have lots of decimal place.

2. Just to clarify, when I say other libraries, what I mean is non-R based methods.

[byapparov]

ok. I can reproduce this, with both data.frame and data.table.

this rounding is done when data is transformed to json, see ratio values:

cars$ratio <- cars$dist / cars$speed
jsonlite::stream_out(cars)
# {"speed":24,"dist":92,"ratio":3.83333}

Solution to this would be to put digits as bigrquery package option to allow users to choose precision:

jsonlite::stream_out(cars, digits = 10)
# {"speed":24,"dist":92,"ratio":3.8333333333}

[jennybc]

Related to r-lib/httr#153, r-lib/httr#455, and r-lib/httr#539.

Yes, some way needs to be chosen to expose toJSON()'s digits to the motivated bigrquery user.

[philbrierley]

Thanks for finding the issue. As my data set has some small numbers then a digits parameter would be much welcomed.

[aldomann]

Any updates regarding this issue?

[ncriswell]

The default package allows for the ability to change the default argument of a function. I have tried default(toJSON) <- list(digits = 10) but that doesn't seem to work. stream_out only has a digits argument in the ... which seems invisible to the default function. Just wanted to add this in case it spurs any thoughts in others.