Batch upload refactored #732
Conversation
A few places make Django ORM queries at the module level and thus run during import of the portal code. This implies that a working connection to a properly set-up database is required to e.g. use the Django management command. This greatly complicates packaging, initial setup, and deployment. This problem is fixed by simple refactorings and caches.
Creation of the SQLAlchemy model caches occurs during import of the portal code. This implies that a working connection to a properly set-up database is required to e.g. use the Django management command. This greatly complicates packaging, initial setup, and deployment. This problem is fixed by ignoring errors creating the cache and reminding the user to check the database.
The `iterkeys()` method on a dict has been replaced by the equivalent `keys()` method in Python 3. Update to use the correct method.
Reject duplicate names in the controlled vocabularies list. It is unknown why such duplicates are being served, but they are as of 2022-12-27.
With release 0.17.0, organization_id is now the identifying field for site registration. I missed this correction in resolving merge conflicts.
Fix initial setup
352KB file (one column): 148s -> 1.6s (cherry picked from commit 0f80fe5)
(cherry picked from commit 832cc38)
…nsor measurements (cherry picked from commit 8f950bc)
(cherry picked from commit aecd2c7)
(cherry picked from commit 6dc8eea)
(cherry picked from commit 4365f6a)
(cherry picked from commit 484ca0f)
(cherry picked from commit 434ab81)
(cherry picked from commit e90b108)
(cherry picked from commit a1e3eaf)
(cherry picked from commit 87a6c53)
(cherry picked from commit 1002c8d)
(cherry picked from commit 5a24db1)
(cherry picked from commit fca8b31)
Request.data is of type QueryDict (https://docs.djangoproject.com/en/3.0/ref/request-response/#django.http.QueryDict) which subclasses the standard Dictionary. One notable change is that QueryDict contains a false mutability flag (true by default), making the pop commands trigger an exception. Changing to get so that this does not require mutability (e.g. removing item from the dictionary)
Remove the undefined variable from message (causing a new NameError). This will stop the exception for reraising a new exception and triggering a 500 response. Update the except generic exception to only catch SQLAlchemy errors.
aufdenkampe
added this to the v0.18 release. EnviroDIY performance improvements milestone
|
@ptomasula just deployed this to staging for @SRGDamia1 to test from: |
|
I'm assuming from the comment on #649 Is this a good place to make comments or where? [2024-10-02 09:22:49] pubDQTR Sending data to [ 0 ] staging.monitormywatershed.org:80 |
There was a problem hiding this comment.
Excited to see this moving forward!
Did some testing on the staging instance and it appears commit 8601a06 has broken the ability to have multiple data points per request. Because the sampling_feature key is no longer removed, the check for an unequal number of data points will fail.
I don't know why I didn't run into this during development (maybe Django versions?) but in any case I have filed another PR #734 to fix this and a couple other minor issues I noticed.
Merging that PR should update this one automatically. Please do that and then re-deploy and I can continue testing.
As described in #674 (comment), @ptomasula created this
batch_uploadbranch by cherrypicking commits from this PR by @tpwrules:This PR reflects that work, and will also close: