Error with data from UK Find a Tender Service

[duncandewhurst]

I tried to use the following command to flatten some data from UK Find a Tender Service: spoonbill release-package.json

The data has some minor structure and format issues, but nothing major.

Data
DRT results

Spoonbill output and traceback:

State file not supplied, going to analyze input file first
Analyze options:
 - threshold                      => 5
 - language                       => en
Processing file: release-package.json
Input file is release package
No schema provided, using version 1__1__5
  Processed 3 objects  []  65536/5970561    1%  00:02:17
Traceback (most recent call last):
  File "/usr/local/bin/spoonbill", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/spoonbill/cli.py", line 230, in cli
    for read, number in analyzer.analyze_file(filename, with_preview=False):
  File "/usr/local/lib/python3.7/dist-packages/spoonbill/__init__.py", line 88, in analyze_file
    for count in self.spec.process_items(items):
  File "/usr/local/lib/python3.7/dist-packages/spoonbill/stats.py", line 321, in process_items
    self.inc_table_rows(item, rows, parent_key, record)
  File "/usr/local/lib/python3.7/dist-packages/spoonbill/stats.py", line 244, in inc_table_rows
    self.add_preview_row(rows, record.get("id", ""), parent_key)
  File "/usr/local/lib/python3.7/dist-packages/spoonbill/stats.py", line 237, in add_preview_row
    row = rows.new_row(table, item_id).as_dict()
  File "/usr/local/lib/python3.7/dist-packages/spoonbill/rowdata.py", line 65, in new_row
    while parent_row.parent:
AttributeError: 'str' object has no attribute 'parent'

[jpmckinney]

@sorenabell Can you provide an estimate for fixing this bug? It is a blocker for launch (see also the update to open-contracting/spoonbill-web#353).

[sorenabell]

@jpmckinney From 1 to 3 days depending on what is the source of the issue.

[jpmckinney]

@sorenabell We will have to postpone the launch of the tool, since per open-contracting/spoonbill-web#353 the integration with the registry is entirely broken by this bug.

[sorenabell]

@jpmckinney Do you approve the work for this bug fix?

We will need more data samples. We can use the one provided by @duncandewhurst, but the more json files we have the better we could test the library.

[jpmckinney]

Yes, I approve the work. Every dataset available from data.open-contracting.org will cause the bug, per @yolile's update in open-contracting/spoonbill-web#353

[sorenabell]

@jpmckinney @yolile The fix was prepared. Please test it on https://ocp12.open-contracting.org/

[yolile]

I tested it with the document that Duncan shared and also with one from the data registry and the tool is able to produce results this time

[sorenabell]

Great.
@jpmckinney should we deploy updates to production or you would take care of that?

[jpmckinney]

I made a deployment with open-contracting/spoonbill-web#394, so I think these changes were also deployed.

[yolile]

I tested them in production now and the bug is still there, so I guess these changes weren't part of open-contracting/spoonbill-web#394

[jpmckinney]

Yeah, I saw the Sentry error come through.

[sorenabell]

Maybe the front-end wasn't deployed/updated?

[jpmckinney]

@sorenabell I did docker-compose pull and docker-compose up -d. I don't know how to trigger a build of the frontend images. That part is not documented in the deployment instructions. Can you clarify?

[sorenabell]

We updated the instructions and sent them to your email.
The necessary information is as follows:

If this is a regular deployment, change tags for images in the .env file. You can check image tags in GitHub (Application back-end, Application front-end).

[yolile]

I deployed the latest version in production now and it is working! I tested the file that Duncan shared with the command line tool and it is working well now too