Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support elasticsearch version 7.x #11

Open
paularmand opened this issue Nov 25, 2019 · 1 comment
Open

support elasticsearch version 7.x #11

paularmand opened this issue Nov 25, 2019 · 1 comment

Comments

@paularmand
Copy link

Trying intake-elasticsearch on ES7. Apparently ES now reports ['hits']['total'] back as a dict instead of a value. We would, or:

  1. need to test the version of ES (not the library, but the server)
  2. just try/catch it

The second approach leads to changing the following in _run_query:

        try:
            # elasticsearch >= version 7
            scroll_size = s['hits']['total']['value']
        except KeyError:
            # elasticsearch <= version 6 
            scroll_size = s['hits']['total']

Okay for a PR like so, or would you prefer not to support ES7 just yet? Or want a different approach?

ps: unsure what to do when s['hits']['total']['relation'] != 'eq' meaning that the total hits is not exact. We can force exact results by setting "track_total_hits": true, see https://www.elastic.co/guide/en/elasticsearch/reference/7.4/search-request-body.html#request-body-search-track-total-hits for more info. (This can be passed via the catalog qargs probably)

@martindurant
Copy link
Member

If you know how to test the server version, that would work fine - otherwise the try/except would be OK as a workaround. I would be happy to see this in a PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants