Skip to content

support elasticsearch version 7.x #11

@paularmand

Description

@paularmand

Trying intake-elasticsearch on ES7. Apparently ES now reports ['hits']['total'] back as a dict instead of a value. We would, or:

  1. need to test the version of ES (not the library, but the server)
  2. just try/catch it

The second approach leads to changing the following in _run_query:

        try:
            # elasticsearch >= version 7
            scroll_size = s['hits']['total']['value']
        except KeyError:
            # elasticsearch <= version 6 
            scroll_size = s['hits']['total']

Okay for a PR like so, or would you prefer not to support ES7 just yet? Or want a different approach?

ps: unsure what to do when s['hits']['total']['relation'] != 'eq' meaning that the total hits is not exact. We can force exact results by setting "track_total_hits": true, see https://www.elastic.co/guide/en/elasticsearch/reference/7.4/search-request-body.html#request-body-search-track-total-hits for more info. (This can be passed via the catalog qargs probably)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions