Skip to content

unorderable types: tuple() < NoneType() (500 INTERNAL SERVER ERROR) #244

@bert2002

Description

@bert2002

Hi,
running into an weird problem. I am collecting data with collectd and push it to graphite-api which has graphite-carbon as backend. Works wonderful for all metrics, except for this one.

Jun 06 15:45:31 monitoring-1 gunicorn3[725]: {"reader": "whisper", "start": 1559806831, "end": 1559807131, "path": "/var/lib/graphite/whisper/collectd/test-node-server-1/processes/ps_state-running.wsp", "metric_path": "collectd.test-node-server-1.processes.ps_state-running", "event": "fetch"}
Jun 06 15:45:31 monitoring-1 gunicorn3[725]: {"event": "Exception on /render [POST]", "exception": "Traceback (most recent call last):\n  File \"/usr/lib/python3/dist-packages/flask/app.py\", line 1982, in wsgi_app\n    response = self.full_dispatch_request()\n  File \"/usr/lib/python3/dist-packages/flask/app.py\", line 1614, in full_dispatch_request\n    rv = self.handle_user_exception(e)\n  File \"/usr/lib/python3/dist-packages/flask/app.py\", line 1517, in handle_user_exception\n    reraise(exc_type, exc_value, tb)\n  File \"/usr/lib/python3/dist-packages/flask/_compat.py\", line 33, in reraise\n    raise value\n  File \"/usr/lib/python3/dist-packages/flask/app.py\", line 1612, in full_dispatch_request\n    rv = self.dispatch_request()\n  File \"/usr/lib/python3/dist-packages/flask/app.py\", line 1598, in dispatch_request\n    return self.view_functions[rule.endpoint](**req.view_args)\n  File \"/usr/lib/python3/dist-packages/graphite_api/app.py\", line 375, in render\n    data_store = fetchData(context, paths)\n  File \"/usr/lib/python3/dist-packages/graphite_api/render/datalib.py\", line 167, in fetchData\n    (node, node.fetch(startTime, endTime)) for node in single_nodes]\n  File \"/usr/lib/python3/dist-packages/graphite_api/render/datalib.py\", line 167, in <listcomp>\n    (node, node.fetch(startTime, endTime)) for node in single_nodes]\n  File \"/usr/lib/python3/dist-packages/graphite_api/node.py\", line 28, in fetch\n    return self.reader.fetch(startTime, endTime)\n  File \"/usr/lib/python3/dist-packages/graphite_api/finders/whisper.py\", line 121, in fetch\n    cached_datapoints = self.carbonlink.query(self.real_metric_path)\n  File \"/usr/lib/python3/dist-packages/graphite_api/carbonlink.py\", line 199, in query\n    results = self.send_request(request)\n  File \"/usr/lib/python3/dist-packages/graphite_api/carbonlink.py\", line 230, in send_request\n    host = self.select_host(metric)\n  File \"/usr/lib/python3/dist-packages/graphite_api/carbonlink.py\", line 160, in select_host\n    for node in self.hash_ring.get_nodes(key):\n  File \"/usr/lib/
Jun 06 15:45:31 monitoring-1 gunicorn3[725]: python3/dist-packages/graphite_api/carbonlink.py\", line 108, in get_nodes\n    index = bisect.bisect_left(self
[ps_state-running.dump.txt](https://github.com/brutasse/graphite-api/files/3260621/ps_state-running.dump.txt)
.ring, search_entry) % self.ring_len\nTypeError: unorderable types: tuple() < NoneType()"}

I already deleted all datasets (stopped all processes before), but the problem seems to be persistent. I attached a dump of the data (from whisper-fetch). I don't see anything wrong and wondering what is happening. because I have 10s of other server which the same configuration and dont run into this problem. Also all other metrics in processes are working as expected. Any idea?

OS: Debian 9.9
graphite-api: 1.1.3-2+deb9u1
graphite-carbon: 0.9.15-1
grafana: 6.2.2

Many thanks,
bert

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions