Will graphite-api support clustering?

[StephenPCG]

Hi,

I'm currently settings up a graphite cluster, with following architecture:

                   |--------------|
                   | carbon-relay |
                   |--------------|
                          |
        +-----------------+-----------------+
        |                 |                 |
|--------------|   |--------------|   |--------------|
|    host-1    |   |    host-2    |   |    host-3    |
| carbon-cache |   | carbon-cache |   | carbon-cache |
| graphite-api |   | graphite-api |   | graphite-api |
|--------------|   |--------------|   |--------------|
        |                 |                 |
        +-----------------+-----------------+
                          |
                   |--------------|
                   | graphite-web |
                   |    grafana   |
                   |--------------|

I'm trying to use graphite-api on the carbon-cache host because graphite-api is much OPS friendly, and has better performance against graphite-web.

However, this does not work now. graphite-web queries upstreams with format=pickle, but graphite-api does not support pickle rendering.

I searched all docs and issues here, just find clarifications on "no pickle rendering support", but no reason. I also searched history of app.py file (git log -Spickle -- graphite_api/app.py), and it seems there was never pickle rendering support.

What is the reason graphite-api does not support pickle rendering? Will graphite-api support clustering like graphite-web? If the reason is lack of user or developers, I would like to contribute. If the reason is something that conflicting with graphite-api's philosophy, can someone explain it?

[pkittenis]

Graphite-web is not compatible with Graphite-API and graphite-api does not have remote rendering. For the configuration above to work, the web app running on the grafana host would need to be graphite-api, not web.

It can however query remote carbon caches.

Since graphite-api does not yet support querying only remote carbon hosts ( #212 ) which would be needed to switch the grafana webapp to graphite-api, you could change configuration on hosts 1 to 3 to have the remote carbon-caches on all other hosts. Eg on host 1 add carbon configuration for hosts 2 and 3 and so on.

Grafana can then use any of host 1 to 3 for queries, without an additional webapp.

[tantra35]

maybe you will be interested in this #169. Solition that @pkittenis to offer will not to work because shared whisper db needed, carbon cache only provide access to memory cached metrics not it all

[pkittenis]

Don't think that is accurate @tantra35 - carbon caches are linked to each other via carbon link in order to query data they do not have available locally. This is part of the distributed hashing of carbon caches. Local shared whisper not required when carbon link hosts are used.

[tantra35]

@pkittenis Sorry but i missunderstend what you mean when talk about "carbon link"? Is it the same as mentioned in carbon/host section of graphite api:

carbon:
  hosts:

?

For example here is code from graphite carbon (https://github.com/graphite-project/carbon/blob/1.0.2/lib/carbon/protocols.py#L183):

  def stringReceived(self, rawRequest):
    request = self.unpickler.loads(rawRequest)
    cache = MetricCache()
    if request['type'] == 'cache-query':
      metric = request['metric'] 

it referecnce to MetricCache singleton (https://github.com/graphite-project/carbon/blob/1.0.2/lib/carbon/cache.py#L210)

And it demonstrate that metrics readed only from cache when we use CARBONLINKS_HOST in graphite-web, so when metrics will be flushed on disk you can't retrieve them via cache query

[pkittenis]

There too, but I meant in configuration of each carbon cache. The webapp needs to know which caches to contact, yes. Each cache also needs to know what other caches are in the cluster or they will only see their local data - it's not a cluster if there is only the local cache configured.

If I understand correctly, if data is not cached in memory carbon-cache never retrieves from disk? I do not see how that could possibly be true for graphite to work at all.

Can test this by restarting caches, thus clearing their memory, and doing a query for the oldest data point in your whisper files - does it get retrieved? Was there a read from disk at the same time?

Our setup prior to influxdb + graphite-api and influxgraph was like that too - multiple carbon caches each with their own disk (not shared disk), webapps + caches all configured with all other caches.

Queries for any data, on disk or cached, could be satisfied by any cache either from local data or by re-directing to another cache. Works fine as long as data distribution is correct and cluster is balanced. InfluxDB simplifies this a lot thankfully, carbon in general is overly complex once clusters are involved.

Likely have a configuration issue in carbon-cache - they also need destinations configuration to know which other caches to contact. See carbon-cache's config example which states this:

Note that if the destinations are all carbon-caches then this should
exactly match the webapp's CARBONLINK_HOSTS setting in terms of
instances listed (order matters!).

In graphite-api, the carbonlink_hosts equivalent is carbon: hosts as above.

Behaviour of caches in general is:

a) Data is cached - return from cache
b) Data is not cached - fetch from < elsewhere >, add to cache, return

With distributed caches this extends to other remote caches as well. See relevant remote cache code and tests. Also, here is the carbon-cache code for reading from disk.

The code linked above is for writing, not reading.