Retrieving the processes is very complicated internally

[christophfriedrich]

Nothing users would care about, just something I noticed under the hood:

The data fed to the Processes component in the DiscoveryToolbar is retrieved from the allProcesses computed property:

openeo-web-editor/src/components/DiscoveryToolbar.vue

Line 30 in 7600f8d

<Processes class="category" :processes="allProcesses" :searchTerm="searchTerm" :offerDetails="false" :collapsed="collapsed" :hideDeprecated="!showDeprecated" :hideExperimental="!showExperimental">

which returns:

openeo-web-editor/src/components/DiscoveryToolbar.vue

Lines 125 to 127 in 7600f8d

	allProcesses() {
	return this.processes.all();
	},

which comes from the Vuex Store:

openeo-web-editor/src/components/DiscoveryToolbar.vue

Line 112 in 7600f8d

...Utils.mapGetters(['supports', 'fileFormats', 'processes']),

So far so good.

In this store, however, processes is not a direct value of the state, but a getter:

openeo-web-editor/src/store/index.js

Line 61 in 7600f8d

getters: {

openeo-web-editor/src/store/index.js

Lines 109 to 118 in 7600f8d

	processes: (state) => {
	let registry
	if (state.processesUpdated && state.connection !== null) {
	registry = state.connection.processes;
	}
	else {
	registry = new ProcessRegistry();
	}
	return Object.assign(registry, ProcessRegistryExtension);
	},

which uses the connection, where the processes property holds a ProcessRegistry:

https://github.com/Open-EO/openeo-js-client/blob/6d0e09bd3373870b6538628518c3c3167bd8070f/src/connection.js#L97

(sadly Github doesn't render such links as code snippets when they are (spoiler) from a different repo, so I'll copy the code here directly for those cases)

		this.processes = new ProcessRegistry([], Boolean(options.addNamespaceToProcess));

Sounds like smart reusing at first, BUT: Because of this, the processes are handled differently than the collections etc., which all have a mutation in the store:

openeo-web-editor/src/store/index.js

Lines 332 to 347 in 7600f8d

	mutations: {
	discoveryCompleted(state, completed = true) {
	state.discoveryCompleted = completed;
	},
	connection(state, connection) {
	state.connection = connection;
	},
	authProviders(state, authProviders) {
	state.authProviders = authProviders;
	},
	userInfo(state, info) {
	state.userInfo = Utils.isObject(info) ? info : {};
	},
	fileFormats(state, fileFormats) {
	state.fileFormats = fileFormats;
	},

Such mutations are triggered when the store's commit function is called. In the Web Editor, this happens in the discover function that is part of the store's actions:

openeo-web-editor/src/store/index.js

Line 132 in 7600f8d

actions: {

openeo-web-editor/src/store/index.js

Lines 178 to 182 in 7600f8d

	if (capabilities.hasFeature('listCollections')) {
	promises.push(cx.state.connection.listCollections()
	.then(response => cx.commit('collections', response))
	.catch(error => errors.push(error)));
	}

For the processes, this cx.commit is missing -- see, nothing is done with the value returned by listProcesses():

openeo-web-editor/src/store/index.js

Lines 189 to 192 in 7600f8d

	if (capabilities.hasFeature('listProcesses')) {
	promises.push(cx.state.connection.listProcesses()
	.catch(error => errors.push(error)));
	}

(the promises in the array are just waited for to resolve and then discarded too)

So how does it even work that the data ends up in the app?! That is because the ProcessRegistry stored in connection.processes is filled as a side-effect of that connection.listProcesses() call. And that is a function in openeo-js-client (!), so the Web Editor relies on the JS Client doing it this way.

This is the whole function:

https://github.com/Open-EO/openeo-js-client/blob/6d0e09bd3373870b6538628518c3c3167bd8070f/src/connection.js#L349-L352

	async listProcesses(namespace = null) {
		const pages = this.paginateProcesses(namespace);
		return await pages.nextPage([], false);
	}

Cryptic and unexpected, right? The actual call to the backend happens in that nextPage:

https://github.com/Open-EO/openeo-js-client/blob/6d0e09bd3373870b6538628518c3c3167bd8070f/src/pages.js#L58-L60

  async nextPage(oldObjects = [], toArray = true) {
    // Request data from server
    const response = await this.connection._get(this.nextUrl, this.params);

But we don't really have to look at what nextPage and thus listProcesses eventually returns, because as you remember, the value is discarded anyway.

The magic is in this little line within nextPage:

https://github.com/Open-EO/openeo-js-client/blob/6d0e09bd3373870b6538628518c3c3167bd8070f/src/pages.js#L83-L84

    // Store objects in cache if needed
    newObjects = this._cache(newObjects);

I wouldn't have guessed that from reading the code and the comment. Or from looking at that function's implementation in the Page class:

https://github.com/Open-EO/openeo-js-client/blob/6d0e09bd3373870b6538628518c3c3167bd8070f/src/pages.js#L140-L148

  /**
   * Caches the plain objects if needed.
   * 
   * @param {Array.<object>} objects
   * @returns {Array.<object>}
   */
  _cache(objects) {
    return objects;
  }

That's because the relevant part is actually in the overriding function of the ProcessPages subclass that inherits from Page:

https://github.com/Open-EO/openeo-js-client/blob/6d0e09bd3373870b6538628518c3c3167bd8070f/src/pages.js#L237

class ProcessPages extends Pages {

https://github.com/Open-EO/openeo-js-client/blob/6d0e09bd3373870b6538628518c3c3167bd8070f/src/pages.js#L272-L282

  _cache(objects) {
    const plainObjects = objects.map(p => (typeof p.toJSON === 'function' ? p.toJSON() : p));
    this.connection.processes.addAll(plainObjects, this.namespace);   // <--- line 274
    if (!this.cls) {
      for (let i in objects) {
        objects[i] = this.connection.processes.get(objects[i].id, this.namespace);
      }
    }
    return objects;
  }
}

Line 274, that's where the ProcessRegistry is filled that eventually feeds the Processes component in the UI.

I found that extremely hard to understand and I partly wrote this down to get it fully clear to myself (and not forget it again), partly because I mentioned the ignored value to Matthias and he told me I should note it in an issue here, and partly because I find that the storing should be a bit more explicit, ideally through a cx.commit('processes', ...) to the Vuex store.

[m-mohr]

Unfortunately, this difficulty comes from how the different parts (Vue components and Model Builder, JS Client, Web Editor) play together. There's no easy solution for this except if we start to rewrite the whole stack, which will not happen anytime soon due to limited funding. We can keep this open for reference, but I don't expect an implementation anytime soon.

[christophfriedrich]

I fully understand this is a complex, evolved architecture that's not going to be refactored quickly, especially not while it doesn't cause 'actual' problems.

But maybe this is an easy fix for the Web Editor part:
Couldn't the JS client explicitly return the ProcessRegistry, which the Web Editor then stores in this.processes through a mutation that is triggered via cx.commit('processes', processRegistryFromJsClient)?

[m-mohr]

I can't tell without further investigation. There are so many side-effects that apply for the processes so that all changes need to be taken with care.

[christophfriedrich]

Fair enough!