Improvement of response matrices

[lfarv]

This discussion follow a mail exchange about possible improvements to the response matrix computations. The idea is to get a specification for useful additions.

[lfarv]

from @swhite2401:

Les fonctionalités de response matrix sont tres appreciable toutefois dans le cadre de pyML on aurait besoin de generaliser leur usage pour des CustomVariable et CustomObservable, cela permettrait d’avoir le meme code pour l’acces au modele et l’acces au systeme de controle (pas d’object ring…)

In your example all QF1 magnets could be varied by an increment or even better by a factor (which in effect is the same as serialized PS).

Another useful addition would be to add IntegratedPolynomA/B attribute to the magnets.

[lfarv]

La classe générique ResponseMatrix est supposée accepter n’importe quels VariableBase et Observable, et donc CustomVariable. Si aucune des variables et observables n’utilise ring, on peut mettre None en argument du constructeur de ResponseMatrix. Pour que ça soit plus clair, on pourrait dans le constructeur de ResponseMatrix mettre l’argument ring en keyword: ResponseMatrix(variables, observables, ring=None), mais il faudrait changer l’ordre des arguments. Possible ?

[swhite2401]

Yes it is possible, I don't think we have so many users to this functionality yet

[lfarv]

from @JeanLucPons:

Ce qu'on voudrait c'est pouvoir faire des matrices de réponses avec des CustomVariable qui peuvent accéder au système de contrôle.
Il faudra peut-être ajouter un hook entre chaque ligne pour un check éventuel, une correction d'orbite pour remettre à plat,etc et pouvoir abandonner la mesure.

Pour les integratedPolynom , je pense pas que ce soit la priorité. les setters/getters de la CustomVariable écriront ou retourneront des strength intégrées de toute façon.
On a pas besoin de pouvoir exclure des éléments, pyaml se chargera de créer dynamiquement les CustomVariable.

def set_field(value):
    amagnet.strength.set(value)

def get_field():
    return amagnet.strength.get()

def hook():
    do something

var = CustomVariable(set_field, get_field, hook, delta=1.0e-6)

Dans l'exemple ci-dessus, amanget est un object pyaml qui peut être connecté soit au modèle soit au système de contrôle.

[lfarv]

from @lfarv/

Le hook est interessant. Ça me semble facile à ajouter. Ce hook sera appelé uniquement par la mesure de matrice de réponse, pas sur les autres appels Variable.set(), Variable.Increment(), Variable.step_up()… Est-ce qu’il doit être membre de Variable ou de ResponseMatrix ?
S’il est spécifique pour chaque variable, c’est évident. Sinon, c’est au choix, à vous de dire.

Un détail: actuellement, il y a deux méthodes pour construire la matrice de réponse: build_tracking() et build_analytical(). Selon les variables et observables utilisés, build_tracking() ne fait pas forcément du tracking. Peut-être faut-il renommer la méthode ?

Quant aux polynômes intégrés, ça ne coûte pas cher…

[swhite2401]

Renaming the build methods would in fact be appropriate, integrated polynoms come for free and would be useful in many cases, I will create a new PR for this.

[swhite2401]

I looked a bit at integrated strength and it is not as simple as I thought, especially when setting one element of the array with the syntax:

IntegratedPolynomB[0] = value
when IntegratedPolynomB returns PolynomB*Length

This fails and I have not clear solution for the moment... let's see

[swhite2401]

For the matching, there is something that is a bit annoying: it is required to provide a ring when building variables to set their initial values, otherwise they are set to None and matching function fails. However, ring is also required as input to the matching function, in case initial values are None, couldn't we just use the Lattice object provided for the matching function to initialize all variables and observables?

[lfarv]

couldn't we just use the Lattice object provided for the matching function to initialize all variables and observables?

Is not it what happens ?

    vini = variables.get(ring=ring, initial=True, check_bounds=True)
    bounds = np.array([var.bounds for var in variables])

    constraints.evaluate(ring, initial=True, dp=dp, dct=dct, df=df)

Can you give an example ? I do not see the problem. You can build the VariableList without a ring keyword: for variables which need it for evaluation (RefptsVariables), the initial value is indeed left at None. But when the matching starts, it calls a variable get, with the ring provided to the match function, so it should initialise all variables properly.

If no variable needs a ring for evaluation (for instance ElementVariables, CustomVariables, …), and if no Observable needs a ring for evaluation (none in the standard set, but one could derive one from the base Observable), one could call match() with ring=None.

[swhite2401]

I always use ReptsVariables I find it way more practical because I do matching in step of various parts of the ring and basically always work with copies so initialization is indeed None.
If these were not previously initialized with ring, the matching would simply fail, I will produce a simple example.

[swhite2401]

I find it very annoying to have to input ring every time an evaluate/get/set function is called, can we find a way to optionally store the lattice somewhere (maybe when initializing the object?) in case we would like to simplify the syntax?

[lfarv]

Good question. Let's try to look separately to Variables and Observables:

Variables

No variable needs explicitly a ring keyword at instantiation. But some need it for setting/getting (RefptsVariable), and some don't (ElementVariable, CustomVariable). Back to the reason to provide both (see the notebook):

• ElementVariables are linked directly to an element object, which does not need to be part of a lattice, or can belong to several lattices. They do not need a ring for evaluation, but they do not follow ring copies. Lattice copies are frequent: they happen for all lattice modifying functions where the copy keyword is set to True. For instance, they can be used for matching if copy is False, but fail if copy is True.
• RefptsVariables refer to a location in the lattice, so they need a ring for setting/getting, but may be used on lattice copies.

My message would be: try to use ElementVariables whenever possible.

Proposed improvements:

• in setting/getting a RefptsVariable with ring==None, improve the error message: "Can't set values if ring is None. Try to use an ElementVariable instead if possible",
• compute the response matrices on the given ring instead of on a copy. For safety, the computation of response matrices makes copies before acting on the variables. This ensures that the provided ring is unchanged but prevents using ElementVariables in the VariableList. In the example given by @JeanLucPons, the computation may act on a real machine: no possible lattice copy. So we have to make sure to restore the given lattice whatever happens. The lattice copy would not be necessary any more, allowing ElementVariables in the VariableList.

[lfarv]

Observables

As it is now, the ring has to be provided for evaluation. The reasons are:

1. For efficiency, all Observables are evaluated simultaneously on a single computation pass over the lattice. Providing the lattice at the instantiation of Observables may lead to conflicts if all are not defined to the same lattice,
2. The same observables may be used on different lattices with different settings,
3. No need for a ring at instantiation (as for variables).

Of course, an observable may not need a ring: using the base Observable class for instance, or deriving from it. For those, the evaluation function does not receive the ring keyword, so ring can be set to anything (None for instance).

Propose improvements:

• modify the response matrix signature to set ring as a keyword argument defaulting to None. However, the use cases where none of the variables and observables are associated with a lattice should be very rare.
• if really necessary, one could imagine giving optionally a ring at instantiation of any Observable. The ring given for evaluation, if present, would have priority. Otherwise we would check that the ring of all observables is the same and that one would be used for evaluation. I don't like this too much: it's complicated and not intuitive. But that's a possibility.

[JeanLucPons]

That sounds good for me. Since I can use Variables and Observables without a ring and without using any refpts, it can do the job.

[swhite2401]

Variables

• I want to use RefptsVariables for practical reasons, a method set_ring to VariableList would be very simple to implement and solve the problem, the "setted" ring would be used only if ring is not provided as input.
• As a first step the improved error message would at least help the user come up with the correct approach.
• I agree that RM should ensure that all settings are restored, so this is a desirable feature in any case.

Observable

• If both variable and observables are coming from control system (very frequent) they will not be linked to a ring
• Alternatively a set_ring() function to ObservableList could be used

That being said, this is not the main priority and the present syntax is probably safer and cleaner, I just find it annoying but can live with it.

So for me, improving error message, making ring optional in RM and cleanly restoring parameters after RM are the priority.

[lfarv]

a method set_ring to VariableList would be very simple

Why not simply an optional ring attribute ? It could be available in the constructor.

[lfarv]

#974 is open an implements some of the points discussed here.

• Fix the use of custom variables,
• allow instantiating a ResponseMatrix with ring=None (the default), if no lattice is necessary for any variable or observable. The signature of the ResponseMatrix instantiation is now ResponseMatrix(variables, observables, ring=None)
• The build_tracking method is renamed as build,
• improvement of the error message when trying to access a RefptsVariable with ring=None,
• ensure that all variables are restored if an exception occurs during building the matrix,
• adds a hook called after each variable variation. This is implemented as a keyword argument checkfun of ResponseMatrix.build and called as checkfun(var) after the variable var has been scanned.