New curve fit acceptance criteria

[nkanazawa1989]

What is the expected behavior?

Currently we evaluate the curve fit quality with chi-squared:= sum(residuals / sigma^2) / dof. Though this is meaningful value, sometime it is not practical because this is quite sensitive to outlier.

For example, both of above fit curves look agree well with the measured Rabi data, but the second result shows larger chi-squared value. According to the current criteria, this data will be rejected, i.e. chi-squared < 3. This difference is caused by the non-linear effect at larger amplitude (you can see tiny offset in |x| > 0.75), which is not considered in the fit model (chi-squared captures this really well).

The parameter of interest here is rabi_rate and indeed its difference is larger than estimated standard error. However, the value from the second experiment is still acceptable to estimate pi amplitude.

• from the experiment 1: pi_amp = 0.1904
• from the experiment 2: pi_amp = 0.1952

Thus we need to develop new quantity to evaluate practicability of fit parameters.

Relevant issue: #397 and PR #401 (closed)

[wshanks]

For the example shown, it might be sufficient to do some specific tests like the ratio of the fit error to the fit value for the Rabi rate and the amplitude and a sanity check of the values (Rabi rate is not much slower than 1/range and not faster than sampling rate / 2). But I think you are hoping for something that is independent of the model like chisq.

Does it work to calculate chisq with an error per point based on our tolerance to deviations from the model instead of the actual measurement error? So for the Rabi example where the data ranges from 0 to 1, we might say we are okay with deviations from the fit model of 0.05 per point.

[nkanazawa1989]

something that is independent of the model like chisq

Yes, otherwise all analysis classes must define unique criteria. This is not only bothersome but also hurts fairness among analysis classes written by different developers.

There was another long discussion in the closed PR #401

@oliverdial suggested

a parameter somewhat whimsically named wonkiness is the sqrt(chisq) scaled by the signal between 0 and 1 squared, so a wonkiness of 0.1 indicates the RMS fit error is 10% of the shots. We adopted it for the reason Naoki mentions above -- that chisq is only a good goodness of fit measure if the underlying model is perfect, and we usually have some small non-model imperfections like ZZ, leading to a tendency to reject curves with the best SNRs. However, if you cut against that you also really need to include some SNR cut so you don't simply accept a fit with absolutely no signal...

Sounds like you are suggesting the idea along this line. But how can we normalize the IQ data? If the measure data covers full dynamic range it is easy to normalize it with min-max. However, in the case of Rabi experiment, it would be hard to do this analysis correctly if the oscillation is less than half period (though most of experiments are based on discriminated data).