Fix tests to accomodate scipy 1.16.0 changes #234

woodsp-ibm · 2025-06-26T19:53:24Z

Summary

Fixes #233

Fixes nightly CI failures that arose when scipy 1.16.0 was released recently

Details and comments

In scipy 1.16.0 the COBYLA implementation was changed

https://docs.scipy.org/doc/scipy/reference/optimize.minimize-cobyla.html

Changed in version 1.16.0: The original Powell implementation was replaced by a pure Python version from the PRIMA package, with bug fixes and improvements being made.

and this seems to have affected a few CI tests here that failed

==============================
Failed 6 tests - output below:
==============================

test.eigensolvers.test_vqd.TestVQD.test_callback_1
--------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/ddt.py", line 221, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^

      File "/home/runner/work/qiskit-algorithms/qiskit-algorithms/test/eigensolvers/test_vqd.py", line 205, in test_callback
    np.testing.assert_array_almost_equal(history["eval_count"], ref_eval_count, decimal=0)

      File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/numpy/testing/_private/utils.py", line 1175, in assert_array_almost_equal
    assert_array_compare(compare, actual, desired, err_msg=err_msg,

      File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/numpy/testing/_private/utils.py", line 813, in assert_array_compare
    raise AssertionError(msg)

    AssertionError: 
Arrays are not almost equal to 0 decimals

(shapes (20,), (6,) mismatch)
 ACTUAL: array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10,  1,  2,  3,  4,  5,  6,  7,
        8,  9, 10])
 DESIRED: array([1, 2, 3, 1, 2, 3])


test.eigensolvers.test_vqd.TestVQD.test_aux_operator_std_dev_1
--------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/ddt.py", line 221, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^

      File "/home/runner/work/qiskit-algorithms/qiskit-algorithms/test/eigensolvers/test_vqd.py", line 425, in test_aux_operator_std_dev
    self.assertAlmostEqual(

      File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/unittest/case.py", line 939, in assertAlmostEqual
    raise self.failureException(msg)

    AssertionError: np.float64(0.743234181389445) != 0.0019531249999999445 within 1 places (np.float64(0.741281056389445) difference)


test.minimum_eigensolvers.test_qaoa.TestQAOA.test_qaoa_initial_point_2
----------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/ddt.py", line 221, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^

      File "/home/runner/work/qiskit-algorithms/qiskit-algorithms/test/minimum_eigensolvers/test_qaoa.py", line 215, in test_qaoa_initial_point
    self.assertIn(graph_solution, solutions)

      File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/unittest/case.py", line 1152, in assertIn
    self.fail(self._formatMessage(msg, standardMsg))

      File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/unittest/case.py", line 715, in fail
    raise self.failureException(msg)

    AssertionError: '1111' not found in {'0100', '1011'}


test.minimum_eigensolvers.test_qaoa.TestQAOA.test_qaoa_qc_mixer_many_parameters
-------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/runner/work/qiskit-algorithms/qiskit-algorithms/test/minimum_eigensolvers/test_qaoa.py", line 137, in test_qaoa_qc_mixer_many_parameters
    self.assertIn(graph_solution, S1)

      File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/unittest/case.py", line 1152, in assertIn
    self.fail(self._formatMessage(msg, standardMsg))

      File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/unittest/case.py", line 715, in fail
    raise self.failureException(msg)

    AssertionError: '110' not found in {'1010', '0101'}


test.minimum_eigensolvers.test_sampling_vqe.TestSamplerVQE.test_optimizer_scipy_callable
----------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/runner/work/qiskit-algorithms/qiskit-algorithms/test/minimum_eigensolvers/test_sampling_vqe.py", line 191, in test_optimizer_scipy_callable
    self.assertEqual(result.cost_function_evals, 2)

      File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/unittest/case.py", line 885, in assertEqual
    assertion_func(first, second, msg=msg)

      File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/unittest/case.py", line 878, in _baseAssertEqual
    raise self.failureException(msg)

    AssertionError: np.int64(6) != 2


test.optimizers.test_optimizers.TestOptimizers.test_cobyla
----------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/runner/work/qiskit-algorithms/qiskit-algorithms/test/optimizers/test_optimizers.py", line 106, in test_cobyla
    self.run_optimizer(optimizer, max_nfev=100000)

      File "/home/runner/work/qiskit-algorithms/qiskit-algorithms/test/optimizers/test_optimizers.py", line 85, in run_optimizer
    np.testing.assert_array_almost_equal(x_opt, [1.0] * len(x_0), decimal=2)

      File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/numpy/testing/_private/utils.py", line 1175, in assert_array_almost_equal
    assert_array_compare(compare, actual, desired, err_msg=err_msg,

      File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/numpy/testing/_private/utils.py", line 926, in assert_array_compare
    raise AssertionError(msg)

    AssertionError: 
Arrays are not almost equal to 2 decimals

Mismatched elements: 5 / 5 (100%)
Max absolute difference among violations: 1.96222778
Max relative difference among violations: 1.96222778
 ACTUAL: array([-0.96,  0.94,  0.88,  0.78,  0.61])
 DESIRED: array([1., 1., 1., 1., 1.])

What I did, in the main, was to adjust tests, say the initial point or some other parameter to have them pass in 1.16 as well as pre that e.g 1.15.3

This failure for example AssertionError: np.int64(6) != 2 seems to be that setting maxiter=2 no longer resulted in 2 functional evals but 6! Setting it to 8 seemed to have it stop at 8 functional evals
test_qaoa and test_optimizers I adjusted the initial point. The test_cobyla test in test_optimizers, I will note that under 1.16.0 it seems to take far longer to run. I tried adjust the tol values given and maxiter to get something that would run/pass faster but to no avail.
test_vqd was more complicated and I could not find a setting that passed for both. I had to increase maxiter so as to get a consistent number of evals but then the mean outputs recorded by the callback being tested did not end up identical despite me trying different initial points etc. In the end I got things close to where only the last 2 entries in the list varied. I was going to check the list up to those entries ie add [0:-2] to subset the list. In the end I chose to detect the scipy version and switch according which reference mean list was used.

It passed locally for me under 3.12 using either 1.15.3 or 1.16.0 - lets see here in CI. I looked in scipy github but did not see any issues that might be relevant. I guess with an implementation switch only time will tell how this works out for the CI tests here.

woodsp-ibm · 2025-06-26T20:17:37Z

Seems this test fails here in CI test.minimum_eigensolvers.test_qaoa.TestQAOA.test_qaoa_qc_mixer_many_parameters but does not when I run locally - Windows + Python 3.12. It seems that its in the set of failures I copied above from nightly CI and I missed it as I expected the same failure locally which it seems did not happen as I fixed all failures - double checked and it passes locally so I will have to figure something out I guess. (Update I changed the maxiter and added an explicit initial point which seems to have "fixed" things so it now passes)

==============================
Failed 1 tests - output below:
==============================

test.minimum_eigensolvers.test_qaoa.TestQAOA.test_qaoa_qc_mixer_many_parameters
-------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/runner/work/qiskit-algorithms/qiskit-algorithms/test/minimum_eigensolvers/test_qaoa.py", line 137, in test_qaoa_qc_mixer_many_parameters
    self.assertIn(graph_solution, S1)

      File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/unittest/case.py", line 1152, in assertIn
    self.fail(self._formatMessage(msg, standardMsg))

      File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/unittest/case.py", line 715, in fail
    raise self.failureException(msg)

    AssertionError: '110' not found in {'0101', '1010'}

woodsp-ibm · 2025-06-26T20:54:13Z

Oh, it seems that there is a tutorials failure too under Python 3.13, where it would be using scipy 1.16.0, in a cell that uses COBYLA()

Notebook error:
CellExecutionError in tutorials/05_qaoa.ipynb:
------------------
from qiskit.circuit.library import TwoLocal

from qiskit_algorithms import SamplingVQE
from qiskit_algorithms.utils import algorithm_globals

algorithm_globals.random_seed = 10598

optimizer = COBYLA()
ansatz = TwoLocal(qubit_op.num_qubits, "ry", "cz", reps=2, entanglement="linear")
sampling_vqe = SamplingVQE(sampler, ansatz, optimizer)

result = sampling_vqe.compute_minimum_eigenvalue(qubit_op)

x = sample_most_likely(result.eigenstate)

print(x)
print(f"Objective value computed by SamplingVQE is {objective_value(x, w)}")

- - - - - - - - - -

File /opt/hostedtoolcache/Python/3.13.3/x64/lib/python3.13/site-packages/numpy/_core/numeric.py:2119, in binary_repr.<locals>.err_if_insufficient(width, binwidth)
   2117 def err_if_insufficient(width, binwidth):
   2118     if width is not None and width < binwidth:
-> 2119         raise ValueError(
   2120             f"Insufficient bit {width=} provided for {binwidth=}"
   2121         )

ValueError: Insufficient bit width=3 provided for binwidth=4

You can ignore this error by setting the following in conf.py:

    nbsphinx_allow_errors = True

make: *** [Makefile:70: html] Error 2

but at least the unit tests seem to be ok now :)

woodsp-ibm · 2025-06-27T14:19:42Z

I updated the 05_qaoa notebook so that the SamplingVQE cell produces the same output for scipy 1.16.0 as 1.15.3 and does not fail under 1.16.0 like it did - I added an explicit initial point.

Hmmm... now it failed under 3.9. I know I tried different values, to both get it to work and get the same result it had before, and I was sure I ran it under 1.15.3 as well.... Yes it does work locally with 1.15.3, I re-checked now, but I am on Python 3.12 and Windows for an env so maybe there is something to that. Will try something different that hopefully works in CI too!

I removed the initial_point as I found a seed that works for both (i.e. underlying random initial point it will create). Hopefully this is good here in CI - locally for me with scipy 1.15.,3 and 1.16.0 it is but with the former I am still using Python 3.12 not 3.9. Update seems that it is :)

coveralls · 2025-06-27T15:02:02Z

Pull Request Test Coverage Report for Build 15929192104

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.01%) to 90.462%

Totals
Change from base Build 15658711776:	0.01%
Covered Lines:	6402
Relevant Lines:	7077

💛 - Coveralls

Fix tests to accomodate scipy 1.16.0 changes

ea438fa

woodsp-ibm requested review from Cryoris and ElePT as code owners June 26, 2025 19:53

woodsp-ibm mentioned this pull request Jun 26, 2025

Added V2 and ISA support #197

Open

1 task

woodsp-ibm added 2 commits June 26, 2025 16:38

Update test to see if any better

64bd376

Fix style

3b10786

Update 05_qaoa notebook SamplingVQE cell for scipy 1.16.0

6b5d25e

Alternative cell change (differnt seed rather than set initial-point)

7566385

woodsp-ibm merged commit bd91342 into qiskit-community:main Jun 27, 2025
16 of 18 checks passed

woodsp-ibm deleted the scipy1_16 branch June 27, 2025 16:09

This was referenced Jun 27, 2025

Nightly actions failing qiskit-community/qiskit-machine-learning#957

Open

Nightly actions failing qiskit-community/qiskit-optimization#667

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix tests to accomodate scipy 1.16.0 changes #234

Fix tests to accomodate scipy 1.16.0 changes #234

Uh oh!

woodsp-ibm commented Jun 26, 2025 •

edited

Loading

Uh oh!

woodsp-ibm commented Jun 26, 2025 •

edited

Loading

Uh oh!

woodsp-ibm commented Jun 26, 2025 •

edited

Loading

Uh oh!

woodsp-ibm commented Jun 27, 2025 •

edited

Loading

Uh oh!

coveralls commented Jun 27, 2025

Uh oh!

Uh oh!

Uh oh!

Fix tests to accomodate scipy 1.16.0 changes #234

Fix tests to accomodate scipy 1.16.0 changes #234

Uh oh!

Conversation

woodsp-ibm commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details and comments

Uh oh!

woodsp-ibm commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

woodsp-ibm commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

woodsp-ibm commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Jun 27, 2025

Pull Request Test Coverage Report for Build 15929192104

Details

💛 - Coveralls

Uh oh!

Uh oh!

Uh oh!

woodsp-ibm commented Jun 26, 2025 •

edited

Loading

woodsp-ibm commented Jun 26, 2025 •

edited

Loading

woodsp-ibm commented Jun 26, 2025 •

edited

Loading

woodsp-ibm commented Jun 27, 2025 •

edited

Loading