multi-modal input in SBI

[MicheleRonchi]

Hi,

I have a simulator that given a set of parameters theta produces an output x which is formed by two input images with different resolutions, 32x32 the first one and 16x16 the second one. The two images carry different information and have different resolutions so I can't stuck them in a two-channel input and process them through a single CNN embedding net.
I was thinking to use a multi-modal CNN as embedding net, formed of two separated CNNs, one for each input image, which then merge together to form a unique feature vector which is passed through the density estimator for NPE.

I tried this with a simple simulator, where I simulated pairs (theta, x) where x is a tuple object containing the two tensors with shape (32, 32) and (16, 16).
Then I tried to run the following, where the embedding net is a multi-modal CNN:

# instantiate the neural density estimator
neural_posterior = posterior_nn(model="mdn", embedding_net=embedding_net)

# setup the inference procedure with NPE
inferer = SNPE(prior=prior, density_estimator=neural_posterior)

density_estimator = inferer.append_simulations(theta, x).train(training_batch_size=256, show_train_summary=True,)

However I got this error:

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[10], line 1
----> 1 density_estimator = inferer.append_simulations(theta, x).train(training_batch_size=256, show_train_summary=True,)
      2 posterior = inferer.build_posterior(density_estimator)

    684 r"""
    685 Checks if the passed $(\theta, x)$ are valid.
    686 
   (...)    704     training_device: Training device for net.
    705 """
    706 assert isinstance(theta, Tensor), "Parameters theta must be a `torch.Tensor`."
--> 707 assert isinstance(x, Tensor), "Simulator output must be a `torch.Tensor`."
    709 assert theta.shape[0] == x.shape[0], (
    710     f"Number of parameter sets (={theta.shape[0]} must match the number of "
    711     f"simulation outputs (={x.shape[0]})"
    712 )
    714 # I did not fuse these asserts with the `isinstance(x, Tensor)` asserts in order
    715 # to give more explicit errors.

AssertionError: Simulator output must be a `torch.Tensor`.

By looking at the way append_simulations works, it seems to accept only a unique tensor x as argument. But in my case I can't see an easy way to build a single tensor x that is composed of two tensors with different shapes.

Do you have any suggestions if there is a way to get around this?

Thanks a lot in advance for the help!

[michaeldeistler]

Hi there,

I you want such a high level of customization, you have to follow the tutorial here. This tutorial shows how you can entirely write your own training loop, allowing full customization of data loading and handling.

I think this should work for you, but feel free to let us know if not!
Michael

[MicheleRonchi]

Hi Michael,

Thanks a lot for the very quick response, I'll have a look at the tutorial you suggests!

thanks for the help,
Michele