PINN model didn't perform good outside the training points

[emonhossainraihan]

I run your simple ode solver using PINN. It seems that the model perform well on the training interval,

But couldn't manage to predict outside the training interval,

I was surprised, as the paper was considered so good (from my theoretical perspective). Isn't it should learn the periodicity of the solution? Do you have tested it and get better performance outside the training interval? Let me know your thought to improve it.

Thanks for your consideration.

[jdtoscano94]

Hello Emon, thank you for your comment.
Yes, you are right. I found that same problem when I tried to evaluate this model outside the initial domain; thanks for noticing it.

I am not sure if we should refer to the collocation points (i.e., where we evaluate our PDE) as "real" training data. We usually call training data to the $(X-f(X))$ data used to tune up our parameters by comparing the input-output values $(f(X), NN(x))$. Please review https://book.sciml.ai/notes/03/. They solve Hooke's equation (i.e., periodic function) using a PINN. Please notice that they also evaluate their PDE along the whole domain.

In other words, you could use a few observations even if they are clustered in a small section of X, but you should evaluate your PDE in the whole domain.

In general, I believe the problem is that our weights are tuned to work in a certain interval. So, if we go outside of the initial domain, it gets messy.

For example:

For the first layer:

Let's assume that the first linear transformation is:
$a1=w1x+b1$
with $w1=1.5$ and $b1=-\pi/8$.

And that we activate it using the sigmoid function:
$z1=\sigma(a1)$

Now, let:
$x1=pi/4$
$x2=pi/4+2\pi$

Ideally, $x1=x2$; however:
$a1(x1)=3\pi/8-\pi/8=\pi/4$
$a1(x2)=27\pi/8-\pi/8=13\pi/4=3\pi+\pi/4$

Notice that $a1(x1)$ and $a1(x2)$ are different, even before activating them.

Then:
$z1(a1(x1))= 0.6868423739506077$
$z1(a1(x2))= 0.9999632073667951$

Since the z1 is input into the next layer, the error propagates through the whole network.

If you are still interested in this topic, please read the following article:

Wang, S., Teng, Y., & Perdikaris, P. (2021). Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing, 43(5), A3055-A3081. https://arxiv.org/abs/2001.04536

I am also still learning, so not sure if this answers your question, but I hope it helped you.

Sincerely,

Diego Toscano

[Arup-nit]

Thank you dear @jdtoscano94 for all these videos and description.
I have some doubts

1. Is PINN supervised model?
2. Shall we need exact solution of DE before solve by PINN?

[jdtoscano94]

Hi @Arup-nit, thank you so much for your kind words.
1. Is a PINN a supervised model?
There is no direct answer to that question; a Physics-Informed (PI) neural network (NN) is a NN with a modified loss function. Consequently, you can make PI almost any type of NN (e.g., convolutional, recurrent, modular, etc.). In other words, if your model is unsupervised and you use a PDE in your loss function, it would ideally become a PI-unsupervised model. We usually refer to supervised learning when we use labeled data (i.e.,input-output) to train our model. Therefore, since we do not use the output data for our PDE loss function, we could make PI any type of model (supervised or unsupervised). Still, in most cases, people usually apply PINNs to solve supervised problems (e.g., the examples on this repository).

2. Shall we need the exact solution of DE before solving by PINN?
No, you only need your PDE and your domain (input data). For traditional PINNs (supervised), you may want to include some initial and boundary conditions (remember that the general solution of a PDE is a family of functions, so you may want to constrain it with some IC/BC).

I hope that answers your questions.

Thank you so much for your comment.

[Arup-nit]

Hi @jdtoscano94 thank you so much for your kind reply.
Yes you have cleared my doubts.
Thank you

Could you plz share your google scholar/ researchgate link.

[jdtoscano94]

Hi @Arup-nit. Sure! I do not have any studies in this field yet; thank you so much for your interest.

Google Scholar: https://scholar.google.com.ec/citations?user=XNvLUKwAAAAJ&hl=en

Next month, I will start my Ph.D. at Brown University and focus on scientific ML. So, (hopefully )in a few months, I will upload a new study related to this field to my google scholar.

Still, thank you so much for asking. I hope you have a nice day.