Visualized simulation based on ground truth data

[EndingCredits]

In the visualised simulations there is the following line

ts_loss_part,velocities=sess.run([mse,P],feed_dict={O:[np.transpose(raw_data[i-1])] ...

which is then fed in to update estimated_data. This means the model is predicting based on the GT data, rather than simulated data. This does mean that the network is doing some long-term prediction, but it's not a true simulation by the model.

Is this intentional or a bug?

[jsikyoon]

Hi EndingCredits,

I think it is to calculate loss with GT velocities at each frame (to be honest, I can't remember why ^^;;).

To calculate the loss at each frame, current code is right, but as your mention, it is not a true simulation on long-term prediction. I had to divide loss calculation and simulation.

Thank you for pointing out this!!

Will fix it when I can do.

Thanks. Regards, Jaesik.

[EndingCredits]

Thanks Jaesik,

As a general note, I don't think this implementation works completely in it's current state, at least in terms of creating a 'simulator'.

I'm happy with the implementation of the network itself, but the validation loss is calculated on the same trajectories, which gives an unrealistic impression of the out-of-sample loss. Of course the test accuracy you report at the end is accurate.

One particular issue is the simulations often end up with the masses flying off in random directions after about 200-500 steps, which probably means that most data is pretty meaningless when training. I have tried fixing this by putting all masses in circular orbits. This produces much nicer simulations, while still involving a good deal of interaction between masses.

Additionally, 10 train trajectories is obviously insufficient, so the full 1000-2000 trajectories are needed (I imagine you have tired this already).

With these changes I get down to about 100 MSE test acc (haven't seen the generated simulation yet) on the 6-body simulation, which is still much higher than reported in the papers.
I've been running training on CPU so I haven't quite built up the number of epochs they say in the paper. I will try a proper GPU run at some point.

Sorry, I don't mean to critique your work, and I suspect you have gotten this working at some point and other changes you've made (for testing purposes) have perhaps broken it. Just trying to find out the settings I need for a good final simulation. And thanks for providing this repo!

Best,
Will

[jsikyoon]

Hi Will,

First off, thank you for your helpful comments.

I checked the issue flying off after hundreds steps, but just used it with the detail tuning (e.g. initial location region, the range of initial velocity).

Could I know how did you make the masses move in orbits?

Agree 10 trajectories is not enough and validation data not just out-of-samples but also from different trajectories is more reasonable.

I remember the training was very sensitive to which model structure is used, hope that your model will saturate.

Thank you for your helpful comments.

Best, Jaesik.

[EndingCredits]

Hi Jaesik,

Using centripetal force eqn a = v^2 / r and gravitational eqn F = ma = GMm / r^2 gives tangential velocity v = sqrt(GM/r). I then set initial velocities via

      data[0][i][3]=-1*sin(theta_rad)*np.sqrt(G*data[0][0][0]/distance);
      data[0][i][4]=cos(theta_rad)*np.sqrt(G*data[0][0][0]/distance);

In your communication with the original authors, did you discuss the environment settings at all?

BW, Will

[jsikyoon]

Thanks!!

No, I just got a confirmation they cannot open their dataset.

Best, Jaesik.

[EndingCredits]

Par the course for DeepMind, unfortunately (although at least they are willing to provide comment).

Did you ever get the loss numbers that they report in the paper?

-Will

[jsikyoon]

Hi,

As I remember, the result which I posted is what I got.

Bests, Jaesik.