The KL divergence of gumbel relaxation

[taufikxu]

Dear Authors:

Thank you for your effort in reproducing this project. It's really useful and helpful.

In this project, I think the KL divergence is something wrong. The KL divergence should be tf.log(prior_temperature + eps) - y_times_prior_temp """- y""" + prior_log_odds -
2.0 * tf.log(1.0 + tf.exp(-y_times_prior_temp + prior_log_odds) + eps), where I use """ to highlight the difference between the original implementation.

The log probability is proportional to -1*(temperature + 1) * pre_log_y.

[aprams]

Hi @taufikxu,
Thanks for your comment and detailled suggestion, will take a look at it!