About the entropy_model in paper “END-TO-END OPTIMIZED IMAGE COMPRESSION”

[TNTWEN]

Hi,
Recently, I am learning the entropy model in paper “END-TO-END OPTIMIZED IMAGE COMPRESSION” and “VARIATIONAL IMAGE COMPRESSION WITH A SCALE HYPERPRIOR”
I want to know could i find the code of the entropy_model in paper “END-TO-END OPTIMIZED IMAGE COMPRESSION” which used piecewise linear density model?

Thanks a lot!

[jonaballe]

Hi,

sorry, we don't have code for the piecewise linear model any more. The model in the "end-to-end" paper was written in Theano, which is not maintained any more, and wasn't in a good state for publication to begin with.

You can consider the NoisyDeepFactorized model a drop-in replacement that also doesn't have some of the problems of the piecewise linear model (e.g. it can model smooth densities more accurately).

Hope this helps!

[TNTWEN]

Thanks ! ^-^

[mharshe]

I'm looking at using the code from model.py to create a model that we can retrain, but I see that the archs.py has this block of code:

class FactorizedPriorLayer(tf.keras.layers.Layer):
  """Factorized prior to code a discrete tensor."""

  def __init__(self):
    """Instantiate layer."""
    super(FactorizedPriorLayer, self).__init__(name="FactorizedPrior")
    self._entropy_model = tfc.EntropyBottleneck(
        name="entropy_model")

and it the class Hyperprior

self._side_entropy_model = FactorizedPriorLayer()

The EntropyBottleneck class was removed in commit 89eefc8 so I'm not sure how this code will run? Should I replace that side_entropy_model with NoisyDeepFactorized?

[jonaballe]

Hi mharshe, please open a new discussion item for this. @fab-jul is a better contact for the HiFiC code base.

[TNTWEN]

@jonycgn
Hi! I have a question about differential cross entropy and Shannon (discrete) entropy.
In Shannon (discrete) entropy, the real probability distribution is unknown.

In loss function we compare

and q is unit length uniform distribution,So we just compute

In "end-to-end " ,you show the empirical evaluation of the approximation of differential cross entropy and discrete entropy.

I am wondering how do you compare the discrete entropy in the horizontal axis?
it's just

divided by the number of pixels or compress latent variables into bitstream and calculate bpp?

If it is the former, I am confused about the approximation,because it seems that this can only explain the approximation between adding noise during training and real quantization. As for Shannon entropy, we don't know the real probability distribution and we can't calculate it ，so how could we evaluate the approximation between differential cross entropy(estimated bpp) and Shannon discrete entropy (actual bpp) ?

[jonaballe]

Hi @TNTWEN,

actually in both cases (discrete and continuous), we are measuring cross entropies. The probability model in both cases is not necessarily the same as the true distribution (but we are of course fitting it to be close).

I hope this answers your question.

[TNTWEN]

@jonycgn

Can I understand it like this：

is the self-information of one point to be coded. we sum all points' self-information to measure the total bits we need to
encode the latent variable.

p comes from the entropy model which is not the true distribution. We could minimize the number of bits if p is identical to the true distribution (which also means reducing the cross entropy )

[jonaballe]

There are two distinctions here:

• differential (relative) entropies vs. Shannon entropies.
• cross-entropy vs. entropy.
  So you can have a differential cross-entropy or a Shannon entropy, for example.

Differential (relative) entropies are defined as integrals over continuous distributions, and are not lower bounded by zero. They are not a measure of bits to be encoded.

Shannon entropies are lower bounded by zero.

The true distribution of y can't be known, because we don't know the distribution of the data. So, we are always forced to use cross entropies here. Shannon cross entropies also express exactly the minimum bits needed to compress the data y using a code which is designed for a probability model p. This is the source coding theorem. An optimal code for p encodes each possible y into a bit sequence of length -log_2(p(y)). So if we assume that we can design an optimal code for any p, we don't need to worry about the code itself, just p.

The cross-entropy is minimal exactly when p is equal to the true distribution of y. In that case, it is equal to the entropy of y.

The reason why we use differential entropies here is that Shannon cross-entropies cannot be optimized with SGD. The gradients vanish due to quantization. But adding uniform noise and then taking the differential cross-entropy, it turns out that this is a differentiable upper bound on the Shannon cross-entropy (for details, see https://doi.org/10.1109/JSTSP.2020.3034501). This is why in the figure above, the differential entropy is always slightly above the entropy.

Hope this helps!

[TNTWEN]

Very helpful! Thank you very much for your patience! 😄