Generate smooth sky from an original image with unsmooth sky

[michaelhuang74]

I have an original image shown as follows. Due to the moon, the lower half of the sky is brighter than the upper half. The part of the sky around the moon is particularly bright.

When I use the "Times" filter in the Vinci app, the output shows an uneven sky as follows, which reflects the unsmooth sky in the original image.

However, when I use a different filter, i.e., "Harvest" filter, the output shows an amazing smooth sky as follows.

I also tried to use texture_nets to stylize the original image using different style images, the output always had the uneven sky like the one with the "Times" filter. I have tried to use different combinations of layers for styles and contents, changed the style sizes and image sizes, no luck so far.

Any one has idea how to generate a smooth sky like the one produced by the "Harvest" filter? Thanks.

[DmitryUlyanov]

Hi, did you try eval for instance norm?

[michaelhuang74]

@DmitryUlyanov

Thanks for the response! Could you provide more details? When you say "try eval for instance norm", what do you exactly mean?
Many thanks.

[DmitryUlyanov]

One possibility is to modify eval function https://github.com/DmitryUlyanov/texture_nets/blob/master/InstanceNormalization.lua#L94 such that batchnorm is set to eval mode.

For a dirty try just add self.bn.evaluate()
in https://github.com/DmitryUlyanov/texture_nets/blob/master/InstanceNormalization.lua#L48

Using this eval mode you can get more flat images, as far as I know Vinci uses it for some of their filters.

[michaelhuang74]

@DmitryUlyanov

Thanks a lot for the quick response. I will give it a try.

[michaelhuang74]

@DmitryUlyanov

I added self.bn:evaluate() to Line 48 in the InstanceNormalization.lua file. Then I used the following image as the style image.

Followings are the parameters for training:
-style_size 600 -image_size 512 -style_weights 10 -content_weights 1 -num_iterations 10000 -batch_size 4 -save_every 1000 -model johnson -style_layers relu1_2,relu2_2,relu3_2,relu4_2 -content_layers relu4_2 -learning_rate 1e-3
The stylized output is as follows.

Do you have suggestions for (1) getting rid of the white strips on the sky, (2) making it colorful? Thanks.

[DmitryUlyanov]

Hi, being a hack it does not work for every style image, but helps sometimes.