GAN generated images after training are different than during training

[benbo]

Hi all,

I am experiencing a curious issue training a GAN with pytorch lightning. Namely, the generated images logged during training are great, while using the generator after training produces basically just a uniformly gray mess.

model = LightningGAN()
trainer = Trainer(gpus=args.gpus, logger=tb_logger, accelerator='ddp', max_epochs=args.max_epochs)
trainer.fit(model, datamodule)
# if I now use model to generate images, they are just uniform gray noise. 

I have checked if this changes if I set the generator to eval() or train() but that doesn't impact the issue. Is there anything else I might be missing?

[rohitgr7]

did you check if the pipeline where you are generating images after training is the same as that of the one used during training?
hard to tell the issue without a reproducible script.

[benbo]

Yes, I'm calling the exact same function to generate images. Adding a script to reproduce this is difficult due to all the dependencies but I'll see if I can put a simple one together.