Export Depthmap and create Side-by-Side Images

[spackonaut]

Somehow I can't download the Depthmap. I can press CTRL+I on Firefox and then go to Media and find it there. I also didn't find a comand line switch to save the Depthmap.
The reason is that I want to watch the result on a 3d capable display (a projector, google cardboard etc). It is even not easy to find a program that can create Side-by-Side Images from the Depthmap and the original image. The search results are cluttered with ai stuff how create the depthmap. But how to get it to view it on the hardware is complete gone. I have some old windows program from 2006 (3d combine) that could do that, but crashes under wine (I'm on linux only).
Would be happy if somebody has better options.

[Tremeschin]

Somehow I can't download the Depthmap. (...)

I've noticed firefox is inconsistent on being able to download the depth map or not when right clicking, I find it weird because show_download_button is True by default in the gradio image component and I'm doing no attempts in blocking it. Will take a look before the next release, just created issue #72

I also didn't find a comand line switch to save the Depthmap.

That's a common requested feature I can easily hook up! Tracking on #73 so I remember it

I'll add a command to export the depth maps to uint16 pngs as they are saved in each of the depth estimator classes of mine! They all should be located in ~/.local/share/BrokenSource/DepthFlow/Cache/DepthEstimator/*, I'm using diskcache https://pypi.org/project/diskcache/ for the easy disk database so it's sharded, a bit inconvenient to search for manually, but they're there

Also, as you're on linux it should be able to identify all the files ending in .val are actualy PNGs because of the magic headers at the file start :)

(...) Side-by-Side Images from the Depthmap (...)

With side by side do you mean rendering like this?

video.mp4

You can press "p" in the realtime window to cycle camera projection modes or programmatically change it in a custom script file like this (copied from Custom.py, I'll add a batch export and SBS demo scripts, and prob rename VirtualReality to SideBySide):

import sys

from DepthFlow.Scene import DepthScene
from ShaderFlow.Modules.Camera import CameraProjection


class CustomScene(DepthScene):
    def setup(self):
        DepthScene.setup(self)

        self.camera.projection = CameraProjection.VirtualReality

def manual():
    scene = CustomScene()
    scene.cli(sys.argv[1:])

def managed():
    scene = CustomScene(backend="headless")
    scene.input(image="image.png")
    scene.main(output="./video.mp4", fps=30, time=5)
    scene.main()

if __name__ == "__main__":
    # managed()
    manual()

Hope that helps! Let me know if this is what you're looking for 🙂

[spackonaut]

Thanks for your fast answer and dedication. I should have tried with chromium, the right click to download works with it.
I haven't look into the python scripts, and I don't know if I have the time to grasp it. I managed to process a sample image with the script (the first hit for "person" :-) ): https://www.indiafilings.com/learn/wp-content/uploads/2023/03/Can-a-single-person-own-a-firm-in-India.jpg. The video generation is not so important, a single sbs image would be fine.

There are unfortunately some problems:

• I guess one would need to tune the depth and offset or some other parameters to have the right depth when viewing it.
• The aspect ratio for most projectors must be halved in width so that both images fit in a 16:9 image. However, halving the width of the SBS image afterwards messes up the depth information. I don't know how to handle images that are not in 16:9 aspect ratio.

With the projector I was not able get it right. With the a google Cardbord thingy the depth was not really recognizable.

[Tremeschin]

Oh good point, on side by side we should double the aspect ratio, we can enforce it manually like depthflow main --ar 32:9 for a 16x9 input image, ideally automatic, will try making so.

There's a hacky way to get screenshots via scripts currently on git main:

from DepthFlow.Scene import DepthScene
from PIL import Image
from ShaderFlow.Modules.Camera import CameraProjection


class CustomScene(DepthScene):
    def setup(self):
        DepthScene.setup(self)
        self.camera.projection = CameraProjection.VirtualReality

        self.scene.on_frame.subscribe(self.save_frame)

    def update(self):
        self.state.height = 0.5

    def save_frame(self):
        data = self.screenshot()
        image = Image.fromarray(data)
        image.save("screenshot.png")

def main():
    scene = CustomScene(backend="headless")
    # scene.input(image=...)
    scene.main(
        freewheel=True,
        fps=1, time=1,
        ratio=(32/9),
        height=1080,
        ssaa=4,
    )

if __name__ == "__main__":
    main()

I was able to get this image out of it:

I should have a better way of manually controlling the scene so we can simply do scene = DepthScene() then scene.next() and data = scene.screenshot(), it fails currently due some stuff that happens before compiling shaders in the main() method, will keep an eye for improving the usability that way!

I guess one would need to tune the depth and offset or some other parameters to have the right depth when viewing it.

Yes!, I have a good sense of what the units actually are, viewing angles etc. Will have the formulas in the docs eventually and how it works / assumptions etc.

Maybe the 3d wasn't that much noticeable because the animations needs to be kinda subtle for people to not complain on the stretchy parts where there's simply not image data to fill in there 🙂

Cranking up the state.height parameter makes the surface pop out a lot more

It might look weird with crossing the eyes because the left image is meant to be the left eye. Let me know how it works on VR equipment, as I don't have any here :^), ^{as it looks good if I make the left image the right eye and cross it}

[Tremeschin]

For reference, the same image above but inverted L/R, if I cross my eyes it looks great instead of weird

Do you happen to know what is the right option that those stereoscopic viewers expect? I feel like the first one (left image is left eye) is the correct one? but looks bad when crossing the eyes ofc, and the (left image is right eye) looks natural

[spackonaut]

I just managed to install the git version. In the script I just changed the ratio to 16/9 i get a nice 3d effect 👍 But the mountain is stretched in the width by the factor 2. So what is needed is a final image with 16/9, where the left and right image are not cropped to 8/9, but compressed to 8/9. (Here is an example of some youtube sbs clip: https://youtu.be/u0hmMt5znX0?t=170)

When I was rerendering with 32/9 in the script and just doubling the height with gimp everything seemed to be fine (except some artefacts on the of the left front mountain ridge):

I have tried a non 16/9 picture, (https://openverse.org/image/6c55dd5c-dceb-42e1-9845-9e5ebd343026?q=portrait&p=19). In this case some of the top and bottom is cropped and left and right is padded with something:

original image:

script with 32/9:

final: stretched the height by factor 2:

This looks also very nice in 3d! I think for some pictures just padding without cropping would be better, maybe padding with black.

I guess for most projectors left image for the left eye, and the right for the right eye is the default. But at my projector I can actually swap that in the settings.