Will DepthFlow implement features similar to Immersity 4.0 (Neural Depth Rendering)?

[indlulzsec]

Description

Hi! I recently checked out the new Immersity 4.0 update which introduces Neural Depth Rendering, a technique that reveals hidden details in images by individually processing depth layers with advanced inpainting. It allows for deeper, more dynamic virtual camera movements and reduces distortion.

Since Immersity is a closed-source platform, I’m curious:
Are there any plans to bring similar features or techniques (like Neural Depth Rendering) to DepthFlow in future updates?

Would love to see these capabilities in an open-source project. Thank you for all your work on DepthFlow, it's an amazing alternative!

[Tremeschin]

Hi there, tricky request, huh?

I think I've seen a video of it. It should be theoretically possible to implement something similar here, yes. But I'm afraid the whole workflow would be too demanding for low end users, and better handled inside ComfyUI or an advanced script.

What I'm thinking is, imagine a textured wall away from the camera and a sphere close by. Let's say the whole sphere is at depth > 0.5 and the wall at depth < 0.2. If we cut the depthmap at every point below 0.5, we will get the wall and a missing mask of the sphere. We then inpaint the missing detail with some AI and get a full layer of the wall itself. Then just about rendering depthflow's math twice, first the background then the sphere layer. Both layers should only be allowed to output colors in their depth range to not overwrite previous layers. Example with the mountain:

^{Another problem to solve: Which AI model? What prompts? How to call their inference code, etc.}

Something like this would tell what is behind the mountain, render it first, then add the other parts of it on top. A shortcoming is that the inpainted areas won't have their proper depth values, re-estimating with models would give inconsistencies in the layers (so does inpaint), so it would pretty much only work for small offsets.

I do, however, have ways of detecting the stretchy parts, maybe a regular AI inpaint per frame (expensive) could solve it the best way. I got this example video that didn't necessarily use it, just feedbacked images to AI in Comfy:

(Click to expand video)

AnimateDiff_00016.mp4

^{Not my creation, colors got cooked because an upscaler was being applied}

Ultimately, it's up to experimentation and finding the best way to do it resource and quality wise. Immersity could be doing something similar, but they can train models to detect objects in the scene more easily, whilst here needs the agnostic layers thing. Any contributions on how to do this or showing tools/packages/knowledge that can help are very much welcome!

[indlulzsec]

Thanks for the detailed response! That makes a lot of sense. I can see how adding multi-layer depth rendering and inpainting per depth range could get heavy, especially for lower-end users.

I appreciate the breakdown of the idea with the wall and sphere. It really helped me visualize the workflow. The part about layering and selectively rendering based on depth is super interesting. I get that the inpainting depth inconsistency could be tricky for bigger camera moves, but even limited offsets could still look impressive.

I'll definitely explore more around ComfyUI workflows and see if I can find any relevant tools or models that could help. If I come across anything useful, I’ll share it here.

Thanks again for taking the time to explain it all. DepthFlow is already awesome, and seeing these kinds of experimental ideas being considered is super exciting!

[onlysharma0852]

Immersity using the integrated gpu of our system through browser directly.

00028_animation.mp4

Its almost perfect.

[Tremeschin]

That's a great result @onlysharma0852, but it weirdly feels like to me they separated it in two or three distinct layers

Could you send the same animation with this Minecraft concept art? It kind of has multiple levels of well defined objects at certain depths that would easily occlude one another in such animation, could give insights how they made it.

I really hope we're not doing a clean room design here, JIC our math/codebases are 100% distinct LOL 😅

[onlysharma0852]

wallhaven-e7651w_animation.mp4

I think it looks like they are filling up some spaces.

[Tremeschin]

There we go, we have our answers right here! 🙂

My first comment turned out to be very precise when reasoning out in my imagination how it can be done:

• Notice how the closest tree and flowers still have the stretchy edges, as all of them are on the first depth separation layer range (no different than depthflow or regular immersity), seems like there are only 2 layers overall.
• I don't fully understand how in between Alex's arm and body got filled. I think they detected such stretchy parts and applied a blur like this one, or their depth estimation failed miserably there (intuition says the latter)
• Yes, they are filling up spaces as expected per range cutoffs as I described in the original comment, otherwise there simply isn't information to render that part and it stretches. The transition is very clear in the closest tree, stretchy at the bottom and inpainted in the middle where the background is visible to the camera on offsets.
• The depth range cut is abrupt seeing the honeycomb and center tree. We can make a much better job on actually doing object detection and splitting it into smarter depth range layers. I guess they chose 0.5 as only one inpainting is done.

Womp, shader code would need to change a bit for this feature. I think it would be better done in a separate file anyways for advanced usage, because inpainting itself would need a decent-ish GPU or patience.

I can make it parametric so any number of these layers would be possible, which would improve quality even more (also they got aliased edges, ew, set sssa=4 in current git main and use 16x more GPU for the smoothest thing hehe)

[onlysharma0852]

I think you are right. There is only a little difference between depthflow and immersity 4.0. They are Just filling up the spaces.

Well all the best bro. Depthflow is still an amazing tool. But we will wait for the future update when depthflow will surpasses immersity💀😌

[Tremeschin]

Another very interesting observation: The background layer is mostly... static?

See, they seem to have zeroed out or multiplied by a small value (~0.2) the depth values there, likely because of the problem I mentioned of the inpainted areas being impractical to estimate their depths and keep relative values to original ones intact.

This makes it so the foreground move at a different speed than the background, causing the honeycomb or center tree discontinuities. The scene is practically a composition of the standard thing nearby and a lesser intensity background.

[Tremeschin]

I hope it so, too @onlysharma0852 🙂

There's quite some work left in my own executables solution before I work on this though, as they're missing from the v0.9 release. No ETAs, just a matter of priorities, but I'm very interested on this!

[onlysharma0852]

Its ok bro take your time! ❤️

…

On Wed, 16 Apr, 2025, 10:57 pm Gabriel Tremeschin, ***@***.***> wrote: I hope it so, too @onlysharma0852 <https://github.com/onlysharma0852> 🙂 I'm working on another banger side project I'll publish soon, then final website work, release new version and I'll try getting this inpainting working in an advanced script. No ETAs, just a matter priorities, haha — Reply to this email directly, view it on GitHub <#82 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A6NM55FBCJOP6KNWFOJF2JT2Z2HG7AVCNFSM6AAAAAB3BQ3QY2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMJQGIZTSNZQG4> . You are receiving this because you were mentioned.Message ID: ***@***.***> *Tremeschin* left a comment (BrokenSource/DepthFlow#82) <#82 (comment)> I hope it so, too @onlysharma0852 <https://github.com/onlysharma0852> 🙂 I'm working on another banger side project I'll publish soon, then final website work, release new version and I'll try getting this inpainting working in an advanced script. No ETAs, just a matter priorities, haha — Reply to this email directly, view it on GitHub <#82 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A6NM55FBCJOP6KNWFOJF2JT2Z2HG7AVCNFSM6AAAAAB3BQ3QY2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMJQGIZTSNZQG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>