General phases of Generation and Processing Question.

[tounsicorp]

So within the context of efficiency. Couldn't the process become more streamline for vid2vid animation if the system processed the Actual frames first, then focused on building the occlusion AND THEN process into the prompt material?

It seems like a waste of resource of formatting when the system is running all the processes at once instead of processing in steps. That's what we do in rendering large formats for Animation, why isn't it being done for the SD Animation? Or is that just an oversight that these developers don't really have insight on, because they aren't into the animation field?

General question/idea for future purposes.

[ricardofeynman]

@tounsicorp Unfortunately, for the time being at least, dev has shifted focus (to a pretty dam cool project - check it out on his other repo).

My guess is if he is ever to return to this, it'll be when he is making bank from his monetizeable endeavours. That leaves us to do the work.

I have been working on an approach to expand/modify functionality for both v2v and t2v. Will leave it at that until I feel I'm closer to delivering the goods.

Can you lay out your idea in more detail?

[tounsicorp]

lol. That's about the same for all cool projects that have validity to be amazing.

All of these stable diffusion models and systems are trying to push and force processes quickly through the render stages and denoise into something you want. But the denoising process can be so much smoother and coherent if the user is able to watch the steps and communicate to the pytortch to change edits in realtime.

I've been modding and building extensions and addons on Maya, Mudbox, Blender, and nukex for years. These systems render slowly for a reason to better format what is being presented for the render. It seems if Stable diffusion and some of these models would stop trying to push the information through the tensors to make up for speed, they could run models with better results. Which could lead to fully rendering animation scenes longer than 5 seconds with the way that they have it sent up.

SD CN Animation tries to morph every part of the scene at once through one continuous process. That's why you get artifacts and stretching. Instead of allowing the Image generator to actually use models for the same movement, but render them at different positions. It just seems like there is a lot more functionality within the software, but it's being blocked by the lack of foresight on how it's going to be used and developed.

[ricardofeynman]

Yeah I would imagine teams of well paid anim industry wizard class programmers will be busy investigating these refined approaches for SD. Probably won't see them open sourced any time soon though.

I remember about 11 years back I spent 8 days rendering 18 frames at 4k in Vue. I almost haven't touched a 3D rendering tool since! I'm sure speed will have improved greatly since then with newer cards, but that was the render that put me off going further down that avenue and got me into exploring game engines. Tonight I'll have finished my longest SD-CN sequence yet, close to 90k frames, only took 15 days (and possibly another day ahead to upscale to 4k)!

My point? Speed is nice, and currently, some quality tradeoffs come with that. Personally I'm more excited at the prospect of being able to use real-time generations for live performance this year, than having comparible quality to a traditional 3D rendering engine, but I take your point.

For me, specifically relating to SD-CN, I'm just trying to wrap my head around exactly which paramaters in which scripts are controlling the extent of per frame motion and warping. The type of flow I'm getting with certain models and prompts is very nice indeed, but with other promps or models, the same parameters produce a lot of 'artifacts'. The stretching (hopefully we're describing the same quality) is one of the signiature aesthetics of SD-CN and one I've put to good use. I just require more granular control.

There's definitely a powerhouse lurking in this extension, but exposing the params has to happen first, and if you're a seasoned programmer for animation tools, I'd be grateful to talk with you about this further at a later stage if you might have the time/inclination to share some thoughts on how you would personally go about expanding the functionality of SD-CN using what has already been provided by the dev.