Batch prompts (or prompts from list compatibility)?

[oxencorp]

I'm loving this and getting interesting results in txt2vid, but I'd love to be able to feed a list of prompts in and come back later to a bunch of videos. I've tried the using the prompts from file script, but I suspect it's jumping in at the frame level instead of waiting for a video to complete. Is there any way to create a list of prompts and run through them instead of manually starting them one at a time? Thanks!

[peteTater]

This would be great, I had just started writing a similar feature request when I spotted your comment. #137 Hopefully a batch option for the same prompt will be fairly trivial to implement, and could open the door to further batch options if volotat is so inclined at some stage. I'd love to be able to go out for the day (or go get some sleep!) and come back to a range of new material to work with.

[jfranecki]

I love this idea! It would be great to have prompts and settings set for each video and then process them through a queue. That'd be awesome to leave my computer alone for a weekend and have 10 videos processed with my prompts and settings.

[peteTater]

Fingers crossed the big chap is up for it, I'm glued to my machine most days presently as I'm generating many shorter clips, to edit together. Not complaining of course - I wouldn't be getting anywhere near as much work done if not for this extension, but it would be so sweet to set up a few hundred clips to render from the same prompt, once I've tested and know the prompt produces the goods each time I generate with it, and then just step away for a day or two while the magic happens.

[jfranecki]

Same here. Ever since I discovered this extension two weeks ago I only turned off my computer once to install a 4090 for better video resolution and faster processing for this extension.

[peteTater]

Nice, yes the only reason I'll be turning off my machine anytime soon is to take the plastic backplate off my 3060 to stop it melting in to the case fans below! Got a second 3060 sitting idle in its packaging for the last few weeks, so hopefully a multi-GPU option for A1111 happens at some stage soon. (Edit: From what I gather though, although data parallelization is possible, and running multiple instances of sd-webui works with enough system RAM, memory pooling seems to be the holy grail for creators with multi-GPU setups, but it's hardly in the low hanging fruit category.) I've been making 2D/3D/stop motion animation for years, but the amount of styles Ive managed to get working well with this extension in such a short period of time is astonishing.

[oxencorp]

This is probably like when a client who knows nothing about animation says 'this should be easy..' but with my complete lack of coding knowledge these are some possibilities for how I see this working, starting with what I assume would be easiest:

vid2vid Batch folder process with a single prompt: Load up a directory of videos and apply the same prompt to each one until they are done.

txt2vid with a single prompt & varied seed: Let user set a N value for number of videos and then output that number of videos from the same prompt with a random or incremented seed value.

Prompt list: a text file list of prompts to generate from. for txt2vid this could just generate one after the other, for vid2vid it could apply sequentially to videos in a source directory or generate videos for each prompt for every video in a source directory.

Wild card: I love this format for still images and am aware it could quickly overwhelm a machine for generating video, but being able to have a wild card prompt with an adjustable number of random outputs would be great. something like (< astronaut | cowboy | rabbit >) (< eating | riding >) a (<bicycle | motorbike | pizza >) (< line art drawing | photograph | oil painting >).

Have I missed anything?

Thanks for considering this request!

[volotat]

How about we do it a little bit differently? I thought a lot about how to implement this feature in a maximally productive way and here is what I think: it is not only prompts that we usually want to change from one run to another, but all available parameters. We also have an export parameters feature and soon there will be parameters import. So, I think the better option will be to have a list of json files with all parameters predefined and then we simply run through all those json files one by one. I just do not see the best way to integrate it into the UI yet.

[peteTater]

Wow exciting! Sounds great, thanks for going maximal! This approach seems like it would be especially useful during the initial exploratory phase for tweaking settings to find the sweet spot for fluid motion and structural consistency of the subject in any given seed/prompt combination.

It's hard to tell from a few frames how the animation will progress, so I'd been using a min 5-10 sec sequence txt2vid generations to get a better sense of how things change per param tweak. Presently that's a hassle for doing 60-120 frames manually. It's the only way I know of to have peace of mind that when leaving the machine unsupervised, a longer sequence won't entropy to a blur or other undersired aesthetic, but it was so time consuming I've taken to leaving every setting at default, and just focus on observing how my choice of prompts and models play to the strengths of the txt2vid tab default settings! For my current use case (and now character-free choice of subject matter) this has worked out to be a time saver, but any and every batching methods that allow for faster variations testing and some extended time away from the screen are most greatly appreciated.

Few questions and related use cases:

Q1. If not already implemented in the latest commit, in future updates, can export param files also include the seed used even if it was initially set to -1 when generation begins? (Thinking this'll also be essential for the per-update quality improvement metrics I have in mind to make the deterministic aspects of testing reproducible between versions.)

// Use case: Re-run same prompt/seed/settings combination for the unique evolutionary variations the optical flow provides per re-run even when using identical settings. Presently if returning to a sequence to do a re-run after restarting A1111, or just generating other sequences, I copy the seed back in to txt2vid tab from the first generated image of the sequence that goes to "txt2img-images" folder. With the caveat that I've not updated from 111711f, the export params feature presently doesn't save the seed used when it was set to -1, so unsure how this approach could work if batch import of params files are to be used, and the seed is always -1 in the settings files. I'm aware of #80 and that export file now goes straight to folder after the update yesterday but I can't test newest update for next few days to know if current export files still lists -1 seed. Apologies if any of these queries are already redundant.

Q2: Will the export/import settings approach accommodate for a user-specified folder path, containing the final frames from the sequences the creator wishes to extend by batch process for txt2vid? Or even better, as oxencorp requested, defining a directory for running through all of its sub-folders' contents?

// Use case: Having generated N sequences, there are generally a smaller subset of sequences where the creator may wish to continue the sequences from their respective final frames to extend the shot duration. Presently I drag in the final generated frame of a sequence on txt2vid tab and re-run several times until I get a continuation of the animation that I'm happy with. Or sometimes, an otherwise beautiful sequence has some frames where an unwanted text or graphic makes it in to the shot despite negative prompts and it becomes far too apparent a visual feature of the frames to manually fix blemishes in an image editor, so the only way to remedy is re-run from the final text/graphic-free frame in a sequence to achieve the desired shot length with consistent visual quality. I have to do this several times manually for every sequence I want to extend, so I copy the final blemish-free frames of N sequences into a folder to keep track of them, and work through each one manually, usually generating several times over until I get the desired results.

Q3: Will this approach allow for N single frame batch generation on the txt2vid tab to test out and easily compare variations of prompts/seeds/settings with the full range of the txt2vid tab settings?

// Use case: Although inconvenient, time wise I find it's still preferrable to source an initial image from a batch run on the main A1111 text2img tab. Run a batch of 100 images and pick the best ones as starting seeds as it works out faster and avoids clogging the SD-CN output folder with a bunch of two frame sequence folders and 0sec length videos from interrupted sequences that result from using seed -1 on the txt2vid tab if not satisfied with aesthetics of the first generated frame. Having to copy over prompt/seed/steps/cfg each time is time consuming and leaves room for human error so it would be a major development if N single frame batch runs with full tab param control for every imagine in the batch "prompt-matrix" or x/y/z plot"-style could be viewed as grids natively somewhere within the extensions UI.

Looking forward to seeing how you proceed with it!