Training Details

[anilbatra2185]

hi @loveofguoke @longqian-zju

I have the following queries on training the model.

• Do we train the model with image-text or video-text pairs? If yes, then which datasets are used for the same.

• How to train with GESM data? The converted data do not have conversations key, however, the source code expect this information. Moreover after conversion, the only present keys are id, data_type, data as used in convert_data_gesm.py.

• I assume we first need to train with GESM data and then perform supervised instruction fine-tuning with Moment-10M. Please confirm.

It would be really helpful if you can share the information to train the model along with appropriate dataset format.

Thanks in advance!

[longqian-zju]

Sorry for the inconvenience.

Do we train the model with image-text or video-text pairs? If yes, then which datasets are used for the same.

By utilizing the checkpoint of Video-ChatGPT, we skip the stage of modality alignment. Some of the results from other projects indicate that it will be better to train from sketch with more recent image-text and video-text datasets, but it will be much more expensive.

How to train with GESM data? The converted data do not have conversations key, however, the source code expect this information. Moreover after conversion, the only present keys are id, data_type, data as used in convert_data_gesm.py.

I will recheck the data format. Thank you for your interest in our work!

[skhcjh231]

Have you checked the GESM data format? I also want to know how to train using GESM annotations!