Multi-view learning (mvlearning)

Framework for different fusion strategies in multi-view learning with PyTorch. Academic purposes

Examples

• Example on how to define and use different fusion strategies in the MVL model
• Example on how to train these MVL models

For definitions on the concepts used here please look at Common Practices and Taxonomy in Deep Multiview Fusion for Remote Sensing Applications.

Fusion types usage

• For Input fusion (with feature concatenation) you can just create it with

from mvlearning.fusion import InputFusion
InputFusion(single_pytorch_model, view_names=["a list of string", "with the names of the views"])

• For Decision fusion (with averaging output) you can just create it with

from mvlearning.fusion import DecisionFusion
DecisionFusion({"view 1": pytorch_model1, "view 2": pytorch_model2, ...})

• For Feature fusion you can just create it with

from mvlearning.fusion import FeatureFusion
FeatureFusion({"view 1": pytorch_encoder1, "view 2": pytorch_encoder2, ...}, pytorch_merge_module, pytorch_model_head)

Details

• The encoders have to had a function called get_output_size where the dimension of the output is returned.
• For detailed examples see the Examples folder.

Install

• For installation you can run:

pip install --editable .

---

Sensor Dropout (SensD)

We include a new component based on our recent paper. This component randomly drop sensors (or views in our case) during training. In practice the drop means it replace the values with 0, but other options can be easily extended.

• In any of the model described (fusion types) the sensor dropout will work by just indicating sensd in the maug argument.

The maug_args with drop_ratio is optional. In case it is not used it will randomly select one missing combination from the list of all possible missing cases.

from mvlearning.fusion import InputFusion
InputFusion(..., maug="sensd", maug_args= {"drop_ratio": 0.3})
... #same for other fusion types

• Example on how to train with this technique

Citation

📜 Mena, Francisco, et al. "Increasing the robustness of model predictions to missing sensors in Earth observation." accepted at the MACLEAN workshop in the ECML/PKDD, 2024.

@article{mena2024increasing,
  title={Increasing the Robustness of Model Predictions to Missing Sensors in Earth Observation},
  author={Mena, Francisco and Arenas, Diego and Dengel, Andreas},
  journal={arXiv preprint arXiv:2407.15512},
  year={2024}
}

---