UForm is a Multi-Modal Modal Inference package, designed to encode Multi-Lingual Texts, Images, and, soon, Audio, Video, and Documents, into a shared vector space! It comes with a set of homonymous pre-trained networks available on HuggingFace portal and extends the transfromers package to support Mid-fusion Models. Late-fusion models encode each modality independently, but into one shared vector space. Due to independent encoding late-fusion models are good at capturing coarse-grained features but often neglect fine-grained ones. This type of models is well-suited for retrieval in large collections. The most famous example of such models is CLIP by OpenAI. Early-fusion models encode both modalities jointly so they can take into account fine-grained features. Usually, these models are used for re-ranking relatively small retrieval results. Mid-fusion models are the golden midpoint between the previous two types. Mid-fusion models consist of two parts – unimodal and multimodal.
Features
- Early-fusion models encode both modalities jointly
- Late-fusion models encode each modality independently
- Mid-fusion models are the golden midpoint between the previous two types
- Mid-fusion models consist of two parts – unimodal and multimodal
- The unimodal part allows encoding each modality separately as late-fusion models do
- Encode Multi-Lingual Texts, Images, and, soon, Audio, Video, and Documents, into a shared vector space
Project Samples
