CogVLM2 is the second generation of the CogVLM vision-language model series, developed by ZhipuAI and released in 2024. Built on Meta-Llama-3-8B-Instruct, CogVLM2 significantly improves over its predecessor by providing stronger performance across multimodal benchmarks such as TextVQA, DocVQA, and ChartQA, while introducing extended context length support of up to 8K tokens and high-resolution image input up to 1344×1344. The series includes models for both image understanding and video understanding, with CogVLM2-Video supporting up to 1-minute videos by analyzing keyframes. It supports bilingual interaction (Chinese and English) and has open-source versions optimized for dialogue and video comprehension. Notably, the Int4 quantized version allows efficient inference on GPUs with only 16GB of memory. The repository offers demos, API servers, fine-tuning examples, and integration with OpenAI API-compatible endpoints, making it accessible for both researchers and developers.
Features
- Supports both image and video understanding with multimodal reasoning
- Up to 8K context length and 1344×1344 image resolution input
- Bilingual support for English and Chinese interactions
- Quantized Int4 version for efficient inference on 16GB GPUs
- Outperforms previous open-source models on TextVQA, DocVQA, and ChartQA
- Provides demos for CLI, Gradio, API server, and fine-tuning workflows
Project Samples
