Multi-Modal Large Language Models (MLLM): Data, Methodology, Benchmark, and Platform

I gave a talk about Multi-Modal Large Language Models (MLLMs) at VALSE 2023, including data, methodology, benchmark and platform. More details please check the slides.

Slides

LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark

(1) We present the LAMM-Dataset and LAMM-Benchmark, which cover almost all high-level vision tasks for 2D and 3D vision. (2) We demonstrate how to construct instruction-tuning datasets and benchmarks for MLLMs, which will enable future research on MLLMs to scale up and extend to other domains, tasks, and modalities faster. (3) We provide a primary but potential MLLM training framework optimized for modalities’ extension.

GitHub Repo

Dolphin: A General Video Interaction Platform Based on LLMs

Dolphin is a general video interaction platform based on large language models. Our team is trying to build a smart chatbot for video understanding, processing and generation.

GitHub Repo

Benchmarking Omni-Vision Representation through the Lens of Visual Realms

Omni-Realm Benchmark (OmniBenchmark) is a diverse (21 semantic realm-wise datasets) and concise (realm-wise datasets have no concepts overlapping) benchmark for evaluating pre-trained model generalization across semantic super-concepts/realms.

X-Learner: Learning Cross Sources and Tasks for Universal Visual Representation

Jointly learning from heterogeneous tasks and multiple data sources contributes to universal visual representation, leading to better transferring results of various downstream tasks.

ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis

We host the ForgeryNet Challenge 2021 in ICCV 2021, The 3rd Workshop on Sensing, Understanding and Synthesizing Humans.

MMEKG: Multi-modal Event Knowledge Graph towards Universal Representation across Modalities

MMEKG unifies different modalities of knowledge via events, which complement and disambiguate each other.

Prompt for Extraction? PAIE: Prompting Argument Interaction for Event Argument Extraction

In this paper, we propose an effective yet efficient model PAIE for both sentence-level and document-level Event Argument Extraction (EAE), which also generalizes well when there is a lack of training data.

CelebA-Spoof: Large-Scale Face Anti-spoofing Dataset with Rich Annotations

CelebA-Spoof is a large-scale face anti-spoofing dataset that has 625,537 images from 10,177 subjects, which includes 43 rich attributes on face, illumination, environment and spoof types.

Jing Shao

Research Scientist

Shanghai Artificial Intelligence Laboratory

Biography

Jing Shao is currently at Shanghai Artificial Intelligence Laboratory as a research scientist focusing on multi-modal foundation models and agents, with special interests in understanding various properties of current models beyond their accuracy, such as explainability, robustness, safety and generalization, towards the reliableness of AI systems. She is also an Adjunt Ph.D. supervisor at the Shanghai Jiao Tong University, and a Co-PI of S-Lab in Nanyang Technological University.

She received her Ph.D. (2016) in Electronic Engineering from The Chinese University of Hong Kong (CUHK), supervised by Prof. Xiaogang Wang, and work closely with Prof. Chen Change Loy and the Multimedia Lab (MMLab) led by Prof. Xiaoou Tang. She has published 40+ peer-reviewed articles (including 20 first/co-first/corresponding author papers) in top-tier conferences and journals such as TPAMI, IJCV, ICML, ICLR, NeurIPS and CVPR, with 7300+ citations in Google Scholar. She serves as the reviewer of IJCV, T-PAMI, T-CSVT, T-MM, T-ITS, and CVIU, and reviewed CVPR, ICCV, ECCV, NeurIPS, ICLR, AAAI, IJCAI, and ACM MM.

We are hiring Full-time Researchers/Interns working together on safety/robustness/explainablity of generative models and agents. I am also looking for talented students targeted to Master or Ph.D. degree. Please drop me an email if you are interested.

News

2023.09: One paper accepted to NeurIPS 2023 Dataset and Benchmarks Track.
2023.08: We have released LAMM, an open-source project for Multi-Modal Large Language Models and Applications as AI agents.
2023.06: I gave a talk about Multi-Modal Large Language Models (MLLMs) at VALSE 2023, Wuxi, including the data, methodology, benchmark and platform. More details please check the slides.
2023.03: One paper accepted to CVPR 2023.
2022.12: One paper accepted to NeurIPS 2022.
2022.08: We are organizing ECCV 2022 (2nd) DeeperAction Challenge and Workshop on detailed video action understanding and anomaly recognition by introducing five new benchmarks.
2022.08: We are hosting ECCV 2022 OmniBenchmark Challenge on evaluation of pre-trained models.
2022.07: Winner of Challenge on RxR-Habitat Vision-and-Language Navigation Competition.
2022.07: Four papers accepted to ECCV 2022.
2022.04: One paper accepted to IJCAI 2022.
2022.03: Two papers accepted to ACL 2022.
2021.10: We are organizing ICCV 2021 (2nd) SenseHuman Workshop on human sensing, understanding and synthesis.
2021.10: We are organizing ICCV 2021 (1st) DeeperAction Workshop on detailed video action understanding by introducing three new benchmarks.
2021.08: One paper accepted to IEEE T-PAMI.
2021.07: We are hosting ForgeryNet - Face Forgery Analysis Challenge in ICCV 2021.
2021.04: Two papers accepted to CVPR 2021 (One Oral and one Poster).
2020.09: We are hosting CelebA-Spoof - Face Anti-Spoofing Challenge in ECCV 2020.
2020.08: Winner of ActivityNet Challenge on Spatio-temporal Action Localization (AVA).
2020.07: 4th Place of DeepFake Detection Challenge (DFDC).
2020.07: One paper accepted to IJCV.
2020.07: Three papers accepted to ECCV 2020.
2019.09: Three papers: 1@ICCV 2019 on Text-image retrieval, 1@NeurIPS 2019 on Layout to image generation, 1@AAAI 2020 on Dense point cloud completion.
2019.06: [Bug (invalid links) fixed] Baidu Disk links of CUHK Crowd Dataset has been renewed!
2019.04: Four papers accepted to CVPR 2019 (one Oral and three Posters)
2018.07: Five papers: 3@ECCV 2018, 1@ACM MM 2018, 1@BMVC 2018.
2018.04: Three papers accepted to CVPR 2018 (one Oral and two Posters)
2017.07: Two papers accepted to ICCV 2017.
2017.04: One paper accepted to CVPR 2017
2016.12: One paper accepted to IEEE TCSVT.
2016.09: I am invited to participate in MSRA Ph.D. Forum 2016.
2016.06: Participate in CVPR 2016 Doctoral Consortium.
2016.06: One paper accepted to IEEE TCSVT.
2016.04: One Spotlight paper accepted to CVPR 2016.
2015.04: One Oral paper accepted to CVPR 2015.
2014.04: One Oral paper accepted to CVPR 2014.

Recent Publications

Quickly discover relevant content by filtering publications.

Zeren Chen, Gengshi Huang, Wei Li, Jianing Teng, Kun Wang, Jing Shao, Chen Change Loy, Lu Sheng (2023). Siamese DETR. CVPR.

PDF Code

Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, Hongsheng Li (2022). ST-Adapter: Parameter-efficient Image-to-Video Transfer Learning. NeurIPS.

PDF Code

Yuanhan Zhang, Qinghong Sun, Yichun Zhou, Zexin He, Zhenfei Yin, Kun Wang, Lu Sheng, Yu Qiao, Jing Shao, Ziwei Liu (2022). Bamboo: Building Mega-Scale Vision Dataset Continually with Human-Machine Synergy. CoRR.

PDF Cite Code Project

Yinan He, Gengshi Huang, Siyu Chen, Jianing Teng, Wang Kun, Zhenfei Yin, Lu Sheng, Ziwei Liu, Yu Qiao, Jing Shao (2022). X-Learner: Learning Cross Sources and Tasks for Universal Visual Representation. European Conference on Computer Vision (ECCV), 2022.

PDF Cite

Yuanhan Zhang, Zhenfei Yin, Jing Shao, Ziwei Liu (2022). Benchmarking Omni-Vision Representation through the Lens of Visual Realms. European Conference on Computer Vision (ECCV), 2022.

PDF Cite

Luya Wang, Feng Liang, Yangguang Li, Honggang Zhang, Wanli Ouyang, Jing Shao (2022). RePre: Improving Self-Supervised Vision Transformer with Reconstructive Pre-training. International Joint Conference on Artificial Intelligence (IJCAI), 2022.

PDF Cite DOI

Dong An, Zun Wang, Yangguang Li, Yi Wang, Yicong Hong, Yan Huang, Liang Wang, Jing Shao (2022). 1st Place Solutions for RxR-Habitat Vision-and-Language Navigation Competition (CVPR 2022). CoRR.

PDF Cite

Yubo Ma, Zehao Wang, Mukai Li, Yixin Cao, Meiqi Chen, Xinze Li, Wenqi Sun, Kunquan Deng, Kun Wang, Aixin Sun, Jing Shao (2022). MMEKG: Multi-modal Event Knowledge Graph towards Universal Representation across Modalities. Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo), 2022.

PDF Cite DOI

Yubo Ma, Zehao Wang, Yixin Cao, Mukai Li, Meiqi Chen, Kun Wang, Jing Shao (2022). Prompt for Extraction? PAIE: Prompting Argument Interaction for Event Argument Extraction. Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL Long Papers), 2022.

PDF Cite Code DOI

Meiqi Chen, Yixin Cao, Kunquan Deng, Mukai Li, Kun Wang, Jing Shao, Yan Zhang (2022). ERGO: Event Relational Graph Transformer for Document-level Event Causality Identification. CoRR.

PDF Cite

See all publications

Contact

amandajshao [at] gmail [dot] com