Click here to download the project base paper of panoptic.
Abstract:
We present Advanced Deep Learning the All-Seeing (AS) project: a large-scale data and model for recognizing and understanding everything in the open world. Using a scalable data engine incorporating human feedback and efficient models in the loop, we create a new dataset (AS-1B) with over 1 billion regions annotated with semantic tags, question-answering pairs, and detailed captions. It covers a wide range of 3.5 million common and rare concepts in the real world and has 132.2 billion tokens that describe the concepts and their attributes. Leveraging this new dataset, we develop the All-Seeing model (ASM), a unified framework for panoptic visual recognition and understanding. The model is trained with open-ended language prompts and locations, which allows it to generalize to various vision and language tasks with remarkable zero-shot performance, including region-text retrieval, region recognition, captioning, and question-answering. We propose a new large-scale dataset (AS-1B) for open-world panoptic visual recognition and
understanding, using an economical semi-automatic data engine that combines the power of off-the-shelf vision/language models and human feedback.
We hope this project can serve as a foundation for vision-language artificial general intelligence research. Models and the dataset shall be released at https://github.com/OpenGVLab/All-Seeing, and the demo can be seen at https://huggingface.co/spaces/OpenGVLab/all-seeing.
