Visual ChatGPT, a novel model from Microsoft, combines ChatGPT with visual foundation models (VFMs) like Transformers, ControlNet, and Stable Diffusion. According to media accounts, the model broadens ChatGPT’s powers and enables engagement that goes beyond language.
ChatGPT has gained interdisciplinary interest for its exceptional conversational competence and reasoning abilities across various fields, making it a top choice for a language interface.
However, its linguistic training prevents it from processing or producing images from the visual environment. Meanwhile, visual foundation models like Visual Transformers or Stable Diffusion excel at tasks with one-round fixed inputs and outputs, showing remarkable visual comprehension and generating capabilities. Combining these two models can lead to a new model, such as Visual ChatGPT, with the ability to process and generate visual inputs beyond language.
Microsoft researchers have created a system known as Visual ChatGPT that includes multiple visual foundation models and allows users to communicate with ChatGPT through graphical user interfaces. The system’s capabilities include:
Visual ChatGPT can transmit and receive not only text but also images.
Visual ChatGPT is capable of handling complex visual queries or editing instructions that require the cooperation of multiple AI models across several phases.
The researchers created a set of prompts that integrate visual model information into ChatGPT to accommodate models with multiple inputs/outputs and those that require visual feedback. Through testing, they have found that Visual ChatGPT enables the exploration of ChatGPT’s visual capabilities using visual foundation models.
However, the researchers identified areas of concern in their work, including the inconsistent generation results due to the failure of visual foundation models (VFMs) and the variability of the prompts. They concluded that a self-correcting module is necessary to ensure execution outputs align with human intentions and to make necessary adjustments. However, incorporating such a module may increase the model’s inference time due to constant course correction. The team plans to investigate this issue further in a future study.
