Have you ever wished you could chat with an AI that could understand your words and create images based on them? Imagine conversing with a virtual friend who can draw anything you ask for or edit existing images according to your instructions. Sounds like science fiction, right?

Well, not anymore. Thanks to a new system called Visual ChatGPT, developed by researchers at Microsoft, you can now interact with a chatbot that can do all that and more. Visual ChatGPT is a multimodal chatbot combining ChatGPT, a state-of-the-art language model that can generate natural and engaging conversations, with Visual Foundation Models (VFMs), a collection of powerful image models that perform various visual tasks.

How does Visual ChatGPT work?

Visual ChatGPT connects ChatGPT with different VFMs through carefully designed prompts that inject the visual information into the language model. For example, if you want to ask the chatbot to draw a cat, you can type something like “Can you draw me a cat?” The chatbot will then use a prompt like “[IMAGE: draw a cat]” to trigger the VFM that can generate images from text, such as Stable Diffusion or ControlNet. The chatbot will then send you an image of a cat and a response like “Sure, here is a cat I drew for you.”

Similarly, you may want to edit an existing image. In that case, you can upload it to the chatbot and type something like “Can you make this image brighter?” The chatbot will then use a prompt like “[IMAGE: edit this image: make it brighter]” to trigger the VFM that can edit images from text, such as InstructPix2Pix or GLIGEN. The chatbot will then send you an edited image with a response like “OK, I made this image brighter for you.”

You can also ask the chatbot complex visual questions or instructions that require multiple steps or models. For example, you can type something like, “Can you draw me a dog wearing glasses and a hat?” The chatbot will then use prompts like “[IMAGE: draw a dog]” and “[IMAGE: edit this image: add glasses and a hat]” to trigger different VFMs and combine their outputs. The chatbot will then send you an image of a dog wearing glasses and a hat and a response like “Here is a dog wearing glasses and a hat I drew for you.”

Similar Article: GPTZero vs Turnitin: Which One is More Effective for Detecting Plagiarism?

You can also provide feedback and ask for corrected results. For example, suppose you are unsatisfied with the image the chatbot generated or edited. In that case, you can type something like “Can you make it more realistic?” or “Can you change the colour of the hat?” The chatbot will then use prompts like “[IMAGE: edit this image: make it more realistic]” or “[IMAGE: edit this image: change the colour of the hat]” to trigger the appropriate VFMs and update their outputs. The chatbot will then send you back an improved image along with a response like “OK, I made it more realistic for you.” or “OK, I changed the colour of the hat for you.”

Why is Visual ChatGPT important?

Visual ChatGPT is an important breakthrough in AI research because it demonstrates that language models can be extended beyond text and integrated with visual models to enable multimodal communication and collaboration. It opens up new possibilities for human-AI interaction and applications across various domains.

For example, Visual ChatGPT can be used as an educational tool to help students learn about different topics through visual examples and explanations. It can also be an entertainment tool to create fun and engaging conversations with images. Moreover, it can be a creative tool to help artists and designers generate or edit images based on their ideas or preferences.

Visual ChatGPT is also important because language models can leverage existing visual models without additional training or data. This reduces the cost and complexity of developing multimodal systems and makes them more accessible and scalable.

How to use Visual ChatGPT?

Visual ChatGPT is publicly available at https://stablediffusionweb.com/Visual-ChatGPT. You can access it from any device with an internet connection and start chatting with the AI immediately. You can also choose from different VFMs depending on your needs and preferences.

To use Visual ChatGPT, enter your message in the text box. You can also upload an image by clicking on the camera icon. The chatbot will reply with an image or text based on your input. You can continue the conversation by typing more messages or uploading more images.

You can also switch between different VFMs by clicking on the screen’s drop-down menu at the top right corner. You can choose from 22 different VFMs that cover various visual tasks such as generation, editing, captioning, classification, segmentation, detection, recognition, etc.

You can also adjust some settings, such as image size, quality, style, etc., by clicking on the screen’s gear icon at the top right corner. You can also clear the chat history by clicking on the trash icon at the top right corner of the screen.

Conclusion

Visual ChatGPT is an innovative system that combines ChatGPT with VFMs to enable multimodal chatting with images. It allows users to interact with an AI that can understand their words and create images based on them. It also allows users to ask complex visual questions or instructions that require multiple steps or models. It also allows users to provide feedback and ask for corrected results.

Visual ChatGPT is an important breakthrough in AI research because it demonstrates that language models can be extended beyond text and integrated with visual models to enable multimodal communication and collaboration. It also shows that language models can leverage existing visual models without additional training or data.

Visual ChatGPT is publicly available at https://stablediffusionweb.com/Visual-ChatGPT. You can access it from any device with an internet connection and start chatting with the AI immediately. You can also choose from different VFMs depending on your needs and preferences.

Suppose you are interested in learning more about Visual ChatGPT. In that case, you can read the original paper by Chenfei Wu et al. five or visit their website at https://visualchatgpt.github.io/.

You might be interested in: Netus AI: AI for social media