Lesson 9: Vision Models: When AI Tries to Understand Images!
Lesson No: 9
Vision Models: When AI Tries to Understand Images!
By Tanzeel-ur-Rahman (CodeHubb & MindCrafts_AI) Farooka (Sargodha)
Assalamu Alaikum wa Rahmatullahi wa Barakatuh!
So far, we have mostly used AI for written tasks. Sometimes we had it write prompts, sometimes summarise documents, and sometimes arrange scattered information into a table.
But our world is not made of words alone. It also includes pictures, sketches, charts, screenshots and many kinds of visual information. If we want AI to help us make sense of these things, we need to get to know one of its important abilities.
This ability is called Vision (the ability to understand visual information).
Models that can work with images or other kinds of information alongside text are generally called Multimodal AI. However, every model has different abilities, so the ability to understand images is not the same in every AI tool or model.
Let's use a practical example to see how this ability can be useful to us.
1. What Is a Vision Model?
Suppose you want to build a website, but at the moment all you have is a rough sketch drawn on paper.
The sketch has a header at the top, a large banner in the middle, and three boxes below it to display products.
Before, you would have had to explain this sketch to a designer, or write out its details yourself to tell an AI.
But if your AI tool can understand images, you can upload a photo of the sketch and ask it questions directly.
For example:
What sections does this sketch contain?
Is the layout of the website suitable?
How could this design be improved for a mobile screen?
Where should the most important button for the user go?
Which widgets will be needed to build this sketch in WordPress and Elementor?
The AI will try to answer based on the shapes, layout, visible text and other visual clues in the image.
Important: being able to understand images does not mean the AI will recognise everything correctly. A blurry photo, small text, a complicated sketch or unclear markings can make things difficult for it.
2. What Can We Do with Vision Models?
The ability to understand images is not limited to web design. It can also help with many different everyday tasks.
Web design and UI/UX
You can upload a screenshot of a website and get suggestions for improving its layout, colours, spacing, buttons and user experience.
Educational work
You can ask for an explanation of an educational diagram, chart or picture. For example, identifying the parts of a scientific diagram, or explaining the trend shown in a graph.
Documents and charts
It can help you understand a table or chart in an image, state its key points and interpret the information that is visible.
Graphic design
You can have a poster or design reviewed and get suggestions about its layout, how prominent the text is, and its visual balance.
Guidance from screenshots
You can give it a screenshot of a software screen and ask what options are visible, or what the next step might be for a particular task.
However, be careful when interpreting information that appears in an image. If a button, a piece of text or a symbol is not clear, the AI may misread it.
3. How Do We Write a Prompt to Go with an Image?
After uploading an image, simply writing "Look at this" is not always enough to get a useful answer.
For better results, tell the AI three things:
1. Role: In what capacity should it work?
For example: a web design consultant, a graphic designer or an educational assistant.
2. Task: What should it do with the image?
For example: identify the sections of a sketch, analyse a design or suggest improvements.
3. Conditions: How do you want the answer?
For example: short points, an analysis based only on what is visible, or a clear marking of any unclear parts.
This way, the AI knows what to take from the image and how to present its answer.
4. Practical Exercise: Show AI Your Hand-Drawn Sketch
Now let's begin the practical part of this lesson.
Step 1: Draw a sketch
Take a sheet of paper and a pen. Draw a rough sketch of a simple website on it.
Include these parts in the sketch:
A header at the very top, with boxes for a logo and a menu.
A large banner below it.
Three boxes below the banner to display products.
A footer at the end, with space for contact information.
Write the name of each part inside its box as well, so that you can tell later whether the AI identified it correctly or not.
Step 2: Upload the image
Take a clear photo of the sketch with your mobile phone. Try to make sure all the boxes are clearly visible, the lighting is good and the photo is not tilted.
Now upload the image to an AI tool that is able to understand images.
Step 3: Use this prompt
> Role: You are an experienced web design and UI/UX consultant.
Task: Analyse the hand-drawn website sketch in the attached image.
First part: Identify all the sections visible in the sketch and say what purpose each one seems to serve.
Second part: Give three suggestions for improvement concerning the layout of the design, the user experience and how it could be arranged on a mobile screen.
Conditions:
1. Base your analysis only on the information visible in the image.
2. If the purpose of any section is unclear, ask for clarification instead of guessing.
3. Try not to overlook any section in the image.
4. Give your answer in short, clear points.
5. Where your advice cannot be proven directly from the image, present it as a suggestion, not as established fact.
Step 4: Check the answer
Put the AI's answer next to your original sketch and compare them.
Did it recognise the header correctly? Did it tell the banner apart from the product boxes? Did it point out the footer? Did it mention any section that is not in the image at all?
This will show you how accurately the AI understood the image.
5. Get Guidance on Building the Same Sketch in WordPress
Now ask the next question in the same conversation:
> "Tell me the basic widgets and layout needed to build this sketch in WordPress and Elementor. Write the purpose of each widget next to it. If any section might need an extra plugin or Elementor Pro, explain that."
The AI will probably suggest suitable widgets for the header, image, button, container and product sections.
But remember that there is no single widget that is required for every design. Some things can be done in basic Elementor, while others may need Elementor Pro, WooCommerce or an additional plugin.
So check the suggestions you receive against your own WordPress setup and the features available to you.
There is also an important distinction to understand here: AI understanding a sketch and AI building a complete, working website from that sketch are two different jobs. A Vision Model can help with planning the design, but the actual build may need further instructions, tools and testing.
Today's Takeaway and Task
Today's task is to draw a simple sketch by hand of a website, a mobile app or some other interface, and show it to a Vision Model.
Then do these three things:
1. Ask the AI to identify the sections in the sketch.
2. Compare its suggestions with the original sketch to see what it understood correctly and where it went wrong.
3. Ask it for the basic widgets and layout needed to build the sketch in WordPress and Elementor.
Write this in your notebook:
"Vision Models are useful helpers for analysing images and sketches, but they can make mistakes in interpreting them. So before making any important decision, it is essential to verify the visual information and the suggestions."
Tell us in the comments: Did the AI understand your hand-drawn sketch correctly? And did any of its suggestions give you a new idea for improving your design?
Tanzeel-ur-Rahman
#Lesson9 #VisionModels #MultimodalAI #ImageUnderstanding #UIUXPlanning #WordPress #Elementor #MindCraftsAI #CodeHubb
Join the WhatsApp channel for all previous lessons, and check the comments for Lesson 8.