In the modern world of artificial intelligence, computers have become remarkably good at two separate things: seeing and speaking. On one side, vision systems can look at a photograph and describe what is happening with surprising accuracy. On the other, language models can write stories, answer questions, and hold conversations by processing vast amounts of text. Recently, scientists have begun to bridge these two abilities, creating systems that can understand an image and a sentence at the same time. These hybrid tools, known as vision-language models, can already perform tasks like answering questions about a picture or writing a caption for a scene. However, most of these systems are trained to look at a single image and describe it, or to look at a picture and a question and provide an answer. They are not yet trained to look at a before-and-after pair of images and figure out exactly what changed between them, especially when that change involves specific adjustments like brightness, contrast, or color balance.

This gap in understanding is the focus of a new study by researchers Vahid Azizi and Fatemeh Koochaki. They set out to see if a powerful, open-source vision-language model called MiniGPT-4 could be taught to perform a task they call "reverse designing." Imagine you have a photo of a sunset, and then you have a second version of that same photo where the sky has been made more orange and the water darker. A human editor might know exactly which sliders were moved to achieve that look. The researchers wanted to know if a computer could look at the original and the edited version, perhaps with a vague hint like "make it more dramatic," and then predict the specific technical steps and numbers used to create that edit. This is a complex challenge because the computer must simultaneously understand two different images and the relationship between them, a task that goes beyond simply describing what is in a single picture.

To test this, the researchers took the MiniGPT-4 model, which was already trained to understand images and text, and fine-tuned it using a specific dataset called I-MAD-Dense. This dataset contains about 22,000 examples, each consisting of a source image, an edited version of that image, and a high-level description of the creative idea behind the edit. Crucially, the dataset also includes the "ground truth," which is the exact list of operations and their numerical values that were applied to the original image to create the new one. The researchers converted these technical lists of operations into sentences so the language part of the model could read them. They then fed the model pairs of images along with optional text descriptions and asked it to predict the list of changes that turned the first image into the second.

The team ran several experiments to see how well the model could learn this skill. In their first attempt, they simply fine-tuned the existing model without changing its structure. The results showed that the model could learn the task, successfully predicting the types of changes about 72 percent of the time on average. However, the researchers noticed the model sometimes guessed the wrong number of changes or got the specific values slightly off. To improve this, they tried adding a mathematical penalty to the training process whenever the model's predicted numbers for the adjustments were far from the correct values. This small tweak helped the model get the numbers more accurate, reducing the average error in its predictions. They also tested whether giving the model a specific command in the text, such as "adjust the brightness," would help. The results confirmed that having this extra textual context significantly improved the model's ability to find the correct changes, especially when the instructions were clear.

The researchers also explored whether adding special markers to the text to separate the images from the words would help the model keep track of the different inputs. They tried inserting a specific token to signal where one image ended and the next began, but this actually made the model's performance worse. This finding suggests that the model's existing way of handling images and text was sufficient and that adding extra, artificial markers confused it rather than clarified it. Another experiment involved trying to teach the model to predict the number of changes more accurately by penalizing it for guessing too many or too few, but this approach also failed to improve the results and in some cases made them worse.

The most successful version of their system, which combined the fine-tuned model with a specific way of handling numerical errors, showed that these vision-language models are indeed capable of learning complex, multi-step relationships between images. The model performed best when it was given a textual description of the edit, highlighting the importance of language in guiding visual understanding. While the system is not yet perfect and still struggles with some complex scenarios, the study demonstrates that off-the-shelf models can be extended to handle sophisticated tasks like reverse engineering image edits. The researchers conclude that while their current results are promising, there is still room for improvement, such as using higher-resolution images to capture finer details or training the model on even more carefully curated human-generated data. This work opens a door for future tools that could help editors understand their own workflows or allow computers to learn from the history of how images are created.