For years, curators have attempted to use the latest technology to help modern audiences connect with centuries-old art — and the paintings of Dutch artist Rembrandt van Rijn are at the forefront. Some curators have even used applications, not unlike the Snapchat filters found on your smartphone, to make the naturalist masterpieces from the Dutch Golden Age seem even more lifelike.
“Our paintings come alive at night!” the official Instagram account for the Rijksmuseum in Amsterdam, which houses most of Rembrandt’s work, joked in one post showing this trick at play. Portraits from the 17th century often already looked viewers in the eye, but now, thanks to deepfake technologies, they blink and move as well. It’s uncanny, to say the least.
Though these deepfakes make for some enticing marketing campaigns, they aren’t exactly groundbreaking from a scientific standpoint. If we want to delve deeper into the relationship between art and artificial intelligence, we must turn to other, more elaborate research projects that used Rembrandt as their personal test subject.
Aggregating Rembrandt
In 2014, a team of Microsoft data scientists teamed up with AI and 3D printing experts to construct a computer program capable of creating a Rembrandt-like picture in the painter’s unparalleled style. Dubbed “The Next Rembrandt,” the project began when a convolutional neural network (a type of artificial neural network used in image recognition) was given access to a massive, 150GB database containing high-resolution graphics of Rembrandt’s work.
While sifting through the course load, the network learned to recognize combinations of shapes and colors for the real-life objects they represented. The program analyzed 67 common features in the paintings — including noses, hands and eyes — and then blended various images together to 3D print what researchers called an “original” Rembrandt.
Whether the portrait can truly be considered original has become a subject of debate. It was, after all, neither created from imagination nor drawn from life, but distilled from Rembrandt’s encyclopedic oeuvre. “How can a computer replicate the humanity of Rembrandt’s portrait of his lover Hendrickje Stoffels?” art critic Jonathan Jones objected in The Guardian. “It would have to go to bed with her first.”
While The Next Rembrandt wasn’t the milestone in creative AI its team was hoping for, the project did help us to see Rembrandt from a different angle. Detailed, 3D scans of the painter’s canvases revealed where he applied the most pressure to his brush. Skin typically had a smooth and subtle texture to it, while Rembrandt tended to apply rough, emotive strokes for clothing, especially later in his career.
Most importantly, perhaps, the two-year project helped lay the groundwork for another, more recent attempt to emulate Rembrandt’s genius using the power of AI. In 2019, the Rijksmuseum launched a multimillion-dollar restoration mission called Operation Night Watch. Earlier this year, conservators used computer technology to recreate sections of the eponymous painting that were lost centuries ago.
Turning a Copy Into an Original
During the 17th century, politicians in charge of Amsterdam’s town hall came into possession of Rembrandt’s The Night Watch. When it wouldn’t fit on the wall its new owners selected, they cut off all four edges of the artist’s most famous painting.
These cosmetic adjustments completely changed its overall effect. The absence of two figures on the far left of the painting pushed Captain Frans Banninck Cocq from the side to the center, upsetting Rembrandt’s composition. Also among the casualties was an elegant balustrade catching the shadow of the aforementioned captain, which would have given his otherwise static squadron a sense of forward momentum.
Wishing visitors could experience The Night Watch the way that Rembrandt had originally intended, the Rijksmuseum called upon its senior scientist, Robert Erdmann, to restore these long-lost sections using a neural network capable of doing what human hands cannot: transforming a painted copy of Rembrandt’s original masterpiece into something that resembled the real deal as closely as possible.
Whereas researchers from The Next Rembrandt could repurpose old paintings, Erdmann had to make do with an imitation of The Night Watch. Gerrit Lundens, a contemporary of Rembrandt, copied the original painting before its outer edges were removed. While Lundens faithfully reproduced every little detail, his artistic style was so different from Rembrandt’s that Erdmann couldn’t simply copy this version onto the original.
After adjusting the perspectives for both paintings to make sure they overlapped perfectly, Erdmann asked a neural network to translate random tiles from Lundens’ copy into Rembrandt’s style. He then graded each of the network’s attempts until its output could fool even an art critic. At this point, Erdmann asked the AI to repaint the outer edges of Lundens’ Night Watch, which he then pasted onto Rembrandt’s.
Different Methods of Machine Learning
While the AI used for The Next Rembrandt project improved itself by tracking similarities across multiple works, Erdmann’s network learned via comparing the works of two separate artists. By noticing what Rembrandt did differently from Lundens, the network deduced what made each painter’s style unique. It then applied this knowledge to the missing sections present in the copy.
By studying Rembrandt through the lens of a computer program, humans can pick up things that we might have missed with the naked eye. When “translating” Lundens’ copy, for example, the network replaced the copyist’s saturated palate with darker and duller colors. It also sharpened the contrast between lights and shadows. In the art world, this technique is known as “chiaroscuro,” and it was one of Rembrandt’s favorite tricks.
Practicing chiaroscuro is one thing, but understanding what it means is another. Does Erdmann’s AI grasp the technique’s aesthetic quality in the same way an artist like Rembrandt did? Probably not. A more likely explanation is that the software discovered that strengthening the contrasts between light and dark resulted in higher grades received from Erdmann, which is why it kept doing it.
Once the AI translated the missing sections of the copy into Rembrandt’s style, they were printed out on canvas and covered with a gloss similar to the original. Instead of stitching the work of man and machine together, the missing sections were attached to plates that were placed less than an inch in front of the original Night Watch, without actually touching it.
Erdmann said that while growing up, he hugely admired the polymath Leonardo da Vinci for his ability to make breakthroughs in multiple academic disciplines. When asked to join Operation Night Watch, the programmer saw a rare opportunity to do the same, by combining his knowledge of coding with his love for art.
