How an Algorithm Draws Unexpected Connections
Anew artificial intelligence algorithm called MosAIc draws unexpected connections between seemingly disparate works of art: for instance, a piece of Persian glassware that resembles a naturalist’s 18th-century illustration of a blue crane, and an 18th-century Dutch garment that pairs well with a woodwork doll of a mother and child.
Researchers from Computer Science and Artificial Intelligence Laboratory (CSAIL) collaborated with to create the tool, which draws inspiration from a recent exhibition at Amsterdam’s Rijksmuseum, according to a statement. The show, titled “Rembrandt and Velázquez,” featured 60 works by Dutch and Spanish Old Masters (including the two who lent the exhibition its name), juxtaposed to highlight similarities in the artists’ lighting, colors and symbols.
MosAIc focuses specifically on objects housed in the collections of New York’s Metropolitan Museum of Art and the Rijksmuseum. Using machine learning and an image-retrieval system, the A.I. can spot connections between works from different cultures, artists and mediums.
“We have used this approach to find incredibly interesting and thought-provoking pairs of images,” Mark Hamilton, who studies artificial intelligence.
The software may help museums curate future exhibitions, but it certainly won’t replace experts: As Hamilton adds, “Art and curators can provide a much deeper understanding and context of these matches.”
One pair of paintings highlighted in “Rembrandt and Velázquez” provided significant inspiration for the project. Spaniard Francisco de Zurbarán’s The Martyrdom of Saint Serapion shows the titular saint with his arms raised, while Dutch artist Jan Asselijn’s The Threatened Swan depicts a bird widely interpreted as a representation of political martyr Johan de Witt. The paintings both center on sacrificial figures clad in white—a symbol of innocence—and standing in similar poses.
“These two artists did not have a correspondence or meet each other during their lives, yet their paintings hinted at a rich, latent structure that underlies both of their works,” says Hamilton in the statement.
To identify relevant couplings users introduce an image like that of the blue-and-white Double Face Banyan, or men’s robe. Then, they need only ask the software to find the musical instrument that most closely resembles the garment. An 18th-century Delft violin made of white earthenware and covered with cobalt blue illustrations.
The most similar piece of ceramic artwork, meanwhile, is a Chinese figurine. As Daphne Leprince-Ringuet points out for ZDNet, this connection can be linked to “the flow of porcelain and iconography” between China and the Netherlands during the 16th through 20th centuries. A paper describing how the software connects paintings is available as a preprint, which means it has not yet been peer reviewed, and a web version of the tool is currently in beta testing.
MosAIc is built around a “k-nearest neighbors” (KNN) algorithm which is typically used to group similar items for tasks like recommending products online, per ZDNet. Usually, KNN algorithms require programs to conduct a time-consuming search through an entire database or ask users to choose filters for the data they’d like to explore raising the possibility of inadvertently excluding results of interest. But the team behind MosAIc built the A.I. to apply filters as it searches through the trove of artwork instead of applying filters before it begins. The researchers describe this approach a “conditional KNN tree,” in which the search starts at the trunk and follows promising paths while pruning fruitless branches.