By Matthew Lincoln

Playing on George Kubler’s anthropology of visual style, The Shape of Time¹, this app finds a path of similar images through the Rijksmuseum collections, at times moving forwards in time, and at times moving backwards.

Sometimes the results are uncanny. Other times they are laughably misguided. But they each offer an alternative pathway into a large art collection bound together by more formal similarities than we usually chart in our textbooks and exhibitions.

It produces both animated GIFs that are tweeted under @MechaKubler. This companion site hosts pages for each tweet with links to all the artworks.

See pathways found so far.

Why?

Inspired by Google Cultural Institute’s “X Degrees of Separation”, I was curious to see if it was possible to recreate that app by hand using a smaller collection of works, and constraining the kinds of paths that would get drawn between two given artworks. Was it possible to make this path move only forward in time? Or only backwards? To only consider a certain set of objects by type or nationality? The idea had been gnawing at me for some time.

How?

To find a “visual path” between two artworks, I used the penultimate max pooling layer of the pre-trained VGG 16 convolutional neural network² to produce multidimensional embeddings for over 120,000 images of artworks in the Rijksmuseum collections. Artworks close to each other in this vector space tend to share visual features (although fine-tuning the network that produced these embeddings would go a long way to fostering similarities more familiar to art historians in this domain.³) By drawing an ideal path between two points in this space, we can find real points (i.e. artworks) close to this path, effectively producing a list of images evenly-spaced in visual similarity - at least, the similarity recognized by VGG 16. The R package pathway powers this search, and offers some functions for constraining the search to move in one direction - in our case, either forward or backwards through the “time” represented by the creation dates given to these objects by Rijksmuseum curators.

A representation of the visual fingerprints of ~120k images, the multidimensional space returned by VGG-18 here boiled down to just two principal components, with the start, end, and intermediate images highlighted.

Mechanical Kubler does not produce real causal chains of visual influence. What it does do is deliberately remove some of the magic from GCI’s “X Degrees of Separation” app, more closely binding it to the idea of historical process. Hopefully this can encourage us to revisit Kubler’s ideas about how to write histories of visual morphologies even in the absence of social context - ideas all the more important in an age where more images are being looked at by machines than by humans.

1. George Kubler, The Shape of Time: Remarks on the History of Things (New Haven: Yale University Press, 1962). ↩

2. Karen Simonyan and Andrew Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” ArXiv:1409.1556 [Cs], September 4, 2014, http://arxiv.org/abs/1409.1556. ↩

3. This avenue is already being explored by Benoit Seguin et al., “Visual Link Retrieval in a Database of Paintings,” in Computer Vision – ECCV 2016 Workshops, ed. Gang Hua and Hervé Jégou, vol. 9913 (Cham: Springer International Publishing, 2016), 753–67, https://doi.org/10.1007/978-3-319-46604-0_52. ↩