Home
The Science of Imagination Laboratory Imagination Engine SOILIE is a program designed to cognitively model 2D visual images that resemble human imaginings given the same input. The goal of the Science of Imagination Labratory is to understand how people generate visual scenes in their heads and to understand it well enough to replicate it with computer software. SOILIE is composed of multiple modules (The Oracle of Objects, Visuo, Renderer, Coherencer) and databases (The Esp Game, Peekaboom, Wordnet, LabelMe) that together take an initial query and generate a rendered image of a 2D scene depicting the initial query and several semantically related objects along with it. For example, the query "dog" would retrieve an image of a dog, but also several associated co-occurring images of objects such as "leash", "man", and "tree".
Used as a proxy for human perceptual memory, SOILIE's dataset is a collection of over fifty thousand images and more than ten thousand object labels derived from a combination of two online games created by Luis von Ahn: the ESP Game and Peekaboom. In the ESP Game, two randomly assigned players who have no direct communication between one another attempt to describe an image of a scene using the same words. When both partners describe the image using the same word, a point would be received and a new image is shown with the game lasting a total of three minutes. The data collected from the ESP Game allows for SOILIE's Oracle of Objects module to create a matrix of co-occurrence which results in semantically related label correlations of objects typically found within the same scene such as "car" and "road". If a query is not found in ESP or Peekaboom, another dataset, Wordnet can be used to find semantic similarity.
In Peekaboom, a similar strategy takes place between two randomly assigned players but instead focuses on the position of objects within the image of a scene. Player one, "the boomer", is given a scene as well as an associated label of an object found within the image. This is all hidden to player two, "the peeker", who sees a blank screen. Using clicks of their mouse, player one reveals portions of the image to player two who attempt to guess the label of the object being revealed by the first player. After a few clicks, the object in the image (e.g."tree") often becomes recognizable. When player two correctly guesses the label, points are awarded. Since the boomer player often reveals the object the label describes in the scene image first, the database of Peekaboom not only contains the labels the ESP database has, but also the pixel locations of images of those labels. Because of this, Peekaboom allows for a second module, Visuo, to implement a theory of quantitative spatial memory which can determine the relative positioning distance and angles between objects. For example, with the query for "car" as seen below, the sky is above the car, and the road below both. Visuo can also predict sizes of analogous objects using size modifying adjective-noun pairs (e.g., 'crow' and 'large crow'). This size modification in Visuo is even possible when estimating unseen concepts, such as a 'raven', from known concepts that offer similar semantic transfer such as a 'crow'.
The final scene is ultimately displayed through the implementation of the module, Renderer, which takes the output from the two previous modules, centers the initial query and processes three or four relating objects spatially around it appropriately. A fourth data set, LabelMe,is used by Renderer which allows for the displayed objects to be more accurately outlined with the inclusion of more distinguishing features than what Peekaboom alone would allow for.
Work continues to be done on SOILIE by The Science of Imagination Laboratory to refine and implement several cognitive capacities thereby creating a more robust model of human visual imagination consistent with empirical findings in cognitive science. See Coherencer for an example of how the incoherence created by the ambiguity of homonyms such as "mouse", computer versus mammal, was resolved in SOILIE. For more on SOILIE and related work, see Relevant Publications.
