Raw material selection for object construction

Abstract

An important step in the construction of novel objects is the ability to recognise combinations of raw materials which are likely to be useful. We aim to exploit the intuition that the visual characteristics of candidate raw materials provide useful cues to their potential combinations. Toward this end, we present a Siamese neural network based model that is able to recognise unseen raw materials present in objects given a list of candidate material images. We demonstrate the utility and efficacy of our model within two domains. The first being a proof-of-concept within Minecraft where we predict the combinations of objects that will result in a target object. The second, more realistic domain, uses the ShapeNet 3D model dataset where we attempt to recover the materials present in a model. We empirically demonstrate that our model is able to learn from a subset of object material pairs and generalise to unseen objects, materials, texture packs. Under these conditions of high visual variation, we show that our model outperforms chance and baseline methods.