LeSSS: Learned Shared Semantic Spaces for Relating Multi-Modal Representations of 3D Shapes

Simple item page
dc.contributor.authorHerzog, Roberten_US
dc.contributor.authorMewes, Danielen_US
dc.contributor.authorWand, Michaelen_US
dc.contributor.authorGuibas, Leonidasen_US
dc.contributor.authorSeidel, Hans-Peteren_US
dc.contributor.editorMirela Ben-Chen and Ligang Liuen_US
dc.date.accessioned2015-07-06T05:00:50Z
dc.date.available2015-07-06T05:00:50Z
dc.date.issued2015en_US
dc.description.abstractIn this paper, we propose a new method for structuring multi-modal representations of shapes according to semantic relations. We learn a metric that links semantically similar objects represented in different modalities. First, 3D-shapes are associated with textual labels by learning how textual attributes are related to the observed geometry. Correlations between similar labels are captured by simultaneously embedding labels and shape descriptors into a common latent space in which an inner product corresponds to similarity. The mapping is learned robustly by optimizing a rank-based loss function under a sparseness prior for the spectrum of the matrix of all classifiers. Second, we extend this framework towards relating multi-modal representations of the geometric objects. The key idea is that weak cues from shared human labels are sufficient to obtain a consistent embedding of related objects even though their representations are not directly comparable. We evaluate our method against common base-line approaches, investigate the influence of different geometric descriptors, and demonstrate a prototypical multi-modal browser that relates 3D-objects with text, photographs, and 2D line sketches.en_US
dc.description.number5en_US
dc.description.sectionheadersCorrespondence and Matchingen_US
dc.description.seriesinformationComputer Graphics Forumen_US
dc.description.volume34en_US
dc.identifier.doi10.1111/cgf.12703en_US
dc.identifier.pages141-151en_US
dc.identifier.urihttps://doi.org/10.1111/cgf.12703en_US
dc.publisherThe Eurographics Association and John Wiley & Sons Ltd.en_US
dc.titleLeSSS: Learned Shared Semantic Spaces for Relating Multi-Modal Representations of 3D Shapesen_US
Files
Collections
34-Issue 5
SGP15: Eurographics Symposium on Geometry Processing (CGF 34-5)