3-D Instance Segmentation of MVS Buildings

Abstract

We present a novel 3D instance segmentation framework for Multi-View Stereo\n(MVS) buildings in urban scenes. Unlike existing works focusing on semantic\nsegmentation of urban scenes, the emphasis of this work lies in detecting and\nsegmenting 3D building instances even if they are attached and embedded in a\nlarge and imprecise 3D surface model. Multi-view RGB images are first enhanced\nto RGBH images by adding a heightmap and are segmented to obtain all roof\ninstances using a fine-tuned 2D instance segmentation neural network. Instance\nmasks from different multi-view images are then clustered into global masks.\nOur mask clustering accounts for spatial occlusion and overlapping, which can\neliminate segmentation ambiguities among multi-view images. Based on these\nglobal masks, 3D roof instances are segmented out by mask back-projections and\nextended to the entire building instances through a Markov random field\noptimization. A new dataset that contains instance-level annotation for both 3D\nurban scenes (roofs and buildings) and drone images (roofs) is provided. To the\nbest of our knowledge, it is the first outdoor dataset dedicated to 3D instance\nsegmentation with much more annotations of attached 3D buildings than existing\ndatasets. Quantitative evaluations and ablation studies have shown the\neffectiveness of all major steps and the advantages of our multi-view framework\nover the orthophoto-based method.\n

Keywords

Affiliated Institutions

• Delft University of Technology NL
• Zhejiang University of Technology CN

Related Publications