SEMANTIC 3D RECONSTRUCTION AND BENCHMARKING IN DYNAMIC ENVIRONMENTS

Abstract

Simultaneous Localisation and Mapping, or SLAM is a key component in many applications, such as autonomous robot and vehicle navigation, motion capture and augmented reality. With new sensors such as the Microsoft Kinect and the wide availability of GPUs, SLAM has evolved towards generating increasingly more meaningful representations of the environment: 3D reconstruction and semantic scene understanding are common tasks addressed by state-of-the-art systems. The core contribution of this thesis is a technique and an evaluation methodology to address the problem of semantic 3D reconstruction in dynamic environments, which aims to unify previous paradigms and simultaneously recover the semantic and geometric aspects of deforming objects and the static background. Firstly, we develop an evaluation method by extending the open-source SLAMBench framework to support new sensors, metrics and datasets. We then propose FullFusion, a framework which uses RGB-D sensors to recover the geometry and semantic information of dynamic scenes, and develop a baseline implementation using KinectFusion, DynamicFusion, and a novel segmentation module which uses geometry and semantic labels. Our method is the first to address the problem of semantic 3D reconstruction in dynamic environments. Additionally, our results show state-of-the-art performance in pose estimation, proving that semantic labels can be used to discard unreliable elements when estimating the pose of a moving sensor.

Date of Award	1 Aug 2020
Original language	English
Awarding Institution	The University of Manchester
Supervisor	Mikel Luján (Supervisor) & Gavin Brown (Supervisor)