Exploring Generalisable Multi-Task Reinforcement Learning Agents using Task Similarity Metrics

Abstract

One of the many issues that has limited the usage of reinforcement learning (RL) in its application to real-world problems, such as robotics, has been its inability to train agents to both learn optimally and adapt to slight changes in the tasks interacted with---this can be described as the agents ability to \textit{generalise} to new experiences. Many of the current state-of-the-art RL methods focus on performing optimally in a single-task setting and therefore suffer from a lack of generalisability, thus limiting their usage in a real-world context. There has been efforts to tackle this using multi-task learning (MTL) approaches, however within RL this has generally focused on specific and limited task adaptations. This results in agents that are unable to generalise to multi-task environments involving problems that are diverse in their properties including rewards, goals, transition dynamics, etc. Therefore, this work focuses on the following aspects: (a) providing a model- and task-agnostic definition of generalisation in RL, which expands on the current definitions that focus on a single-task case to provide a metric that quantitatively judges an agents ability to deal with adaptations in a set of tasks, (b) analyse how the current state-of-the-art methods attempt to remedy the challenges presented by a MTL setting through the understanding of the similarity of tasks, and, (c) proposes a new method that leverages task similarity to form a curriculum that encourages an agent to generalise effectively in an environment containing tasks with diverse and adaptive characteristics. Across all of this work, we find the importance of task similarity with regard to investigating generalisability, forming adaptive curricula to encourage task relevance that results in greater agent generalisability whilst also understanding the limits of current methods to transfer knowledge across tasks.

Date of Award	1 Aug 2022
Original language	English
Awarding Institution	The University of Manchester
Supervisor	Xiaojun Zeng (Supervisor) & Ke Chen (Supervisor)