Let's Look Out For Each Other: A Distributed Framework for Botcloud Detection

  • Twisha .

Student thesis: Phd


Cloud Computing (CC) has gained increasing attention from the industry for on-demand rapid provisioning of shared pool of resources and services (R&S). This set of R&S is configured as per the users’ requirement and are accessible through virtual machines (VMs). This infrastructure enables VMs to access user data, thus increasing the risk of losing it, particularly given the fact that VMs maybe much more vulnerable to theft or loss in comparison with conventional computing devices such as workstations. Therefore, more stringent security provision is needed in this environment. The initial literature study in the topic of VM security shows that VMs are susceptible to a number of security attacks such as distributed denial of service (DDoS), side channel, man-in-the middle attacks and more, with DDoS as the most common. These attacks can be performed by a malware-infected VM residing on the same system or by an outsider. The attacks performed from within the system are difficult to track. Moreover, the strength of these attacks increases drastically when a group of malicious VMs attack simultaneously. These are the VMs infected by the same malware (bot) known as botVMs and the attack is known as botCloud attack. The high impact of a botCloud attack has motivated us to investigate how to strengthen the security of VMs and minimize the effect of such attacks. To this end, the thesis makes the following novel contributions. The thesis proposes and evaluates a novel BotVM Detection (bVMD) framework to detect a set of botVMs in an effective and efficient manner. The novelty of the framework lies in that it uses a two-staged approach to botCloud detection: in Stage-1, a peer-VM mutual monitoring based, suspected botVM identification method is used to identify suspected malicious VMs (S-VMs) and, in Stage-2, a detailed examination of run-time state is carried out on each identified suspected botVM. This two-staged approach to botCloud detection reduces the number of VMs on which run-time state examinations are carried out, thus reducing overhead costs and making the detection more efficient. The peer-VM mutual monitoring based, suspected botVM identification method allows each VM being watched by any peer VM, potentially increasing the true positive detection rate, thus making the detection more effective. In addition, the run-time state examination of each S-VM is done by using an improved method to minimize the number of examiners, thus reducing the overhead costs. The bVMD framework consists of four types of components: (i) a VMWatcher, which is a VM monitoring component residing in each VM monitoring and recording the behaviour of each peer VM communicating with this VM, (ii) an S-VM Detector (S-VMD), an analysis component, collecting data from VMWatchers, analysing the collected data to identify any S-VMs, (iii) a Forensic VM (FVM), a mini-VM dispatched to any identified S-VM to analyse its run-time state and (iv) a BotVM Detector (BotVMD), the component that controls the FVMs and makes decision as which S-VM is confirmed as a botVM. Components (i) and (ii) work in Stage-1 and (iii) and (iv) in Stage-2. In proposing this framework, we have also studied different parameter value settings and feature extraction techniques to increase detection accuracy, while, at the same time, minimizing overhead costs. The bVMD framework is implemented and evaluated using the Omnet++ simulation tool. The evaluation results are compared against the most relevant work in the literature. The simulation study shows that bVMD outperforms the relevant protocol in terms of the true positive and false negative detection. These enhancements make bVMD more effective in detecting a group of botVM(s). In addition, the bVMD framework minimizes the associated overhead costs, thus improving the efficiency of the system.
Date of Award31 Dec 2022
Original languageEnglish
Awarding Institution
  • The University of Manchester
SupervisorNing Zhang (Supervisor) & Mustafa Mustafa (Supervisor)


  • Virtual Machine Introspection
  • VM Security
  • Trust based solution
  • Intrusion Detection System
  • Cloud computing
  • Botcloud attacks
  • cloud security
  • Botnet attacks

Cite this