Corpus Analysis and Annotation for Helpful Sentences in Product Reviews

Hana Almagrabi, Areej Malibari, John McNaught

Research output: Contribution to journalArticlepeer-review


For the last two decades, various studies on determining the quality of online product reviews have been concerned with the classification of complete documents into helpful or unhelpful classes using supervised learning methods. As in any supervised machine-learning task, a manually annotated corpus is required to train a model. Corpora annotated for helpful product reviews are an important resource for the understanding of what makes online product reviews helpful and of how to rank them according to their quality. However, most corpora for helpfulness are annotated on the document level: the full review. Little attention has been paid to carrying out a deeper analysis of helpful comments in reviews. In this article, a new annotation scheme is proposed to identify helpful sentences from each product review in the dataset. The annotation scheme, guidelines and the inter-annotator agreement scores are presented and discussed. A high level of inter-annotator agreement is obtained, indicating that the annotated corpus is suitable to support subsequent research.
Original languageEnglish
Pages (from-to)76-87
Number of pages12
JournalComputer and Information Science
Issue number2
Early online date30 Apr 2018
Publication statusPublished - May 2018


  • content analysis
  • corpus annotation
  • sentiment analysis
  • opinion mining


Dive into the research topics of 'Corpus Analysis and Annotation for Helpful Sentences in Product Reviews'. Together they form a unique fingerprint.

Cite this