Distantly Supervised Named Entity Recognition with Category-Oriented Confidence Calibration

Named entity recognition plays an important role in extracting valuable information from digital libraries, which can help stakeholders to take full advantage of large quantities of documents to boost the development of scholarly knowledge discovery. Nevertheless, there aren’t many annotated NER datasets aiming at scientific literature except medical domain, restricting to utilize abundant of advanced deep learning models. As an alternative solution, distant supervision provides a feasible way to eliminate the need of human annotations by automatically generating annotated datasets based on external resources such as knowledge base, while introducing noise inevitably. In this work, we study the noisy-labeled named entity recognition under distant supervision setting. Considering that most NER systems based on confidence estimation deal with noisy labels ignoring the fact that model has different levels of confidence towards different categories, we propose a Category-oriented confidence calibration (Coca) strategy with an automatically confidence threshold calculation module. We integrate our method into a teacher-student self-training framework to improve the model performance. Our proposed approach achieves promising performance among advanced baseline models and can be easily integrated into other confidence based model frameworks (Our code is publicly available at: https://github.com/possible1402/BOND_Coca ).