Quantifying Contextual Aspects of Inter-annotator Agreement in Intertextuality Research - HAL-SHS - Sciences de l'Homme et de la Société Accéder directement au contenu
Communication Dans Un Congrès Année : 2021

Quantifying Contextual Aspects of Inter-annotator Agreement in Intertextuality Research

Enrique Manjavacas
  • Fonction : Auteur
  • PersonId : 1128496
Mike Kestemont
  • Fonction : Auteur
  • PersonId : 1132227

Résumé

We report on an inter-annotator agreement experiment involving instances of text reuse focusing on the well-known case of biblical intertextuality in medieval literature. We target the application use case of literary scholars whose aim is to document instances of biblical references in the 'apparatus fontium' of a prospective digital edition. We develop a Bayesian implementation of Cohen's  for multiple annotators that allows us to assess the influence of various contextual effects on the inter-annotator agreement, producing both more robust estimates of the agreement indices as well as insights into the annotation process that leads to the estimated indices. As a result, we are able to produce a novel and nuanced estimation of inter-annotator agreement in the context of intertextuality, exploring the challenges that arise from manually annotating a dataset of biblical references in the writings of Bernard of Clairvaux. Among others, our method is able to unveil the fact that the obtained agreement depends heavily on the biblical source book of the proposed reference, as well as the underlying algorithm used to retrieve the candidate match. Finally, a discussion of the hurdles encountered by annotators supplements the results of the statistical analysis, contributing a qualitative insight into the difficulties involved in the identification of literary text reuse.

Domaines

Religions
Fichier principal
Vignette du fichier
2021.latechclfl-1.4.pdf (582.02 Ko) Télécharger le fichier
Origine : Fichiers éditeurs autorisés sur une archive ouverte

Dates et versions

halshs-03636967 , version 1 (11-04-2022)

Identifiants

Citer

Enrique Manjavacas, Laurence Mellerin, Mike Kestemont. Quantifying Contextual Aspects of Inter-annotator Agreement in Intertextuality Research. LaTeCH-CLfL 2021, 2021, Punta Cana, Dominican Republic. ⟨10.18653/v1/2021.latechclfl-1.4⟩. ⟨halshs-03636967⟩
22 Consultations
39 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More