Challenges in web corpus construction for low-resource languages in a post-BootCaT world - HAL-SHS - Sciences de l'Homme et de la Société
Communication Dans Un Congrès Année : 2013

Challenges in web corpus construction for low-resource languages in a post-BootCaT world

Résumé

The state of the art tools of the "web as corpus" framework rely heavily on URLs obtained from search engines. Recently, this querying process became very slow or impossible to perform on a low budget. Trying to find reliable data sources for Indonesian, we perform a case study of different kinds of URL sources and crawling strategies. First, we classify URLs extracted from the Open Directory Project and Wikipedia for Indonesian, Malay, Danish and Swedish in order to enable comparisons. Then we perform web crawls focusing on Indonesian and using the mentioned sources as start URLs. Our scouting approach using open-source software leads to a URL database with metadata which can be used to replace or at least to complement the BootCaT approach.
Fichier principal
Vignette du fichier
ABarbaresi_LTC13_Challenges-LRL_paper.pdf (77.85 Ko) Télécharger le fichier
ABarbaresi_LTC13_Challenges-LRL_slides.pdf (263.86 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Format Autre

Dates et versions

halshs-00919410 , version 1 (16-12-2013)
halshs-00919410 , version 2 (05-08-2014)

Identifiants

  • HAL Id : halshs-00919410 , version 1

Citer

Adrien Barbaresi. Challenges in web corpus construction for low-resource languages in a post-BootCaT world. 6th Language & Technology Conference, Less Resourced Languages special track, Dec 2013, Poznan, Poland. pp.69-73. ⟨halshs-00919410v1⟩
400 Consultations
1122 Téléchargements

Partager

More