Text Corpora and the Challenge of Newly Written Languages

Alice Millour (1) , Karën Fort (1, 2)

1 STIH - Sens, Texte, Informatique, Histoire
2 SEMAGRAMME - Semantic Analysis of Natural Language

Text corpora represent the foundation on which most natural language processing systems rely. However, for many languages, collecting or building a text corpus of a sufficient size still remains a complex issue, especially for corpora that are accessible and distributed under a clear license allowing modification (such as annotation) and further resharing. In this paper, we review the sources of text corpora usually called upon to fill the gap in low-resource contexts, and how crowdsourcing has been used to build linguistic resources. Then, we present our own experiments with crowdsourcing text corpora and an analysis of the obstacles we encountered. Although the results obtained in terms of participation are still unsatisfactory, we advocate that the effort towards a greater involvement of the speakers should be pursued, especially when the language of interest is newly written.

Mots clés: text corpora; dialectal variants; spelling; crowdsourcing

Domaines: Informatique et langage [cs.CL]

Alice Millour, Karën Fort. Text Corpora and the Challenge of Newly Written Languages. 1st Joint SLTU and CCURL Workshop (SLTU-CCURL 2020), May 2020, Marseille, France. ⟨hal-02611209⟩