Corpora: Difference between revisions

Content added Content deleted

Inline

Revision as of 18:09, 9 April 2023

type: varied
quality: varied (i’ve heard jan Kipo tampered with his corpus to conform to his idea of toki pona grammar)
dialect: mostly old/pu
size: 4675k but a bunch of it is in english or duplicated, gotta count it properly eventually
license: none, some under various cc licenses (247k, mostly by jan Kipo)
preprocessing: probably a lot
where: https://github.com/davidar/nltk-tp/tree/master/Corpus

note that this corpus contains the entirety of:

type: single sentences, some with translations
quality: varied (can be filtered with Users' sentence reviews)
dialect: mostly pu
size: 1896k (as of 2021-06-12)
license: cc-by 2.5 fr
preprocessing: extract last column of tsv file, replace Ton and Mali/Mewi/Mawi with random names
where: https://tatoeba.org/en/downloads

audio recordings sorta available (under cc-by-sa 4.0) but you’ll have to scrape them yourself

type: articles, some definitions, a bit of discussion in talk pages
quality: varied
dialect: varied
size: 1084k raw, probably half? when stripped (as of 2021-06-13)
license: cc-by-sa 3.0 (some 4.0)
preprocessing: strip templates and markup
where:
- https://archive.org/download/wiki-wikipesija.org
- https://wikipesija.org/wiki/ilo:Export (for chosen categories)

type: screenplay
quality: good, though grammar funnies are abound
dialect: ma ponian
size: 54.2k
1. toki ala o e toki Inli - 1.5k
2. nanpa pi kipisi ala - 0.5k (not even really toki pona, not worth it)
3. tu kuntu - 35.6k
4. mijomi telo - 10.somethingk (can’t copy first page for some reason)
5. wi lon - 7.1k
license: none
preprocessing: extract 3 and 4 from pdf, uppercase names in 4, probably move names from header to somewhere else
where:

type: essays
quality: good
dialect: ma ponian
size: ~400k as of 2022-06-25 (assuming all are similar to lipu tenpo nanpa pan (~25k per issue))
license: cc-by-sa 4.0
preprocessing: surprisingly easy thanks to pdftotext, some manual checking still required though (e.g. nanpa pan has a sitelen pona thing)
where: https://liputenpo.org/ (also: https://wikisource.org/wiki/Category:Lipu_tenpo)

type: prose, dialogue
quality: varied
dialect: varied
size: ~650k when raw as of 2021-06-29, far less once the empty parts are filtered out
license: cc-by 4.0
where: http://antetokipona.infinityfreeapp.com/csv/

includes pepper & carrot

type: spoken dialogue
quality: as good as spontaneous speech can be
dialect: varied, mostly ma ponian
size: ~97k as of 2022-01-30 (assuming 50% formatting overhead)
license: none that i know of
preprocessing: convert from .srt to .txt (remove the first 2 lines after each blank line), remove speaker labels, deal with square brackets
where: https://drive.google.com/drive/folders/12H2xY06Wtwh4V6zoPOhDnE_R4aZ5sVJp
clean version: https://cdn.discordapp.com/attachments/773267909903908894/1094680735069257798/file

the english is a literal translation of the toki pona, with all the awkwardness that implies; this may or may not be desired

type: sentences
quality: varied
dialect: varied
size: 230k as of 2022-06-27
license: cc0
where: https://github.com/common-voice/common-voice/blob/main/server/data/tok/sentence-collector.txt

note that this corpus largely consists of these already listed sources:

audio recordings (of varying quality) available here: https://commonvoice.mozilla.org/en/datasets

@@ Line 113: / Line 113: @@
 * preprocessing: convert from .srt to .txt (remove the first 2 lines after each blank line), remove speaker labels, deal with square brackets
 * where: https://drive.google.com/drive/folders/12H2xY06Wtwh4V6zoPOhDnE_R4aZ5sVJp
+* clean version: https://cdn.discordapp.com/attachments/773267909903908894/1094680735069257798/file
 == [todo] library ==