EduSenti: Education Review Sentiment in Albanian#

Pretraining and sentiment student to instructor review corpora and analysis in Albanian. This repository contains the code base to be used for the paper RoBERTa Low Resource Fine Tuning for Sentiment Analysis in Albanian. To reproduce the results, see the paper reproduction repository. If you use our model or API, please cite our paper.

Obtaining#

The library can be installed with pip from the pypi repository:

pip3 install zensols.edusenti

The models are downloaded on the first use of the command-line or API.

Usage#

Command line:

$ edusenti predict sq.txt
(+): <Per shkak të gjendjes së krijuar si pasojë e pandemisë edhe ne sikur [...]>
(-): <Fillimisht isha e shqetësuar se si do ti mbanim kuizet, si do të [...]>
(+): <Kjo gjendje ka vazhduar edhe në kohën e provimeve>
...

Use the csv action to write all predictions to a comma-delimited file (use edusent --help).

API#

>>> from zensols.edusenti import (
>>>     ApplicationFactory, Application, SentimentFeatureDocument
>>> )
>>> app: Application = ApplicationFactory.get_application()
>>> doc: SentimentFeatureDocument
>>> for doc in app.predict(['Kjo gjendje ka vazhduar edhe në kohën e provimeve']):
>>>     print(f'sentence: {doc.text}')
>>>     print(f'prediction: {doc.pred}')
>>>     print(f'prediction: {doc.softmax_logit}')

sentence: Kjo gjendje ka vazhduar edhe në kohën e provimeve
prediction: +
logits: {'+': 0.70292175, '-': 0.17432323, 'n': 0.12275504}

Models#

The models are downloaded the first time the API is used. To change the model (by default xlm-roberta-base is used) on the command-line, use --override esi_default.model_namel=xlm-roberta-large. You can also create a ~/.edusentirc file with the following:

[esi_default]
model_namel = xlm-roberta-large

Performance of the models on the test set when trained and validated are below.

| Model | F1 | Precision | Recall | |:——————–|—–:|———-:|——-:| | xlm-roberta-base | 78.1 | 80.7 | 79.7 | | xlm-roberta-large | 83.5 | 84.9 | 84.7 |

However, the distributed models were trained on the training and test sets combined. The validation metrics of those trained models are available on the command line with edusenti info.

Differences from the Paper Repository#

The paper reproduction repository has quite a few differences, mostly around reproducibility. However, this repository is designed to be a package used for research that applies the model. To reproduce the results of the paper, please refer to the reproduction repository. To use the best performing model (XLM-RoBERTa Large) from that paper, then use this repository.

The primary difference is this repo has significantly better performance in Albanian, which climbed from from F1 71.9 to 83.5 (see models). However, this repository has no English sentiment model since it was only used for comparing methods.

Changes include:

Python was upgraded from 3.9.9 to 3.11.6
PyTorch was upgraded from 1.12.1 to 2.1.1
HuggingFace transformers was upgraded from 4.19 to 4.35
zensols.deepnlp was upgraded from 1.8 to 1.13
The dataset was re-split and stratified.

Documentation#

See the full documentation. The API reference is also available.

Changelog#

An extensive changelog is available here.

Citation#

If you use this project in your research please use the following BibTeX entry:

@inproceedings{nuci-etal-2024-roberta-low,
    title = "{R}o{BERT}a Low Resource Fine Tuning for Sentiment Analysis in {A}lbanian",
    author = "Nuci, Krenare Pireva  and
      Landes, Paul  and
      Di Eugenio, Barbara",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italy",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1233",
    pages = "14146--14151"
}

License#

MIT License