Pretrained Language Models v. Court Ruling Predictions
Olivia Vaudaux
(1, 2)
,
Caroline Bazzoli
(1)
,
Maximin Coavoux
(3)
,
Géraldine Vial
(4)
,
Étienne Vergès
(4)
1
TIMC -
Translational Innovation in Medicine and Complexity / Recherche Translationnelle et Innovation en Médecine et Complexité - UMR 5525
2 LIG - Laboratoire d'Informatique de Grenoble
3 GETALP - Groupe d’Étude en Traduction Automatique/Traitement Automatisé des Langues et de la Parole
4 CRJ - Centre de Recherches Juridiques
2 LIG - Laboratoire d'Informatique de Grenoble
3 GETALP - Groupe d’Étude en Traduction Automatique/Traitement Automatisé des Langues et de la Parole
4 CRJ - Centre de Recherches Juridiques
Caroline Bazzoli
- Fonction : Auteur
- PersonId : 12671
- IdHAL : carolinebazzoli
- ORCID : 0000-0002-5785-3827
- IdRef : 144651564
Maximin Coavoux
- Fonction : Auteur
- PersonId : 13643
- IdHAL : maximin-coavoux
- ORCID : 0000-0003-4089-4558
Géraldine Vial
- Fonction : Auteur
- PersonId : 172878
- IdHAL : geraldine-vial
- IdRef : 11811090X
Étienne Vergès
- Fonction : Auteur
- PersonId : 173338
- IdHAL : etienne-verges
- IdRef : 081416946
Résumé
NLP systems are increasingly used in the law domain, either by legal institutions or by the industry. As a result there is a pressing need to characterize their strengths and weaknesses and understand their inner workings. This article presents a case study on the task of judicial decision prediction, on a small dataset from French Courts of Appeal. Specifically, our dataset of around 1000 decisions is about the habitual place of residency of children from divorced parents. The task consists in predicting, from the facts and reasons of the documents, whether the court rules that children should live with their mother or their father. Instead of feeding the whole document to a classifier, we carefully construct the dataset to make sure that the input to the classifier does not contain any 'spoilers' (it is often the case in court rulings that information all along the document mentions the final decision). Our results are mostly negative: even classifiers based on French pretrained language models (Flaubert, JuriBERT) do not classify the decisions with a reasonable accuracy. However, they can extract the decision when it is part of the input. With regards to these results, we argue that there is a strong caveat when constructing legal NLP datasets automatically.
Format du dépôt | Fichier |
---|---|
Type de dépôt | Communication dans un congrès |
Résumé |
en
NLP systems are increasingly used in the law domain, either by legal institutions or by the industry. As a result there is a pressing need to characterize their strengths and weaknesses and understand their inner workings. This article presents a case study on the task of judicial decision prediction, on a small dataset from French Courts of Appeal. Specifically, our dataset of around 1000 decisions is about the habitual place of residency of children from divorced parents. The task consists in predicting, from the facts and reasons of the documents, whether the court rules that children should live with their mother or their father. Instead of feeding the whole document to a classifier, we carefully construct the dataset to make sure that the input to the classifier does not contain any 'spoilers' (it is often the case in court rulings that information all along the document mentions the final decision). Our results are mostly negative: even classifiers based on French pretrained language models (Flaubert, JuriBERT) do not classify the decisions with a reasonable accuracy. However, they can extract the decision when it is part of the input. With regards to these results, we argue that there is a strong caveat when constructing legal NLP datasets automatically.
|
Titre |
en
Pretrained Language Models v. Court Ruling Predictions
|
Sous-Titre |
en
A Case Study on a Small Dataset of French Court of Appeal Rulings
|
Auteur(s) |
Olivia Vaudaux
1, 2
, Caroline Bazzoli
1
, Maximin Coavoux
3
, Géraldine Vial
4
, Étienne Vergès
4
1
TIMC -
Translational Innovation in Medicine and Complexity / Recherche Translationnelle et Innovation en Médecine et Complexité - UMR 5525
( 1043049 )
- Domaine de la Merci, 38706 La Tronche, France
- France
2
LIG -
Laboratoire d'Informatique de Grenoble
( 1043301 )
- UMR 5217 - Laboratoire LIG - Bâtiment IMAG - 700 avenue Centrale - Domaine Universitaire de Saint-Martin-d’Hères
Adresse postale : CS 40700 - 38058 Grenoble cedex 9
Tél. : 04 57 42 14 00
- France
3
GETALP -
Groupe d’Étude en Traduction Automatique/Traitement Automatisé des Langues et de la Parole
( 1043313 )
- Laboratoire LIG - Bâtiment IMAG - 700 avenue Centrale, CS 40700 - 38058 Grenoble cedex 9
- France
4
CRJ -
Centre de Recherches Juridiques
( 1043153 )
- Faculté de droit - Aile B - 1133, rue des Résidences - 38400 Saint-Martin-d'Hères
- France
|
URL du congrès ou éditeur |
https://aclanthology.org/2023.nllp-1.5/
|
Langue du document |
Anglais
|
Titre du congrès |
Natural Legal Language Processing Workshop 2023
|
Date début congrès |
2023-12-07
|
Ville |
Singapore
|
Pays |
Singapour
|
Actes |
Oui
|
Vulgarisation |
Non
|
Comité de lecture |
Oui
|
Invité |
Non
|
Audience |
Internationale
|
Titre de la collection |
Proceedings of the Natural Legal Language Processing Workshop 2023
|
Page/Identifiant |
38-43
|
Domaine(s) |
|
Projet(s) ANR |
|
Éditeur scientifique |
|
Éditeur commercial |
|
DOI | 10.18653/v1/2023.nllp-1.5 |
Origine :
Fichiers produits par l'(les) auteur(s)
Loading...