Preprocess BRAT to BIO

Table of Contents

• Usage
• Requirements

This python file contains the necessary code to convert annotated data in Brat format to BIO format.

The files must be in a specific structure. The directory dataset must contain at least one set (train, dev, test). Each set directory must contain two main directories:

1. First one with the text files (raw data)

2. Second one with the file with the annotation. Only one file with all the annotations. The structure of this file is as follows (each column is sep by a tab, so the pandas.DataFrame is readed as a tsv):

        filename    mark  label    off0    off1    span
        f1_example  T1	  HUMAN    112     118	  hombre
        f1_example  T2	  HUMAN    1025    1033	  paciente
        f2_example  T1	  HUMAN    112     118	  mujer
        f3_example  T1	  SPECIE   1025    1033	  coronavirus
   

Example of dataset organization for a dataset called 'cantemist', Each set has a directory 'text-files' with the raw data and a directory 'cantemist-ner' where the annotated data is located.

/path/to/dataset
└── cantemist
    ├── train-set
    │   ├── cantemist-ner
    │   │   └── ner_annotations.tsv
    │   └── text-files
    │       ├── text-file_1.txt
    │       ├── text-file_2.txt
    │       └── ...
    │
    ├── dev-set
    │   ├── cantemist-ner
    │   │   └── ner_annotations.tsv
    │   └── text-files
    │       ├── text-file_1.txt
    │       ├── text-file_2.txt
    │       └── ...
    │
    └── test-set
        ├── cantemist-ner
        │   └── ner_annotations.tsv
        └── text-files
            ├── text-file_1.txt
            ├── text-file_2.txt
            └── ...

Requirements

The requirements are already in the requirements.txt file:

   pandas==2.1.2
   pysbd==0.3.4
   tqdm==4.66.1

Usage

python preprocess.py <path-to-dataset> -tr <train-set-dir-name> \
-de <dev-set-dir-name> -te <test-set-dir-name> -txt <text-files-dir-name> \
-an <annotations-dir-name> -hd <list-of-columns> -sn <output-dir> \
-n <number-of-process>

Example from the dataset tree in above:

• Dataset: "/home/carlos/datasets/cantemist"
  • Train: "train-set"
  • Evaluation: "dev-set"
  • Test: "test-set"
• Text files: "text-files"
• Annotations: "cantemist-ner"
• Header: "filename,mark,label,off0,off1,span"
• Output file: "processed_data",
• Number of processes: "8"

Example command:

python preprocess.py /home/carlos/datasets/cantemist -tr train-set \
-de dev-set -te test-set -txt text-files -an cantemist-ner \
-hd filename,mark,label,off0,off1,span -sn processed_data -n 8