Skip to content
/ ARTS Public

ARTS: Assessing Readability & Text Simplicity

License

Notifications You must be signed in to change notification settings

irgroup/ARTS

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ARTS (Assessing Readability & Text Simplicity)

Welcome to ARTS — a project that focuses on assessing text readability and simplicity using various models and statistical techniques.

This repository contains scripts and notebooks aimed to analyze text simplicity, generate annotations, develop simplicity regression models, and run simulations—all while ensuring ease of understanding.


Table of Contents


Project Overview

This work alleviates the problem of Assessing Readability & Text Simplicity. We present ARTS, a method for language-independent construction of datasets for simplicity assessment. We propose using pairwise comparisons of texts in conjunction with an Elo algorithm to produce a simplicity ranking and simplicity scores. Additionally, we provide a high-quality human-labeled and three GPT-labeled simplicity datasets. Our results show a high correlation between human and LLM-based labels, allowing for an effective and cost-efficient way to construct large synthetic datasets.


Features

  • Text Simplicity Scoring: Tools to leverage models, including regression techniques, for predicting the simplicity of text segments.
  • GPT Annotation: Automated annotation using GPT for ranking and scoring simplicity across different text samples.
  • Statistical Analysis: Scripts and notebooks for performing in-depth statistics on the textual data.
  • Simulation: Simulate various readability scenarios to test effectiveness.

Folder and File Structure

Here's an overview of the most important files and folders in this repository:

/workspace

Contains all the core resources:

  • data/: Placeholder folder where you place your input datasets.
  • Histories/: Records of past model performance histories.
  • pages/: Various notebook segments or pages with content to analyze and review.
  • Scores/: Folder to store score outputs from the assessment scripts.
  • 5.5_Simplicity Regression Model.ipynb: Notebook containing the regression model that predicts simplicity.
  • A.7_Statistics.ipynb: Notebook to perform statistical analysis.
  • data_preparation.ipynb: Notebook to prepare the raw text data for further analysis.
  • gpt_annotation.ipynb: Notebook to generate annotations using GPT.
  • gpt_annotation_rank_all.ipynb: Notebook to generate annotations using GPT based on ranked texts.
  • gpt_annotation_score_individual.ipynb: Notebook for scoring individual text segments using GPT.
  • rating_interface.py: Streamlit application to run the labeling interface for humans.
  • simulation.ipynb: Notebook for running oracle and user simulations.
  • utils.py: Utility functions for the project.

Root Directory Files

  • docker-compose.yml: Docker Compose configuration.
  • Dockerfile: Defines the Docker image used in this project.
  • LICENSE: License information.
  • requirements.txt: List of Python dependencies required by the project.
  • README.md: This readme file.

Setup Instructions

To run this project, we highly recommend using Docker. Docker simplifies environment setup and ensures consistency across different machines.

Docker Setup

  1. Install Docker:

  2. Build the ARTS Docker Image:

    • Navigate to the root of the project directory (where the Dockerfile is located).

    • Run the following command to build the Docker image:

      docker build -t arts .
  3. Run the Container:

    • After successfully building the image, spin up a container using:

      docker compose up
    • This command mounts the current directory into the Docker container and opens the required ports for Jupyter Notebook/IPython interface interaction.


Contributing

We welcome contributions of all kinds to improve ARTS. To contribute:

  1. Fork the repository.
  2. Create a new branch for your feature or bug fix.
  3. Submit a pull request.

Please make sure your contribution adheres to defined coding convention and includes necessary tests.


License

This project is licensed under the MIT License - see the LICENSE file for details.


That's it! 👩‍💻👨‍💻 Happy Scoring with ARTS!

About

ARTS: Assessing Readability & Text Simplicity

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published