Updated scripts for preparing subgraphs input data

[mdsalnikov]

No description provided.

[highly0]

Few comments, nothing to major:)

[highly0]

I think for the sake of reproducibility, include 2 lines to create the folder if not exist

from pathlib import Path
output_path = './to_subgraphs'
Path(output_path).mkdir(parents=True, exist_ok=True)

with open(Path(output_path) / "mintaka_mixtral_valid", 'w+') as f:
    for data_line in data_to_subgraphs(valid_df):
        f.write(ujson.dumps(data_line)+'\n')

[mdsalnikov]

Done

[highly0]

Same thing as above

with open(Path(output_path) / "mintaka_mixtral_test", 'w+') as f:
    for data_line in data_to_subgraphs(valid_df):
        f.write(ujson.dumps(data_line)+'\n')

[mdsalnikov]

Done

[highly0]

I think in this case, it is better to use pandas.read_json, since we want to load in a dataframe anyway. Get rids of 2 unnecessary steps.

import pandas as pd

valid_predictions = pd.read_json('./mintaka_mixtral_50_valid_predictions.json')
test_predictions = pd.read_json('./mintaka_mixtral_50_test_predictions.json')

ds = datasets.load_dataset("AmazonScience/mintaka")

valid_df = pd.merge(
    valid_predictions,
    ds['validation'].to_pandas(),
    on=['id', 'question'],
)

test_df = pd.merge(
    test_predictions.rename(columns={'mixtral answers': 'model_answer'}),
    ds['test'].to_pandas(),
    on=['id', 'question'],
)

[mdsalnikov]

Yes, am agree! Thank you
Done