Graph data in separate config

.github/workflows/ci-cd.yaml

@@ -79,8 +79,12 @@ jobs:
 
           git lfs install
           git lfs track "*-wiktextract-data.jsonl"
+          git lfs track "word-definition-graph-data.jsonl"
+
           git add *-wiktextract-data.jsonl
-          git commit -m "Extract raw-wiktextract-data.jsonl into per-language wiktextract-data.jsonl"
+          git add word-definition-graph-data.jsonl
+
+          git commit -m "Extract raw-wiktextract-data.jsonl into per-language wiktextract-data.jsonl and generate graph data"
           git push https://QubitPi:$HF_TOKEN@huggingface.co/datasets/QubitPi/wiktionary-data master:main -f
         env:
           HF_TOKEN: ${{ secrets.HF_TOKEN }}

README.md

@@ -24,6 +24,10 @@ configs:
       path: old-persian-wiktextract-data.jsonl
     - split: Akkadian
       path: akkadian-wiktextract-data.jsonl
+  - config_name: Graph
+    data_files:
+    - split: All
+      path: word-definition-graph-data.jsonl
 tags:
   - Wiktionary
   - German
@@ -33,6 +37,7 @@ tags:
   - Old Persian
   - Akkadian
   - Vocabulary
+  - Knowledge Graph
 size_categories:
   - 1M<n<10M
 ---
@@ -78,6 +83,12 @@ The available splits are
 - `OldPersian`
 - `Akkadian`
 
+In addition, a separate split for graph data is offered:
+
+- `GraphData`
+
+This split contains all the languages and puts everything in a giant graph
+
 Development
 -----------
 

extract.py

@@ -23,3 +23,4 @@
     args = vars(parser.parse_args())
 
     extract_data(args["input"])
+    extract_graph(args["input"])