Sql example

[tmarkovich]

Note, this PR is a work in progress. I'm putting it up now to get some guidance on code organization -- where should I put these functions, refactoring, etc?

Types of changes

• [] Docs change / refactoring / dependency upgrade
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Motivation and Context / Related issue

This PR provides a set of scripts to process data stored in SQL tables to get them into the right format for use with PBG.

Checklist

• The documentation is up-to-date with the changes I made.
• I have read the CONTRIBUTING document and completed the CLA (see CONTRIBUTING).
• All tests passed, and additional code has been covered with new tests.

[adamlerer]

Hey Thomas, this looks good to me! You are going to now add a script that does this end to end (downloads data, converts to sql, runs PBG, processes the outputs back to sql)?

Maybe we call it sql_end2end or something?

[adamlerer]

Your sql script takes a CSV file as input. I know that's just as an example and not the "intended use", but for this setup is it faster than our existing scripts? If so, should we be replacing those scripts with this code, rather than just using it as an example?

[adamlerer]

The config is an actual python function... having python code that generates python code seems overly complex, no?

Either you can call train directly from within an end-to-end script, with the config, or perhaps you can write a config with json.dump or something?

Here's a schematic that might make this more clear:

>>> cfg = dict(
...   entity_path="/foo/bar",
...   edge_paths=[
...     "path1",
...     "path2",
...   ],
...   relations={
...    "rel1": 3
...   }
... )
>>> import json
>>> print(json.dumps(cfg, indent=2))
{
  "entity_path": "/foo/bar",
  "edge_paths": [
    "path1",
    "path2"
  ],
  "relations": {
    "rel1": 3
  }
}

So you would do something like:

cfg = copy.deepcopy(DEFAULT_CFG)
cfg.relations = relations
cfg.entities = entities

with open(cfg_file, "w") as f:
  f.write(f"def get_torchbiggraph_config():\n  {json.dumps(cfg, indent=4)}")

[tmarkovich]

This makes sense.

[tmarkovich]

In a single test it was faster, but I'm not sure that I'd advocate for replacing the conversion script yet. I'd probably want to spend a little more time optimizing and evaluating first before doing that.