Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Epjson support #4199

Merged
merged 36 commits into from
Mar 4, 2021
Merged

Epjson support #4199

merged 36 commits into from
Mar 4, 2021

Conversation

lefticus
Copy link
Contributor

@lefticus lefticus commented Feb 3, 2021

Pull request overview

Pull Request Author

Add to this list or remove from it as applicable. This is a simple templated set of guidelines.

  • Verified that C# bindings built fine on Windows, partial classes used as needed, etc.
  • All new and existing tests passes

Labels:

  • If deemed ready, add label Pull Request - Ready for CI so that CI builds your PR

Review Checklist

This will not be exhaustively relevant to every PR.

  • Perform a Code Review on GitHub
  • Code Style, strip trailing whitespace, etc.
  • All related changes have been implemented: model changes, model tests, FT changes, FT tests, VersionTranslation, OS App
  • Labeling is ok
  • If defect, verify by running develop branch and reproducing defect, then running PR and reproducing fix
  • If feature, test running new feature, try creative ways to break it
  • CI status: all green or justified

@lefticus lefticus added Pull Request - Ready for CI This pull request if finalized and is ready for continuous integration verification prior to merge. component - epJSON labels Feb 3, 2021
@lefticus lefticus requested a review from kbenne February 3, 2021 01:38
@tijcolem
Copy link
Collaborator

tijcolem commented Feb 5, 2021

This is awesome @lefticus! I ran a clean, full build and things look good on Windows and Linux.

I see this on mac. Looks like it picked up two unused variables (below). Using apple-clang10 on mac.
Mac Config:

[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=apple-clang
compiler.libcxx=libc++
compiler.version=10.0
os=Macos
os_build=Macos
[  3%] Building CXX object src/radiance/CMakeFiles/openstudio_radiance.dir/AnnualIlluminanceMap.cpp.o
/Users/jenkins/git/OpenStudioFull/epjson_support/src/epjson/epJSONTranslator.cpp:292:17: error: unused variable 'type' [-Werror,-Wunused-variable]
    const auto& type = obj.iddObject().type().valueName();
                ^
/Users/jenkins/git/OpenStudioFull/epjson_support/src/epjson/epJSONTranslator.cpp:291:17: error: unused variable 'group' [-Werror,-Wunused-variable]
    const auto& group = obj.iddObject().group();
                ^
2 errors generated.

@lefticus
Copy link
Contributor Author

lefticus commented Feb 5, 2021

@tijcolem I'll clean those up. I'm most fascinated that it wasn't caught on any of the platforms I run it on.

@arif-hanif
Copy link
Contributor

@lefticus this is awesome will give it a test and look over the code

@kbenne
Copy link
Contributor

kbenne commented Mar 2, 2021

Fantastic @lefticus. The next steps that I can think of are.

  1. I wonder if the functions
    EPJSON_API Json::Value toJSON(const openstudio::model::Model& model, const openstudio::path& schema = schemaPath());
    that take Model and return a JSON instance would be more logical as a method of the EnergyPlus translator, similar to this
    Workspace translateModel(const model::Model& model, ProgressBar* progressBar = nullptr);
  2. Update the OpenStudio Workflow to either translate to JSON by default or expose a config parameter. (or both)
  3. Switch all or most of the simulation tests to use JSON input to EnergyPlus https://github.com/NREL/OpenStudio-resources/tree/develop/model/simulationtests

Once we conclude item 1 (either deciding to make a change or not), then I'm inclined to go ahead and merge this and work on items 2 and 3 in separate PRs. I think maybe @jmarrec would be a candidate for these tasks.

One more thing. This is schema driven and we have an EnergyPlus JSON schema. Is it possible to generate an OpenStudio Model JSON schema, potentially using the OpenStudio.idd? If that is the case then I think it would be possible to serialize the OS Model as JSON and that might be a nice thing, even if we aren't building the model on JSON natively. Basically a Model::saveAs(...) feature.

cc @shorowit

jmarrec added 3 commits March 3, 2021 10:33
example:
```
m = Model.new; z = ThermalZone.new(m); space = Space.new(m); space.setThermalZone(z);
 json_hash = OpenStudio::EPJSON::toJSON(m) 
```
Copy link
Collaborator

@jmarrec jmarrec left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 this is cool @lefticus!

(One day we'll just have this for both model and idf. and then in IdfFile::m_load, we could easily count the number of objects, call m_objects.resize(numObjects) and parse stuff much faster)

src/epjson/epJSON.i Show resolved Hide resolved

EPJSON_API openstudio::path schemaPath();

EPJSON_API Json::Value loadJSON(const openstudio::path& path);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really expected to see a model / Workspace fromJSONString/loadJSONString here after seeing the swig interface first.

I think I'd like that. so in ruby tests I could create a test example via a hash without having to serialize it to a file then read it back in.

I'm also confused: is there nothing to load a JSON object then? It's swig-ignored. I just got here, so I'm not sure what was the scope and what this achieves, I'll have more clarity once I'm done looking at everything.
I thought this PR was to do IDF => epJSON, but since I see toJSON(const openstudio::model::Model& model) I suppose I expected the reverse to be avalaible. In the future we'd do stuff like VersionTranslator::loadModel(epJSON), etc

Copy link
Collaborator

@jmarrec jmarrec Mar 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I see looking at CPP file that this is just pain loading a json file. This should probably be made a utility function somewhere else, so it could be reused in utilities/filetypes/WorkflowJSON.cpp and others. (It shouldn't be exposed publicly in this class at least)

Otherwise the toJSON(const Model& model) stuff has been addressed by kbenne's comment above already. So this is a no-op comment here.

Comment on lines +42 to +49
openstudio::path setupIdftoEPJSONTest(const openstudio::path& location) {
const auto basename = openstudio::toPath(openstudio::filesystem::basename(location));
const auto working_directory = openstudio::filesystem::complete(openstudio::toPath("epjson_tests") / basename);
const auto idf_path = working_directory / openstudio::toPath("eplus.idf");
openstudio::filesystem::create_directories(working_directory);
openstudio::filesystem::copy_file(location, idf_path, openstudio::filesystem::copy_option::overwrite_if_exists);
return idf_path;
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(looks like something that should be in the Fixture instead no?)

Comment on lines +20 to +21
template <typename... KeyType>
const Json::Value& safeLookupValue(const Json::Value& base, const KeyType&... keys) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally wouldn't mind a comment or two in there :)


JSONValueType schemaPropertyTypeDecode(const Json::Value& type) {
if (type.isNull() || !type.isString()) {
LOG_FREE(LogLevel::Warn, "epJSONTranslator", "Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm getting a few of these:

[1] OS-build-release(main)> m = Model.new; z = ThermalZone.new(m); space = Space.new(m); space.setThermalZone(z);
[2] OS-build-release(main)> s = OpenStudio::EPJSON::toJSONString(m)
[openstudio.model.YearDescription] <-1> 'UseWeatherFile' is not yet a supported option for YearDescription
[openstudio.model.YearDescription] <-1> 'UseWeatherFile' is not yet a supported option for YearDescription
[utilities.idf.WorkspaceObject] <0> Object of type 'Schedule:Constant' and named 'Always On Discrete', points to an object named OnOff from field 1, but that object cannot be located.
[utilities.idf.WorkspaceObject] <0> Object of type 'Schedule:Constant' and named 'Always Off Discrete', points to an object named OnOff 1 from field 1, but that object cannot be located.
[utilities.idf.WorkspaceObject] <0> Object of type 'Schedule:Constant' and named 'Always On Continuous', points to an object named Fractional from field 1, but that object cannot be located.
[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option
[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option
[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option

That's not a terribly helpful log message, I think it'd be better to catch the NumberOfString return type at calling point so we can tell which field_name does that

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 71f7455 :

[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option. Occurred for property_name=ceiling_height
[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option. Occurred for property_name=volume
[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option. Occurred for property_name=floor_area

Comment on lines 434 to 441
Json::Value toJSON(const openstudio::model::Model& model, const openstudio::path& schemaToLoad) {
openstudio::energyplus::ForwardTranslator trans;

openstudio::Workspace workspace = trans.translateModel(model);
workspace.toIdfFile();

return toJSON(workspace.toIdfFile(), schemaToLoad);
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @kbenne this should be in openstudio::energyplus::forwardtranslator. Reading the method I assumed it would serialize my Model into JSON form, not translate to IDF then serialize that.

@jmarrec
Copy link
Collaborator

jmarrec commented Mar 3, 2021

I made a couple commits to address some stuff I'd like, but I won't be offended if they get reverted because it's trash :)

@jmarrec
Copy link
Collaborator

jmarrec commented Mar 3, 2021

For C#, in the current state, the Model class isn't swig'ed properly:

OS-build-release$ find ./csharp_wrapper/generated_sources/OpenStudioEPJSON/ -name "*SWIGTYPE_p*"
./csharp_wrapper/generated_sources/OpenStudioEPJSON/SWIGTYPE_p_openstudio__model__Model.cs

Edit fixed: 4063f50

@jmarrec
Copy link
Collaborator

jmarrec commented Mar 3, 2021

@kbenne What I'm proposing in terms of API is to leave this in epjson, but not provide something that takes model. Take Workspace instead: https://github.com/NREL/OpenStudio/compare/epjson_support...epjson_mod?expand=1

Then, to translate a model to epJSON (IDF) you would be responsible to call the FT yourself first:

m = Model.new; z = ThermalZone.new(m); space = Space.new(m); space.setThermalZone(z);
ft = OpenStudio::EnergyPlus::ForwardTranslator.new
w = ft.translateModel(m)
j = OpenStudio::EPJSON::toJSON(w);

j["Zone"]

=> {"Thermal Zone 1"=>{"direction_of_relative_north"=>0.0, "multiplier"=>1.0, "part_of_total_floor_area"=>"Yes", "x_origin"=>0.0, "y_origin"=>0.0, "z_origin"=>0.0}}

At one point, if we allow serializing a Model to a JSON format, you could do j = OpenStudio::EPJSON::toJSON(); but currently it'll be nil

[5] utilities(main)> j = OpenStudio::EPJSON::toJSON(m);
[epJSONTranslator] <1> At the moment, only IddFileType::EnergyPlus is supported
[6] utilities(main)> j
=> nil

Then we can go and add helpers in IdfFile, Workspace, and Model such as saveAs if we want to, but this keeps the logic inside the epjson namespace.

@jmarrec
Copy link
Collaborator

jmarrec commented Mar 3, 2021

If we think we'll allow Model to be serialized to epJSON, then I would actually prefer this then

- EPJSON_API Json::Value toJSON(const openstudio::IdfFile& inputFile, const openstudio::path& schema = schemaPath());
+ EPJSON_API Json::Value toJSON(const openstudio::IdfFile& inputFile, boost::optional<openstudio::path> schema = boost::none);

(Or just default schema_path to an empty path).

Then if not provided, we call schemaPath(openstudio::IddFileType) and figure out if we need the Energy+.epJSON or the OpenStudio.epJSON (when that's going to be a thing)

Edit: I addressed that on my branch too. see c630048


Additionally, I think these two (or at least OpenStudio.idd) should be embedded instead of trying to locate them on disk.

@kbenne
Copy link
Contributor

kbenne commented Mar 3, 2021

To summarize as I know it. @jmarrec has a branch https://github.com/NREL/OpenStudio/compare/epjson_mod where the principle change is that the toJSON(...) functions that take Model, now take Workspace instead. This accomplishes two things.

  1. There is no longer an implicit behind the scenes call to the EnergyPlus translator, which was happening with toJSON(Model...)
  2. Clients must explicitly call the EnergyPlus translator and then convert to JSON if they desire. The EP translator returns a Workspace (Not IdfFile), which is why it is handy to now have toJSON(Workspace...).

Just repeating what @jmarrec stubbed out. Here is the workflow.

m = Model.new
ft = OpenStudio::EnergyPlus::ForwardTranslator.new
w = ft.translateModel(m)
j = OpenStudio::EPJSON::toJSON(w);

I think these are both good changes, and I'm prepared to merge to @jmarrec's changes into this PR. I have a few more questions.

  1. What purpose remains for the toJSON(IdfFile ) functions? I'm inclined to just remove them.
  2. Maybe the json schema(s) should be embedded. We currently do that for IDD, and it means that the api is fully functional without external resources, up until the point that you try to run EnergyPlus. I think that is a good thing.
  3. Should toJson require a schema argument at all? Can't we just internally look at the given Workspace, identify the supporting idd, and the find the matching JSON schema if it is available. I'm not sure providing the schema is something the client should worry about.
  4. How hard is it to generate a schema for the OpenStudio.idd?

@jmarrec
Copy link
Collaborator

jmarrec commented Mar 3, 2021

Thanks for summary that brings order @kbenne

  1. What purpose remains for the toJSON(IdfFile ) functions? I'm inclined to just remove them.

I suppose you could just do idf_file = IdfFile::load('my.idf'); EPSON::toJSON(idf_file)

  1. Should toJson require a schema argument at all? Can't we just internally look at the given Workspace, identify the supporting idd, and the find the matching JSON schema if it is available. I'm not sure providing the schema is something the client should worry about.

This is an optional argument on my branch. passing it would allow you to supply a custom schema, which granted shouldn't be used often, but can't hurt.

  1. How hard is it to generate a schema for the OpenStudio.idd?

I am actually doing that right now.

@kbenne
Copy link
Contributor

kbenne commented Mar 3, 2021

Thanks for summary that brings order @kbenne

  1. What purpose remains for the toJSON(IdfFile ) functions? I'm inclined to just remove them.

I suppose you could just do idf_file = IdfFile::load('my.idf'); EPSON::toJSON(idf_file)

I typically subscribe to the theory of less is more, but I guess these functions are pretty harmless so I'm happy to let them be.

  1. Should toJson require a schema argument at all? Can't we just internally look at the given Workspace, identify the supporting idd, and the find the matching JSON schema if it is available. I'm not sure providing the schema is something the client should worry about.

This is an optional argument on my branch. passing it would allow you to supply a custom schema, which granted shouldn't be used often, but can't hurt.

It is optional, but can we add some intelligence to the defaulting by looking at the given Workspace idd, and map to the appropriate schema?

  1. How hard is it to generate a schema for the OpenStudio.idd?

I am actually doing that right now.

Of course you are. :) cool!

Can GitHub please give us proper threaded conversations?

@jmarrec
Copy link
Collaborator

jmarrec commented Mar 3, 2021

I hijacked the E+ scripts on this branch: https://github.com/jmarrec/EnergyPlus/tree/OpenStudio.epJSON reusing some portions from this failed PR: NREL/EnergyPlus#7969

So I could produce an OpenStudio.epJSON quickly. I'm assuming there's a lot of clean up that should happen, but that's what I could come up with in under an hour... It seem I can translate a model to epJSON then... Anyways, this seems possible.

OpenStudio.epJSON.txt

m = Model.new; z = ThermalZone.new(m); space = Space.new(m); space.setThermalZone(z);
[13] model(main)> j = OpenStudio::EPJSON::toJSON(m, "OpenStudio.epJSON");
[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option. Occurred for property_name=handle
[utilities.idf.IdfObject] <1> Could not convert '{71fd1c12-1bae-49d0-99fe-09fc3c007235}' to double
[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option. Occurred for property_name=ceiling_height
[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option. Occurred for property_name=volume
[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option. Occurred for property_name=floor_area
[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option. Occurred for property_name=handle
[utilities.idf.IdfObject] <1> Could not convert '{aabaa92e-6d73-42c2-8019-af5feabf5664}' to double
[utilities.idf.IdfObject] <1> Could not convert '{4d84a881-e4aa-4b18-97f6-66120afc4504}' to double
[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option. Occurred for property_name=handle
[utilities.idf.IdfObject] <1> Could not convert '{4d84a881-e4aa-4b18-97f6-66120afc4504}' to double
[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option. Occurred for property_name=handle
[utilities.idf.IdfObject] <1> Could not convert '{8684cf56-42eb-42ba-8aa5-a78dad95060b}' to double
[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option. Occurred for property_name=handle
[utilities.idf.IdfObject] <1> Could not convert '{911b4f25-8bed-4153-91ad-3ef13eb4103b}' to double
[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option. Occurred for property_name=handle
[utilities.idf.IdfObject] <1> Could not convert '{3283bf29-ff2f-4449-b2ba-7b6a669b2f24}' to double
[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option. Occurred for property_name=handle
[utilities.idf.IdfObject] <1> Could not convert '{7993b5ca-039f-4c85-a7f6-9b9e8880bc6e}' to double
[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option. Occurred for property_name=dedicated_outdoor_air_low_setpoint_temperature_for_design
[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option. Occurred for property_name=dedicated_outdoor_air_high_setpoint_temperature_for_design
[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option. Occurred for property_name=handle
[utilities.idf.IdfObject] <1> Could not convert '{821d079e-8b6a-492a-a463-72d18eef72f7}' to double
[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option. Occurred for property_name=handle
[utilities.idf.IdfObject] <1> Could not convert '{aa1cce0c-780d-4ba9-b51c-b6a601c7bfc6}' to double

Using the resulting json:

[14] model(main)> j["OS:ThermalZone"]
=> {"Thermal Zone 1"=>
  {"handle"=>"{71fd1c12-1bae-49d0-99fe-09fc3c007235}",
   "use_ideal_air_loads"=>"No",
   "zone_air_exhaust_port_list"=>"{911b4f25-8bed-4153-91ad-3ef13eb4103b}",
   "zone_air_inlet_port_list"=>"{8684cf56-42eb-42ba-8aa5-a78dad95060b}",
   "zone_air_node_name"=>"{4d84a881-e4aa-4b18-97f6-66120afc4504}",
   "zone_return_air_port_list"=>"{3283bf29-ff2f-4449-b2ba-7b6a669b2f24}"}}

… OpenStudio.epJSON

Two defaults for one object, otherwise min_fields > num_fields
@jmarrec
Copy link
Collaborator

jmarrec commented Mar 3, 2021

Looking at some warnings I get

[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option. Occurred for property_name=ceiling_height
[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option. Occurred for property_name=volume
[epJSONTranslator] <0> Unknown value passed to schemaPropertyTypeDecode, returning generic 'NumberOrString' Option. Occurred for property_name=floor_area

This occurs for the anyOf fields. It would be nice to clean up these warnings, so we can catch actual problems

        "Zone": {
            "patternProperties": {
                "^.*\\S.*$": {
                    "type": "object",
                    "properties": {
                     [...]
                        "ceiling_height": {
                            "note": "If this field is 0.0, negative or autocalculate, then the average height of the zone is automatically calculated and used in subsequent calculations. If this field is positive, then the number entered here will be used. Note that the Zone Ceiling Height is the distance from the Floor to the Ceiling in the Zone, not an absolute height from the ground.",
                            "units": "m",
                            "default": "Autocalculate",
                            "anyOf": [
                                {
                                    "type": "number"
                                },
                                {
                                    "type": "string",
                                    "enum": [
                                        "",
                                        "Autocalculate"
                                    ]
                                }
                            ]
                        },

```
In file included from ../src/epjson/epJSONTranslator.cpp:5:
../src/epjson/../utilities/idd/IddEnums.hpp:56:1: error: 'IddFileType' defined as a struct here but previously declared as a class; this is valid, but may result in linker errors under the Microsoft C++ ABI [-Werror,-Wmismatched-tags]
```
@jmarrec jmarrec added this to the OpenStudio SDK 3.2.0 milestone Mar 4, 2021
@kbenne kbenne merged commit 8d30dc2 into develop Mar 4, 2021
@kbenne kbenne deleted the epjson_support branch March 4, 2021 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component - epJSON Pull Request - Ready for CI This pull request if finalized and is ready for continuous integration verification prior to merge.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Forward translate to epJSON
6 participants