read_xmile problem (case sensitivity?)

[bpowers]

Hi folks -- I'm working on a tool that would let you edit a model from inside a jupyter notebook, among other things.

I have an example project here: https://github.com/bpowers/simlin-jupyter-example/blob/main/Lotka.ipynb

but when I call read_xmile on the file I produce, pysd seems to have a problem I believe because the case of the identifier in the variable equation is different from the case of text in the variable name attribute:

KeyError                                  Traceback (most recent call last)
~/src/test_jup/env/lib64/python3.9/site-packages/parsimonious/nodes.py in visit(self, node)
    216         try:
--> 217             return method(node, [self.visit(n) for n in node])
    218         except (VisitationError, UndefinedLabel):

~/src/test_jup/env/lib64/python3.9/site-packages/pysd/py_backend/xmile/SMILE2Py.py in visit_identifier(self, n, vc)
    271     def visit_identifier(self, n, vc):
--> 272         return self.extended_model_namespace[n.text] + '()'
    273 

KeyError: 'Prey'

During handling of the above exception, another exception occurred:

VisitationError                           Traceback (most recent call last)
<ipython-input-16-05ac85772edd> in <module>
----> 1 pysd.read_xmile('Lotka_Volterra.mdl.simlin.xmile')

~/src/test_jup/env/lib64/python3.9/site-packages/pysd/pysd.py in read_xmile(xmile_file)
     18     from . import py_backend
     19     from .py_backend.xmile.xmile2py import translate_xmile
---> 20     py_model_file = translate_xmile(xmile_file)
     21     model = load(py_model_file)
     22     model.xmile_file = xmile_file

~/src/test_jup/env/lib64/python3.9/site-packages/pysd/py_backend/xmile/xmile2py.py in translate_xmile(xmile_file)
    121         }
    122 
--> 123         tranlation, new_structure = smile_parser.parse(eqn, element)
    124         element.update(tranlation)
    125         if is_constant_expression(element['py_expr']):

~/src/test_jup/env/lib64/python3.9/site-packages/pysd/py_backend/xmile/SMILE2Py.py in parse(self, text, element, context)
    253         self.new_structure = []
    254 
--> 255         py_expr = self.visit(self.ast)
    256 
    257         return ({

~/src/test_jup/env/lib64/python3.9/site-packages/parsimonious/nodes.py in visit(self, node)
    215         # up.
    216         try:
--> 217             return method(node, [self.visit(n) for n in node])
    218         except (VisitationError, UndefinedLabel):
    219             # Don't catch and re-wrap already-wrapped exceptions.

~/src/test_jup/env/lib64/python3.9/site-packages/parsimonious/nodes.py in <listcomp>(.0)
    215         # up.
    216         try:
--> 217             return method(node, [self.visit(n) for n in node])
    218         except (VisitationError, UndefinedLabel):
    219             # Don't catch and re-wrap already-wrapped exceptions.

~/src/test_jup/env/lib64/python3.9/site-packages/parsimonious/nodes.py in visit(self, node)
    215         # up.
    216         try:
--> 217             return method(node, [self.visit(n) for n in node])
    218         except (VisitationError, UndefinedLabel):
    219             # Don't catch and re-wrap already-wrapped exceptions.

~/src/test_jup/env/lib64/python3.9/site-packages/parsimonious/nodes.py in <listcomp>(.0)
    215         # up.
    216         try:
--> 217             return method(node, [self.visit(n) for n in node])
    218         except (VisitationError, UndefinedLabel):
    219             # Don't catch and re-wrap already-wrapped exceptions.

~/src/test_jup/env/lib64/python3.9/site-packages/parsimonious/nodes.py in visit(self, node)
    215         # up.
    216         try:
--> 217             return method(node, [self.visit(n) for n in node])
    218         except (VisitationError, UndefinedLabel):
    219             # Don't catch and re-wrap already-wrapped exceptions.

~/src/test_jup/env/lib64/python3.9/site-packages/parsimonious/nodes.py in <listcomp>(.0)
    215         # up.
    216         try:
--> 217             return method(node, [self.visit(n) for n in node])
    218         except (VisitationError, UndefinedLabel):
    219             # Don't catch and re-wrap already-wrapped exceptions.

~/src/test_jup/env/lib64/python3.9/site-packages/parsimonious/nodes.py in visit(self, node)
    225             # see where it went wrong.
    226             exc_class, exc, tb = exc_info()
--> 227             reraise(VisitationError, VisitationError(exc, exc_class, node), tb)
    228 
    229     def generic_visit(self, node, visited_children):

~/src/test_jup/env/lib64/python3.9/site-packages/six.py in reraise(tp, value, tb)
    700                 value = tp()
    701             if value.__traceback__ is not tb:
--> 702                 raise value.with_traceback(tb)
    703             raise value
    704         finally:

~/src/test_jup/env/lib64/python3.9/site-packages/parsimonious/nodes.py in visit(self, node)
    215         # up.
    216         try:
--> 217             return method(node, [self.visit(n) for n in node])
    218         except (VisitationError, UndefinedLabel):
    219             # Don't catch and re-wrap already-wrapped exceptions.

~/src/test_jup/env/lib64/python3.9/site-packages/pysd/py_backend/xmile/SMILE2Py.py in visit_identifier(self, n, vc)
    270 
    271     def visit_identifier(self, n, vc):
--> 272         return self.extended_model_namespace[n.text] + '()'
    273 
    274     def visit_quoted_identifier(self, n, vc):

VisitationError: KeyError: 'Prey'

Parse tree:
<RegexNode called "identifier" matching "Prey">  <-- *** We were here. ***

[bpowers]

(in particular this is using 0.10.0 from PyPI -- not sure if this is expected fixed as of the latest unpublished)

[JamesPHoughton]

Hey @bpowers - good to hear from you! The tool looks really cool - I'm always impressed with your front-end work.

Case sensitivity would make sense as a diagnosis, given my read-over. We haven't done much xmile dev work recently, so I'm a bit out of touch with it. @alexprey may know better.

I think we're already forcing some patterns in the namespace dictionary (here: https://github.com/JamesPHoughton/pysd/blob/master/pysd/py_backend/xmile/SMILE2Py.py#L226) so maybe we should add one to lowercase everything. I should check what we're doing on the vensim side. The other option is to enforce case sensitivity in xmile file generation. What does the standard say? Regardless, if you figure out what's happening on the XMILE parsing side of pysd and want to make changes, I don't think anyone would mind. =)

Either way, this would be an interesting test case to add to the test-models repo, with both an xmile and vensim example.

[enekomartinmartinez]

Hi @bpowers
Case sensitivity should be solved in the version of PR #312
I have made a test run with the file you were using and seems that everything works fine.
It would be nice to have a full integration test of case sensitivity in XMile file in the test-models repo.