Enable Hy code in the debugger

[brandonwillard]

With the context manager provided here, pdb (ipdb and Pdb++) sessions started from the Hy REPL can evaluate Hy code. For example, the following is possible:

=> (pdb.run "(setv x 1)")                                                                                                                                                                                           
[1] > <string>(1)<module>()
(Pdb++) p (+ 1 1)                                                                                                                                                                                                   
2L
None
(Pdb++) !(print "hi")                                                                                                                                                                                               
hi
(Pdb++) up                                                                                                                                                                                                          
[0] > /home/bwillard/projects/code/python/hy/hy/cmdline.py(150)_hy_bdb_run()
-> exec(cmd, globals, locals)
(Pdb++) (.keys locals)                                                                                                                                                                                              
['__builtins__', 'pdb', 'x', u'hyx_XasteriskX1', '__name__', u'hyx_XasteriskX3', u'hyx_XasteriskX2', '__doc__']
(Pdb++) c                                                                                                                                                                                                           
=> x                                                                                                                                                                                                                
1L
=> (import [hy.contrib.walk [walk postwalk]])                                                                                                                                                                            
=> (pdb.run "(postwalk identity '(+ x 2))")                                                                                                                                                                         
[1] > <string>(0)<module>()
(Pdb++) b walk
Breakpoint 1 at /home/bwillard/projects/code/python/hy/hy/contrib/walk.hy:17
(Pdb++) c                                                                                                                                                                                                           
[3] > /home/bwillard/projects/code/python/hy/hy/contrib/walk.hy(17)walk()
-> [(instance? HyExpression form)
(Pdb++) ll                                                                                                                                                                                                          
   1     ;;; Hy AST walker                                                                                                                                                                                          
   2     ;; Copyright 2018 the authors.                                                                                                                                                                             
   3     ;; This file is part of Hy, which is free software licensed under the Expat                                                                                                                                
   4     ;; license. See the LICENSE.                                                                                                                                                                               
   5                                                                                                                                                                                                                
   6     (import [hy [HyExpression HyDict]]                                                                                                                                                                         
   7             [functools [partial]]                                                                                                                                                                              
   8             [collections [OrderedDict]]                                                                                                                                                                        
   9             [hy.macros [macroexpand :as mexpand]]                                                                                                                                                              
  10             [hy.compiler [HyASTCompiler]])                                                                                                                                                                     
  11                                                                                                                                                                                                                
  12     (defn walk [inner outer form]                                                                                                                                                                              
  13       "Traverses form, an arbitrary data structure. Applies inner to each                                                                                                                                      
  14       element of form, building up a data structure of the same type.                                                                                                                                          
  15       Applies outer to the result."                                                                                                                                                                            
  16       (cond                                                                                                                                                                                                    
  17  ->    [(instance? HyExpression form)                                                                                                                                                                          
  18         (outer (HyExpression (map inner form)))]                                                                                                                                                               
  19        [(instance? HyDict form)                                                                                                                                                                                
  20         (HyDict (outer (HyExpression (map inner form))))]                                                                                                                                                      
  21        [(instance? list form)                                                                                                                                                                                  
  22         ((type form) (outer (HyExpression (map inner form))))]                                                                                                                                                 
  23        [(coll? form)                                                                                                                                                                                           
  24         (walk inner outer (list form))]                                                                                                                                                                        
  25        [True (outer form)]))                                                                                                                                                                                   
  26                                                                                                                                                                                                                
  27     ...
(Pdb++) 

[brandonwillard]

After adding tests, this should at least close

• Add support for post-mortem debugging from the REPL #142
• hy -i should support the debugger #1395 (I think this was already closed by New patch importer #1672)

With a little extra (and perhaps tangential REPL) work, we could get #1397 in place, as well. Also, after some upcoming macro/require rehauls, some of the ideas in #741 might be do-able.

[Kodiologist]

Lookin' good. Let's see some tests.

[Kodiologist]

I think the condition here is equivalent to line.startswith('!'), which seems like a clearer way to write it.

[brandonwillard]

All these patched functions are direct copies from the [bp]db source (a couple have slight cross-version modifications, though), but, yeah, they could be cleaned up a bit.

[Kodiologist]

In that case, you should attribute the source of the copied code in the commit message.

[Kodiologist]

I think you could just write t if isinstance(t, str) else t.__name__ instead of defining exc_type_name.

[Kodiologist]

Likewise, there's no need for a local exc_info.

[Kodiologist]

You probably don't want to commit this big block of commented-out code.

[brandonwillard]

PyPy raises exceptions during calls to inspect.getsource:

hy 0.15.0+32.g4af87dc.dirty using PyPy(fdd60ed87e941677e8ea11acf9f1819466521bf2) 3.5.3 on Linux
=> (import inspect [tests.resources.bin.pdb [*]])
=> (inspect.findsource func2)
(['(defn func1 [x]\n', '  (print "func1")\n', '  (+ 1 x))\n', '(defn func2 [x]\n', '  (print "func2")\n', '  (func1 x))\n'], -1)
=> (inspect.getsource func2)
Traceback (most recent call last):
  File "/home/bwillard/projects/code/python/hy/hy/importer.py", line 140, in hy_eval
    return eval(ast_compile(expr, "<eval>", "eval"), namespace)
  File "<eval>", line 1, in <module>
  File "/home/bwillard/apps/anaconda3/envs/hy-dev-pypy35/lib-python/3/inspect.py", line 947, in getsource
    lines, lnum = getsourcelines(object)
  File "/home/bwillard/apps/anaconda3/envs/hy-dev-pypy35/lib-python/3/inspect.py", line 939, in getsourcelines
    return getblock(lines[lnum:]), lnum + 1
  File "/home/bwillard/apps/anaconda3/envs/hy-dev-pypy35/lib-python/3/inspect.py", line 919, in getblock
    for _token in tokens:
  File "/home/bwillard/apps/anaconda3/envs/hy-dev-pypy35/lib-python/3/tokenize.py", line 597, in _tokenize
    raise TokenError("EOF in multi-line statement", (lnum, 0))
tokenize.TokenError: ('EOF in multi-line statement', (2, 0))

while CPython raises no exception (notice, however, that the result it gives is incomplete):

hy 0.15.0+32.g4af87dc.dirty using CPython(default) 2.7.15 on Linux
=> (import inspect [tests.resources.bin.pdb [*]])
=> (inspect.findsource func2)
(['(defn func1 [x]\n', '  (print "func1")\n', '  (+ 1 x))\n', '(defn func2 [x]\n', '  (print "func2")\n', '  (func1 x))\n'], 0)
=> (inspect.getsource func2)
'(defn func1 [x]\n'

This appears to be what's causing the current test failures in Travis, and it's because the debugger triggers a call to inspect during code listing, then a -1 start-of-line is returned by PyPy's inspect.findsource, which causes inspect.getsourcelines to send inspect.getblock only the last line and, ultimately, the Python tokenizer to barf. These problems can be fixed by patching inspect, and I image that entire module should probably be patched/made compatible with Hy for other reasons, as well.

In the course of researching and testing said patches, I noticed that defn forms do not completely preserve/map source line-number information (at least not in PyPy). It's possible to work around this — to some extent — via special inspect logic, but a true solution leads right back to fundamental macro questions and, most likely, changes.

Comment on Patching for Hy

On a somewhat related note, all this patching has me rethinking how we could accomplish the same compatibility and core Python code re-use with much less effort. In quite a few instances, the necessary patches simply wrap the use of eval, exec and compile so that they handle Hy code. While patching those bulit-ins directly is possible (well, for exec, in Python 3.x, at least), simply replacing the regular Python handling with Hy is not sufficient; however, there might be other work-arounds available that allow those functions to handle Hy source strings as a special case.

For instance, if Hy source strings were created by the Hy interpreter and loader as a special, identifiable type (e.g. class HySource(UserString)), then eval, exec and compile could be patched with only a simple condition handling those.

Overall, patching every piece of relevant code that calls eval, exec and/or compile sounds like an uphill battle, so it's better to address this now.

[Kodiologist]

Oddly, it seems that no Travis run is showing for this PR.

…however, there might be other work-arounds available that allow those functions to handle Hy source strings as a special case.

You're certainly welcome to try.

[Kodiologist]

I'm closing this because of the lack of follow-up. Please reopen if you want to pick this up again.