[IPython] Support IDLE and Blender scripting module as interactive shell #1308

archibate · 2020-06-23T08:38:53Z

Related issue = #1483

codecov · 2020-07-13T03:56:01Z

Codecov Report

Merging #1308 into master will decrease coverage by 0.57%.
The diff coverage is 61.23%.

@@            Coverage Diff             @@
##           master    #1308      +/-   ##
==========================================
- Coverage   66.84%   66.26%   -0.58%     
==========================================
  Files          38       41       +3     
  Lines        5450     5775     +325     
  Branches      956      998      +42     
==========================================
+ Hits         3643     3827     +184     
- Misses       1644     1777     +133     
- Partials      163      171       +8

Impacted Files	Coverage Δ
python/taichi/idle_hacker.py	`0.00% <0.00%> (ø)`
python/taichi/lang/kernel.py	`82.77% <ø> (+1.20%)`	⬆️
python/taichi/tools/np2ply.py	`13.37% <ø> (ø)`
python/taichi/misc/gui.py	`25.36% <16.12%> (-1.15%)`	⬇️
python/taichi/main.py	`42.18% <17.64%> (+0.10%)`	⬆️
python/taichi/lang/shell.py	`42.51% <39.47%> (+3.81%)`	⬆️
python/taichi/core/util.py	`21.30% <42.85%> (-0.43%)`	⬇️
python/taichi/core/settings.py	`48.14% <45.45%> (ø)`
python/taichi/lang/impl.py	`89.41% <63.15%> (-1.14%)`	⬇️
python/taichi/__init__.py	`75.00% <75.00%> (-17.00%)`	⬇️
... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0f4ce50...d04fbf7. Read the comment docs.

archibate · 2020-07-14T02:13:16Z

python/taichi/idle_hacker.py

+    taichi_dir = os.path.dirname(os.path.abspath(ti.__file__))
+    filename = os.path.join(taichi_dir, '.tidle_' + str(os.getpid()))
+    with open(filename, 'a') as f:
+        f.write('\n===\n' + source)


We have to use this dirty method since idle and taichi is running in separate process... any idea?

IIUIC you want to inject part of the process info of taichi into IDLE so they could work together? I'm not so sure but would ZeroMQ work in this scenario without the dirty hack? Its binding with Python is really easy to work with and is pretty fast at IPC.

or we can directly wrap sockets and pass arbitrary python objects using multiprocessing? Correct me if I'm wrong, I feel like I'm missing sth here lol

sockets

In fact, the IDLE already using socket as IPC, and that could be an ultimate solution, however:

Not sure if it's invarient over different versions.

I'm not super familiar about sockets :(

So I'd rather write a portable file IPC instead, WDYT?

ZeroMQ

No, Taichi is already a highly dedicated package, having too heavy dependencies could easily ban it from everyday people, sorry.

Ahh I see. Then I think this solution's good enough. The one that works is always the best one.

archibate · 2020-07-14T07:59:47Z

I don't have a Blender, could you confirm this PR works for the Blender scripting module too?

Eydcao · 2020-07-16T02:29:03Z

I don't have a Blender, could you confirm this PR works for the Blender scripting module too?

Hi Yubin,

My first time hearing taichi script in Blender. Does Blender have sth like IDLE for running interactive python? I could help confirm when free later.

Thx

I read that in the #1483 that there is a Blender Python environment. :>

archibate · 2020-07-23T13:41:40Z

Why nobody review this.. Here're tons of ad-hoc engineering effort to make the stupid IDLE happy :(

archibate · 2020-07-24T13:14:43Z

Is this going to be closed?

EaryChow · 2020-07-24T15:41:24Z

Please don't close this (╯︵╰)

archibate · 2020-07-24T15:50:59Z

Please don't close this (╯︵╰)

Of course I won't! ❤️ But given that people never review our PR (why??), I think we can merge on our own if they don't hit the Request Changes button after next release.

yuanming-hu · 2020-07-25T01:48:18Z

Thanks for all the contributions here! This is a useful feature and the PR should stay open. I just need more time to review given the large change (and my packed daily schedule...) Sorry about that.

@rexwangcc

* [Misc] Use 'dill.source' of 'inspect' to run codes in interactive shell * [skip ci] Add dill to CI * _ShellInspectorWrapper * Hack IDLE to make it happy * Fix IDLE * [skip ci] cache: we do care user experience!! * improve stability * [skip ci] fix example exit * [skip ci] fix exit in IPython * [skip ci] improve comment * fix exec risk in ti debug (@rexwangcc) * [skip ci] Fix ti debug too verbose * Better line * fix test_cli * [skip ci] idle_hacker.py * fix typo and improve hacker * [skip ci] reimprov * improve idle inspector * [skip ci] add tag [IPython] * revert idle * [skip ci] Revert "revert idle" This reverts commit 794bf05. * tmp save * well * clean * [skip ci] re * [skip ci] al * t * improve hack * nice message * [skip ci] enforce code format * ti idle_hacker -r * [skip ci] enforce code format * fix ipython * revert Co-authored-by: Taichi Gardener <taichigardener@gmail.com>

This reverts commit 202c235.

isdanni

The injector LGTM so far! If the other solutions wouldn't work I think this is a good choice

isdanni · 2020-08-07T06:59:47Z

python/taichi/idle_hacker.py

+    taichi_dir = os.path.dirname(os.path.abspath(ti.__file__))
+    filename = os.path.join(taichi_dir, '.tidle_' + str(os.getpid()))
+    with open(filename, 'a') as f:
+        f.write('\n===\n' + source)


IIUIC you want to inject part of the process info of taichi into IDLE so they could work together? I'm not so sure but would ZeroMQ work in this scenario without the dirty hack? Its binding with Python is really easy to work with and is pretty fast at IPC.

python/taichi/idle_hacker.py

isdanni · 2020-08-07T07:21:52Z

python/taichi/idle_hacker.py

+    taichi_dir = os.path.dirname(os.path.abspath(ti.__file__))
+    filename = os.path.join(taichi_dir, '.tidle_' + str(os.getpid()))
+    with open(filename, 'a') as f:
+        f.write('\n===\n' + source)


or we can directly wrap sockets and pass arbitrary python objects using multiprocessing? Correct me if I'm wrong, I feel like I'm missing sth here lol

Co-authored-by: Danni <38550500+isdanni@users.noreply.github.com>

archibate · 2020-08-08T03:27:37Z

Hi! I searched for the Internet and realized that there are no such package capable of inspecting these various shells yet.
Instead of waiting for somebody to create it, let me create it for serving taichi, and other people suffered from the same issue.
How about this:

Let me create a separate package sourceinspect, and deploy it to PyPI.
The package is able to deal with IDLE, IPython, Jupyter, Blender, etc.
Move all current mocks in taichi, including this PR, to that external package.
No maintaining cost in taichi, we maintain the mocks in sourceinspect separately.
Include sourceinspect as a hard dependency of taichi.

WDYT about this solution? @yuanming-hu @EaryChow @isdanni Inputs are welcome!

archibate added waiting for reply wontfix We won't fix this issue or merge this PR and removed waiting for reply labels Jun 27, 2020

archibate closed this Jul 1, 2020

archibate reopened this Jul 13, 2020

archibate mentioned this pull request Jul 13, 2020

IDLE still doesn't work for me #1483

Open

archibate changed the title ~~[IPython] Support IDLE as interactive shell (experimental)~~ [IPython] Support IDLE and Blender scripting module as interactive shell Jul 13, 2020

archibate marked this pull request as ready for review July 13, 2020 05:30

archibate requested review from rexwangcc, yuanming-hu, k-ye and taichi-gardener and removed request for rexwangcc and yuanming-hu July 13, 2020 05:30

archibate added large changeset The PR contains a large changeset and reviewing may take times and removed wontfix We won't fix this issue or merge this PR labels Jul 13, 2020

archibate force-pushed the dillidle branch 2 times, most recently from d4aa3a2 to 73bff6c Compare July 14, 2020 01:22

archibate commented Jul 14, 2020

View reviewed changes

archibate force-pushed the dillidle branch 2 times, most recently from 1121ab3 to 7847dc3 Compare July 14, 2020 04:24

archibate added the waiting for reply label Jul 14, 2020

archibate requested review from xumingkuan and Eydcao July 14, 2020 12:59

archibate added the enhancement Make existing things or codebases better label Jul 16, 2020

archibate added the python Python engineering related label Jul 16, 2020

archibate requested a review from PavelBlend July 22, 2020 01:50

archibate assigned yuanming-hu and archibate Jul 25, 2020

archibate mentioned this pull request Jul 25, 2020

[ipython] [refactor] Misc tweaks to make #1308 easier to review #1584

Merged

archibate changed the base branch from master to archibate-idle July 25, 2020 06:05

archibate changed the base branch from archibate-idle to master July 26, 2020 09:08

Revert "revert"

8133fcd

This reverts commit 202c235.

archibate force-pushed the dillidle branch from 5627d16 to 8133fcd Compare July 26, 2020 09:11

FantasyVR mentioned this pull request Jul 29, 2020

[release] v0.6.23 #1607

Merged

archibate requested review from isdanni and JYLeeLYJ August 6, 2020 07:41

isdanni reviewed Aug 7, 2020

View reviewed changes

[skip ci] Update python/taichi/idle_hacker.py

d04fbf7

Co-authored-by: Danni <38550500+isdanni@users.noreply.github.com>

archibate added the wontfix We won't fix this issue or merge this PR label Aug 8, 2020

archibate mentioned this pull request Aug 8, 2020

[IPython] Use 'sourceinspect' to support Blender scripting module and IDLE shell #1664

Merged

archibate closed this Aug 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[IPython] Support IDLE and Blender scripting module as interactive shell #1308

[IPython] Support IDLE and Blender scripting module as interactive shell #1308

archibate commented Jun 23, 2020 •

edited

Loading

codecov bot commented Jul 13, 2020 •

edited

Loading

archibate Jul 14, 2020

isdanni Aug 7, 2020

isdanni Aug 7, 2020

archibate Aug 7, 2020

isdanni Aug 7, 2020

archibate commented Jul 14, 2020

Eydcao commented Jul 16, 2020 •

edited

Loading

archibate commented Jul 23, 2020

archibate commented Jul 24, 2020

EaryChow commented Jul 24, 2020

archibate commented Jul 24, 2020

yuanming-hu commented Jul 25, 2020

isdanni left a comment

isdanni Aug 7, 2020

isdanni Aug 7, 2020

archibate commented Aug 8, 2020 •

edited

Loading

[IPython] Support IDLE and Blender scripting module as interactive shell #1308

[IPython] Support IDLE and Blender scripting module as interactive shell #1308

Conversation

archibate commented Jun 23, 2020 • edited Loading

codecov bot commented Jul 13, 2020 • edited Loading

Codecov Report

archibate Jul 14, 2020

Choose a reason for hiding this comment

isdanni Aug 7, 2020

Choose a reason for hiding this comment

isdanni Aug 7, 2020

Choose a reason for hiding this comment

archibate Aug 7, 2020

Choose a reason for hiding this comment

isdanni Aug 7, 2020

Choose a reason for hiding this comment

archibate commented Jul 14, 2020

Eydcao commented Jul 16, 2020 • edited Loading

archibate commented Jul 23, 2020

archibate commented Jul 24, 2020

EaryChow commented Jul 24, 2020

archibate commented Jul 24, 2020

yuanming-hu commented Jul 25, 2020

isdanni left a comment

Choose a reason for hiding this comment

isdanni Aug 7, 2020

Choose a reason for hiding this comment

isdanni Aug 7, 2020

Choose a reason for hiding this comment

archibate commented Aug 8, 2020 • edited Loading

archibate commented Jun 23, 2020 •

edited

Loading

codecov bot commented Jul 13, 2020 •

edited

Loading

Eydcao commented Jul 16, 2020 •

edited

Loading

archibate commented Aug 8, 2020 •

edited

Loading