Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[doc] Add user utilities doc #1295

Merged
merged 18 commits into from
Jul 13, 2020

Conversation

Rullec
Copy link
Contributor

@Rullec Rullec commented Jun 20, 2020

Related issue = Close #1269

[Click here for the format server]

This draft PR is for user utilities. I simply add a user_utilities in MISCELLANEOUS part. Current Changes shows below.

  1. More Explanations on ti.info, this logging module
  2. Document our Profiler module for the first time
  3. say something about assertation: finished in [Lang] Add ti.static_assert for compile-time assertations #1344

I still need to do:

1. Logging

  1. I am not satisfied with ti.info this part. I think it can do more, for example display a tensor's value (under the help of precessor AST)
  2. A potential bug in ti.error or ti.critical, maybe we need to add a test script for this functionality.
  3. ti.critical and ti.CRITICAL are missing, why?
  4. ti.info is only executed once in compile-time. So what are the suitable scenarios for this command?: WIP, the same reason as ti.static_print in [Lang] Add ti.static_assert for compile-time assertations #1344
  5. ti.static_print is executed in compile-time as well. What are the suitable scenarios? : finished in [Lang] Add ti.static_assert for compile-time assertations #1344
  6. I think we need to say that print doesn't support f-string, otherwise lots of guys need to try it and get confused. the same as ti.info and ti.static_print: maybe finished in [Lang] Add ti.static_assert for compile-time assertations #1344

2. Profiling

  1. Our profiler is a little weak relatively. It can not do some very basic profiling
  2. Profiler doesn't work in ti.cpu: Need to test.
  3. Two profilers are needed to be documented.
  4. To distinguish our 2 profilers ScopedProfiler and ProfilerBase [perf] Refactor kernel profiler #1261, I used these 2 names directly in this doc. But they are hidden deeply in Taichi's core and should not be exposed to the end-user. Is there a better way to do that?
  5. We need to explain the output of profilers, explain what these statistics stand for in detail. (Sadly I cannot understand all of them as well.)

3. Asserting

  1. doc python built-in assert
  2. implement ti.static_assert. What feature do we want exactly in ti.static_assert, I am a little confused... :( Is it an assertation that works in runtime as well??: finished in [Lang] Add ti.static_assert for compile-time assertations #1344
  3. doc ti.static_assert. : finished in [Lang] Add ti.static_assert for compile-time assertations #1344

All of these conculsions are my own and only temporary. Some of them may not be true but I need to verify it again and dig deeper.

@codecov
Copy link

codecov bot commented Jun 20, 2020

Codecov Report

Merging #1295 into master will increase coverage by 0.57%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1295      +/-   ##
==========================================
+ Coverage   85.57%   86.14%   +0.57%     
==========================================
  Files          19       19              
  Lines        3368     3543     +175     
  Branches      623      631       +8     
==========================================
+ Hits         2882     3052     +170     
- Misses        356      363       +7     
+ Partials      130      128       -2     
Impacted Files Coverage Δ
python/taichi/lang/common_ops.py 91.90% <0.00%> (-1.97%) ⬇️
python/taichi/lang/expr.py 88.95% <0.00%> (-0.87%) ⬇️
python/taichi/lang/ops.py 93.12% <0.00%> (-0.63%) ⬇️
python/taichi/lang/snode.py 87.03% <0.00%> (-0.47%) ⬇️
python/taichi/lang/transformer.py 93.62% <0.00%> (-0.02%) ⬇️
python/taichi/lang/impl.py 90.54% <0.00%> (+0.45%) ⬆️
python/taichi/lang/util.py 64.70% <0.00%> (+0.63%) ⬆️
python/taichi/lang/matrix.py 91.41% <0.00%> (+0.70%) ⬆️
python/taichi/lang/kernel.py 81.57% <0.00%> (+0.74%) ⬆️
python/taichi/lang/__init__.py 78.60% <0.00%> (+3.25%) ⬆️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c83a7e3...5436c9d. Read the comment docs.

@Rullec Rullec changed the title [doc] add user utilities doc [doc] Add user utilities doc Jun 21, 2020
@@ -0,0 +1,109 @@
User utilities
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: Debugging utilities

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good suggestion

@ti.kernel
def compute():
var[0] = 1.0
ti.info(f"set var[0] = 1.0")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's be clear that this is only called at compile-time. i.e. If you recall compute, it won't reprint the string. (same as ti.static_print)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree.

I have an observation, would you please help me to check whether my understanding is correct?

According to the Debugging part in this doc, It seems that all of the print will be transfered to ti.static_print by the AST preprocessor, so that they will be executed in run-time. That's good.

But it doesn't include logging utils such ti.info, these ti.info are only called at compiled-time but not runtime.


So we cannot do any logging ti.info in kernel runtime, right? :( only print is supported in runtime is a little trivial, and IMO the debugging experince can be nicer if we support ti.info in runtime.

Basically, why wasn't ti.info be supported in runtime? It looks not very hard. Would you be happy to introduce the idea behind this design? :D

Copy link
Collaborator

@archibate archibate Jun 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks not very hard

No, copying strings from GPU to CPU is harder than you may think. Ancient GPUs doesn't even support integer, not to mention f-string...

introduce the idea behind this design?

The OpenGL backend for example, maintains

  1. An array of integer on GPU end (to prevent string operation on GPU)
  2. An string table on CPU end (CPU is rich, able to perform string operation)

The integers in GPU end are indices to the string table.
And when GPU kernel ends, we copy the array of integer, and according to string table, translate back to real strings.

However, ti.info needs f-string to work (e.g. ti.info("x = {:02x}", x)), which needs formatting integer to string according to the string content, this could obviously break the string table attempt.


In fact, the only difference between ti.info and print is its color and f-string.. I think print('x =', x) is good enough, not sure why ti.info superior.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your clear explanation! I get the point. Yes, I fully understand your meaning. ;-)

The difference is just the timestamp(color is relatively unimportant). Let's assume we want to do a simulation which can cost for a whole day (It's quite common in mechanical and chemstry simulation, most of them are in reseach cases), If we want to find the bug after waking up in the morning, a timestamp or screenshot can be helpful IMAO.

Copy link
Collaborator

@archibate archibate Jun 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think they could do ti.info between kernel launches.
In fact, print in kernel also won't print until kernel ends as my explanation goes above.
So there's no difference for this usage.

If you want a timestamp in print, I think print(ti.timestamp(), 'x =', x) is a good approach (where ti.timestamp() is a runtime timestamp getter TBD).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

I will invesigate this case further more. I need to get a global view of Taichi in order to understand it, instead of floating on the surface. XD

docs/user_utilities.rst Outdated Show resolved Hide resolved
docs/user_utilities.rst Outdated Show resolved Hide resolved

TODO:

1. add ``ti.static_assert``
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also add non-static assert doc.


1. more explanations and examples in this logging section. can we log a taichi tensor here, why and how?

2. ``ti.error`` and ``ti.critical`` tested fail. I need to check it again
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In fact, ti.error will purposely terminate the program, IIRC.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think you are correct according to my experiment. ti.error will cause a crash.

I can fully understand this design, but not very agree with it. Our loggging utility should focus on logging but not try to control the workflow of ANY Taichi program.

For example, If I say:

ti.error("here is an error!")

I simply want to show this error to the screen but not crash in this place. It's also spdlog recommending.

HDYT?

Copy link
Collaborator

@archibate archibate Jun 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That could be hard, since there's already many C++ code assuming TI_ERROR to be program-terminating. IMHO if user want to show error without terminate, they may use ti.warn or ti.critical instead.
Or, if you insist to, feel free to add another higher-level like TI_PANIC in another PR (not this PR).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, so let's keep ti.error terminte the program and I will documted it on stress emphasize. :D

@yuanming-hu
Copy link
Member

Sorry about my delayed comments.

  • Our profiler is a little weak relatively. It can not do some very basic profiling
  • Profiler doesn't work in ti.cpu

We have two sets of profilers: #1261

  • implement ti.static_assert

This will be very helpful!

  • ti.critical and ti.CRITICAL are missing, why?

I guess that's because we rarely use them. For now, ti.error seems to be the highest level of erroneous logging. Adding critical would be helpful. Thank you!

@Rullec
Copy link
Contributor Author

Rullec commented Jun 25, 2020

Thanks a lot for your powerful guide! So that I can investigate these 2 profilers deeply and documente them!

Agree, I also think ti.ERROR is enough for us because it even can crash the program :O

Btw, #1312 simply fix some typos in export_results.rst and the changeset is quite small. Would you be happy to check that in your spare time?

@Rullec
Copy link
Contributor Author

Rullec commented Jun 26, 2020

We have two sets of profilers: #1261

The design of ScopedProfiler is so exquisite! Very enjoyable to read it.

But I can only see its work in a single thread. Would you please give me some guides on a multi-threads case?

@Rullec
Copy link
Contributor Author

Rullec commented Jun 26, 2020

Update:

First sorry: I do git merge --rebase master in my local repo in order to catch up with the latest change, but now the commit history is messed out. Do I need to fix it? If yes, what can I do? :(

Here is a new version user_utilities.rst! Though it is far less perfect, but it covers more than the latter version.

I met with some new questions and got stuck when writing this doc. @archibate Would you be happy to give me some guidence? The questions are maintained in the beginning part. Thanks in advance!

@archibate archibate self-requested a review June 27, 2020 16:51
@archibate
Copy link
Collaborator

archibate commented Jun 27, 2020

On my way!
EDIT: a little tricky, will first go fix this tmr, good night!

@Rullec
Copy link
Contributor Author

Rullec commented Jun 28, 2020

On my way!
EDIT: a little tricky, will first go fix this tmr, good night!

#1344 is very amazing! So this issue #1295 can simply focus on the other 2 parts. :-)

@archibate archibate force-pushed the add_user_utilities_doc branch from e87630e to f3ef2d6 Compare June 28, 2020 02:50
@archibate
Copy link
Collaborator

@Rullec Fixed now, run git pull --force to go ahead.

@Rullec
Copy link
Contributor Author

Rullec commented Jun 28, 2020 via email

@Rullec
Copy link
Contributor Author

Rullec commented Jun 29, 2020

Update: Now there is a new version. Would be please give me some suggestions?

@Rullec Rullec marked this pull request as ready for review June 29, 2020 05:19
@Rullec
Copy link
Contributor Author

Rullec commented Jul 1, 2020

Hi all,
Is there anything I can improve now? Suggestions are welcome!

import taichi as ti

ti.init()
ti.set_logging_level(ti.ERROR)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove this or user would think loglev has to be error to make error work.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great!

compute()
ti.core.print_profile_info()

``ti.core.print_profile_info()`` will output statistics in a hierarchical format, with different colors for different levels.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OFT: Can we make this ti.print_profile_info?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, update comming soon.

::

[ 22.73%] jit_evaluator_0_kernel_0_serial min 0.001 ms avg 0.001 ms max 0.001 ms total 0.000 s [ 1x]
[ 0.00%] jit_evaluator_1_kernel_1_serial min 0.000 ms avg 0.000 ms max 0.000 ms total 0.000 s [ 1x]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OFT: Can we make jit&accessor kernels invisible by default? They are usually 0 and confusing to end-users.

.. code-block :: python
:emphasize-lines: 1, 9

ti.init(ti.cpu, kernel_profiler = True)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please leave no space around =, see yapf.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, a formatter has been introduced in now.

@archibate
Copy link
Collaborator

Hello? @Rullec are you still there?

@Rullec
Copy link
Contributor Author

Rullec commented Jul 5, 2020 via email

@Rullec
Copy link
Contributor Author

Rullec commented Jul 5, 2020

Update:

  1. Move all contents to utilities.rst
  2. fix some typos

Copy link
Member

@yuanming-hu yuanming-hu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool! LGTM in general. I took a pass and fixed a few wording issues. Please take your time and no rush at all.

Hello? @Rullec are you still there?

@archibate please be courteous and friendly. We are an open-source community and nobody is obligated to contribute. Keep in mind that most people have their daily work to do. Thanks! :-)

docs/utilities.rst Outdated Show resolved Hide resolved
docs/utilities.rst Outdated Show resolved Hide resolved
docs/utilities.rst Outdated Show resolved Hide resolved
docs/utilities.rst Outdated Show resolved Hide resolved
docs/utilities.rst Outdated Show resolved Hide resolved
docs/utilities.rst Outdated Show resolved Hide resolved
docs/utilities.rst Outdated Show resolved Hide resolved
docs/utilities.rst Outdated Show resolved Hide resolved
docs/utilities.rst Outdated Show resolved Hide resolved
docs/utilities.rst Outdated Show resolved Hide resolved
@Rullec
Copy link
Contributor Author

Rullec commented Jul 7, 2020

Cool! LGTM in general. I took a pass and fixed a few wording issues. Please take your time and no rush at all.

Hello? @Rullec are you still there?

@archibate please be courteous and friendly. We are an open-source community and nobody is obligated to contribute. Keep in mind that most people have their daily work to do. Thanks! :-)

Thanks both @archibate and @yuanming-hu , for your kindly comprehension and all of these powerful modifications. :D

I will consider how to revise this doc again, and hopefully a new version can come soon.

Rullec and others added 2 commits July 9, 2020 00:31
updated by yuanming

Co-authored-by: Yuanming Hu <yuanming-hu@users.noreply.github.com>
@Rullec Rullec requested review from archibate and yuanming-hu July 8, 2020 16:51
@Rullec
Copy link
Contributor Author

Rullec commented Jul 8, 2020

Hi all, here is a new version. Most of problems we mentioned above has been treated in it.

But I still have a question, @yuanming-hu said that ti.info should not be used in ti.kernel, would you please tell me the story behind it? Because it doesn't throw an exception and also works the same as my expection. Thanks!

@archibate
Copy link
Collaborator

#1295 (comment)

Just invoke the kernel twice, and you'll find it's only printed once.

@Rullec
Copy link
Contributor Author

Rullec commented Jul 8, 2020

Just invoke the kernel twice, and you'll find it's only printed once.

But isn't that what we want? For example,

import taichi as ti
ti.init(arch=ti.cpu)

ti.set_logging_level(ti.INFO)
mat = ti.var(dt=ti.f32, shape=(5, 5))


@ti.func
def calc(i: ti.int32, j: ti.int32):
    ti.info(f"set var in ti.func")
    mat[i, j] = i * j


@ti.kernel
def compute():
    calc(0, 0)
    calc(1, 1)
    calc(2, 2)
    ti.info(f"set var in ti.kernel")


compute()

output:

[I 07/09/20 01:29:58.039] [main.py:calc@11] set var in ti.func
[I 07/09/20 01:29:58.040] [main.py:calc@11] set var in ti.func
[I 07/09/20 01:29:58.040] [main.py:calc@11] set var in ti.func
[I 07/09/20 01:29:58.040] [main.py:calc@11] set var in ti.func
[I 07/09/20 01:29:58.041] [main.py:compute@22] set var in ti.kernel

It has been pointed out in our doc These functions should only be called at compile-time instead of run-time.
which indicates that ti.info inside of ti.kernel only runs for a single time when kernels are invoked, aka compile-time.

So I feel a little hard to understand why @yuanming-hu said using ti.info in ti.kernel is misleading :).

@archibate
Copy link
Collaborator

archibate commented Jul 9, 2020

import taichi as ti
ti.init(arch=ti.cpu)

ti.set_logging_level(ti.INFO)
mat = ti.var(dt=ti.f32, shape=(5, 5))


@ti.func
def calc(i: ti.int32, j: ti.int32):
    ti.info(f"set var in ti.func")
    mat[i, j] = i * j


@ti.kernel
def compute():
    calc(0, 0)
    calc(1, 1)
    calc(2, 2)
    ti.info(f"set var in ti.kernel")


compute()
compute()
compute()
compute()
compute()
compute()
compute()
compute()

users' expected output:

[Taichi] mode=development
[Taichi] preparing sandbox at /tmp/taichi-cudg5ylh
[Taichi] <dev mode>, llvm 10.0.0, commit 8dbcc3b8, python 3.8.3
[Taichi] Starting on arch=x64
[I 07/09/20 11:52:15.340] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.341] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.342] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.342] [a.py:compute@19] set var in ti.kernel
[I 07/09/20 11:52:15.340] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.341] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.342] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.342] [a.py:compute@19] set var in ti.kernel
[I 07/09/20 11:52:15.340] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.341] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.342] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.342] [a.py:compute@19] set var in ti.kernel
[I 07/09/20 11:52:15.340] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.341] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.342] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.342] [a.py:compute@19] set var in ti.kernel
[I 07/09/20 11:52:15.340] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.341] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.342] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.342] [a.py:compute@19] set var in ti.kernel
[I 07/09/20 11:52:15.340] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.341] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.342] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.342] [a.py:compute@19] set var in ti.kernel
[I 07/09/20 11:52:15.340] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.341] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.342] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.342] [a.py:compute@19] set var in ti.kernel
[I 07/09/20 11:52:15.340] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.341] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.342] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.342] [a.py:compute@19] set var in ti.kernel

real output:

[Taichi] mode=development
[Taichi] preparing sandbox at /tmp/taichi-cudg5ylh
[Taichi] <dev mode>, llvm 10.0.0, commit 8dbcc3b8, python 3.8.3
[Taichi] Starting on arch=x64
[I 07/09/20 11:52:15.340] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.341] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.342] [a.py:calc@10] set var in ti.func
[I 07/09/20 11:52:15.342] [a.py:compute@19] set var in ti.kernel

@Rullec
Copy link
Contributor Author

Rullec commented Jul 9, 2020

Yes, It seems we reaches an agreement. Here is a new example, would it be better?

Copy link
Collaborator

@archibate archibate left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTMig + section position nits

print(err)


Profiler
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Profiler should be shared by devs and end-users, maybe we can have a separate .rst for it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO I think it's enough in utilities.rst at this moment.

@@ -1,25 +1,185 @@
Developer utilities
===================


This section provides a detailed description of some commonly used utilities for Taichi developers.

Logging
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, sorry about dismissing your changes, I have already done with Logging in #1475 :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May you merge these 2 changes into a single one? I also have no idea about how to deal with this conflict.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To prevent conflicts with #1475, you may move the Profiler to another section like profiler.rst :) I'll try to merge conflicts in Logging section once one of these two PR is merged.

@Rullec
Copy link
Contributor Author

Rullec commented Jul 13, 2020

Finished, as @archibate adviced, I remove all contents about logging :(. It is not a good feeling...I hope next time maybe I can avoid this duplicate work

Copy link
Collaborator

@archibate archibate left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank for breaking the changeset down! This really help me review quicker ❤️ LGTMig

Copy link
Member

@yuanming-hu yuanming-hu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool! Ready to merge after the wording issues are addressed. Thank you so much!

(The logging part is probably still useful as long as it's made more concise.)

docs/profiler.rst Outdated Show resolved Hide resolved
docs/profiler.rst Outdated Show resolved Hide resolved
@yuanming-hu yuanming-hu merged commit 5213b0d into taichi-dev:master Jul 13, 2020
@Rullec
Copy link
Contributor Author

Rullec commented Jul 14, 2020

Thank you all for your work as well!

Rullec added a commit to Rullec/taichi that referenced this pull request Jul 14, 2020
@Rullec Rullec deleted the add_user_utilities_doc branch July 14, 2020 03:35
@FantasyVR FantasyVR mentioned this pull request Jul 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Doc] add more details for debug utils and developer utils
4 participants