Precompiled brython script measurements

[ed2050]

As title says, this post is about time measurements for precompiled brython scripts.

Rationale

As Pierre said in another discussion:

It would sure be interesting to develop eg a Flask app that does what you suggest, and measure the improvement on the overall speed compared to the current situation (with the indexedDB cache)

Before building such an app, I wanted to measure how much time is saved by using precompiled brython scripts (i.e. running brython.pythonToJS () on the python code and loading the resulting js code in the web page, instead of having brython parse and generate each time).

Some of this can already be achieved with exec option <brython-options debug="1" cache="true"> and with compiling stdlib modules. But there's a gap on first script load these tools don't cover. The savings can potentially be significant: think hundreds or thousands of users a day on an internal network visiting a brython page.

Data

Let's get straight to the data. Methodology is below for anyone who wants details.

I tested 5 scripts of python code of varying sizes. First 2 from Gallery on brython website, other 3 from pypi projects (they aren't brython code per se, just straight python, but it doesn't matter. the scripts don't need to run successfully, just be parsed and generate js).

Script	Code	Line Count	Compile time
pie chart	source	128	75
3d walker	source	398	123
bs4 element	source	2430	228
pyparsing	source	7107	502
tifffile	source	24523	1151

• Script name, Code, and Line Count should be self explanatory
• Compile time is extrapolated time in ms it takes brython.js to parse the script and generate js code, i.e. the amount that could be saved by precompilation. See Methodology below for more explanation how this was calculated.

Data gathered on chrome with webserver running on localhost.

Observations

1. Brython parsing scales well. There's a fixed penalty to parsing and generating code, so short files like pie chart have a high cost relative to their line count. But overall the fixed penalty is quite modest <100ms. Larger files compile much faster, with cost decreasing faster than line count increases (appears logarithmic).

2. Despite this, compile time becomes noticeable around 5000 lines when it approaches nearly half a second. At 20,000 lines parsing time is nearly 1 second.

3. Let's consider energy usage across a network of users. A 500 ms delay for 1000 users is 500 secs of duplicated processor time, when the script could be compiled once on the server instead of separately in each user's browser. Intel CPUs can consume 100 W more power at high load, let's assume +50W for brython parsing. That's 0.007 kwh per day x 250 working days = 1.7 kwh per year. Not a significant amount until you get into millions of users (1 million users a day = 1,736 kwh per year x 0.40 $ / kwh = $694). Negligible in most cases.

4. Precompilation only seems to be worth it under these conditions:

   • lines of brython code is rather large
   • savings of 0.5 to 1.0 seconds in startup time would be a noticeable improvement, considering other sources of delay like network transit, DOM rendering, etc
   • most page visits are first time users or once-per-day users, where enabling indexedDB cache won't help

Methodology

Measuring the time it takes a script to compile is tricky without access to brython internals. In theory, I could read through brython.js source, find the entry and exit points for parsing and js code generation, and measure time between those points. I tried that approach and quickly gave up as it takes too long in practice.

Instead I decided to measure two points as a proxy:

1. The last moment before brython.js runs, which happens at the DOMContentLoaded event.
2. The first moment code in the user script runs, which happens after brython compiles the script.

To measure #1, I inserted this at the bottom of each html page in the test, right before the closing tag:

<script type="text/javascript">
    start = Date.now () ;
    console.log ('page loaded : ' + start) ;
</script>

To measure #2, I inserted this code at the top of every brython script in the test:

from browser import window, console
stop    = window.Date.now ()
elapsed = stop - window.start

console.log (f'script loaded : { stop }')
console.log (f'TIME elapsed  : { elapsed } ms')

These points aren't perfect. A bit more happens after the start time before brython is invoked: browser finishes parsing DOM, might do cleanup, might fetch remote resources, etc, before issuing DOMContentLoaded event and invoking brython (). And brython does more than just parse the user script and generate js code: general setup and bookkeeping, environment detection, searching DOM for script tags, among other tasks.

To account for this, I took time measurements for both the raw (normal) brython script and the precompiled js version. The difference between them should reflect the time it takes brython to parse and compile the script. Here's the full version of the table above, showing the values Compile time was derived from.

Script	Code	Line Count	Compile time	Raw ms	Precomp ms
pie chart	source	128	75	78	3
3d walker	source	398	123	127	4
bs4 element	source	2430	228	234	6
pyparsing	source	7107	502	564	62
tifffile	source	24523	1151	1246	95

• Script name, Code, and Line Count should be self explanatory
• Compile time is extrapolated time in ms it takes brython.js to parse the script and generate js code - i.e. the amount that could be saved by precompilation. Computed as difference between Raw ms and Precomp ms.
• Raw ms is the time to load the script normally, where brython.js processes <script type="text/python">
• Precomp ms is the time to load the script when already precompiled to js

For sake of completeness:

• tests were with brython 3.13.0
• server was fastapi running on localhost
• compiled js versions were generated with Editor page on brython website. Paste in brython script, then hit javascript button to get compiled js.
• these lines were manually added at the top of compiled js to get it to run with brython:

__BRYTHON__.imported["exec"] = {};
__BRYTHON__.frames_stack = [];

• compiled script was included in the page as <script src="script.py.js" type="text/javascript" defer>. note that defer is required to start script after DOM loads (and after brython () starts)
• huge thanks to @denis-migdal for walking me through the steps to get compiled brython code to actually run

Conclusion

This post is a bit long, but I wanted to thoroughly describe what I did. Hope it makes sense for anyone interested.

[denis-migdal]

Thanks for your measurements.

I think you should instead :

• measure the time taken by pythonToJS().
• measure the size of the input and output files in kB (not in lines) [knowing that the files has comments]
• measure the size of the input and output files compressed (lzma or br).
• compute the tradeoff between pre-transcription time gain and the generated file extra download time (compressed).

Also one good thing with pre-transcription is that you don't need to whole Brython.js to run it (>600kB removed?). So less to download, and to startup.

[denis-migdal]

* pythonToJS () is just part of the time brython takes loading scripts.  It also searches DOM for script tags and does other processing on them. 

Well, pre-transcripted or not, you'd always need to load Brython for the runtime, so Brython will always search DOM for script tags and do other operation.

Also, avoid doing a console.log inside code you are benchmarking (this is a very slow operation), avoid also having the developer tools opened (it also slow execution time).

* Compression time is interesting but a separate topic.  Compression is optional, webmaster can turn it on or off.  It's not necessary to run precompiled brython.  And it varies from user to user, depending on client processor, memory, load, etc.

I'm not talking about compression time, but about the dataload on the network.
The generated JS is ~5-6x bigger than the original source file. This has a cost.

* For a true accounting of all network factors, you would have measure in actual conditions your users encounter: latency, bandwidth, server distance, packet loss & retransmission, network congestion, etc, etc.  Those vary from user to user.  Very hard to recreate real conditions.

You do not need anything complex, e.g.:

• I win 500ms by pre-transcripting, but generate 1MB more raw data, so +166kB to download once compressed.
• Meaning that if my connection speed is < 332kB/s, I may be loosing time.

Ofc there might be variations.
You'd likely use defer to download all scripts while the browser is loading the DOM, etc.
But it gives you an indication of the tradeoff.

For web serving, compression of small text files is about saving bandwidth for provider serving many clients, not saving load time on one individual client.

Brython stdlib is ~4MB uncompressed.
If you have a 300Mb/s connection, i.e. 37.5MB/s, it'd likely load in > 107ms.
This is way too slow.

Compressed, it'd load ~5.6x faster, so ~19ms. That's still quite slow, but better.
Decompression is often at several GB/s (so < 4ms ?), and can be done in parallel of loading things and other operations.

[ed2050]

Thanks for the feedback. 😊 Discussion is interesting. We come from different needs and perspectives.

If the data you want can be measured simply, I'm open to collecting it. The complexity worries me though.

• pythonToJS () is just part of the time brython takes loading scripts. It also searches DOM for script tags and does other processing on them.

Well, pre-transcripted or not, you'd always need to load Brython for the runtime, so Brython will always search DOM for script tags and do other operation.

There's a fixed component and a per-script component to brython. Yes you pay fixed cost in both. My measurements capture the fixed cost in both cases and subtract it out. Variable costs include:

• searching the DOM for "text/python" scripts (querySelectAll may return faster when there are no results i.e. precompiled scripts are text/js not text/python)
• extracting the code of those scripts. brython has to fetch src url via ajax. browser also has to fetch src of precompiled js scripts, but it can be done in parallel while rest of page loads. brython interpreter has to wait for ajax results. even for inline scripts, script.innerText has a cost.
• brython seems to do some internal bookkeeping to keep track of scripts, whether they're loaded, whether they've been run, etc. this is probably de minimis compared to parse and generate time, but it's there.

Also, avoid doing a console.log inside code you are benchmarking (this is a very slow operation), avoid also having the developer tools opened (it also slow execution time).

Good to know. I don't think it significantly affects my results, because :

• there's only one console.log inside the timing measurements (between start time and stop time)
• console.log happens in both raw and precompiled versions, so it gets subtracted out (e.g. it's part of the 3ms load time for precompiled pie chart)

If console.log has a variable delay, or if there are more in one test than another, then it would bias results. Anyway my measurements are an upper bound on time savings, to see if it's worth proceeding further.

• Compression time is interesting but a separate topic. Compression is optional, webmaster can turn it on or off. It's not necessary to run precompiled brython. And it varies from user to user, depending on client processor, memory, load, etc.

I'm not talking about compression time, but about the dataload on the network.
The generated JS is ~5-6x bigger than the original source file. This has a cost.

Yes everything has a cost. Processor time to compress source (hopefully cached by server). Network time to transmit data, whether compressed or not. Processor time to decompress. The question is are these costs significant enough to be worth measuring separately?

• I win 500ms by pre-transcripting, but generate 1MB more raw data, so +166kB to download once compressed.

• Meaning that if my connection speed is < 332kB/s, I may be loosing time.

Also have to account for compression time on server (possibly cached) and decompression time. Though those are usually much smaller than network time. 332 kb/s is rather slow in many scenarios these days.

But it all depends on client. A low-spec mobile phone with a slow processor may find compression more burden than a bigger download. Hard to say without knowing the environment.

Ofc there might be variations.
You'd likely use defer to download all scripts while the browser is loading the DOM, etc.
But it gives you an indication of the tradeoff.

I completely agree that ideally you want to measure all components separately to figure out what improvements are best for any particular scenario. In practice, getting all those precise measurements and making them generalizable across many different environments is quite a bit of work. I'm looking for measurements that are easy to implement.

Personally I'm concerned about end to end time. If compressed vs uncompressed saves 50 ms then I don't consider that significant for my purposes (for one-off events. if you have 20 such events per page then it matters). I understand why system designers would want to know and save that time.

Brython stdlib is ~4MB uncompressed.
If you have a 300Mb/s connection, i.e. 37.5MB/s, it'd likely load in > 107ms.
This is way too slow.

Our perspectives are very different. 😄 To me 100ms is nothing, end user won't even notice. Plus transmission time will probably be swamped by time to execute and load stdlib in this scenario.

I get your point though. 30Mb/s is reasonable these days, and theoretical transmission time could be around 1000 ms in such circumstances (real world performance may be significantly better when you factor in burst speed, local CDN caches, etc. Or much worse with lost packets, dropped TCP connections, and the like. so hard to predict). 1 second delay is very significant.

I was thinking user scripts not stdlib. My largest example above is 848k. Personally my largest brython scripts are around 10k lines counting all modules. Maybe 250k bytes. Different class than stdlib.

Anyway stdlib is already optimized as js, and can be optimized further with brython-cli make_modules to reduce the size greatly. I know you're just making a point about compression. But seems very rare a brython app needs to download more than 1MB of code (not counting
interpreter) unless it naively includes all of stdlib.

[denis-migdal]

To me 100ms is nothing, end user won't even notice.

End user will notice it, but may not consciously care.
However, if your page startup time is >100ms, you'll get penalized at the SEO level.

Ofc, you'll have sometime worse Internet, e.g. on mobile phone.

My largest example above is 848k

Of Python code ? Once converted to JS it should be 5-6 times bigger (ofc excluding the comments).

But seems very rare a brython app needs to download more than 1MB of code

You may write big app in Brython. I have some code written in JS (not in Brython), that can easily weight 10MB.
If it was written in Brython, the generated JS would likely weight 50MB.

[ed2050]

if your page startup time is >100ms, you'll get penalized at the SEO level.

Do they include js in those measurements? Would brython scripts affect "startup time" however that's measured for SEO purposes?

You can always improve startup by preloading static content and delaying dynamic scripts to run later. Seems easy to get around. But I don't deal with SEO, so maybe not.

---

Of Python code ? Once converted to JS it should be 5-6 times bigger (ofc excluding the comments).

Good point

You may write big app in Brython. I have some code written in JS (not in Brython), that can easily weight 10MB. If it was written in Brython, the generated JS would likely weight 50MB.

Yes, I know web apps that size exist. I just didn't expect brython apps to reach that size, given lack of commercial support. But I'd love to be wrong. Let's shoot for the moon! 🙂

[denis-migdal]

Do they include js in those measurements? Would brython scripts affect "startup time" however that's measured for SEO purposes?

They measure the time it takes, from the browser asking for the page, to the time the document is ready to interact with the user.

delaying dynamic scripts to run later.
This is indeed a common strategy, but it depends on what your script is doing.

Although other metrics are used (not all are public/known).

[PierreQuentel]

most page visits are first time users or once-per-day users, where enabling indexedDB cache won't help

This assertion is at least surprising (if it was true, why are there browser caches ?) and a little unfair in the context of this discussion, since it allows you to ignore the built-in tools used by Brython to avoid explicit pre-translation to Javascript.

[ed2050]

most page visits are first time users or once-per-day users, where enabling indexedDB cache won't help

This assertion is at least surprising (if it was true, why are there browser caches ?) and a little unfair in the context of this discussion, since it allows you to ignore the built-in tools used by Brython to avoid explicit pre-translation to Javascript.

I'm not asserting that it's true in general. I'm stating conditions where precompilation would have a significant benefit. Whether it applies to any particular project depends on circumstances.

Many sites like google and social media thrive on repeated visits per day. Others like wikipedia get a mix of traffic: heavy regular users and sporadic visits by occasional users. And some sites primarily get sporadic visits.

Example

Think of a payroll app where users have to login at the end of a biweek to record their time and get paid (I've worked at several places with systems like this). 13 days in a row the site gets almost no visits. Then on the due date, the site gets 13,000+ visitors all visiting once for a single transaction (to enter their hours). Assuming computers reboot at least weekly, then browser caches will be empty.

In general, I agree - indexedDB cache probably applies in most cases. I'm simply looking at cases where it doesn't help.

[PierreQuentel]

I'm not asserting that it's true in general. I'm stating conditions where precompilation would have a significant benefit. Whether it applies to any particular project depends on circumstances.

Oh yes, sorry, it was clear in your initial post

[PierreQuentel]

@ed2050 I tested the biggest of your examples (tifffile.py)

There was a specific issue with multi-line annotations, which are present in many places in this script. I fixed it in commit 0d4fe31.

Then I tried another approach to generate the tokens fed to the parser. So far it was done by a Javascript generator that yields tokens or raises an error; replacing it by a function that returns the list of tokens, stopping if an error is detected, reduced the parsing time by around 30%. This is implemented in commit 003c1c7.

With these commits the compile time of this script should be significantly smaller. The second commit should also bring visible improvements for scripts that have a few thousands of lines.

[PierreQuentel]

I measure the time taken to process tifffile.py with this script:

var $B = __BRYTHON__
var xhr = new XMLHttpRequest()
xhr.open('GET', 'tifffile.py')
xhr.onreadystatechange = function(ev){
  if(this.readyState == 4){
    var src = this.responseText,
        filename = "test",
        imported,
        timer = window.performance

    var t0 = timer.now()
    var tokens = $B.tokenizer(src, 'test')
    console.log('tokenizer time', timer.now() - t0, 'for', tokens.length, 'tokens')
    
    var t0 = timer.now(),
      nb = 0
    var parser = new $B.Parser(src, filename, 'file'),
        _ast = $B._PyPegen.run_parser(parser)

    var t1 = timer.now()
    console.log('parse_time, including tokenizer', t1 - t0)
    var future = $B.future_features(_ast, filename)
    var t2 = timer.now()
    console.log('future time', t2 - t1)

    var symtable = $B._PySymtable_Build(_ast, filename, future)
    var t2 = timer.now()
    console.log('symtable time', t2 - t1)
    var js_obj = $B.js_from_root({ast: _ast,
                                  symtable,
                                  filename,
                                  src,
                                  imported})
    var t3 = timer.now()
    console.log('ast to js time', t3 - t2)
    console.log('TOTAL', t3 - t0)
  }
}
xhr.send()

Here are the results on my PC (Windows 10, Chrome)

tokenizer time 488.69999998807907 for 147506 tokens
parse_time, including tokenizer 1288.2999999821186
future time 0.20000001788139343
symtable time 87.90000000596046
ast to js time 175.5
TOTAL 1551.699999988079

[denis-migdal]

Oh thanks a lot, that's exactly the code I needed for some of my benchmarks ^^.
I'll also have to update the Brython version I'm currently using in my tests.

The fact I can get a token list would help me a lot to implement my own ASTNode generation without requiring the current conversions I'm making.

However, I'll have lot of work for ~5 weeks, so I won't be able to work on it for quite some time.

[ed2050]

Had to lookup what those numbers mean. window.performance.now () uses ms resolution for integer part. So 1551.69 is ms = 1.551 sec.

I'm curious, what type of processor did you run those tests on?

That's your 30% faster code. So old parser code was prob around 2.2 sec on your machine. Why so much slower than my numbers above? Mine was 1.1 secs for all the steps you measured and more (since my measurement points were less precise than yours).

My machine is relatively old, a 2018 mac with 3 GHz Intel i5. Even going back a few more cpu generations shouldn't halve performance. Could a memory bottleneck be causing page swaps or something?

[PierreQuentel]

Yes, I saw the figures in your tests and thought that you had a much more powerful computer than mine... My PC is also relatively old (2018 too), with Intel i5 at 2,5 GHz and 8 MB of RAM. When I run these tests the CPU and memory usage remain low, there is no sign of swapping.

[ed2050]

8 MB of RAM, well there's your problem. 😉 I know you mean GB.

Looking at geekbench data for 2018 core i5 single core results:

• 2.3 GHz usually runs 1100-1200 (couldn't find 2.5 GHz)
• 3.0 GHz is usually 1250-1350

Sizable difference, but even at the outer bounds there shouldn't be x2 factor in performance. Maybe that's the "Windows tax" at work. 🙂

[PierreQuentel]

Not directly related to this discussion, but in recent commits I have modified ast_to_js.js to generate indented Javascript code. This has a very limited impact on performance (less than 25ms for the huge tifffile.py example). The indentation level defaults to 2 and can be set by a new option called js_tab