deprecation: remove all coffeescript from cocalc

[williamstein]

Motivation: Coffeescript is dangerous, as compared to Typescript.

What remains:

~/cocalc/src/packages$ git ls-files |grep -e cjsx$ -e coffee$
database/filesystem-bucket.coffee
database/postgres-base.coffee
database/postgres-blobs.coffee
database/postgres-ops.coffee
database/postgres-server-queries.coffee
database/postgres-synctable.coffee
database/postgres-user-queries.coffee
database/postgres-user-query-queue.coffee
frontend/assistant/actions.coffee
frontend/assistant/body.cjsx
frontend/assistant/common.coffee
frontend/assistant/dialog.cjsx
frontend/assistant/footer.cjsx
frontend/assistant/header.cjsx
frontend/assistant/legacy.cjsx
frontend/assistant/main.coffee
frontend/assistant/store.coffee
frontend/console.coffee
frontend/course/test/project-upgrades.coffee
frontend/editor.coffee
frontend/editor_history.coffee
frontend/editor_jupyter.coffee
frontend/printing.coffee
frontend/sagews/3d.coffee
frontend/sagews/d3.coffee
frontend/sagews/interact.coffee
frontend/sagews/sagews-eval.coffee
frontend/sagews/sagews.coffee
frontend/syncdoc.coffee
hub/access.coffee
hub/api/handler.coffee
hub/auth-token.coffee
hub/blobs.coffee
hub/client.coffee
hub/local_hub_connection.coffee
hub/metrics-recorder.coffee
hub/p.coffee
hub/password.coffee
hub/projects.coffee
hub/sign-in.coffee
hub/smc-manager.coffee
hub/user-query-stats.coffee
hub/user-remember-me.coffee
project/blobs.coffee
project/client.coffee
project/common.coffee
project/exec_shell_code.coffee
project/kucalc.coffee
project/port_manager.coffee
project/print_to_pdf.coffee
project/read_write_files.coffee
project/sage_session.coffee
project/watcher.coffee
util/test/defaults.coffee
util/test/key-value-store.coffee
util/test/misc-test.coffee
util/test/types.coffee

And the same, but with line counts, sorted from largest to smallest:

~/cocalc/src/packages$ git ls-files |grep -e cjsx$ -e coffee$
~/cocalc/src/packages$ git ls-files |grep -e cjsx$ -e coffee$ | xargs wc -l |sort -nr
   26969 total
    2947 database/postgres-server-queries.coffee
    2484 frontend/sagews/sagews.coffee
    2072 hub/client.coffee
    1888 frontend/editor.coffee
    1763 database/postgres-user-queries.coffee
    1494 frontend/editor_jupyter.coffee
    1139 database/postgres-base.coffee
    1041 util/test/misc-test.coffee
    1014 frontend/console.coffee
     964 frontend/sagews/3d.coffee
     887 database/postgres-blobs.coffee
     716 hub/local_hub_connection.coffee
     627 frontend/printing.coffee
     604 database/postgres-synctable.coffee
     591 frontend/editor_history.coffee
     589 project/client.coffee
     577 frontend/syncdoc.coffee
     478 frontend/sagews/interact.coffee
     352 hub/password.coffee
     327 hub/sign-in.coffee
     323 frontend/assistant/actions.coffee
     321 project/sage_session.coffee
     273 project/kucalc.coffee
     256 frontend/assistant/body.cjsx
     238 frontend/sagews/d3.coffee
     218 database/postgres-user-query-queue.coffee
     193 frontend/course/test/project-upgrades.coffee
     181 hub/metrics-recorder.coffee
     171 hub/api/handler.coffee
     168 database/postgres-ops.coffee
     163 util/test/types.coffee
     131 frontend/assistant/dialog.cjsx
     123 project/read_write_files.coffee
     122 hub/auth-token.coffee
     110 frontend/assistant/header.cjsx
     107 hub/projects.coffee
     103 util/test/defaults.coffee
     100 project/print_to_pdf.coffee
     100 frontend/assistant/store.coffee
      97 frontend/sagews/sagews-eval.coffee
      89 frontend/assistant/footer.cjsx
      81 hub/user-query-stats.coffee
      77 hub/access.coffee
      74 project/watcher.coffee
      73 hub/smc-manager.coffee
      70 project/blobs.coffee
      66 hub/blobs.coffee
      58 util/test/key-value-store.coffee
      51 project/exec_shell_code.coffee
      49 frontend/assistant/main.coffee
      48 database/filesystem-bucket.coffee
      45 hub/user-remember-me.coffee
      36 project/port_manager.coffee
      36 frontend/assistant/legacy.cjsx
      28 frontend/assistant/common.coffee
      21 project/common.coffee
      15 hub/p.coffee

It's still around 20K lines of code (or about 10% of the cocalc codebase):

~/cocalc/src/packages$ git ls-files |grep -e cjsx$ -e coffee$ | xargs cat | wc -l
   26969
~/cocalc/src/packages$ git ls-files |grep -e cjsx$ -e coffee$ | xargs cat | sort | uniq | wc -l
   17864
~/cocalc/src/packages$ git ls-files |grep -e ts$ -e tsx$ | xargs cat | sort | uniq | wc -l
  160965

[williamstein]

I asked ChatGPT for better code for counting non-blank non-comment lines of code in all our coffeescript files and it suggested this, which may be useful for keeping track of our status:

~/cocalc/src$ git ls-files | grep '\.coffee$' | xargs cat | grep -vE '^\s*$|^\s*#' | wc -l
20117

[williamstein]

Status after @haraldschilly did a ton of work on decaf of the project:

~/cocalc/src/packages$ git ls-files | grep '\.coffee$' | xargs cat | grep -vE '^\s*$|^\s*#' | wc -l
19256

[haraldschilly]

yes. next up is a "final push" for the project, then removing coffeescript itself from it, etc. IMHO the next logical step after this is to transcribe some coffee files in the hub, which are calling code in the project.

[williamstein]

Current status is actually getting worse than a year ago, somehow. The problem is that we have a bunch of important backend coffeescript code used by the hub still and sometimes have to improve or add to it. There's a lot of hard thankless work to be done...

~/cocalc/src$ git ls-files |grep -e cjsx$ -e coffee$ | xargs wc  |sort -nr
 21061  77495 856906 total
  2621  10204 107844 packages/database/postgres-server-queries.coffee
  2504   8879 100752 packages/frontend/sagews/sagews.coffee
  2149   7598  90197 packages/hub/client.coffee
  1790   5917  72479 packages/frontend/editor.coffee
  1779   7353  79724 packages/database/postgres-user-queries.coffee
  1146   4821  49855 packages/database/postgres-base.coffee
  1041   4205  41416 packages/util/test/misc-test.coffee
  1014   3207  38178 packages/frontend/console.coffee
   964   3325  36848 packages/frontend/sagews/3d.coffee
   910   3959  41383 packages/database/postgres-blobs.coffee
   716   2615  29355 packages/hub/local_hub_connection.coffee
   627   2220  27746 packages/frontend/printing.coffee
   604   2236  22604 packages/database/postgres-synctable.coffee
   577   1840  21900 packages/frontend/syncdoc.coffee
   478   1405  18103 packages/frontend/sagews/interact.coffee
   238    695   7678 packages/frontend/sagews/d3.coffee
   218    934   8800 packages/database/postgres-user-query-queue.coffee
   181    925   7936 packages/hub/metrics-recorder.coffee
   171    441   5084 packages/hub/api/handler.coffee
   168    681   6502 packages/database/postgres-ops.coffee
   163    617   6411 packages/util/test/types.coffee
   122    408   4512 packages/hub/auth-token.coffee
   108    428   4658 packages/hub/password.coffee
   107    296   3392 packages/hub/projects.coffee
   103    365   3278 packages/util/test/defaults.coffee
    97    366   3527 packages/frontend/sagews/sagews-eval.coffee
    81    305   3211 packages/hub/access.coffee
    81    225   2904 packages/hub/user-query-stats.coffee
    73    221   2652 packages/hub/smc-manager.coffee
    64    341   2902 packages/hub/blobs.coffee
    58    121   1525 packages/util/test/key-value-store.coffee
    48    141   1393 packages/database/filesystem-bucket.coffee
    45    132   1481 packages/hub/user-remember-me.coffee
    15     69    676 packages/hub/p.coffee

[williamstein]

In this branch #8346 after a massive effort, we are finally down to 17K lines:

~/cocalc/src/packages$ git ls-files |grep -e cjsx$ -e coffee$ | xargs wc  |sort -nr
 17322  64844 707802 total
  2518   9822 103958 database/postgres-server-queries.coffee
  2507   8894 100959 frontend/sagews/sagews.coffee
  1790   7420  80519 database/postgres-user-queries.coffee
  1732   5786  71010 frontend/editor.coffee
  1156   4888  50562 database/postgres-base.coffee
  1041   4205  41416 util/test/misc-test.coffee
   964   3325  36848 frontend/sagews/3d.coffee
   760   3389  34618 database/postgres-blobs.coffee
   704   2569  28709 hub/local_hub_connection.coffee
   627   2220  27746 frontend/printing.coffee
   604   2236  22604 database/postgres-synctable.coffee
   577   1840  21919 frontend/syncdoc.coffee
   478   1405  18103 frontend/sagews/interact.coffee
   238    695   7678 frontend/sagews/d3.coffee
   218    934   8800 database/postgres-user-query-queue.coffee
   181    925   7936 hub/metrics-recorder.coffee
   168    681   6502 database/postgres-ops.coffee
   163    617   6411 util/test/types.coffee
   122    408   4512 hub/auth-token.coffee
   107    296   3392 hub/projects.coffee
   103    365   3278 util/test/defaults.coffee
    97    366   3527 frontend/sagews/sagews-eval.coffee
    81    305   3211 hub/access.coffee
    81    225   2904 hub/user-query-stats.coffee
    73    221   2652 hub/smc-manager.coffee
    66    344   2953 hub/blobs.coffee
    58    121   1525 util/test/key-value-store.coffee
    48    141   1393 database/filesystem-bucket.coffee
    45    132   1481 hub/user-remember-me.coffee
    15     69    676 hub/p.coffee

[williamstein]

Current status as of July 2025:

~/cocalc/src$ git ls-files |grep -e cjsx$ -e coffee$ | xargs wc  |sort -nr
 15773  58588 647048 total
  2518   9822 103958 packages/database/postgres-server-queries.coffee
  2507   8894 100959 packages/frontend/sagews/sagews.coffee
  1790   7420  80519 packages/database/postgres-user-queries.coffee
  1732   5786  71010 packages/frontend/editor.coffee
  1156   4888  50562 packages/database/postgres-base.coffee
   964   3325  36848 packages/frontend/sagews/3d.coffee
   760   3389  34618 packages/database/postgres-blobs.coffee
   701   2546  28521 packages/hub/local_hub_connection.coffee
   627   2220  27746 packages/frontend/printing.coffee
   604   2236  22604 packages/database/postgres-synctable.coffee
   577   1840  21919 packages/frontend/syncdoc.coffee
   478   1405  18103 packages/frontend/sagews/interact.coffee
   238    695   7678 packages/frontend/sagews/d3.coffee
   218    934   8800 packages/database/postgres-user-query-queue.coffee
   168    681   6502 packages/database/postgres-ops.coffee
   122    408   4512 packages/hub/auth-token.coffee
   107    296   3392 packages/hub/projects.coffee
    97    366   3527 packages/frontend/sagews/sagews-eval.coffee
    81    305   3211 packages/hub/access.coffee
    81    225   2904 packages/hub/user-query-stats.coffee
    73    221   2652 packages/hub/smc-manager.coffee
    66    344   2953 packages/hub/blobs.coffee
    48    141   1393 packages/database/filesystem-bucket.coffee
    45    132   1481 packages/hub/user-remember-me.coffee
    15     69    676 packages/hub/p.coffee

[williamstein]

I just stumbled onto this github page that links to all coffeescript files:

https://github.com/search?q=repo%3Asagemathinc%2Fcocalc++language%3ACoffeeScript&type=code

Also, here is the current language breakdown:

[williamstein]

With a46ab15 I just got this down to almost the only remaining coffeescript is the backend database code, which is used almost only for user_query functionality (since almost all other queries use the standard pg driver directly):

wstein@lite:~/build/cocalc-lite/src$ git ls-files |grep -e cjsx$ -e coffee$ | xargs wc  |sort -nr
  7741  31185 327123 total
  2518   9814 103907 packages/database/postgres-server-queries.coffee
  1783   7382  80179 packages/database/postgres-user-queries.coffee
  1159   4904  50731 packages/database/postgres-base.coffee
   760   3389  34618 packages/database/postgres-blobs.coffee
   604   2236  22604 packages/database/postgres-synctable.coffee
   218    934   8800 packages/database/postgres-user-query-queue.coffee
   168    681   6502 packages/database/postgres-ops.coffee
   122    408   4512 packages/hub/auth-token.coffee
    81    305   3211 packages/hub/access.coffee
    81    225   2904 packages/hub/user-query-stats.coffee
    73    221   2652 packages/hub/smc-manager.coffee
    66    344   2953 packages/hub/blobs.coffee
    48    141   1393 packages/database/filesystem-bucket.coffee
    45    132   1481 packages/hub/user-remember-me.coffee
    15     69    676 packages/hub/p.coffee

So this ticket is down to pretty much finishing re-implementing the user_query stuff in typescript.