NodeJS Support

[jlarmstrongiv]

It’d be great to see support for running these upscaler models in NodeJS 😄

This will require changing how images are loaded (https://github.com/thekevinscott/UpscalerJS/blob/master/src/image.ts#L4). It would be great to support both image paths and buffers. There may be other changes too, but that’s where my error was thrown 😂

While UpscalerJS already supports webworkers, it may be good to use threads.js to support worker threads in NodeJS.

Supporting @tensorflow/tfjs-node would also substantially speed up processing. I have previously used this package, and found it difficult to navigate the native code dependencies and package size when using it in serverless functions. The package I made https://github.com/jlarmstrongiv/tfjs-node-lambda may provide some workarounds in serverless environments. Personally, I think being able to pass in the preferred version of tfjs (@tensorflow/tfjs, @tensorflow/tfjs-node, tfjs-node-lambda) as a constructor parameter would give a lot of flexibility.

I am interested in contributing. While I have experience with typescript and tensorflow, it would be great to collaborate with writing tests.

Anyway, thank you for making such a helpful library 👍

[danwexler]

Upscaler already works in Node, both using tfjs-node and tfjs-node-gpu. I have it running using the CPU on OSX and the GPU on GKE.

[jlarmstrongiv]

That’s great news! I tried searching the documentation for NodeJS, but came up empty.

@danwexler Could you post a short example snippet? I tried following the quickstart, but ran into the error with HTMLImageElement from loadImage. Is there a flag or option I’m missing?

[danwexler]

Here's an example:

// Returns a PNG-encoded UInt8Array
const upscaleImageToUInt8Array = async (filename, upscaler, progress) => {
  const file = fs.readFileSync(filename)
  const image = tf.node.decodeImage(file, 3)
  const options = { output: 'tensor', patchSize: 64, padding: 6, progress }
  const tensor = await upscaler.upscale(image, options)
  const upscaledImage = await tf.node.encodePng(tensor)
  return upscaledImage
}

This code works in the browser and in Node.

[jlarmstrongiv]

Thank you @danwexler I’ll try it out later this evening 😄

[danwexler]

Let me know if you hit the realDivide issue I just posted, #12

[thekevinscott]

Jumping onto this thread late - thank you @danwexler for providing guidance and runnable code!

I'm not super familiar with TFJS in Node, but it sounds like y'all have been able to get it running. From a maintenance perspective it'd also be useful to:

• Add tests for browser and node environments like you mentioned @jlarmstrongiv
• Add documentation for Node support
• integrate upscaleImagetoUInt8Array into the library, and switch based on whether we're in the browser or Node

We're currently running tests via Jest, which afaik is running in a Node-like environment, but we don't currently have any end-to-end tests. I believe Jest supports a browser-like environment, but it may be better to use something like Puppeteer to bootstrap an actual (headless) browser for tests.

I'm also open if others have pointers on effectively testing JS libraries across both Node and the browser!

[thekevinscott]

Question for the group - would you (if willing) speak more to your use case for running in Node?

For background, I assumed that the main use case for UpscalerJS would be via the browser, thus the lack of Node support out of the box. But if there's appetite for Node, it also makes me wonder whether the training code should be ported over; e.g., whether we should rewrite ESRGAN in Javascript so anyone could train it via Node. (As an aside, I'd love to make it easier for the community to contribute new models to the repo).

[danwexler]

I'm building a tool that uses TFJS in both the client and the server. It allows you to connect to your cloud storage, or load local files, and then apply a TF image processing model. You can run the models in either locally the browser or at scale on the cloud. It can also generate annotations (labels) which can be used to search for images or videos across your cloud storage drives. In my case, the browser is nice for testing out the operations locally, especially if the customer has a nice GPU. But even a nice GPU will take awhile on a movie file, and even longer if you want to process a folder filled with dozens of videos. That's when running in the cloud will be nice, since we can break up the video into batches of N frames and process them in parallel before re-encoding the final movie.

[thekevinscott]

That makes a lot of sense, and is very cool. Thanks @danwexler!

[jlarmstrongiv]

Oh! I missed must’ve missed your last reply—whoops. I ended up learning to train custom models with https://github.com/victorca25/BasicSR I tried, but was unable to convert models to tfjs. @thekevinscott do you have any tips for converting models?

[thekevinscott]

Well, the main issue with that particular repo is it's using Pytorch, whereas you'd want a Tensorflow version. I've seen some folks discuss converting Pytorch models to Tensorflow (sometimes with ONNX as an in-between) but I haven't tried that myself. I'm not sure how the two codebases manage the core ops, so I could see it getting pretty hairy for anything non-trivial.

Assuming you find a Tensorflow library, I did write a bit on the conversion process here. It's unfortunately a bit touch-and-go, particularly when a library implements custom ops (as the RRDN implementation in ESRGAN does) but it's certainly possible.

[jlarmstrongiv]

Ahh alrighty. I was able to get from pytorch => onnx => tensorflow, but the tensorflow => tfjs failed. I also tried onnx => onnxjs, but that library isn’t compatible with the latest versions.

[thekevinscott]

Gotcha. Yeah, I haven't spent too much time with onnxjs, but with PyTorch picking up so much mindshare it may be worth a second look.

[thekevinscott]

Just wanted to follow up on the ONNX.js point - although commits are still being made to the repo, issues like this and this make it seem like there's some big gaps (and the last release was 7 months ago). So it's probably not worth supporting at the moment.

Still, if we can figure out a way to go from Pytorch => tfjs, I think that'd be huge. I'm just not sure how feasible it will be.

[jlarmstrongiv]

Yep, the lack of onnxjs support worried me. One other solution I didn’t mention was compiling to WASM. Pytorch has a c++ lib which can be ported. Since c++ is officially supported, there would be fewer edge cases compared to converting the models 3+ times to js. But my lack of c++ knowledge means that solution isn’t feasible for me.

[jlarmstrongiv]

More alternatives include NodeJS C++ addons, with example implmentations:

• https://github.com/vvmnnnkv/libtorchjs
• https://github.com/torch-js/torch-js

[weingaunity]

Still, if we can figure out a way to go from Pytorch => tfjs, I think that'd be huge. I'm just not sure how feasible it will be.

At the end i did the same. My solution was now converting the ONNX file to a tensorflow.js based on this i'm using now following lines:

pip install tensorflowjs
don't use: pip install onnx_tf
pip install git+https://github.com/onnx/onnx-tensorflow.git

import tensorflow as tf
import tensorflowjs as tfjs
import onnx
from onnx_tf.backend import prepare
from tensorflow.python.tools.import_pb_to_tensorboard import import_to_tensorboard

onnx_model = onnx.load("output/pytorch_mobilenetv2.onnx")
tf_rep = prepare(onnx_model) # returns: A TensorflowRep class object representing the ONNX model
tf_rep.export_graph("output/pytorch_mobilenetv2.pb")
#import_to_tensorboard("output/pytorch_mobilenetv2.pb", "tb_log","")
mdl2=tf.saved_model.load("output/pytorch_mobilenetv2.pb")
tfjs.converters.convert_tf_saved_model("output/pytorch_mobilenetv2.pb", "output/js")

[RyukAD]

Hi,
On I'm on node 10.24, and when i try to specify a custom model like this :
const upscaleed = new Upscaler({ model: '/home/agam/flash_images/node_modules/@upscalerjs/models/models/psnr-small-quant-uint8/image-super-resolution-psnr-small.h5', scale: 2, });

it says :
(node:21262) UnhandledPromiseRejectionWarning: TypeError: Only absolute URLs are supported at getNodeRequestOptions (/home/agam/flash_images/node_modules/node-fetch/lib/index.js:1305:9) at /home/agam/flash_images/node_modules/node-fetch/lib/index.js:1410:19 at new Promise (<anonymous>) at fetch (/home/agam/flash_images/node_modules/node-fetch/lib/index.js:1407:9) at HTTPRequest.PlatformNode.fetch (/home/agam/flash_images/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:5818:16) at HTTPRequest.<anonymous> (/home/agam/flash_images/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:6768:55) at step (/home/agam/flash_images/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:82:23) at Object.next (/home/agam/flash_images/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:63:53) at /home/agam/flash_images/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:56:71 at new Promise (<anonymous>)

I would be really happy if you could help me out

[thekevinscott]

I haven't used this library for Node myself, I'm curious if others in the community can chime in with their solutions.

I don't think Tensorflow.js supports paths to relative models like you're specifying. You may be able to try something like this - though I'm not 100% sure if the loadModel code is set up to correctly handle non-URLs. If not, we (or I) should modify it to handle that scenario.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[thekevinscott]

Node support is being tracked in #185.

Canary releases are available on npm at 0.10.0-canary.x (currently 0.10.0-canary.1)

[thekevinscott]

Latest Node release is on npm as upscaler@0.10.0-canary.5. This adds support for Node via an import of upscaler/node, and addresses a few memory leak issues.

This will be the last canary release before an official 0.10.0 release, pending discovery of any show-stopping bugs. 0.10.0 will be released once documentation is ready.

[thekevinscott]

#185 was merged and Node support is live.

UpscalerJS with Node support has been published on npm as 0.10.0.

Make sure to import the correct package for your system (and have the correct peer dependency of TFJS installed):

// CPU only
import Upscaler from 'upscaler/node';

// GPU
import Upscaler from 'upscaler/node-gpu';

If you run into any bugs, please make a new issue!