Running Sequencer on gCloud #755

tech4GT · 2019-02-09T21:00:48Z

Current Service

We are running a google cloud function which processes the image and returns the output

// This is the current script
/**
 * Responds to any HTTP request.
 *
 * @param {!express:Request} req HTTP request context.
 * @param {!express:Response} res HTTP response context.
 */
const sequencer = require('image-sequencer')();

exports.helloWorld = (req, res) => {

    const img = req.query.url,
        sequence = req.query.sequence;
  
    sequencer.loadImages(img, () => {
        sequencer.importString(sequence);
        sequencer.run(out => {
            res.status(200).send({ data: out });
        });
    });
}

Process:

Ideas

Parallelize the processing of images using more cloud functions or maybe deploying on cloud engine
Optimize the core APIs to make the processing faster and more memory efficient
Try to run Image sequencer on a stream based input

tech4GT · 2019-02-09T21:00:59Z

cc @jywarren @icarito

jywarren · 2019-02-09T21:10:53Z

This is great; can you share the script for how you set it up, and maybe some screenshots?

…

On Sat, Feb 9, 2019, 4:01 PM Varun Gupta ***@***.*** wrote: Current Service - We are running a google cloud function which processes the image and returns the output Ideas - Parallelize the processing of images using more cloud functions or maybe deploying on cloud engine - Optimize the core APIs to make the processing faster and more memory efficient - Try to run Image sequencer on a stream based input — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#755>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ6t7YtEDoHI76f3P5UWEBkyy7yYSks5vLzcAgaJpZM4ayvPE> .

tech4GT · 2019-02-09T21:11:59Z

Sure!

jywarren · 2019-02-10T03:29:40Z

Also let's set it up to fwd to the dataurl if output=forward?

…

On Sat, Feb 9, 2019, 5:33 PM Varun Gupta ***@***.*** wrote: cc @jywarren <https://github.com/jywarren> @icarito <https://github.com/icarito> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#755 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ13Po57Z0WTa-D53xYglx8tM3HBaks5vLzcMgaJpZM4ayvPE> .

jywarren · 2019-02-10T13:43:00Z

To try out!

https://us-central1-public-lab.cloudfunctions.net/is-function/?url=https://publiclab.org/system/images/photos/000/024/686/original/valverde.jpg&upload=true&sequence=ndvi,colormap

harshkhandeparkar · 2019-02-10T13:48:29Z

@jywarren I checked it out, It took around 2mins to load the page itself. I haven't checked the dataURI it returned is there any way to ckeck it out? Also will it work for any steps?

jywarren · 2019-02-10T13:57:13Z

Anyone should be able to set up a free trial Google cloud account... Try it out! I think what I wrote about redirecting could work to view the dataurl.

jywarren · 2019-02-10T16:54:39Z

Is there a way to git track the script?

jywarren · 2019-02-10T16:54:59Z

https://us-central1-public-lab.cloudfunctions.net/is-function/?url=https://publiclab.org/system/images/photos/000/024/686/original/valverde.jpg&upload=true&sequence=ndvi,colormap&redirect=true

tech4GT · 2019-02-10T16:57:21Z

I think this is working!

tech4GT · 2019-02-10T16:58:03Z

For some reason it is loading a landing page though?

jywarren · 2019-02-10T16:59:51Z

I just changed it to return "out" instead of {data:out} to see.

…

On Sun, Feb 10, 2019, 11:57 AM Varun Gupta ***@***.*** wrote: I think this is working! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#755 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ6DRYgiXgEopf6zTxjto3gpNq-zAks5vME9xgaJpZM4ayvPE> .

tech4GT · 2019-02-10T17:00:34Z

Oh, I already fixed it to redirect, we can do a res.redirect(out) to redirect to the image

jywarren · 2019-02-10T17:01:24Z

Yeah there might be a limitation on redirecting... Maybe we can somehow override it by returning an HTML page with a built in redirect or something... Anyways it's just for convenience. Many programs will want the Json response.

…

On Sun, Feb 10, 2019, 11:59 AM Jeffrey Warren ***@***.*** wrote: I just changed it to return "out" instead of {data:out} to see. On Sun, Feb 10, 2019, 11:57 AM Varun Gupta ***@***.*** wrote: > I think this is working! > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#755 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AABfJ6DRYgiXgEopf6zTxjto3gpNq-zAks5vME9xgaJpZM4ayvPE> > . >

tech4GT · 2019-02-10T17:02:26Z

Yeah, I'll figure out the redirect right now! But this looks promising!!
So what are our next steps?
I mean should I jump to core optimizations now?

tech4GT · 2019-02-10T17:06:48Z

@jywarren I updated it!

jywarren · 2019-02-10T17:07:12Z

Yes perhaps so! Let's start benchmarking too. Can we actually build in a "execution time" attribute to the Json response so we can benchmark more easily? You can make a copy of the gc function pointed at a branch of IS on your GitHub repo, and keep running it as you push commits. And annotate runtimes in a PR so we can track progress empirically, how does that sound?

…

On Sun, Feb 10, 2019, 12:02 PM Varun Gupta ***@***.*** wrote: Yeah, I'll figure out the redirect right now! But this looks promising!! So what are our next steps? I mean should I jump to core optimizations now? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#755 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJxkkQb40f-lzpFWJyBNTP1r7aHUqks5vMFCjgaJpZM4ayvPE> .

tech4GT · 2019-02-10T17:10:41Z

@jywarren redirect is now working, you can try it out at the previous url! 🎉
Also, can you explain this process to me in a little more detail?
I think we can send the execution time as a part of json response from the cloud function but how do I make the copy of gc function run on the github repo?

jywarren · 2019-02-10T17:28:06Z

It looks like we can point the script at a cloud repository instead of entering a string: https://cloud.google.com/source-repositories/

…

On Sun, Feb 10, 2019, 12:11 PM Varun Gupta ***@***.*** wrote: @jywarren <https://github.com/jywarren> redirect is now working, you can try it out at the previous url! 🎉 Also, can you explain this process to me in a little more detail? I think we can send the execution time as a part of json response from the cloud function but how do I make the copy of gc function run on the github repo? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#755 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ-u052lwVWsi2S_ydpEjmmN7BN9Wks5vMFKRgaJpZM4ayvPE> .

tech4GT · 2019-02-10T17:29:05Z

Okay so do we make a new repository for this, or should I add this script to is-app?

tech4GT · 2019-02-10T17:30:20Z

And as new code is pushed into image-sequencer we need to invoke the cloud function, correct? And where do we store the performance results?

jywarren · 2019-02-10T17:31:28Z

Not sure, maybe -app for now?

…

On Sun, Feb 10, 2019, 12:29 PM Varun Gupta ***@***.*** wrote: Okay so do we make a new repository for this, or should I add this script to is-app? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#755 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ4I6DeeH6JlXyPhzAxhSITji3-Bwks5vMFbigaJpZM4ayvPE> .

tech4GT · 2019-02-10T17:32:25Z

Okay, will do that right now!

tech4GT · 2019-02-10T17:33:49Z

I'll also add the performance to response!
But maybe you can help with annotating runtimes?

jywarren · 2019-02-10T17:41:54Z

We can just track speeds in PR comments as improvements are made. Actually I guess maybe you want to link it to your remote is branch directly? So you can test the optimizations as directly as possible?

…

On Sun, Feb 10, 2019, 12:34 PM Varun Gupta ***@***.*** wrote: I'll also add the performance to response! But maybe you can help with annotating runtimes? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#755 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ34O_k5OyiAEWdy8rg_Em6WC9Dz5ks5vMFf9gaJpZM4ayvPE> .

tech4GT · 2019-02-10T17:47:13Z

@jywarren I don't have permission to create repository for the project 😅

tech4GT · 2019-02-10T17:48:34Z

We can just track speeds in PR comments as improvements are made. Actually I guess maybe you want to link it to your remote is branch directly? So you can test the optimizations as directly as possible?
…
On Sun, Feb 10, 2019, 12:34 PM Varun Gupta @.*** wrote: I'll also add the performance to response! But maybe you can help with annotating runtimes? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#755 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AABfJ34O_k5OyiAEWdy8rg_Em6WC9Dz5ks5vMFf9gaJpZM4ayvPE .

SO is this done using some sort of bot which runs the function and comments the runtime in the pr?
Sorry if I am asking stupid questions, I don't really have a lot of experience in this area 😅

harshkhandeparkar · 2019-02-10T17:50:48Z

@tech4GT I think he meant we could give progress reports in the comments. 😅. Not sure though.
Edit: Sorry for commenting without any real reason. I was actually following the discussion. I'm quite pumped up here

tech4GT · 2019-02-10T17:51:40Z

@tech4GT I think he meant we could give progress reports in the comments.

Do you mean do it manually?

jywarren · 2019-02-10T17:52:29Z

Well, honestly we can just do optimization testing locally. Let's do that for now. Eventually we will also have to look at webgl emulation for image distortion too... if thats interesting too, go ahead!

…

On Sun, Feb 10, 2019, 12:47 PM Varun Gupta ***@***.*** wrote: @jywarren <https://github.com/jywarren> I don't have permission to create repository for the project 😅 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#755 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ4FPnpgIaj7XIgKuAma2n_9FRopfks5vMFshgaJpZM4ayvPE> .

jywarren · 2019-02-10T17:52:36Z

#216

jywarren · 2019-02-10T17:53:29Z

I just mean we should keep an empirical log of runtime so we can track progress quantitatively... However you want to do it!

tech4GT · 2019-02-10T17:53:52Z

Oh, Got it!!

tech4GT · 2019-02-10T17:55:40Z

Also can you please grant me the permission to create cloud repositories on public-lab project, I'll link the is-app to it then, so that we can get the cloud function from there.
Also what track should we follow now? As in should I now dive into optimizations? Or we are thinking of deploying to app-engine??

jywarren · 2019-02-10T17:57:10Z

Let's do optimization and webgl now. I think we should demonstrate the whole intended workflow and then look at the serverless options later. This already works!

tech4GT · 2019-02-10T17:59:39Z

Okay awesome! I'll dive into that from tomorrow! (it's pretty late here lol)

jywarren · 2019-02-10T18:01:12Z

Go rest! Great work!

…

On Sun, Feb 10, 2019, 1:00 PM Varun Gupta ***@***.*** wrote: Okay awesome! I'll dive into that from tomorrow! (it's pretty late here lol) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#755 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJxHNr3tx4mMr3sbSIPOo6HKME96tks5vMF4LgaJpZM4ayvPE> .

icarito · 2019-02-20T05:00:28Z

I had missed this, very cool indeed!

grvsachdeva added the project label Feb 21, 2019

Running Sequencer on gCloud #755

Running Sequencer on gCloud #755

Comments

tech4GT commented Feb 9, 2019 • edited Loading

Current Service

Process:

Ideas

tech4GT commented Feb 9, 2019

jywarren commented Feb 9, 2019 via email

tech4GT commented Feb 9, 2019

jywarren commented Feb 10, 2019 via email

jywarren commented Feb 10, 2019

harshkhandeparkar commented Feb 10, 2019

jywarren commented Feb 10, 2019

jywarren commented Feb 10, 2019

jywarren commented Feb 10, 2019

tech4GT commented Feb 10, 2019

tech4GT commented Feb 10, 2019

jywarren commented Feb 10, 2019 via email

tech4GT commented Feb 10, 2019

jywarren commented Feb 10, 2019 via email

tech4GT commented Feb 10, 2019

tech4GT commented Feb 10, 2019

jywarren commented Feb 10, 2019 via email

tech4GT commented Feb 10, 2019

jywarren commented Feb 10, 2019 via email

tech4GT commented Feb 10, 2019

tech4GT commented Feb 10, 2019

jywarren commented Feb 10, 2019 via email

tech4GT commented Feb 10, 2019

tech4GT commented Feb 10, 2019

jywarren commented Feb 10, 2019 via email

tech4GT commented Feb 10, 2019

tech4GT commented Feb 10, 2019

harshkhandeparkar commented Feb 10, 2019 • edited Loading

tech4GT commented Feb 10, 2019

jywarren commented Feb 10, 2019 via email

jywarren commented Feb 10, 2019

jywarren commented Feb 10, 2019

tech4GT commented Feb 10, 2019

tech4GT commented Feb 10, 2019

jywarren commented Feb 10, 2019

tech4GT commented Feb 10, 2019

jywarren commented Feb 10, 2019 via email

icarito commented Feb 20, 2019

tech4GT commented Feb 9, 2019 •

edited

Loading

harshkhandeparkar commented Feb 10, 2019 •

edited

Loading