rsync endpoint to mirror the releases

[davglass]

Any thought on adding an rsync endpoint for this:

https://iojs.org/download/release/

[mikeal]

+1

[bengl]

+1! (but for http://nodejs.org/dist)

[rvagg]

/dist is an alias for /download/release now fwiw

[fhemberger]

Can we please add the rsync endpoint for /dist?
nodejs/nodejs.org#662 (comment)

[jbergstroem]

I can look at doing this if others in the build group agree.

[rvagg]

So I had a thought (:boom:): one of the blockers we have here is hesitation about exposing an unencrypted endpoint while we're trying to move everything to TLS (http://nodejs.org will be fully redirected soon). So, how about we be upfront about this and stand up a host named insecure.nodejs.org and expose stuff via http, rsync and even maybe ftp (does anyone still use ftp?). The name points to our opinions re security, you opt in to the risk by putting "insecure" in your scripts. We can isolate it by having the primary server push updates to it so the new host doesn't have any special access to anything. When we go fully https for nodejs.org, having an http fallback host if people absolutely need it might be a nice touch.

[jbergstroem]

@rvagg I like it. How about adding a http header about expiration as well; putting a final nail in the coffin to plaintext stuff? X-Service-Will-Expire: 2020-01-01 (or the rsync equivalent in message)

[jbergstroem]

(as well as serving the same message with /index.html)

[mhdawson]

Would unencrypted.nodejs.org work ? Not to bikeshed but insecure implies a lot more to me than just not protected by encryption which I think would be the main difference.

[jbergstroem]

Just a quick update: We have an endpoint set up for testing; pm me on irc (jbergstroem) or shoot me an email (bugs at bergstroem dot nu) if you want to give it a whim. If everything works as intended we'll make this public shortly.

[rvagg]

@jbergstroem do we have it syncing in cron yet? last I checked that wasn't enabled.

[jbergstroem]

@rvagg nope, still todo. Also, adding it to the deploy script for 0day.

[jbergstroem]

We're nearly there. I've gotten som feedback and based on that I'd like to suggest extending our EOL/reconsider to 2022 (+2y). I'll land the sync script we'll run from both deploy and cronjob this week.

[jsumners]

I've been noticing an issue where the release files will sometimes not be available. For example, rsync://unencrypted.nodejs.org/nodejs/release/v6.9.2/ may have the docs directory but nothing else. This definitely causes a problem when the --delete option is passed to rsync.

[jbergstroem]

@jsumners the rsync routine isn't fully up to par. Apologies about that. When we moved web servers a few things changed which made it break. It will obviously work pristine once deemed 'production' worthy, but that doesn't mean we shouldn't fit it as soon as possible. I'll have a look in within a day or so; just got mixed up amongst my other todo's.

[jbergstroem]

@jsumners hey; I've redeployed it now. It should be populating and be up to speed pretty soon; updating dns as we speak. If you could verify that things look better in the hour or so, that'd be cool!

Edit: update. Almost done with initial rsync. I moved from cron to systemd timers (yes, i know -- sigh) to get better visibility of running state. Should be done in 5 minutes and it should sync every 15.

[jsumners]

@jbergstroem permissions are broken:

[jsumners@node] ~/ $ rsync rsync://unencrypted.nodejs.org/nodejs/
drwxrwxr-x           8 2016/11/17 13:02:06 .
drwxr-xr-x          46 2017/01/03 05:10:02 chakracore-nightly
drwxr-xr-x           2 2015/08/30 04:29:25 next-nightly
drwxr-xr-x         492 2017/01/03 13:05:02 nightly
drwx------           2 2017/01/03 16:05:29 rc
drwx------           2 2017/01/03 16:05:29 release
drwx------           2 2017/01/03 16:05:29 test

[jbergstroem]

@jsumners that's because rsync hasn't finished processing those folders just yet. Thanks for checking!

[jbergstroem]

Looks like it's done:

NEXT                         LEFT       LAST                         PASSED       UNIT                         ACTIVATES
Tue 2017-01-03 22:45:00 UTC  10min left Tue 2017-01-03 22:30:01 UTC  4min 20s ago rsyncmirror.timer            rsyncmirror.service

[jsumners]

@jbergstroem looking good over here. I'll keep an eye on it.

[ofrobots]

Would it be possible to allow rsync over ssh?

[jbergstroem]

@ofrobots that means we'd have to figure out some kind of ssh jail, right?

[ofrobots]

Yup. Has this been considered? I haven't seen this discussed anywhere.

[jbergstroem]

@ofrobots http://nginx.org/en/docs/http/ngx_http_autoindex_module.html

[gibfahn]

Using https://nodejs.org/dist/index.json was suggested in the meeting (#737), @ofrobots let us know whether that works for you.

[jbergstroem]

[slight hijack here] According to documentation, autoindex is compiled in by default, so I'll see what we can do here.

[jsumners]

The latest release currently available on the rsync server is v8.6.0. It looks like it isn't being updated again.

[joaocgreis]

The timer that synchronizes the server was not enabled by default, and was disabled by a reboot. Should be fixed now.

[jsumners]

👍

[AndersTrier]

I pulled a copy to mirrors.dotsrc.org which is hosted at Aalborg University, Denmark:
http://mirrors.dotsrc.org/nodejs
https://mirrors.dotsrc.org/nodejs
ftp://mirrors.dotsrc.org/nodejs
rsync://mirrors.dotsrc.org/nodejs

Fell free to send some traffic our way. We sync every 6 hours.

@ofrobots @jbergstroem
If you want to tunnel rsync over ssh, have a look at the rrsync restricted shell:
https://www.samba.org/ftp/unpacked/rsync/support/rrsync
https://www.guyrutenberg.com/2014/01/14/restricting-ssh-access-to-rsync/

[sam-github]

Closing as stale, but if anyone wants to take this up feel free to reopen.