Add source content

[zeke]

This PR is a first step toward collecting source content from the nodejs/node repo to be synced with Crowdin. The collect script uses semver.io to determine the "Current" version of Node.js, but we should discuss which versions we want to collect, and how to structure them in this repo.

The diff is a bit noisy because of the imported content. To see just the code changes, check out this commit: 2c2c14c

Considerations

• This is only bringing in docs for one version of Node.js (9.9.0 Current). We'll need to agree on a strategy for what versions we want to localize.
• As is, we are importing everything in node's /doc directory, but it might be prudent to only import .md files and ignore things like images. This is what we do on electron/18n, and we rewrite relative src and href attributes when generated the electron-i18n module that's consumed by the website.

cc @nodejs/i18n

🍐 with @vanessayuenn

[obensource]

@zeke @vanessayuenn holy smokes! This is amazing!!! Thank you for starting us off on such a great foot! Can't wait to get this synced with Crowdin asap! 🙌

[zeke]

Note that as we add new versions of node's docs (e.g 9.9.1 when that comes out) we can configure a Crowdin workflow that applies all existing translations from the Translation Memory. This way translators will only need to translate content that has changed between versions.

[zeke]

See also this Electron issue where we ponder what Electron versions to localize: electron/i18n#288

[zeke]

I made a few updates:

• Updated the collect script to:
  • support fetching multiple versions
  • skip versions that have already been downloaded
  • simplified the tarball work, as the download module can decompress on the fly
  • only keep .md files, ignoring images and any other files
• Took the fetched docs out of this PR for now, to make reviewing easier
• Added an explicit list of target Node.js versions for starters (LTS and Current)
• Added linting and tests

[zeke]

Taking WIP off the title. This is ready for review.

[lukaszewczak]

Hi @zeke , I try to run this script (on Windows), but I receive 404 (Not Found) because of the wrong url

downloading https://github.com/nodejs/node/archive/vv8.11.0.tar.gz
(node:17156) UnhandledPromiseRejectionWarning: HTTPError: Response code 404 (Not Found)

If I remove v from node.js versions in package.json in section nodeVersions it runs.

 "nodeVersions": [
    "8.11.0",
    "9.10.0"
  ],

But then I receive this error during moving docs from temp dir, it probably should by copied to the destination folder

downloading https://github.com/nodejs/node/archive/v8.11.0.tar.gz
(node:17100) UnhandledPromiseRejectionWarning: Error: EXDEV: cross-device link not permitted, rename 'C:\Users\Dom\AppData\Local\Temp\d-118229-17100-1103929.rp0z\node-8.11.0\doc' -> 'D:
\Projekty\Learn\i18n\content\v8.11.0\en-US\doc'
    at Object.fs.renameSync (fs.js:788:18)
    at getDocsForNodeVersion (D:\Projekty\Learn\i18n\script\collect.js:37:6)
    at <anonymous>

[zeke]

Thanks for giving it a go, @lukaszewczak and @FranzDeCopenhague. Should be fixed now. Can you give it another try?

I was waffling between having the v prefix everywhere, and decided to keep it because it's generally safer for variables, filenames, etc to start with a character than a number. Also semver gracefully tolerates a v prefix:

semver.valid('v1.2.3') // true

[zeke]

But then I receive this error during moving docs from temp dir, it probably should by copied to the destination folder

@lukaszewczak I wanna see if this is indeed a permissions error, or just a byproduct of the v issue. If the problem persists, we can download straight to the target directory and clean up unneeded files afterwards.

[zeke]

Thanks @FranzDeCopenhague, though I kind of wanted to get more feedback from other reviewers, but I suppose we can iterate from here in new pull requests.

I opened #59 where we can discuss PR review requirements.

[FranzDeCopenhague]

@zeke, I did wait, a little, for @lukaszewczak feedback and move on because the issue looks like a windows problem where the temp folder is at a different unit drive to the i18n project

[zeke]

But then I receive this error during moving docs from temp dir, it probably should by copied to the destination folder

This sounds familiar. See electron/get#36 -- I agree that we should download to the repo directory.

[lukaszewczak]

Hi, I'm sorry but I wasn't able to look at your changes yesterday @zeke. Download directly to the repo folder sounds good and should make script platform independent. Will it be ok for you @zeke, if try to prepare PR with this change?

[zeke]

Will it be ok for you @zeke, if try to prepare PR with this change?

Of course! Did you mean me or you, though? :)

[lukaszewczak]

Me :-) I will try to prepare it:-)