Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NPM HA support with Kappa, Nginx and Registry-Static #70

Open
marcellodesales opened this issue Aug 6, 2016 · 7 comments
Open

NPM HA support with Kappa, Nginx and Registry-Static #70

marcellodesales opened this issue Aug 6, 2016 · 7 comments

Comments

@marcellodesales
Copy link

Hi guys,

After today's outage on NPM skimdb.npmjs.com unavailable http://status.npmjs.org/incidents/dh3p7jb3ljn5, our internal NPM registry failed to replicate and, as a consequence, all NPM commands failed. I'm thinking on the following strategy to support HA and resilience on the READS:

  • Kappa: Front of all requests
  • Internal NPM: First server that proxies all public libraries from skimdb
  • nginx + Registry Static: Serves from the same directory our Internal NPM registry is writing to.

Questions

  • Could I use this application to support my HA requirements?
  • Could you service use the file-system built by NPM? In order words, are the URL paths from the skimdb.npmjs.com the same to the URI path in the file-system that registry-static reads?
  • Is there support for @scope libraries

thanks a lot
Marcello
@davglass
Copy link
Owner

davglass commented Aug 8, 2016

registry-static follows skim-db and writes them to the filesystem, placing something like nginx in front of it you could make the URL's anything you want.

However, the skimdb feed doesn't support @scope'd packages tho :(

@marcellodesales
Copy link
Author

@davglass https://replicate.npmjs.com does support scoped libraries... ;) Could we work on it so that you can follow a scoped-enabled URL?

Just make URL-encoded to the scoped libraries. If not, you won't be able to retrieve the library:

  • Without URL-encoding
$ curl https://registry.npmjs.org/@exponent/react-native-responsive-image | jq
{
  "error": "Not found"
}
  • With URL Encoding
$ curl https://registry.npmjs.org/@exponent%2freact-native-responsive-image | jq
{
  "_id": "@exponent/react-native-responsive-image",
  "_rev": "9-1fc037eaf17975b1e8994ea006620d8f",
  "name": "@exponent/react-native-responsive-image",
  "description": "A responsive Image component that chooses the best-resolution image for the current screen",
  "dist-tags": {
    "latest": "1.0.0"
  },
  "versions": {
    "0.0.1": {
      "name": "@exponent/react-native-responsive-image",
      "version": "0.0.1",
      "description": "A responsive Image component that chooses the best-resolution image for the current screen",
      "main": "ResponsiveImage.js",
      "scripts": {
        "test": "echo \"Error: no test specified\" && exit 1"
      },
      "repository": {
        "type": "git",
        "url": "git+https://github.com/exponentjs/react-native-responsive-image.git"
      },
      "keywords": [
        "react-native",
        "responsive",
        "image",
        "pixel-ratio"
      ],
      "author": {
        "name": "Charlie Cheever",
        "email": "ccheever@exp.host"
      },
      "license": "MIT",
      "bugs": {
        "url": "https://github.com/exponentjs/react-native-responsive-image/issues"
      },
      "homepage": "https://github.com/exponentjs/react-native-responsive-image#readme",
      "gitHead": "e726c45233413a2198b501ac978afbb06f2ef858",
      "_id": "@exponent/react-native-responsive-image@0.0.1",
      "_shasum": "89028fe28ba2cf5eddf5290081f8c2b49c1c19a9",
      "_from": ".",
      "_npmVersion": "2.11.3",
      "_nodeVersion": "2.3.3",
      "_npmUser": {
        "name": "exponent",
        "email": "exponent.team@gmail.com"
      },
      "dist": {
        "shasum": "89028fe28ba2cf5eddf5290081f8c2b49c1c19a9",
        "tarball": "https://registry.npmjs.org/@exponent/react-native-responsive-image/-/react-native-responsive-image-0.0.1.tgz"
      },
      "maintainers": [
        {
          "name": "exponent",
          "email": "exponent.team@gmail.com"
        }
      ],
      "directories": {}
    },
    "0.0.2": {
      "name": "@exponent/react-native-responsive-image",
      "version": "0.0.2",
      "description": "A responsive Image component that chooses the best-resolution image for the current screen",
      "main": "ResponsiveImage.js",
      "scripts": {
        "test": "echo \"Error: no test specified\" && exit 1"
      },
      "repository": {
        "type": "git",
        "url": "git+https://github.com/exponentjs/react-native-responsive-image.git"
      },
      "keywords": [
        "react-native",
        "responsive",
        "image",
        "pixel-ratio"
      ],
      "author": {
        "name": "Charlie Cheever",
        "email": "ccheever@exp.host"
      },
      "license": "MIT",
      "bugs": {
        "url": "https://github.com/exponentjs/react-native-responsive-image/issues"
      },
      "homepage": "https://github.com/exponentjs/react-native-responsive-image#readme",
      "gitHead": "eb5f0bc64dcf9544511219e2daeca1ae6dda962b",
      "_id": "@exponent/react-native-responsive-image@0.0.2",
      "_shasum": "a3d9769f44db5e0cff9df43035346a4aec9981a7",
      "_from": ".",
      "_npmVersion": "2.11.3",
      "_nodeVersion": "2.3.3",
      "_npmUser": {
        "name": "exponent",
        "email": "exponent.team@gmail.com"
      },
      "dist": {
        "shasum": "a3d9769f44db5e0cff9df43035346a4aec9981a7",
        "tarball": "https://registry.npmjs.org/@exponent/react-native-responsive-image/-/react-native-responsive-image-0.0.2.tgz"
      },
      "maintainers": [
        {
          "name": "exponent",
          "email": "exponent.team@gmail.com"
        },
        {
          "name": "ide",
          "email": "ide+npm@jameside.com"
        },
        {
          "name": "ccheever",
          "email": "ccheever@gmail.com"
        }
      ],
      "directories": {}
    },
    "0.0.3": {
      "name": "@exponent/react-native-responsive-image",
      "version": "0.0.3",
      "description": "A responsive Image component that chooses the best-resolution image for the current screen",
      "main": "ResponsiveImage.js",
      "scripts": {
        "test": "echo \"Error: no test specified\" && exit 1"
      },
      "repository": {
        "type": "git",
        "url": "git+https://github.com/exponentjs/react-native-responsive-image.git"
      },
      "keywords": [
        "react-native",
        "responsive",
        "image",
        "pixel-ratio"
      ],
      "author": {
        "name": "Charlie Cheever",
        "email": "ccheever@exp.host"
      },
      "license": "MIT",
      "bugs": {
        "url": "https://github.com/exponentjs/react-native-responsive-image/issues"
      },
      "homepage": "https://github.com/exponentjs/react-native-responsive-image#readme",
      "gitHead": "bb5b9933417634abeb00c7b7e5e81293bca3b728",
      "_id": "@exponent/react-native-responsive-image@0.0.3",
      "_shasum": "8f8ef4870f0aa554b17ef820c56c91d755397b00",
      "_from": ".",
      "_npmVersion": "2.14.2",
      "_nodeVersion": "4.0.0",
      "_npmUser": {
        "name": "ide",
        "email": "ide+npm@jameside.com"
      },
      "dist": {
        "shasum": "8f8ef4870f0aa554b17ef820c56c91d755397b00",
        "tarball": "https://registry.npmjs.org/@exponent/react-native-responsive-image/-/react-native-responsive-image-0.0.3.tgz"
      },
      "maintainers": [
        {
          "name": "exponent",
          "email": "exponent.team@gmail.com"
        },
        {
          "name": "ide",
          "email": "ide+npm@jameside.com"
        },
        {
          "name": "ccheever",
          "email": "ccheever@gmail.com"
        }
      ],
      "directories": {}
    },
    "0.0.4": {
      "name": "@exponent/react-native-responsive-image",
      "version": "0.0.4",
      "description": "A responsive Image component that chooses the best-resolution image for the current screen",
      "main": "ResponsiveImage.js",
      "scripts": {
        "test": "echo \"Error: no test specified\" && exit 1"
      },
      "repository": {
        "type": "git",
        "url": "git+https://github.com/exponentjs/react-native-responsive-image.git"
      },
      "keywords": [
        "react-native",
        "responsive",
        "image",
        "pixel-ratio"
      ],
      "author": {
        "name": "Charlie Cheever",
        "email": "ccheever@exp.host"
      },
      "license": "MIT",
      "bugs": {
        "url": "https://github.com/exponentjs/react-native-responsive-image/issues"
      },
      "homepage": "https://github.com/exponentjs/react-native-responsive-image#readme",
      "gitHead": "3e9aa840efc294ae827efac865d37b26ad629026",
      "_id": "@exponent/react-native-responsive-image@0.0.4",
      "_shasum": "6f655cc85234e1865fa41235c8e2298eb98a6a46",
      "_from": ".",
      "_npmVersion": "2.14.2",
      "_nodeVersion": "4.0.0",
      "_npmUser": {
        "name": "ide",
        "email": "ide+npm@jameside.com"
      },
      "dist": {
        "shasum": "6f655cc85234e1865fa41235c8e2298eb98a6a46",
        "tarball": "https://registry.npmjs.org/@exponent/react-native-responsive-image/-/react-native-responsive-image-0.0.4.tgz"
      },
      "maintainers": [
        {
          "name": "exponent",
          "email": "exponent.team@gmail.com"
        },
        {
          "name": "ide",
          "email": "ide+npm@jameside.com"
        },
        {
          "name": "ccheever",
          "email": "ccheever@gmail.com"
        }
      ],
      "directories": {}
    },
    "0.1.0": {
      "name": "@exponent/react-native-responsive-image",
      "version": "0.1.0",
      "description": "A responsive Image component that chooses the best-resolution image for the current screen",
      "main": "ResponsiveImage.js",
      "scripts": {
        "test": "echo \"Error: no test specified\" && exit 1"
      },
      "repository": {
        "type": "git",
        "url": "git+https://github.com/exponentjs/react-native-responsive-image.git"
      },
      "keywords": [
        "react-native",
        "responsive",
        "image",
        "pixel-ratio"
      ],
      "author": {
        "name": "Charlie Cheever",
        "email": "ccheever@exp.host"
      },
      "license": "MIT",
      "bugs": {
        "url": "https://github.com/exponentjs/react-native-responsive-image/issues"
      },
      "homepage": "https://github.com/exponentjs/react-native-responsive-image#readme",
      "gitHead": "37881a18dfda3b8e970624a354e28f8478d722f7",
      "_id": "@exponent/react-native-responsive-image@0.1.0",
      "_shasum": "ccac0ff49e717c1ec8235d04d1dc2f97f355bef3",
      "_from": ".",
      "_npmVersion": "3.3.12",
      "_nodeVersion": "5.3.0",
      "_npmUser": {
        "name": "brentvatne",
        "email": "brentvatne@gmail.com"
      },
      "dist": {
        "shasum": "ccac0ff49e717c1ec8235d04d1dc2f97f355bef3",
        "tarball": "https://registry.npmjs.org/@exponent/react-native-responsive-image/-/react-native-responsive-image-0.1.0.tgz"
      },
      "maintainers": [
        {
          "name": "brentvatne",
          "email": "brentvatne@gmail.com"
        },
        {
          "name": "ccheever",
          "email": "ccheever@gmail.com"
        },
        {
          "name": "exponent",
          "email": "exponent.team@gmail.com"
        },
        {
          "name": "ide",
          "email": "ide+npm@jameside.com"
        }
      ],
      "_npmOperationalInternal": {
        "host": "packages-5-east.internal.npmjs.com",
        "tmp": "tmp/react-native-responsive-image-0.1.0.tgz_1455304081677_0.08004394848830998"
      },
      "directories": {}
    },
    "1.0.0": {
      "name": "@exponent/react-native-responsive-image",
      "version": "1.0.0",
      "description": "A responsive Image component that chooses the best-resolution image for the current screen",
      "main": "ResponsiveImage.js",
      "scripts": {
        "test": "echo \"Error: no test specified\" && exit 1"
      },
      "repository": {
        "type": "git",
        "url": "git+https://github.com/exponentjs/react-native-responsive-image.git"
      },
      "keywords": [
        "react-native",
        "responsive",
        "image",
        "pixel-ratio"
      ],
      "author": {
        "name": "Charlie Cheever",
        "email": "ccheever@exp.host"
      },
      "license": "MIT",
      "bugs": {
        "url": "https://github.com/exponentjs/react-native-responsive-image/issues"
      },
      "homepage": "https://github.com/exponentjs/react-native-responsive-image#readme",
      "gitHead": "d7e83ce8ed482048d775f799f8e285527421658f",
      "_id": "@exponent/react-native-responsive-image@1.0.0",
      "_shasum": "f0fe180db766c8c03a73ea710b59665d82140c68",
      "_from": ".",
      "_npmVersion": "3.8.3",
      "_nodeVersion": "5.10.0",
      "_npmUser": {
        "name": "ide",
        "email": "ide+npm@jameside.com"
      },
      "dist": {
        "shasum": "f0fe180db766c8c03a73ea710b59665d82140c68",
        "tarball": "https://registry.npmjs.org/@exponent/react-native-responsive-image/-/react-native-responsive-image-1.0.0.tgz"
      },
      "maintainers": [
        {
          "name": "brentvatne",
          "email": "brentvatne@gmail.com"
        },
        {
          "name": "ccheever",
          "email": "ccheever@gmail.com"
        },
        {
          "name": "exponent",
          "email": "exponent.team@gmail.com"
        },
        {
          "name": "ide",
          "email": "ide+npm@jameside.com"
        }
      ],
      "_npmOperationalInternal": {
        "host": "packages-16-east.internal.npmjs.com",
        "tmp": "tmp/react-native-responsive-image-1.0.0.tgz_1460186803521_0.8547379600349814"
      },
      "directories": {}
    }
  },
  "readme": "# ResponsiveImage [![Slack](http://slack.exponentjs.com/badge.svg)](http://slack.exponentjs.com)\nA responsive Image component that chooses the best-resolution image for the current screen.\n\n## Installation\n\n```\nnpm install @exponent/react-native-responsive-image --save\n```\n\nRequire it with:\n\n```js\nlet ResponsiveImage = require('@exponent/react-native-responsive-image');\n```\n\n## Usage\n\nResponsiveImage accepts the same props as Image plus a new prop called `sources`. The `sources` prop is an object whose keys are pixel ratios (that is, screen scales like \"2\" or \"3\"). Its values are Image sources to display on screens with the respective pixel ratio. This is how you use it:\n\n```js\n<ResponsiveImage\n  sources={{\n    // The values are anything that Image's source prop accepts\n    2: require('image!icon@2x.png'),\n    3: { uri: 'https://example.com/icon@3x.png' },\n  }}\n/>\n```\n\n## Implementation\n\nResponsiveImage chooses the image source for the closest pixel ratio that is greater than or equal to screen's pixel ratio. You will get a crisp image while conserving bandwidth and system resources.\n\nAs an example, if you were to provide 2x and 3x sources as in the Usage example, these are the pixel ratios that ResponsiveImage will choose for various screens:\n\nScreen Scale  | Chosen Pixel Ratio\n------------- | --------------------------\n1x            | 2x (use lowest available)\n2x            | 2x (exact match)\n2.5x          | 3x (round up to nearest)\n3x            | 3x (exact match)\n4x            | 3x (use highest available)\n",
  "maintainers": [
    {
      "name": "brentvatne",
      "email": "brentvatne@gmail.com"
    },
    {
      "name": "ccheever",
      "email": "ccheever@gmail.com"
    },
    {
      "name": "exponent",
      "email": "exponent.team@gmail.com"
    },
    {
      "name": "ide",
      "email": "ide+npm@jameside.com"
    }
  ],
  "time": {
    "modified": "2016-04-09T07:26:45.762Z",
    "created": "2015-07-05T08:09:54.118Z",
    "0.0.1": "2015-07-05T08:09:54.118Z",
    "0.0.2": "2015-07-05T10:53:43.774Z",
    "0.0.3": "2015-09-10T09:54:59.137Z",
    "0.0.4": "2015-09-10T10:09:56.781Z",
    "0.1.0": "2016-02-12T19:08:05.170Z",
    "1.0.0": "2016-04-09T07:26:45.762Z"
  },
  "homepage": "https://github.com/exponentjs/react-native-responsive-image#readme",
  "keywords": [
    "react-native",
    "responsive",
    "image",
    "pixel-ratio"
  ],
  "repository": {
    "type": "git",
    "url": "git+https://github.com/exponentjs/react-native-responsive-image.git"
  },
  "author": {
    "name": "Charlie Cheever",
    "email": "ccheever@exp.host"
  },
  "bugs": {
    "url": "https://github.com/exponentjs/react-native-responsive-image/issues"
  },
  "license": "MIT",
  "readmeFilename": "README.md",
  "_attachments": {}
}

@drewfish
Copy link
Contributor

drewfish commented Aug 8, 2016

Do the npmjs.com folks suggest/prefer that we use replicate.npmjs.com over skimdb.npmjs.com?

@davglass
Copy link
Owner

As long as the follower will fetch from it, it shouldn't be more than just updating this config:

https://github.com/davglass/follow-registry/blob/master/lib/config.json#L3-L4

@marcellodesales
Copy link
Author

@drewfish Yes, we have https://www.npmjs.com/enterprise in our environment and we fixed the outages last week http://status.npmjs.org/incidents/669mzfs17fkr with the skimdb with this new URL.

marcellodesales added a commit to marcellodesales/follow-registry that referenced this issue Aug 11, 2016
@marcellodesales
@scope support: replace urls: replicate.npmjs.com
db80e6f 
This commit replaces the URLs as discussed at davglass/registry-static#70
@marcellodesales marcellodesales mentioned this issue Aug 11, 2016
Merged
@drewfish
Copy link
Contributor

My question is not so much "what can we do?" but more "what do the npm folks tell us is the best approach?". I'd imagine that they'd rather only support one endpoint, and only have long-term plans for one. Just curious which approach works well for both us and them.

@marcellodesales
Copy link
Author

@drewfish Yeah, we have NPMO pointing to replicate.npmjs.com now and all our problems have gone away. In addition, it provided support for customers to load public @scoped libraries.

What I'm looking for here is that the registry-static can provide me the full visibility of the file-system. I could even turn-off the follower because all I wanna have is a service that's running in the background and serving the same files in the file-system that npmo is serving. Architecturally, that would be even simpler and we would avoid worrying about the follower.

billiegoose added a commit to billiegoose/modserv that referenced this issue Nov 14, 2016
@billiegoose
Replace skimdb.npmjs.com/registry with the newer URI replicate.npmjs.…
7123386 
…com (see davglass/registry-static#70)
billiegoose added a commit to wmhilton-contrib/package-stream that referenced this issue Jan 14, 2017
@billiegoose
replace 'skimdb.npmjs.com/registry' with 'replicate.npmjs.com'
6281c26 
There is a subtle "bug" with skimdb.npmjs.com. I am not entirely
sure what it is, but having run my own registry mirror for 6
months, I can say that when I used skimdb.npmjs.com/registry, I
would find that after a few weeks, it would not have the latest
versions of some packages. After switching to this other URL,
I have had zero problems with my mirror / registry follower. I
believe that the difference has something to do with scoped
packages.

Here is where I got the newer URL:
davglass/registry-static#70
@billiegoose billiegoose mentioned this issue Jan 14, 2017
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@davglass @marcellodesales @drewfish