npm replication API changes and migration guide

[monishcm]

Overview

As part of our ongoing improvements, we are making changes to our replication API services. This is to ensure availability and performance of our feeds. Moving forward, only a limited set of API endpoints will be supported. Additionally, requests to skimdb.npmjs.com will now be redirected to replicate.npmjs.com via a 301 Moved Permanently response. All other endpoints will be deprecated and return a 404 Not Found response.

To facilitate a seamless transition, the new endpoints will be available starting Tuesday, March 18, 2025 (12:00 UTC), operating in parallel with the existing endpoints. The existing endpoints will be fully deprecated on Thursday, May 29, 2025 (12:00 UTC). During the transition period, you may access the new endpoints by including the npm-replication-opt-in header with the value true in your requests (wherein they will be ignored thereafter)

Supported Endpoints

Going forward, the following API endpoints will be supported (with some limitations):

• GET https://replicate.npmjs.com/registry/_changes
• GET/POST https://replicate.npmjs.com/registry/_all_docs
• HEAD/GET https://replicate.npmjs.com/

Changes and Limitations

Each supported endpoint will now have a limited set of supported parameters. Requests using unsupported parameters will not function as expected.

_changes Endpoint

The _changes endpoint will continue to function for paginating through package changes. The following parameters will be supported:

Supported Parameters

• doc_ids
• descending
• last-event-id (and Last-Event-ID header)
• limit
• since

Deprecated Parameters

The following parameters will no longer be supported:

• conflicts
• feed
• filter
• heartbeat
• include_docs
• attachments
• att_encoding_info
• style
• timeout
• view
• seq_interval

Behavioral Changes

• Supplying doc_ids will now behave as if filter=_doc_ids is also supplied.
• feed will behave as if normal was passed.
• style will behave as if main_only was passed.

_all_docs Endpoint

The _all_docs endpoint will continue to support both GET and POST methods for paginating through packages in the registry.

Supported Parameters

• descending
• endkey, end_key, endkey_docid, end_key_doc_id
• startkey, start_key, startkey_docid, start_key_doc_id
• inclusive_end
• key
• keys
• limit
• skip
• sorted

Deprecated Parameters

The following parameters will no longer be supported:

• conflicts
• group
• group_level
• include_docs
• attachments
• att_encoding_info
• reduce
• stable
• stale
• update
• update_seq

Root Endpoint (/)

The root endpoint will continue to supply the following parameters in JSON output:

• db_name
• engine
• doc_count
• update_seq

Important Notes

include_docs Deprecation and Direct packument Access

The include_docs parameter will not be supported in any endpoint. Users should instead fetch package metadata directly from https://registry.npmjs.org/<package-name> to ensure consistency.

Note: Users should transition from fetching package metadata from https://replicate.npmjs.com/ to https://registry.npmjs.org/<package-name>. https://replicate.npmjs.com/ will no longer be supported for fetching packuments.

limit Parameter

The limit parameter will have a default value of 1000 and a maximum value of 10000 across all endpoints.

Streaming Responses

Support for streaming responses via feed=longpoll, feed=continuous, and feed=eventsource will be removed. Users should implement manual pagination using sequence IDs via the since parameter.

Migration Steps

1. Update API Endpoints: Ensure all API requests are directed to https://replicate.npmjs.com/registry/ instead of skimdb.npmjs.com/.
2. Adjust Query Parameters: Remove unsupported parameters and update requests accordingly.
3. Fetch Metadata Separately: Use https://registry.npmjs.org/<package_name> instead of include_docs for retrieving package metadata.
4. Implement Pagination: Switch from streaming responses to manual pagination using since.
5. Enforce New Rate Limits: Ensure that limit values in requests adhere to the new default (1000) and maximum (10000) constraints.

Timelines for the migration

The newly migrated endpoints will be accessible beginning Tuesday, March 18, 2025 (12:00 UTC). These endpoints will operate concurrently with the existing endpoints until the latter are deprecated on Thursday, May 29, 2025 (12:00 UTC). To access the new endpoints, please include the npm-replication-opt-in header with the value true in your requests (the header will be ignored thereafter).

Brownout Phases for Testing

A series of brownout phases will begin on Wednesday, April 02, 2025 (12:00 UTC), and continue until the complete deprecation of the old endpoints on Thursday, May 29 , 2025 (12:00 UTC). During this period, deprecated endpoints will be intermittently blocked. To avoid disruptions, you can access the new endpoints by including the npm-replication-opt-in header with the value true in your requests (until Thursday, May 29, 2025 (12:00 UTC) wherein the header will be ignored thereafter).

The Brownout is phased into 3 blocks.

The planned schedule is as follows:
Phase 1: April 2 to April 22 (21 days) - Recurring periods of 2 hours of unavailability from 12:00 UTC, followed by 2 hours of availability
Phase 2: April 23 to May 13 (21 days) - Recurring periods of 4 hours of unavailability from 12:00 UTC, followed by 2 hours of availability
Phase 3: May 14 to May 29 (16 days) - Recurring periods of 10 hours of unavailability from 12:00 UTC, followed by 2 hours of availability

We strongly recommend using this transition period to test your updated replication implementations and provide feedback.

Additional Support

For more details, join the discussion in this post (literally this post 😄). This is the go-to place for questions and comments. If you have specific questions or need assistance, please reach out to our support team.

Thank you for your cooperation as we work to enhance the replication API experience!

[mrgrain]

Hey @monishcm We are saying a huge drop of incoming events. Is that an expected change due to the optimizations you are making?

[monishcm]

You should not see any change till 29th May as the old APIs will continue till then. This could well be an operational issue, and you can always ping our support if it is really a problem.

[mrgrain]

Thanks for the quick response. We are investigating and will contact support if we believe it's an issue.