Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
GitHub Community
Replies: 33 comments 58 replies
-
|
Hey @monishcm We are saying a huge drop of incoming events. Is that an expected change due to the optimizations you are making? |
Beta Was this translation helpful? Give feedback.
-
|
You should not see any change till 29th May as the old APIs will continue till then. This could well be an operational issue, and you can always ping our support if it is really a problem. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the quick response. We are investigating and will contact support if we believe it's an issue. |
Beta Was this translation helpful? Give feedback.
-
|
When building and maintaining a replica, I would expect to first paginate through the This approach worked great with the old endpoints. The old What is the recommended approach to do a bulk download and then transition to fetching updates? Is there a supported endpoint to check the current sequence value? |
Beta Was this translation helpful? Give feedback.
-
|
@monishcm @leobalter any insight on the above? |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I had a word with my engineering team, and they gave me a response on the feasible options. Can you help if this works for you? The latest sequence number can be found from the root endpoint: |
Beta Was this translation helpful? Give feedback.
-
|
That did start working for me, yes. I think that when I originally tested it that endpoint was returning an error. Now it's down again but I understand that's temporary. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
When I skimmed this initially I was under the impression that there were no breaking changes here if already using
The sequence numbers also seem to have jumped again. Last month the sequence numbers were around the The new API is definitely way better though and I'm sure a huge operational saving there, congrats on shipping these improvements! |
Beta Was this translation helpful? Give feedback.
-
|
The suggested (new) endpoints now return 401’s? {
"message": "This API is currently unavailable. Please see this for more information: https://github.blog/changelog/2025-02-27-changes-and-deprecation-notice-for-npm-replication-apis"
} |
Beta Was this translation helpful? Give feedback.
-
|
Can confirm it's currently returning errors: |
Beta Was this translation helpful? Give feedback.
-
|
Hi @guybedford and @jvandenaardweg, please see my reply here: https://github.com/orgs/community/discussions/152515#discussioncomment-12737872 |
Beta Was this translation helpful? Give feedback.
-
|
Hi there all! npm engineer here, we had to disable the new opt-in replication api for the weekend to investigate an incident that occurred this morning. We will be able to take a further look at it monday morning, thanks for your understanding! ❤️ |
Beta Was this translation helpful? Give feedback.
-
|
Hey, if the new API isn't ready yet, could you please postpone the brownout schedule? Right now, the old API is already down half the time, while the new one isn't available at all. |
Beta Was this translation helpful? Give feedback.
-
|
Hello! We are still working on some changes to the new replication api's related to Saturday's incident. In the meantime we disabled the brownout so everyone will still be able to access the old replication feed until the new one is back up. Thanks again for your understanding! |
Beta Was this translation helpful? Give feedback.
-
|
@shmam The replication endpoint returns 503 with or without opt-in header. Which endpoint should we use ? |
Beta Was this translation helpful? Give feedback.
-
|
The issue when accessing https://replicate.npmjs.com/registry is still happening. Have there been any updates? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @shmam, Are there any updates on this? I still can't use the APIs with the header. |
Beta Was this translation helpful? Give feedback.
-
|
I'm also experiencing issues when accessing the legacy APIs (503) |
Beta Was this translation helpful? Give feedback.
-
|
no news about this issue? |
Beta Was this translation helpful? Give feedback.
-
|
Hey all! Just posted an update: https://github.com/orgs/community/discussions/152515#discussioncomment-12753679 |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
As we are temporarily falling back to old replication feed here, any workaround for the |
Beta Was this translation helpful? Give feedback.
-
|
Is it just for me or are all the replication feed ( Tried: |
Beta Was this translation helpful? Give feedback.
-
|
Yes they are all down. |
Beta Was this translation helpful? Give feedback.
-
|
Seems like endpoints are back up, but as mentioned here by @jvandenaardweg seems like something is wrong with the seq ids. @shmam do you have any details about this? |
Beta Was this translation helpful? Give feedback.
-
|
I still have 503 for both APIs |
Beta Was this translation helpful? Give feedback.
-
|
Yup, seems like it is still coming up and going down over time. |
Beta Was this translation helpful? Give feedback.
-
|
@p-brito sorry for the question, but what are the two endpoints? I've been accessing with the https://replicate.npmjs.com/registry for over a year. I don't know the second one. |
Beta Was this translation helpful? Give feedback.
-
|
Here https://github.com/orgs/community/discussions/152515#discussioncomment-12761477 |
Beta Was this translation helpful? Give feedback.
-
|
Any update @shmam? How long will it take to fix this? |
Beta Was this translation helpful? Give feedback.
-
|
Hi all! We've re-enabled our new replication api after implementing some repair items from the previous incident. Please resume using it by adding the We've also resumed the brownout schedule for the old replication api. We are still investigating an issue with this legacy api and working to restore full access to that when the brownout schedule is not in effect. Thank you again for your patience. |
Beta Was this translation helpful? Give feedback.
-
|
Do we need to reset to an older sequence number as mentioned here https://github.com/orgs/community/discussions/152515#discussioncomment-12754432 ? |
Beta Was this translation helpful? Give feedback.
-
|
@monishcm Can you please clarify this? |
Beta Was this translation helpful? Give feedback.
-
|
It looks like there's something wrong with the But if I try to go to a later page with a request like It's even worse if I pick a later Am I doing something wrong or is there something broken with the |
Beta Was this translation helpful? Give feedback.
-
|
On the other hand using |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for reporting this! We're investigating. |
Beta Was this translation helpful? Give feedback.
-
|
@jhiesey This should be fixed now. |
Beta Was this translation helpful? Give feedback.
-
|
An update we will not support |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I'm getting "BadRequest" while using the "sorted" parameter in the _all_docs endpoint. |
Beta Was this translation helpful? Give feedback.
-
|
I just updated the guide. |
Beta Was this translation helpful? Give feedback.
-
|
The last seq of the old address is 41274032. |
Beta Was this translation helpful? Give feedback.
-
|
The curl -H 'npm-replication-opt-in: true' 'https://replicate.npmjs.com/registry/_changes?since=62200000'
{"results":[{"seq":62209148,"id":"@eth-optimism/tokenlist","changes":[{"rev":"1141-fafb58153a80dbafaec6c949c3e67e58"}]},{"seq":62209154,"id":"@believemotion/agents","changes":[{"rev":"8-946cb58bc3f9f7ba7e08e7509bd79de1"}]},{"seq":62209157,"id":"@questlabs/react-sdk","changes":[{"rev":"114-67cfc8179754b14b53b526b474a3b674"}]},{"seq":62209159,"id":"ngx-mat-table-toolkit","changes":[{"rev":"6-d4a4b2fa9f7f6479b723ba461c9672c7"}]}],"last_seq":62209159}curl -H 'npm-replication-opt-in: true' 'https://replicate.npmjs.com/registry/_changes?since=62200000'
{"results":[{"seq":62209157,"id":"@questlabs/react-sdk","changes":[{"rev":"114-67cfc8179754b14b53b526b474a3b674"}]},{"seq":62209159,"id":"ngx-mat-table-toolkit","changes":[{"rev":"6-d4a4b2fa9f7f6479b723ba461c9672c7"}]}],"last_seq":62209159} |
Beta Was this translation helpful? Give feedback.
-
|
What the heck? That's disgusting. |
Beta Was this translation helpful? Give feedback.
-
|
As far as I can tell these differences may seem to happen when once of the packages gets a newer revision later on in the change feed. |
Beta Was this translation helpful? Give feedback.
-
|
I think this has always been the case - the _changes API only returns the latest revision of a document. If foo@1.0.0 is published at seq id 001, and foo@1.0.1 is published in 002, then _changes would only include [002] if you called it after 002 was created. |
Beta Was this translation helpful? Give feedback.
This comment was marked as spam.
This comment was marked as spam.
-
|
Why is registry.npmjs.org telling us by an npm-notice response header that https://registry.npmjs.org is deprecated and to use replicate.npmjs.com instead? Is registry.npmjs.org going away? Is it not still the official registry endpoint? See npm-notice: This endpoint is deprecated. Use https://replicate.npmjs.com instead. |
Beta Was this translation helpful? Give feedback.
-
|
That seems like a bug. I only see it on HEAD requests (try curl -i to show headers with GET). |
Beta Was this translation helpful? Give feedback.
-
|
This was actually only the |
Beta Was this translation helpful? Give feedback.
-
|
Is the |
Beta Was this translation helpful? Give feedback.
-
|
https://github.com/orgs/community/discussions/152515#discussioncomment-12870976 This is not a problem, it's how this endpoint works at db level - deprecated versions of project record are removing. |
Beta Was this translation helpful? Give feedback.
-
|
I'm trying to catch up from sequence ID ~40,000,000 but there are events for package's that return 404. There's no later deleted: true message, I'm not sure what to do with these packages. I don't want to just skip them, I'm worried that when I'm caught up and checking just-published packages there will be a race between the replica and registry APIs and I might get 404s. |
Beta Was this translation helpful? Give feedback.
-
|
I have observed that the |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for this, we'll take a look! |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for reporting this, @jhiesey! This should be addressed, please let us know if this issue persists. |
Beta Was this translation helpful? Give feedback.
-
|
Why is seq one moment over 4 million and the next over 6 million？ |
Beta Was this translation helpful? Give feedback.
-
|
I used to have actual CouchDB replication (into CouchDB!) set up and struggled for days to find a configuration that actually worked and did not error all the time. Now thanks to this change, I need to do the same trial and error once again. I would love to have actual documentation for the replication settings somewhere. In addition, I don't see the reason for removing the include_docs parameter. This just leads to WAY more spam on the GET metadata endpoint. Maybe I am missing the point of this change, but to me it seems that actual replication is made much more difficult this way. Edit: I'm also using PouchDB and so far that has been a bad experience. All the deprecated parameters require tuning the config, and if you get anything wrong you get a very helpful |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately, you won't be able to perform a regular CouchDB replication going forwards. You can use |
Beta Was this translation helpful? Give feedback.
-
|
Then why still call this "replication" instead of "change feed"? Seems misleading. Also, what is the rationale behind removing a useful feature like this? |
Beta Was this translation helpful? Give feedback.
-
|
How are replicas meant to avoid race conditions when fetching pacuments from changes? |
Beta Was this translation helpful? Give feedback.
-
|
I've noticed that sometimes the _changes feed reports a new revision before the packument is actually updated on the registry side. I think this might be due to how things are eventually synced — probably some delay between the database and the API/cache. One thing that helps is checking if the rev in the packument matches the one from the change event. If it doesn't, waiting a bit and retrying usually works. The delay seems to vary, but it's usually short. |
Beta Was this translation helpful? Give feedback.
-
|
That's what I've settled on as well. If I get a packument with _rev that doesn't match the change I stop my replicator for a few minutes, then resume using the last successful seq id. However, I'm a little worried that very active package versions would never be replicated. The _changes API skips stale changes, so you won't get rev 1- if 2- was created. Probably not a real concern - the package would have to be published every 5 minutes (based on cloudflare response headers). |
Beta Was this translation helpful? Give feedback.
-
|
When I see a _rev mismatch between the packument and the change event, I also pause the replicator like you mentioned. But in addition, I try re-fetching the packument after a short delay, adding a unique query param (like ?t=timestamp) to bypass any cache. That usually helps get the correct revision. If the mismatch happens multiple times in a row, I mark that package for a retry later, without moving the seq_id forward. That way I don’t risk skipping over fast-moving updates, especially for packages that are getting published frequently. It’s not a perfect solution, but it’s been fairly reliable so far. |
Beta Was this translation helpful? Give feedback.
-
|
Another issue I found in implementing this approach is catching the items with a |
Beta Was this translation helpful? Give feedback.
-
This is currently nót the case, see: https://github.com/orgs/community/discussions/152515#discussioncomment-13120752 |
Beta Was this translation helpful? Give feedback.
-
|
Is there currently an issue with the service? When calling |
Beta Was this translation helpful? Give feedback.
-
|
Oh damn, what a mess is this migration 😅 Wasting too much time on this crap Sorry for the wording. But what are we supposed to do now? It went from 40m to 60m, and now back in the 40m range again? |
Beta Was this translation helpful? Give feedback.
-
|
The update_seq dropped from ~60m to ~40m, and all the changes around the current (42111860) Not that Script to compare changes to registryconst [
since = (42_000_000).toString(),
limit = (100).toString(),
] = process.argv.slice(2)
const changes = await fetch(`https://replicate.npmjs.com/_changes?since=${since}&limit=${limit}`, { headers: { 'npm-replication-opt-in': 'true' } })
.then(res => res.json())
const row = (pacumentRev, change) => {
const changeRevs = change.changes.map(c => c.rev)
const mark = changeRevs.some(r => r === pacumentRev) ? '✔️': '❌'
console.log([mark, change.id, pacumentRev, ...changeRevs].join("\t"))
}
console.log("match\tpkg\tnpm\treplica")
for (const change of changes.results) {
const { _rev } = await fetch(`https://registry.npmjs.com/${change.id}`).then(res => res.json())
row(_rev, change)
} |
Beta Was this translation helpful? Give feedback.
-
|
We discovered a race condition, which is what has been causing the sequence numbers jumping back and forth. That should be fixed now, so the sequence numbers are consistent. |
Beta Was this translation helpful? Give feedback.
-
|
@jumoel I'm still seeing miss match between the replica and registry server _rev. The current update_seq is 42155665. If I shave that down to 42155500 to get older, but still recent, changes I get Look at I only replicate a change with the same _rev because I need to know if I'm replicating an older cached version, which would cause me to miss the revision the event was actually for. |
Beta Was this translation helpful? Give feedback.
-
|
I'm seeing the same. The |
Beta Was this translation helpful? Give feedback.
-
|
As |
Beta Was this translation helpful? Give feedback.
-
|
@chendelbari That's correct, there's no way to bulk fetch docs anymore. |
Beta Was this translation helpful? Give feedback.
-
@jumoel so what's the allowed RPM for https://registry.npmjs.org/ |
Beta Was this translation helpful? Give feedback.
-
|
Dont give them ideas, next thing you know it becomes half of whatever the current rate of change is. This whole situation seems designed to hinder replication (starting from the fact that it no longer stays true to the name stemming from CouchDb which is no longer supported). It was probably abused by AI companies scraping their registry for code samples. |
Beta Was this translation helpful? Give feedback.
-
|
Question, how are we suppose to use the I assume the When I noticed this I started logging the mismatches, below are just a few. There are thousands of these happening. "processed revision" is the revision from https://registry.npmjs.com/, the "input revision" is from https://replicate.npmjs.com/ . Even after a few hours, the registry does not reflect the changes revision. There's not only a mismatch in the number part, but sometimes even the hash part does not match. Here is just one example of the many i see in my logs: |
Beta Was this translation helpful? Give feedback.
-
|
The revision value in the current change feed is a legacy revision value and will remain as such until the full migration to the new replication feed is completed. Starting May 29, 2025, at 12:00 UTC (the date of the full deprecation of the old api), the new replication feed will provide revision values that align with the packument served by Please keep in mind that the npm registry operates asynchronously, so there might be a delay between the event showing up in the replication feed and the same change being reflected in the packument. |
Beta Was this translation helpful? Give feedback.
-
|
@shmam thanks for clarifying! Would have helped alot if that info was included in the migration details in the first post. There are a few comments here talking about these rev issues. |
Beta Was this translation helpful? Give feedback.
-
Based on the above, the Root( What is going on? which api should we use to fetch this data? |
Beta Was this translation helpful? Give feedback.
-
You need to add the header to opt in |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the update. We'll ensure our systems use the new endpoints with the opt-in header before March 18, and fully switch over ahead of the May 29 deprecation deadline. |
Beta Was this translation helpful? Give feedback.
-
|
Something appears to be seriously wrong now. Because of the seq change mentioned in https://github.com/orgs/community/discussions/152515#discussioncomment-13053455, I tried to rerun our replication from scratch, and going from seq 0 up to 42347999, I received only around 260000 changes, which is less than 10% of all the packages. |
Beta Was this translation helpful? Give feedback.
and others
-
Overview
As part of our ongoing improvements, we are making changes to our replication API services. This is to ensure availability and performance of our feeds. Moving forward, only a limited set of API endpoints will be supported. Additionally, requests to
skimdb.npmjs.comwill now be redirected toreplicate.npmjs.comvia a301 Moved Permanentlyresponse. All other endpoints will be deprecated and return a404 Not Foundresponse.To facilitate a seamless transition, the new endpoints will be available starting
Tuesday, March 18, 2025 (12:00 UTC), operating in parallel with the existing endpoints. The existing endpoints will be fully deprecated onThursday, May 29, 2025 (12:00 UTC). During the transition period, you may access the new endpoints by including thenpm-replication-opt-inheader with the valuetruein your requests (wherein they will be ignored thereafter)Supported Endpoints
Going forward, the following API endpoints will be supported (with some limitations):
GET https://replicate.npmjs.com/registry/_changesGET/POST https://replicate.npmjs.com/registry/_all_docsHEAD/GET https://replicate.npmjs.com/Changes and Limitations
Each supported endpoint will now have a limited set of supported parameters. Requests using unsupported parameters will not function as expected.
_changesEndpointThe
_changesendpoint will continue to function for paginating through package changes. The following parameters will be supported:Supported Parameters
doc_idsdescendinglast-event-id(andLast-Event-IDheader)limitsinceDeprecated Parameters
The following parameters will no longer be supported:
conflictsfeedfilterheartbeatinclude_docsattachmentsatt_encoding_infostyletimeoutviewseq_intervalBehavioral Changes
doc_idswill now behave as iffilter=_doc_idsis also supplied.feedwill behave as ifnormalwas passed.stylewill behave as ifmain_onlywas passed._all_docsEndpointThe
_all_docsendpoint will continue to support bothGETandPOSTmethods for paginating through packages in the registry.Supported Parameters
descendingendkey,end_key,endkey_docid,end_key_doc_idstartkey,start_key,startkey_docid,start_key_doc_idinclusive_endkeykeyslimitDeprecated Parameters
The following parameters will no longer be supported:
conflictsgroupgroup_levelinclude_docsattachmentsatt_encoding_inforeducestablestaleupdateupdate_seqsorted(The results are always sorted)skip(Usestartkey,start_key,startkey_docid,start_key_doc_idinstead)Root Endpoint (
/)The root endpoint will continue to supply the following parameters in JSON output:
db_nameenginedoc_countupdate_seqImportant Notes
include_docsDeprecation and Direct packument AccessThe
include_docsparameter will not be supported in any endpoint. Users should instead fetch package metadata directly fromhttps://registry.npmjs.org/<package-name>to ensure consistency.Note: Users should transition from fetching package metadata from
https://replicate.npmjs.com/tohttps://registry.npmjs.org/<package-name>.https://replicate.npmjs.com/will no longer be supported for fetching packuments.limitParameterThe
limitparameter will have a default value of 1000 and a maximum value of 10000 across all endpoints.Streaming Responses
Support for streaming responses via
feed=longpoll,feed=continuous, andfeed=eventsourcewill be removed. Users should implement manual pagination using sequence IDs via thesinceparameter.Migration Steps
https://replicate.npmjs.com/registry/instead ofskimdb.npmjs.com/.https://registry.npmjs.org/<package_name>instead ofinclude_docsfor retrieving package metadata.since.limitvalues in requests adhere to the new default (1000) and maximum (10000) constraints.Timelines for the migration
The newly migrated endpoints will be accessible beginning
Tuesday, March 18, 2025 (12:00 UTC). These endpoints will operate concurrently with the existing endpoints until the latter are deprecated onThursday, May 29, 2025 (12:00 UTC). To access the new endpoints, please include thenpm-replication-opt-inheader with the valuetruein your requests (the header will be ignored thereafter).Brownout Phases for Testing
A series of brownout phases will begin on
Wednesday, April 02, 2025 (12:00 UTC), and continue until the complete deprecation of the old endpoints onThursday, May 29 , 2025 (12:00 UTC). During this period, deprecated endpoints will be intermittently blocked. To avoid disruptions, you can access the new endpoints by including thenpm-replication-opt-inheader with the valuetruein your requests (untilThursday, May 29, 2025 (12:00 UTC)wherein the header will be ignored thereafter).The Brownout is phased into 3 blocks.
The planned schedule is as follows:
Phase 1:
April 2 to April 22 (21 days)- Recurring periods of2 hoursof unavailability from12:00 UTC, followed by2 hoursof availabilityPhase 2:
April 23 to May 13 (21 days)- Recurring periods of4 hoursof unavailability from12:00 UTC, followed by2 hoursof availabilityPhase 3:
May 14 to May 29 (16 days)- Recurring periods of10 hoursof unavailability from12:00 UTC, followed by2 hoursof availabilityWe strongly recommend using this transition period to test your updated replication implementations and provide feedback.
Additional Support
For more details, join the discussion in this post (literally this post 😄). This is the go-to place for questions and comments. If you have specific questions or need assistance, please reach out to our support team.
Thank you for your cooperation as we work to enhance the replication API experience!
Beta Was this translation helpful? Give feedback.
All reactions