Extractor Freezing on "Fetching series order"

[footbehindhead]

The extractor keeps freezing on the "Fetching Series Order for books in series" (see below image).

I have tried running the extractor multiple times, but it can never get past this phase.

[joonaspaakko]

@footbehindhead, can you do the extraction with the browser console open and take a screenshot of the red error message after it stops. The error message I'm after is likely the last one and slightly different than the rest (there should be quite a few that don't cause any kind of freeze).

[footbehindhead]

Below is the screenshot. Please let me know if you need anything else.

[footbehindhead]

Somehow the issue resolved itself. I had it run over an hour without any success. I uninstalled the add-in and reinstalled it and it still wouldn't work. I waited a week and tried again, and it still froze. But when I was running it to get this screenshot, it finally worked.

[joonaspaakko]

Good! Unfortunately the error log doesn't quite help nail it down... It migh end up as an unsolved mystery. I'll keep this issue open and hope I manage to recreate it.

[footbehindhead]

I removed my extracted data and reran the extractor to see if it would freeze again. It did. I waited about an hour to see if it would progress, and it did not, so I cancelled and re-ran it and it froze again. Below is the error log screenshot. There are more errors than what is shown here, but these are the last ones. If you need anything else, please let me know.

[joonaspaakko]

So in the first picture you were getting this err_http2_protocol_error and now err_connection_closed 200, both of which I'm entirely unfamiliar with. They seem to have something to do with the internet connection and fixes are saying all kinds of things like: use a vpn, disable vpn, make sure antivirus or firewall isn't blocking the website... And like a million other things. Is it possible something was different with your connection when it did work? It's definitely working for me. Maybe try another browser?

You probably don't have to wait for an hour. If it's stuck in a single item for like 1-5 minutes, it's likely stuck forever.

[footbehindhead]

It could be the firewall on my work computer. But every time that I have ran it, it was from the same computer, the same browser (Chrome), at the same location. I'm not sure what was different the one time it didn't hang up. I don't know if it helps, but I re-ran it and it hung up again, but it looks like the error message is different (pic below.) I will try to run it in firefox and see if that works.

[footbehindhead]

I ran it in firefox and it did work there (from the same computer connected to the same network).

[joonaspaakko]

@footbehindhead, haven't found anything obvious yet. It's kind of a mystery why it works occasionally.

I'd like for you to test a build I made that will print out extra console logs. Take a screenshot of the console if and when it gets stuck.

Maybe you've done this already, but in case you still have a gallery with data successfully exported, you can save the raw data on your hard drive and then import it back later... or in another browser. Here's what documentation has to say about it.

0. Download this build audible-library-extractor-v0.0.0-series-test.zip
1. Go to the address: chrome:extensions.
   • Or: Window > Extensions
2. Disable the current extension (to avoid conflicts). No need to remove it, in fact it might be better to keep it.
3. Turn on the developer mode from the top right
4. Drag the downloaded .zip file in the browser window to install
5. Try running the extraction with the browser console open and take a screenshot if it gets stuck.
   • No need to take a screenshot if it doesn't get stuck.
   • I should only need to see the last few of the console messages (just like the screenshots you've taken before).
   • If you could actually also right-click the console window and save as... and save the console log and send it to me via email. Just in case there's something important that didn't fit the screen.

I'm hoping this would tell me which series it was trying to process right when it got stuck, so I can try running it through the extension by myself and recreate the issue you're having.

[footbehindhead]

I sent you the log yesterday, but I forgot to post the screen shot. I reran it today and it still froze on the library step. Below is the screenshot. I don't know if you will need it, but I will e-mail you the log from today's attempt as well.

[joonaspaakko]

No luck. I tried to extract every single series and book from both console logs, but got nothing from either test. But then again, given that I'm from Europe, it could even be caused by a book or a series that I can't access, in which case the extraction would just skip any potential issues there. That said I'm very suspicious about those weird connection errors you're getting.

I'll see if I can come up with other ideas, but I wonder if you could see if it works in Firefox consistently or perhaps another Chromium based browser like (Brave or even the new Edge) — The Firefox (or Chrome) web store version should be enough and if it works, it'll probably work better than the dev build I asked you to test before. I'm seeing yet another new error message ERR_NETWORK_CHANGED and quick googling gives me similar descriptions and solutions as the other two, like all kinds of network related things and Chrome seems to always be mentioned in these, but I'm not sure if it's a coincidence.

As far as your Chrome goes, I wonder if updating it would help and perhaps disabling other extensions. If you have many extensions, the easy way is to go to chrome:extension, open Audible Library Extractor → "details" and enable "Allow in incognito". Then open a new incognito window and ALE should be the only extension enabled in that window... The only issue with that is that it is then kinda hard to use more than just that one time because you have to login every time and it's just inconvenient.

[footbehindhead]

No worries. It works from Firefox, and if I keep trying over and over, it eventually will work in Chrome. Once it works once in Chrome, as long as I don't remove the extracted data it seems to keep working even as I add new books.

I don't know if it is a related issue of not, but I have 10 collections (around 50 to 200 books each) and it does not always extract all of the books. If I keep running the extractor, it will get all of the books, but not every time. It's not a huge issue and if I just keep running the extractor, I can get the data I am looking for, but just wanted to let you know in case you haven't heard of this before and in case it might be related.

If you need any additional information from me to help, let me know.

[joonaspaakko]

Hey @footbehindhead, when you said it also happened in collections I felt I got a bit closer to a possible cause so I started remaking how both of those are extracted.

Series extraction is now a 3 step process.

It's not really doing anything more than before, in fact, it's doing a little bit less than it was before, but it's done in a bit more controlled way now, which also makes it a bit slower and evenly distributed so that there won't be insane bursts of requests all of a sudden. And because of this "slow down" I also let up a little on limiting parallel requests in general, because I feel like it might be able to handle it now.

Collections actually had an issue anyways, because Audible recently took away the archive collection from the collections page on the website...

This worked out even better than before because now the collections ids are fetched from the library pages and all that is left to do in the collections step is to check the first page of every collection to find out the title and description. It was relatively speedy before, but it's much faster now and if this makes it freeze, I'd be surprised.

So if you could try this build:

0. Download ⬇️ audible-library-extractor-v0.2.9-series-test-v2.1.zip
1. Go to the address: chrome:extensions.
   • Or: Window > Extensions
2. If you still have the previous test build installed, remove that, but disable any other builds.
3. Turn on the developer mode from the top right (If needed)
4. Drag the downloaded .zip file in the browser window to install
5. Try running the extraction. Extract library + collections only (wishlist still doesn't work for unrelated reasons)
   • Keep the dev tools console closed. At this point, I'm not expecting to see anything interesting there, but if it does freeze, open the console then and take a screenshot of the tail end.

[joonaspaakko]

@footbehindhead, there's a new (similar) issue with the build I posted about. It turns out that quick testing past midnight might not always be sufficient... Anyways I'll do some of my own tests and I'll probably ping you with another comment once that is done. If by any chance you downloaded this one already, delete it and I'll come back to you with the new build to test later this week probably.

[joonaspaakko]

@footbehindhead, I've now updated the download link 2 comments up ↑

[footbehindhead]

I uninstalled the previous builds and ran your new build. The first time, it worked perfectly, but the then I cleared the data and re-ran the extension and it froze again. Below is the screen shot.

[footbehindhead]

Not sure if it matters, but I ran it again, and it worked.

So the first time, it worked
The second time (after I cleared the extracted data), it hung up on step 5
The third time (after I cleared the extracted data), it worked again.

[joonaspaakko]

@footbehindhead, thanks for testing. I think the issue is some kind of unfulfilled request... I need to do some more testing on my end.

[joonaspaakko]

@footbehindhead, I have a new build for you to test. I think this should do it.

There's now a hard 1 minute time limit per request. This could potentially cause chunks of data to be lost, but realistically even a slow connection should finish a single request way before that minute is up. So now if it seems to get stuck, if you wait a minute, it should move on. I'm feeling incredibly optimistic about this one. I super hope it works.

Here's the new build: audible-library-extractor-v0.2.9-series-test-v3.1.zip (updated 21.04.2022)

[footbehindhead]

I just ran it three times with clearing the extracted data before each time. All three were successful. None of them froze and all of the extracted all of the items for all of my collections.

There are 2 additional collections. I was able to figure out that the one labeled "All You Can Listen" are the free ones from the Audible Plus catalogue. This is a super cool addition. I'm not sure what the other collection labelled "Item a User has Purchased" is for. It only contains one book, the last one I bought.

It appears that with this build, the issues are fixed. Let me know if you would like for me to do any more testing.

[joonaspaakko]

Hey, @footbehindhead, just to be sure... Was the build you tested the one with the v3.1 in the filename? Not the actual extension version, just the filename. If you tested that one, it's all good. I updated the link twice yesterday and the earlier build v3 had an issue where it just straight up skipped fetching most things to do with series due to a typo :/

---

The collections thing is an interesting one. They got rid of the "archive" in the collections page so none of the public releases of the extension can fetch archives. I went to fix it (#89) and found out that they list all collections a book is in in the library pages (it's just hidden), so I just picked up everything it was giving me and turns out there are hidden collections. You can access them in audible if you know the id: https://www.audible.com/library/collections/__AYCL (and if you have any books there) — With this discovery I could've also fetched archive using the old method if I had just force fed it the right address, but this new way of extracting collections is way faster.

The only somewhat useful is the All you can listen, but even that one is perhaps unnecessary since the gallery already has a filter for Plus Catalog books. But if it's there, I'm not going to hide it in the gallery. Another one I had was User wishlist, which has books that are in the library and in the wishlist, but that doesn't seem too useful.

Some of the hidden collections, like the user has purchased seem a little broken. Like I'm pretty sure you've purchased more than one book, right? That collection didn't surface in my library, so I guess I haven't purchased anything :) If you can give me the id for it, I'd like to exclude it. In ALE's gallery the collection is the last thin in the address bar when you are viewing the collection page: chrome-extension://gbi7qpbsrc3gywuwzq6p/gallery/index.html#/collections/__FAVORITES ← __FAVORITES is the id.

I also had (__PENDING) "Pending Items Not In Library" → "Titles that the user has ordered but are awaiting fulfillment" but I already excluded it because it didn't offer anything useful. It listed like 200 of "pending purchases"... all of which sit in my library. I think some of these hidden collections are meant for internal stuff... perhaps for the support to look at.

---

This is only tangentially related, but I also added a new Goodreads "genres" sidebar inspired "drawer" in book details → left sidebar where you can see what collections a book is in. Of course this is only based on your own collections. Clicking the collection name opens the collection inside the gallery. The number is how many books are in the collection.

[footbehindhead]

I think I used the correct one. This is the version I used:

The ID for the Purchased that is currently only showing the last book I bought is: __PURCHASE

The added in collection feature is working on my extract. It's neat because then I can see how many times I've listened to a book (I make a separate collection for each year)

Another thing that I noticed is that the collections (and the library) extractions are excluding podcasts that are in the library. I don't know if this is something that really matters, but it did make my extractions have fewer entries than my audible collection. I listen to very few podcast via Audible, so it's not a big deal to me, but it's something I noticed so I thought I would share it with you.

Below is an example:

The Dr. Katz: The Audiobook is a normal audiobook and works fine. The Dr. Katz: The Audio Files is a podcast. You can see the arrow in place of the listen now and down load buttons. When you click it, it pulls up a separate page with all of the episodes (see screen shot below):

When I add these to my collection, I have to add each episode separately. They show up in my collection that way, but they are currently being excluded in the extraction. Again, for me personally, this is fine. I don't listen to a lot of podcasts via Audible, but it might be something you wanted to look at.

[joonaspaakko]

Looks to be the correct version. I'll probably close this issue whenever I release the next version. And whenI do, you can still comment in here if it comes back or something.

Yea, podcasts have been purposely ignored due to the different structure... I've also not listened to a single one and you're the first to mention it, I think. It could definitely be done. I'll probably make a new (feature request) issue about that... But I wouldn't expect it to be in the next release... Or maybe ever, unless it becomes a huge request. I will look into it though.

[TheSidXXX]

I am now having this problem as well. I will try again tomorrow to see if it is just a temporary glitch on Audible's site. I did download the 0.2.9 v3.1 linked above, but it had the same issue, albeit at a different spot. 0.2.8 hung up on the series extraction, 0.2.9 hung up on the wishlist extraction.

[joonaspaakko]

Did you get this #110? Well, I think in the 0.2.9 draft wishlist is extracted first, so probably not. I did recently have a thing where it stopped after wishlist was done because Audible decided I had to login to see my library.

[TheSidXXX]

Did not get an error message, it said "fetching series order... 38/39" and was on 38 for a good half hour before I killed it. Once I killed it, it made me log in again, so that may be part of it. When I tried 0.2.9 v3.1 it was Wishlist 2/2 that it got stuck on, but again, no error message.

I deleted 0.2.9 and reinstalled 0.2.8, and the behavior was identical, except the number was much higher, because it was extracting from scratch. So it got stuck on "fetching series order..." again. I don't remember the actual number this time (a couple hundred at least, and it got stuck on the second to last one).

[joonaspaakko]

0.2.8 extraction is definitely way more broken and it might seem like it gets further, but to me it really just sounds like that's only (technically) because wishlist is extracted first in 0.2.9. I haven't tried in almost a year, but last time I checked, 0.2.8 stopped during wishlist extraction, since it's still trying to extract the old wishlist layout that hasn't existed since last summer.

I would suggest you try extracting library and wishlist separately in 0.2.9 and see how that goes. Or just try both together for the second time. If it asked you to login afterwards it seems pretty clear that what happened was that you still had access to the wishlist during the extraction, but when it started extracting the library, your library auth had expired and there's no way around that for the extension. You need to be logged in to extract stuff in your account. These types of issues happen when you haven't logged in for a while. After you do login, it shouldn't be a problem for a good while, which is why I would hazard a guess that a second try would've gone through.

This has always been an issue in every version, just the other way around, where it stops just before the wishlist extraction starts. Because I do the wishlist access check first, I thought extracting it first would solve the problem where the wishlist auth drops midway, but I suppose it only flipped the issue. The extension could potentially warn you about what happened, but it's really elusive and I haven't been able to catch the type of error it triggers, or anything that I can latch onto.

It would help if you could get a screenshot of your browser console after it stops. Like at least a few of the last messages, error or not.

Chrome's console opens with:

• Windows Ctrl + Shift + J
• Mac Cmd + Option + J

[joonaspaakko]

Oh and I was somewhat confused about v.3.1 until just now that I remembered I had posted a hotfix version in this thread, which is super old at this point. I ignored my confusion because I was pretty sure I had given you a link to the most current version before. This zip in the repository root is the one I would recommend you to use: audible-library-extractor-0.2.9.zip

I'm repeating myself here (sorry), but now that I know which versions we had been discussing, this 0.2.9 version should be the only one that has any hope to extract your wishlist. If that is the only issue with it, I'd suggest you try logout, login and try extraction again (just in case... though since you probably did recently login it should be unnecessary). Or alternatively try to extract wishlist and library separately: first just one of them and when that extraction is done back on the website to extract the other.

[TheSidXXX]

I forgot you gave me 0.2.9 already, lol! I tried that today, and appears to have worked. Thanks!

One note on the issue of being logged out -- I tried multiple times last time, and every time, it would ask me to log in when it finished running, or when I killed it. So, it was almost like something about the process was forcing a re-login.

I will play around with 0.2.9 and let you know if I have any other issues.

[dreamweaver529]

I was having the same issue while running 0.2.8 with the following error freasing the process

After installing 0.2.9 it worked like a charm. Thanks for all your hard work.

[J45gal]

First time here, getting the same freeze at series fetching. I'm running it on the newer Edge. Could you send me v.0.2.9? Thanks!

[joonaspaakko]

@J45gal

Until the new version actually comes out... For anyone who needs it, installation instructions for the draft:

1. Download the 0.2.9 draft (This file sits in the repository root and may change as I upload new drafts occasionally)
2. Go to the address: chrome:extensions (or from the menu in top right: ⋮ > Extensions > Manage extensions)
3. Turn on the developer mode from the top right
4. Drag the downloaded .zip file into the browser window to install

The new version is coming out one of these days.

[J45gal]

Thanks for your labor of love! It'll take me an evening to figure it out but I hope it works.  emily*

…

-----Original Message----- From: Joonas Pääkkö ***@***.***> To: joonaspaakko/audible-library-extractor ***@***.***> Cc: J45gal ***@***.***>; Mention ***@***.***> Sent: Thu, Jun 15, 2023 3:29 am Subject: Re: [joonaspaakko/audible-library-extractor] Extractor Freezing on "Fetching series order" (Issue #87) @J45galUntil the new version actually comes out... For anyone who needs it, installation instructions for the draft: - Download the 0.2.9 draft (This file sits in the repository root) - Go to the address: chrome:extensions (or: Window > Extensions) - Turn on the developer mode from the top right - Drag the downloaded .zip file in the browser window to install The new version is coming out one of these days. If I put off adding new stuff I think it's almost there. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[brianfreud]

0.2.9 did the trick, thanks! First time I've been able to successfully extract in a few weeks. :)

[samlevan]

Ditto here. The extension was freezing but 0.2.9 worked perfectly. Thank you so much for such a beautiful piece of work. 🙏

[boisjere]

Just to add to the growing chorus. I had this exact problem on 0.2.8, and it went away on 0.2.9

[MatthewAlner]

0.2.9 fixed my "Fetching series order for books in series..." 🥶 freezing issue too. Thanks!

[JamesP7]

Thank you for this lovely Audible library extractor! 0.2.9 also fixed the "Fetching Series..." freeze for me :)

[kulwych]

yay! glad i found this post. 0.2.9 also did it here for me to. Was starting to get bummed with it getting stuck on the last scan.

[joonaspaakko]

@kulwych (and others), I pushed new changes yesterday (8h ago) that introduced a new bug in the gallery search bar where you can't insert a space unless you type the first letter of the next word really fast. I suspect I will get that fixed pretty soon.