[3.3.3] The feed is empty

[bristow]

Checklist

• I am able to reproduce the bug with the latest version.
• I made sure that there are no existing issues - open or closed - which I could contribute my information to.
• I have taken the time to fill in all the required details. I understand that the bug report will be dismissed otherwise.
• This issue contains only one bug, that only occurs in Squawker

Affected version

3.3.3+300003187

Steps to reproduce the bug

1. Go to the feeds tab
2. Look at the page

Thanks for this app !

Expected behavior

No response

Actual behavior

No response

Screenshots/Screen recordings

No response

Logs

No response

Affected Android/Custom ROM version

No response

Affected device model

No response

Additional information

No response

[j-fbriere]

Thank you for bringing this up.
It just happened, an hour ago everything was ok.
Twitter/X is going mad... again!
** Sigh **
I'll look into an alternative request for the feeds... again.

[j-fbriere]

I just looked at the twitter search requests (the "feed" like requests) from nitter, which were working today.
Right now it is not working.
Grr...
Maybe we will have to be a little bit patient.
Last time it lasted about a week.
Hopefully it won't be that long this time.

Still looking for alternative requests..

[zedeus]

I tested all GraphQL endpoints and the older 1.1/2 endpoints ever used by Nitter, and they all return 404 to the 9 bearer tokens I tried. This is a tough one, not really sure where to go from here.

[ghost]

I'm also working on a fix, I also have the same findings. I feel like twitter shut down the https://api.twitter.com/1.1/guest/activate.json endpoint. You can also scrape from the syndication endpoints.

I get an error 403.

[zedeus]

I can refresh the guest token just fine using the activate.json endpoint, but all the other endpoints say "Sorry, that page does not exist".

[zer0fl0w]

zedeus/nitter#983 (comment)

\o/

[j-fbriere]

zedeus/nitter#983 (comment)

Yes I already saw it yesterday, thank you.

I'm looking for a way to implement and test this alternative approach, which has many restrictions but seems workable, at least to some extent.
Nitter is a server. Its solution has created many guest accounts and is using many proxies (to use many different IPs).
So it's a totally different architecture than squawker (or fritter, quacker), which is a client application.
There are limitations to create Twitter guest accounts (I tested on my machine and I could only create one per day).
Also, those guest accounts are limited in time (a month before expiration) and in the number of requests for each Twitter endpoint.
I must implement some kind of internal management of the guest accounts and the number requests, which is complicated.

But I'm working on it.

[yozachar]

I'm afraid this might be unpopular, but why can't Fritter/Squaker piggyback Nitter, instead of re-doing what the Nitter does?

This might reduce the direct load on X/Twitter servers and help more developers focus on improving Nitter's 'scraping' capabilities.

[j-fbriere]

Version 3.4.0 fixes the issue.

[zer0fl0w]

sadly i can not confirm.

do i have to do something special with v3.4.0 in order to get it to work?

[zer0fl0w]

v3.4.1 - no difference. not working for me.

[j-fbriere]

@zer0fl0w just for testing, can you do a user search with elonmusk ?
Does a list of suggestions appear?
If a list of suggestion appears and you select elonmusk, does it show its profile?

[zer0fl0w]

i did.

same problem as here

even with elonmusk...

fyi: do you use blocklists, like nitter? zedeus had to manually whitelist vpn providers... i use mullvad...

[Saroumane]

i did.

same problem as here

even with elonmusk...

fyi: do you use blocklists, like nitter? zedeus had to manually whitelist vpn providers... i use mullvad...

I also use Mullvad and I had to disable the VPN to 'bootstrap' Squawker.
After the initial reloading it seems to work, even with the VPN re-enabled.

[j-fbriere]

@Saroumane Well it seems to show that the problem is to create a temporary Twitter/X guest account when on a VPN.
Once the Twitter/X guest account is created (given that there is no VPN), the Twitter/X API requests seem to work afterwards even with the VPN activated.
So it seems that there is a workaround here.
I made the temporary Twitter/X guest accounts valid for 10 days. So if this hypothesis is right, then this workaround should be applied every 10 days, unless I find a solution for the VPN problem.
I could make the temporary Twitter/X guest accounts valid for 20 days though, as long as its less than a month, which seems to be the limit imposed by Twitter/X.

[zer0fl0w]

can confirm.

initializing squawker without vpn works and continuoues working afterwards with vpn enabled.

thx saroumane.

and many many thanks jean for your work. hype.
sorry i blamed your update when i was just to stupid before.

[j-fbriere]

@zer0fl0w thanks to confirm the workaround.
You are certainly not stupid and your feedback is well appreciated and very important because you are testing the fix for use cases that I did not think of or test.
Also thanks @Saroumane.

[Saroumane]

Well, it's a collaborative effort :)
Thanks @j-fbriere for all the coding work obviously, but I would never have thought of disabling the VPN if @zer0fl0w didn't mention it.

[yozachar]

@j-fbriere

Feed	Profile	Search (Profile)	Search (Tweet)	Replies

Videos are working. Same results, with or without VPN.

[j-fbriere]

@joe733 Thank you for the feedback.
Just to be clear, when you show that "Profile" is not working, you mean when you tap on a subscription, right?
When you show that "Replies" is not working, you mean when you tap on a tweet, right?
Finally, do you get those errors with the version 3.4.1?

[yozachar]

Just to be clear, when you show that "Profile" is not working, you mean when you tap on a subscription, right?

Yes.

When you show that "Replies" is not working, you mean when you tap on a tweet, right?

Yes.

Finally, do you get those errors with the version 3.4.1?

Nope, all of these work for now in 3.4.1.

Feed	Profile	Search (Profile)	Search (Tweet)	Replies

---

You can tabulate screenshots in README.md. Feel free these images as well.

| Feed | Profile | Search (Profile) | Search (Tweet) | Replies |
| --- | --- | --- | --- | --- |
| ![img-2023-08-23-063806_002](https://github.com/j-fbriere/squawker/assets/38415384/2361ca6d-b109-4960-bd3a-34d3f1bee8a1) | ![img-2023-08-23-110642_002](https://github.com/j-fbriere/squawker/assets/38415384/e4405bff-331a-4c55-8390-d7d89d77d597) | ![img-2023-08-23-063806_003](https://github.com/j-fbriere/squawker/assets/38415384/3de7dbc1-0e6e-4747-b328-1c0f5eee5fd5) | ![img-2023-08-23-063806_004](https://github.com/j-fbriere/squawker/assets/38415384/7991c01b-ad6c-4624-b719-a3c184a34dee) | ![img-2023-08-23-110642_003](https://github.com/j-fbriere/squawker/assets/38415384/ef69fbea-df88-4207-800e-64c81f645248) |

[dovah19]

Feed is always empty on 3.4.1
Profile or searches throw error. Once in 10 tries the profile loads.
I also tried disabling the adaway VPN, but that gave same results.

[j-fbriere]

@dovah19 how many subscriptions do you have in squawker?

Also, did you try the following:

• Close squawker (important!)
• Disable VPN
• Open squawker

[dovah19]

@dovah19 how many subscriptions do you have in squawker?

Also, did you try the following:

• Close squawker (important!)
• Disable VPN
• Open squawker

I have 14 subscriptions.
I tried your method but same results. Here are ss after disabling VPN.

[j-fbriere]

Right now I'm experiencing the same behavior as you with the feeds and the search users.
I checked in debug and the Twitter/X requests are literally returning empty responses.
There is no authentication problem, it's just that the requests return nothing. Very strange.
But right now for me the subscription profiles are working.

Must be something that is happening (again) with Twitter/X right now...
I'll continue to monitor it.

[zer0fl0w]

confirmed jean.
i have exactly the same behavior as you described.
feed is empty, single profiles under subscriptions work.

i checked with nitter. there also multi-accounts queries (tweets tab) with nitter.net/bla,lol,narf fail. dont know if it is the same issue.

also in the main-nitter-thread there are some comments on rate-limits which describes this behavior iirc and that they can hit really fast.

also someone just posted a modified guest-accounts-script to get them proxified.

cat and mouse, cat and mouse, cat and mouse.

[zer0fl0w]

not squawkers fault.

you could try connecting with orbot over your vpn.
i could imagine it working. if it does not work first try, circle endpoints. there are a lot. it is worth a shot. well, hopefully.

[DearFox]

I use my own private VPN because I have Twitter blocked. But alas, I get this error and it doesn’t matter whether I use it or not.
I don’t know if my mistake is related to the discussions here, but it seemed to me that it was.

video_2023-09-14_14-36-14.mp4

[yozachar]

Can this issue be closed now?

[zer0fl0w]

Can this issue be closed now?

yes from my side.

[j-fbriere]

@DearFox The errors show that Twitter/X is blocked in your location (connection reset by peer ... address = api.twitter.com ...).

[scubajeff]

just to let you know that the latest version works even when using VPN, no need to turn VPN off before creating a guest account

[zer0fl0w]

hate to stress it again...

is feed working for you guys?

profiles and tweets are working, but feed is empty.

:/

[j-fbriere]

Right now I have the same problem.
I'm investigating it.

[zer0fl0w]

thanks jean. search is also affected.

[j-fbriere]

Twitter/X has outdone itself.
The requests from the Twitter/X API right now are returning empty responses (empty as in 0 byte empty) with a status code 200 which means OK !?!?
Wow. Simply wow. Completely unexpected.
Anyway I will adjust the code to take into account this kind of non-standard REST API responses from Twitter/X.

[j-fbriere]

Indeed the user and tweet searches return empty responses, like the feed.
The profiles/subscriptions are returning normal responses.
I don't know when Twitter/X will be fully functional again.

[zer0fl0w]

what a mess.

[j-fbriere]

The version 5.3.2 takes care of this case.
The feed is now filled with the cached and unread tweets.
Hope that Twitter/X API requests will work again soon.

[zer0fl0w]

thanks for the quick fix.

nitter is working curiously.

[zer0fl0w]

i cleared cache/data for squawker to be sure problem is not on my side before reporting. so i guess it is normal that my feed stays dark even with 3.5.2?

[j-fbriere]

If you don't have any cached tweets left, it is normal.

[zer0fl0w]

thank you.

[j-fbriere]

I'm wondering how it is possible that nitter is still working though.
I thought that the reason is that nitter use a pool of Twitter/X guest accounts and the ones that gives empty feed responses are discarded,
but since you cleared the app data, a new Twitter/X guest account must have been created when opening Squawker, to no avail.
Still investigating.

[zer0fl0w]

feed shows the old tweets now again. latest tweet is from an hour ago or so.

[j-fbriere]

ok.
search is working again.

[zer0fl0w]

search is also working again.
i think it works again!?

[zer0fl0w]

i am too old for this.

(:

thanks for the quick help and assistance.

[zer0fl0w]

#151

[j-fbriere]

In relation to errors for feeds, the new version v3.6.0 has an option to switch the type of Twitter/X requests for feeds to "enhanced requests", which might resolve those types of problems.
This setting is activated by default.
Note though that this new type of request as a lower rate limit (50 requests per 15 min instead of 150).
The option to switch is found under 'Settings/General/X API' subsection.

[anyone00]

@j-fbriere that's great to hear!

I'm failing to generate new guest accounts it seems.
Can you please add a way to insert a list of custom guest account credentials?

I can generate tokens on my network (or proxies) on other devices using the curl or the js script in nitter discussions. But squawker somehow fails.

Appreciate your hard work <3

related: #59 #72

[j-fbriere]

@anyone00 The guest account token generation in Squawker uses the same algorithm as the one of the js script in nitter discussions.
It's not clear to me why it failed.
As for the idea of inserting a list of guest account credentials, yeah this might be a good idea.
Pease create a feature request about it.

[j-fbriere]

The new version v3.6.1 is creating a new Twitter/X guest account each day (Twitter/X forbids to create more than one per day), so that after some time the pool of Twitter/X guest accounts will be large enough to avoid any rate limit problem for the enhanced requests.

[sysadminpower2019]

I am getting this no guest token error again

[j-fbriere]

@sysadminpower2019 Maybe it's because you are of of guest tokens.
Which version of Squawker are you using?
If you are using the latest version (v2.3.7) than you should configure a regular account (in Settings / Account).
It's easy to create a regular account, as explained in the wiki.