Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Various X bug fixes #435

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
Open

Various X bug fixes #435

wants to merge 12 commits into from

Conversation

micahflee
Copy link
Contributor

@micahflee micahflee commented Feb 28, 2025

Fixes #432.

Also fixes #416, which I just realized was introduced in #384.

I also merged #434 and #430 into this branch, so we should review/merge those first.

For some reason, X returned a tweet (technically a RT) with entities that had urls that looked like this:

"urls": [
    {
        "display_url": "t.co",
        "url": "https://t.co/",
        "indices": [
            130,
            143
        ]
    }
],

While indexing, this now only saves URLs from tweets if all of the correct fields are present.

During indexTweets, X sometimes responds with 429 even when it is not really rated limited. This leads the indexing to stop early because it cannot find the retry button. This PR checks to make sure this is not the case before looking for the retry button, so the indexing doesn't stop early. I think this should make indexing more robust.

There were also various problems I noticed with saving DMs. When trying to save conversations it sometimes would timeout too soon and move to saving messages, so I increased the timeout. I fixed that by increasing the timeout, and also fiddling around with scrolling after the page loads to encourage it to show conversations. It seems to work.

I also added a "Enabling Clicking in Browser" link to the job status component. When you click it, you can click around in the browser during automation. If you click again, you can disable it.

Screenshot 2025-02-28 at 4 47 39 PM

It was difficult for me to reproduce, but I did get to a point when indexing DMs where there was a "retry" button that Cyd wasn't clicking and it was stuck, so this allows the user to click it themselves if they want.

@micahflee micahflee marked this pull request as ready for review March 1, 2025 00:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

X: error indexing tweets X: error saving DM users
2 participants