Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Block twitter advertisements #4790

Closed
musjj opened this issue Nov 9, 2023 · 4 comments
Closed

Block twitter advertisements #4790

musjj opened this issue Nov 9, 2023 · 4 comments

Comments

@musjj
Copy link

musjj commented Nov 9, 2023

It looks like that twitter injects advertisements that looks and acts like normal tweets. So when you download a tweet with "replies": true, media from those advertisements will get downloaded alongside the desired tweet. Can we have some kind of ad-blocking functionality to circumvent this?

I've managed to block some of them by using image-filter and filtering sources that comes from advertiser-interface and Twitter for Advertisers. But it turns out that many ad tweets sets their source to Twitter Web App, which makes them indistinguishable from normal tweets.

Right now I'm working around this by using "image-filter": [ "author is user" ], but this might not be suitable for all use cases.

@mikf
Copy link
Owner

mikf commented Nov 10, 2023

Could you run gallery-dl with --write-pages and post the generated file that contains such an ad Tweet? It'll be located in your current working directory.

01_https_twitter.com_i_api_graphql_JlLZj42Ltr2qwjasw-l5lQ_TweetDetail_…

I personally don't seem to get ad injections. At least your example from #3894 (comment) only contains regular Tweets for me. And replies=self works as expected.

@musjj
Copy link
Author

musjj commented Nov 10, 2023

I'm basically getting a lot of ads, with many of them targeting my local location. This is one of the ads that I got: https://twitter.com/i/web/status/1722761553605193827

Here's an entry from data.instructions.entries that seems to represent that tweet:

JSON
{
  "entryId": "conversationthread-1722761553605193827-1ceb3b467bf943a0",
  "sortIndex": "7577152360052002585",
  "content": {
    "entryType": "TimelineTimelineModule",
    "__typename": "TimelineTimelineModule",
    "items": [
      {
        "entryId": "conversationthread-1722761553605193827-tweet-1722761553605193827-1ceb3b467bf943a0",
        "item": {
          "itemContent": {
            "itemType": "TimelineTweet",
            "__typename": "TimelineTweet",
            "tweet_results": {
              "result": {
                "__typename": "Tweet",
                "rest_id": "1722761553605193827",
                "has_birdwatch_notes": false,
                "core": {
                  "user_results": {
                    "result": {
                      "__typename": "User",
                      "id": "VXNlcjoxMTA0NDQ5NTkz",
                      "rest_id": "1104449593",
                      "affiliates_highlighted_label": {},
                      "has_graduated_access": true,
                      "is_blue_verified": true,
                      "profile_image_shape": "Circle",
                      "legacy": {
                        "can_dm": true,
                        "can_media_tag": true,
                        "created_at": "Sat Jan 19 19:17:26 +0000 2013",
                        "default_profile": false,
                        "default_profile_image": false,
                        "description": "Discover a new way to explore YouTube with our chrome extension @_youshare \nBrowse friends' homepages & share videos in a single click.",
                        "entities": {
                          "description": {
                            "urls": []
                          },
                          "url": {
                            "urls": [
                              {
                                "display_url": "youshare.media",
                                "expanded_url": "https://youshare.media",
                                "url": "https://t.co/hV7f8YwQ61",
                                "indices": [
                                  0,
                                  23
                                ]
                              }
                            ]
                          }
                        },
                        "fast_followers_count": 0,
                        "favourites_count": 1174,
                        "followers_count": 3011,
                        "friends_count": 1234,
                        "has_custom_timelines": false,
                        "is_translator": false,
                        "listed_count": 35,
                        "location": "world",
                        "media_count": 46,
                        "name": "Simon Biasotto",
                        "normal_followers_count": 3011,
                        "pinned_tweet_ids_str": [
                          "1720986981214138447"
                        ],
                        "possibly_sensitive": false,
                        "profile_banner_url": "https://pbs.twimg.com/profile_banners/1104449593/1695645486",
                        "profile_image_url_https": "https://pbs.twimg.com/profile_images/843870825094807561/kiW9T9pZ_normal.jpg",
                        "profile_interstitial_type": "",
                        "screen_name": "BiasottoSimon",
                        "statuses_count": 1359,
                        "translator_type": "none",
                        "url": "https://t.co/hV7f8YwQ61",
                        "verified": false,
                        "want_retweets": false,
                        "withheld_in_countries": []
                      },
                      "professional": {
                        "rest_id": "1683295629710114816",
                        "professional_type": "Creator",
                        "category": [
                          {
                            "id": 1055,
                            "name": "Software developer/Programmer/Software engineer",
                            "icon_name": "IconBriefcaseStroke"
                          }
                        ]
                      }
                    }
                  }
                },
                "card": {
                  "rest_id": "card://1722761551797440512",
                  "legacy": {
                    "binding_values": [
                      {
                        "key": "unified_card",
                        "value": {
                          "string_value": "{\"type\":\"video_website\",\"component_objects\":{\"details_1\":{\"type\":\"details\",\"data\":{\"title\":{\"content\":\"Install YouShare right now, it's free!\",\"is_rtl\":false},\"subtitle\":{\"content\":\"google.com\",\"is_rtl\":false},\"destination\":\"override_browser_with_docked_media_1\"}},\"media_1\":{\"type\":\"media\",\"data\":{\"id\":\"13_1702802624615956480\",\"destination\":\"browser_with_docked_media_1\"}}},\"destination_objects\":{\"browser_with_docked_media_1\":{\"type\":\"browser_with_docked_media\",\"data\":{\"url_data\":{\"url\":\"https://chromewebstore.google.com/detail/youshare-for-youtube/kmijndmdcmdfhajeagolfiepokkdoilg\",\"vanity\":\"google.com\"},\"media_id\":\"13_1702802624615956480\"}},\"override_browser_with_docked_media_1\":{\"type\":\"browser\",\"data\":{\"url_data\":{\"url\":\"https://chromewebstore.google.com/detail/youshare-for-youtube/kmijndmdcmdfhajeagolfiepokkdoilg\",\"vanity\":\"google.com\"}}}},\"components\":[\"media_1\",\"details_1\"],\"media_entities\":{\"13_1702802624615956480\":{\"id\":1702802624615956480,\"id_str\":\"1702802624615956480\",\"indices\":[0,0],\"media_url\":\"\",\"media_url_https\":\"https://pbs.twimg.com/amplify_video_thumb/1702802624615956480/img/qW5A9wga4T0mOpN5.jpg\",\"url\":\"\",\"display_url\":\"\",\"expanded_url\":\"\",\"type\":\"video\",\"original_info\":{\"width\":1920,\"height\":1080},\"sizes\":{\"large\":{\"w\":1920,\"h\":1080,\"resize\":\"fit\"},\"medium\":{\"w\":1200,\"h\":675,\"resize\":\"fit\"},\"thumb\":{\"w\":150,\"h\":150,\"resize\":\"crop\"},\"small\":{\"w\":680,\"h\":383,\"resize\":\"fit\"}},\"source_user_id\":1104449593,\"source_user_id_str\":\"1104449593\",\"video_info\":{\"aspect_ratio\":[16,9],\"duration_millis\":55722,\"variants\":[{\"bitrate\":10368000,\"content_type\":\"video/mp4\",\"url\":\"https://video.twimg.com/amplify_video/1702802624615956480/vid/avc1/1920x1080/aLj9ORmRhXvtGG0Z.mp4?tag=16\"},{\"bitrate\":832000,\"content_type\":\"video/mp4\",\"url\":\"https://video.twimg.com/amplify_video/1702802624615956480/vid/avc1/640x360/LfYkUFpqYZK8uoop.mp4?tag=16\"},{\"bitrate\":288000,\"content_type\":\"video/mp4\",\"url\":\"https://video.twimg.com/amplify_video/1702802624615956480/vid/avc1/480x270/t29drvKPyNYtJKw9.mp4?tag=16\"},{\"content_type\":\"application/x-mpegURL\",\"url\":\"https://video.twimg.com/amplify_video/1702802624615956480/pl/fsTUfIWQochO59mH.m3u8?tag=16&container=fmp4\"},{\"bitrate\":2176000,\"content_type\":\"video/mp4\",\"url\":\"https://video.twimg.com/amplify_video/1702802624615956480/vid/avc1/1280x720/KmVJFB-S8H-n8mk6.mp4?tag=16\"}]},\"media_key\":\"13_1702802624615956480\",\"ext\":{\"mediaColor\":{\"r\":{\"ok\":{\"palette\":[{\"rgb\":{\"red\":10,\"green\":7,\"blue\":9},\"percentage\":47.11},{\"rgb\":{\"red\":212,\"green\":75,\"blue\":41},\"percentage\":14.88},{\"rgb\":{\"red\":102,\"green\":26,\"blue\":23},\"percentage\":11.59},{\"rgb\":{\"red\":242,\"green\":146,\"blue\":79},\"percentage\":4.43},{\"rgb\":{\"red\":85,\"green\":30,\"blue\":40},\"percentage\":3.52}]}},\"ttl\":-1}}}}}",
                          "type": "STRING"
                        }
                      },
                      {
                        "key": "card_url",
                        "value": {
                          "scribe_key": "card_url",
                          "string_value": "https://twitter.com",
                          "type": "STRING"
                        }
                      }
                    ],
                    "card_platform": {
                      "platform": {
                        "audience": {
                          "name": "production"
                        },
                        "device": {
                          "name": "Swift",
                          "version": "12"
                        }
                      }
                    },
                    "name": "unified_card",
                    "url": "card://1722761551797440512",
                    "user_refs_results": []
                  }
                },
                "unmention_data": {},
                "unified_card": {
                  "card_fetch_state": "NoCard"
                },
                "edit_control": {
                  "edit_tweet_ids": [
                    "1722761553605193827"
                  ],
                  "editable_until_msecs": "1699576936000",
                  "is_edit_eligible": false,
                  "edits_remaining": "5"
                },
                "is_translatable": false,
                "views": {
                  "count": "572789",
                  "state": "EnabledWithCount"
                },
                "source": "<a href=\"https://help.twitter.com/en/using-twitter/how-to-tweet#source-labels\" rel=\"nofollow\">advertiser-interface</a>",
                "legacy": {
                  "bookmark_count": 33,
                  "bookmarked": false,
                  "created_at": "Thu Nov 09 23:42:16 +0000 2023",
                  "conversation_id_str": "1722761553605193827",
                  "display_text_range": [
                    0,
                    255
                  ],
                  "entities": {
                    "user_mentions": [],
                    "urls": [],
                    "hashtags": [],
                    "symbols": []
                  },
                  "favorite_count": 674,
                  "favorited": false,
                  "full_text": "Discover a new dimension of YouTube content with YouShare \uD83C\uDF0C\n\n✨ Key Features ✨\n\nAccess Friends' YouTube Homepages: Discover content beyond your own algorithm. \uD83D\uDD0D\n\nOne-Click Sharing: Share videos with friends directly on YouTube in a single, elegant click. \uD83D\uDC46",
                  "is_quote_status": false,
                  "lang": "en",
                  "possibly_sensitive": false,
                  "possibly_sensitive_editable": true,
                  "quote_count": 0,
                  "reply_count": 55,
                  "retweet_count": 30,
                  "retweeted": false,
                  "scopes": {
                    "followers": false
                  },
                  "user_id_str": "1104449593",
                  "id_str": "1722761553605193827"
                },
                "quick_promote_eligibility": {
                  "eligibility": "IneligibleNotProfessional"
                }
              }
            },
            "tweetDisplayType": "Tweet",
            "promotedMetadata": {
              "advertiser_results": {
                "result": {
                  "__typename": "User",
                  "id": "VXNlcjoxMTA0NDQ5NTkz",
                  "rest_id": "1104449593",
                  "affiliates_highlighted_label": {},
                  "has_graduated_access": true,
                  "is_blue_verified": true,
                  "profile_image_shape": "Circle",
                  "legacy": {
                    "can_dm": true,
                    "can_media_tag": true,
                    "created_at": "Sat Jan 19 19:17:26 +0000 2013",
                    "default_profile": false,
                    "default_profile_image": false,
                    "description": "Discover a new way to explore YouTube with our chrome extension @_youshare \nBrowse friends' homepages & share videos in a single click.",
                    "entities": {
                      "description": {
                        "urls": []
                      },
                      "url": {
                        "urls": [
                          {
                            "display_url": "youshare.media",
                            "expanded_url": "https://youshare.media",
                            "url": "https://t.co/hV7f8YwQ61",
                            "indices": [
                              0,
                              23
                            ]
                          }
                        ]
                      }
                    },
                    "fast_followers_count": 0,
                    "favourites_count": 1174,
                    "followers_count": 3011,
                    "friends_count": 1234,
                    "has_custom_timelines": false,
                    "is_translator": false,
                    "listed_count": 35,
                    "location": "world",
                    "media_count": 46,
                    "name": "Simon Biasotto",
                    "normal_followers_count": 3011,
                    "pinned_tweet_ids_str": [
                      "1720986981214138447"
                    ],
                    "possibly_sensitive": false,
                    "profile_banner_url": "https://pbs.twimg.com/profile_banners/1104449593/1695645486",
                    "profile_image_url_https": "https://pbs.twimg.com/profile_images/843870825094807561/kiW9T9pZ_normal.jpg",
                    "profile_interstitial_type": "",
                    "screen_name": "BiasottoSimon",
                    "statuses_count": 1359,
                    "translator_type": "none",
                    "url": "https://t.co/hV7f8YwQ61",
                    "verified": false,
                    "want_retweets": false,
                    "withheld_in_countries": []
                  },
                  "professional": {
                    "rest_id": "1683295629710114816",
                    "professional_type": "Creator",
                    "category": [
                      {
                        "id": 1055,
                        "name": "Software developer/Programmer/Software engineer",
                        "icon_name": "IconBriefcaseStroke"
                      }
                    ]
                  }
                }
              },
              "disclosureType": "NoDisclosure",
              "experimentValues": [],
              "impressionId": "1ceb3b467bf943a0",
              "impressionString": "1ceb3b467bf943a0",
              "clickTrackingInfo": {
                "urlParams": [
                  {
                    "key": "twclid",
                    "value": "2-796xshlzlbdjafz4n0w9dtnin"
                  }
                ]
              }
            }
          },
          "clientEventInfo": {
            "details": {
              "conversationDetails": {
                "conversationSection": "HighQuality"
              }
            }
          }
        }
      }
    ],
    "displayType": "VerticalConversation",
    "clientEventInfo": {
      "details": {
        "conversationDetails": {
          "conversationSection": "HighQuality"
        }
      }
    }
  }
}

(let me know if you still need the full output)

promotedMetadata.advertiser_results might have some useful information.

The filename of the of the output from --write-pages is also interesting:

07_https_twitter.com_i_api_graphql_JlLZj42Ltr2qwjasw-l5lQ_TweetDetail_variables_%7B%22focalTweetId%22%3A%221646219676802772992%22%2C%22referrer%22%3A%22profile%22%2C%22with_rux_injections%22%3Afalse%2C%22includePromotedContent%22%3Atrue%2C%22withCommu.txt

includePromotedContent=true seems like an obvious culprit.

@mikf
Copy link
Owner

mikf commented Nov 10, 2023

This is more than good enough, thanks.

includePromotedContent=true seems like an obvious culprit.

Yeah, that's probably it. I never really looked at what exactly all these parameters are and just copy-pasted them from the network inspector as my web browser sent them.

mikf added a commit that referenced this issue Nov 11, 2023
add 'ads' option in case someone actually wants to
download promoted content for whatever reason
@mikf mikf closed this as completed Nov 11, 2023
@musjj
Copy link
Author

musjj commented Nov 11, 2023

Works perfectly, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants