Googlebot

[paribaker]

Googlebot agent does not seem to be rerouted to the prerender solution (hosted or default)

According to Prerender.io

In May of 2018, Google introduced "Dynamic Rendering" for serving normal JavaScript to your users and serving Prerendered pages to search engines. This is exactly what we do at Prerender.io.

The change introduced by Google for Dynamic Rendering means they will stop crawling the ?escaped_fragment= URLs and you can now serve a prerendered page to Googlebot by checking their user agent directly. That might sound like cloaking, but Google introduced a policy change where they are allowing you to send prerendered pages to Googlebot by checking their user agent.

Due to this Dynamic Rendering announcement, we have changed our middleware to add Googlebot to the list of user agents being checked directly.

That being said when looking at the useragent.py file

     # These first three should be disabled, since they support escaped 
      fragments, and
    # and leaving them enabled will penalize a website as "cloaked".
    # "Googlebot",
    # "Yahoo",
    # "bingbot",

Googlebot is still commented out in django-seo-js, to fix this I had to add it to the list manually in settings.py

     SEO_JS_USER_AGENTS=(
    "Googlebot",
    # "Yahoo",
    # "bingbot",

    "Ask Jeeves",
    "baiduspider",
    "facebookexternalhit",
    "twitterbot",
    "rogerbot",
    "linkedinbot",
    "embedly",
    "quoralink preview'",
    "showyoubot",
    "outbrain",
    "pinterest",
    "developersgoogle.com/+/web/snippet",
    )

Since the variable value is not appended to an existing list we have to add them all.

[paribaker]

It seems there is a PR for this

Added Googlebot etc. user agent since they don't support escaped fragment anymore. #37