Getting working links as dead in result !

[subratsf]

The package is considering some active links as dead URLs. for example :

Udemy : Modern React with Redux
Have I Been Pwned

Using the remark-lint package as below:

import {reporter} from 'vfile-reporter'
import {remark} from 'remark'
import remarkLintNoDeadUrls from 'remark-lint-no-dead-urls'

const result = await remark()
          .use(remarkLintNoDeadUrls)
          .process(`# remark-lint-no-dead-urls
          
          [Udemy link](https://www.udemy.com/course/react-redux)
          [Have I been Pwned](http://haveibeenpwned.com)`)

console.log(reporter(result));

and getting the result:

[subratsf]

@transitive-bullshit @davidtheclark Need your help with this issue.

[wooorm]

hey! thanks for your patience! In this upcoming version, that’s improved. For this code:

import remarkLint from 'remark-lint'
import remarkLintNoDeadUrls from 'remark-lint-no-dead-urls'
import remarkParse from 'remark-parse'
import remarkStringify from 'remark-stringify'
import {unified} from 'unified'
import {reporter} from 'vfile-reporter'

const document = `# remark-lint-no-dead-urls

[Udemy link](https://www.udemy.com/course/react-redux)
[Have I been Pwned](http://haveibeenpwned.com)`

const file = await unified()
  .use(remarkParse)
  .use(remarkLint)
  .use(remarkLintNoDeadUrls)
  .use(remarkStringify)
  .process(document)

console.error(reporter(file))

You’d get:

3:1-3:55 warning Unexpected dead URL `https://www.udemy.com/course/react-redux`, expected live URL                         no-dead-urls remark-lint
  [cause]:
         error   Unexpected not ok response `403` (`Forbidden`) on `https://www.udemy.com/course/react-redux`              dead         dead-or-alive
4:1-4:47 warning Unexpected redirecting URL `http://haveibeenpwned.com/`, expected final URL `https://haveibeenpwned.com/` no-dead-urls remark-lint

⚠ 2 warnings

The second error is simple, and useful: switch to the HTTPS URL in your markdown and it’s solved.

The other error happens because udemy has some advanced security measures in place.
You’d typically get around it with:

+  .use(remarkLintNoDeadUrls, {
+    deadOrAliveOptions: {
+      userAgent:
+        'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0'
+    }
+  })

...but here that doesn’t work, they still know it’s automated access.
So, I’d recommend:

     deadOrAliveOptions: {
       userAgent:
         'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0'
-    }
+    },
+    skipUrlPatterns: [/^https:\/\/www.udemy.com\//i]
   })
   .use(remarkStringify)
   .process(document)

Closing this for now, will release in the coming days!