refactor(captcha): use screenshot instead of passing image url

docs/reference/captcha.md

@@ -3,7 +3,7 @@
 A mechanism has been implemented to allow users to interactively handle captcha challenges without being directly connected to their streetmerchant instance. This works by sending the captcha challenge to the user directly via their preferred messaging service and waiting for their response, which is then input to the captcha page, allowing streetmerchant to proceed with its processing. 
 
 ???+ attention
-    This implementation has only been tested/used on Amazon (North America). Please submit an issue if you're facing captcha on other stores so we can get it integrated.
+    This implementation has only been tested/used on Amazon sites. Please submit an issue if you're facing captcha on other stores so we can get it integrated.
 
 ## How to use
 
@@ -21,7 +21,7 @@ To use this feature, you will have to set up a bot user on your desired messagin
 You can test your notification configuration by running `npm run test:captcha`.
 
 ???+ info
-    The test command will allow the user up to 30 seconds to enter a response before timing out. This is not directly configurable.
+    The test command will use the values from the dotenv configuration file, including timeout and poll interval.
 
 ## Configuration variables
 
@@ -68,28 +68,36 @@ You have to enable Developer Mode in the Advanced settings. Once that's enabled,
 Create an app [here](https://api.slack.com/apps) and copy the token you get once the setup is complete. Put the token in the dotenv file.
 
 ???+ info
-    The app will need `chat:write`, `im:history`, `im:write`, and `reactions:write` permissions.
+    The app will need `chat:write`, `im:history`, `im:write`, `files:write`, and `reactions:write` permissions.
 
 #### Discord
 
 Create an app [here](https://discord.com/developers/applications) and copy the token, client ID, and permissions integer (I used `518208`). Then use the url [here](https://discord.com/developers/docs/topics/oauth2#bot-authorization-flow-url-example), replacing the `client_id` and `permissions` values with your own to add the bot to your server. Paste the token into your dotenv file.
 
-### The bot didn't send a message when I got a captcha page.
+### The bot didn't send a message when it detected a captcha page.
 
 That isn't a question. This is an FAQ.
 
-### The bot didn't send a message when I got a captcha page?
+### The bot didn't send a message when it detected a captcha page?
 
 Much better. This could either be a configuration error in streetmerchant (not completed, wrong values, etc) or the bot user isn't configured correctly in your messaging service. Double-check the configuration variables you've entered and use `npm run test:captcha` to help find out the root cause.
 
+### Why are the bot images coming through broken?
+
+If you're seeing broken image link icons, the image has been blocked somehow. If you are running in low bandwidth mode, disable it to ensure captcha images load. Otherwise, file an issue.
+
 ### The bot doesn't do anything when I respond to the message and eventually times out. What's happening?
 
 When streetmerchant sends a message via Slack/Discord, it keeps a reference to that message and listens only for direct replies to it until either a response is obtained or the timeout threshold is reached. This allows the interactive captcha process to be used with multiple concurrent streetmerchant instances. Please review the warning in the [How to use](#how-to-use) section, which discusses specifically how to respond to the bot for a successful interaction.
 
 ### Why isn't captcha being detected on some of the stores I'm monitoring?
 
-Not sure, but we'll want to get that fixed! Submit an issue and we can look into it.
+Captcha is detected by looking for elements on the page that someone has defined in the streetmerchant code. These elements can change over time, or something else could be going on. Either way, submit an issue and we can look into it.
+
+### Does the interactive captcha handler process work on every store?
+
+Not yet. It's only implemented for a subset of stores. If you're facing captchas (detected or not) that aren't being handled, submit an issue and we can work on integrating it.
 
-### Will this work on every store's captcha system?
+### Does this work against (insert captcha implementation here)?
 
-Not likely. There are a plethora of captcha implementations that retailers can utilize to protect their sites. As of this writing, the interactive captcha handler has only been tested/used for Amazon (North America), and this is where the vast majority of captcha complaints come from. Any other store can implement a different captcha approach, and even Amazon can change their captcha at any time. All that said, if you're running into captcha issues with a store, submit an issue so we can work on a solution and getting it integrated, if possible.
+Not likely (yet). There are a plethora of captcha implementations that retailers can utilize to protect their sites. Any store can pick from existing captcha solutions, make their own, and obviously change it at any time.

src/config.ts

@@ -2,7 +2,6 @@ import {existsSync, readFileSync} from 'fs';
 import {banner} from './banner';
 import dotenv from 'dotenv';
 import path from 'path';
-import * as console from 'console';
 
 if (process.env.npm_config_conf) {
   if (

src/index.ts

@@ -28,48 +28,7 @@ async function restartMain() {
  * Starts the bot.
  */
 async function main() {
-  const args: string[] = [];
-
-  // Skip Chromium Linux Sandbox
-  // https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md#setting-up-chrome-linux-sandbox
-  if (config.browser.isTrusted) {
-    args.push('--no-sandbox');
-    args.push('--disable-setuid-sandbox');
-  }
-
-  // https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md#tips
-  // https://stackoverflow.com/questions/48230901/docker-alpine-with-node-js-and-chromium-headless-puppeter-failed-to-launch-c
-  if (config.docker) {
-    args.push('--disable-dev-shm-usage');
-    args.push('--no-sandbox');
-    args.push('--disable-setuid-sandbox');
-    args.push('--headless');
-    args.push('--disable-gpu');
-    config.browser.open = false;
-  }
-
-  // Add the address of the proxy server if defined
-  if (config.proxy.address) {
-    args.push(
-      `--proxy-server=${config.proxy.protocol}://${config.proxy.address}:${config.proxy.port}`
-    );
-  }
-
-  if (args.length > 0) {
-    logger.info('ℹ puppeteer config: ', args);
-  }
-
-  await stop();
-  browser = await launch({
-    args,
-    defaultViewport: {
-      height: config.page.height,
-      width: config.page.width,
-    },
-    headless: config.browser.isHeadless,
-  });
-
-  config.browser.userAgent = await browser.userAgent();
+  browser = await launchBrowser();
 
   for (const store of storeList.values()) {
     logger.debug('store links', {meta: {links: store.links}});
@@ -115,6 +74,54 @@ async function loopMain() {
   }
 }
 
+export async function launchBrowser(): Promise<Browser> {
+  console.warn('launch browser called');
+  const args: string[] = [];
+
+  // Skip Chromium Linux Sandbox
+  // https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md#setting-up-chrome-linux-sandbox
+  if (config.browser.isTrusted) {
+    args.push('--no-sandbox');
+    args.push('--disable-setuid-sandbox');
+  }
+
+  // https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md#tips
+  // https://stackoverflow.com/questions/48230901/docker-alpine-with-node-js-and-chromium-headless-puppeter-failed-to-launch-c
+  if (config.docker) {
+    args.push('--disable-dev-shm-usage');
+    args.push('--no-sandbox');
+    args.push('--disable-setuid-sandbox');
+    args.push('--headless');
+    args.push('--disable-gpu');
+    config.browser.open = false;
+  }
+
+  // Add the address of the proxy server if defined
+  if (config.proxy.address) {
+    args.push(
+      `--proxy-server=${config.proxy.protocol}://${config.proxy.address}:${config.proxy.port}`
+    );
+  }
+
+  if (args.length > 0) {
+    logger.info('ℹ puppeteer config: ', args);
+  }
+
+  await stop();
+  const browser = await launch({
+    args,
+    defaultViewport: {
+      height: config.page.height,
+      width: config.page.width,
+    },
+    headless: config.browser.isHeadless,
+  });
+
+  config.browser.userAgent = await browser.userAgent();
+
+  return browser;
+}
+
 void loopMain();
 
 process.on('SIGINT', stopAndExit);

src/messaging/captcha.ts

@@ -1,11 +1,12 @@
 import {config} from '../config';
 import {getDiscordCaptchaInputAsync} from './discord';
 import {getSlackCaptchaInputAsync} from './slack';
+import {DMPayload} from '.';
 
 const {service} = config.captchaHandler;
 
 export async function getCaptchaInputAsync(
-  payload: string,
+  payload: DMPayload,
   timeout?: number
 ): Promise<string> {
   switch (service) {

src/messaging/discord.ts

@@ -2,13 +2,11 @@ import {Link, Store} from '../store/model';
 import Discord from 'discord.js';
 import {config} from '../config';
 import {logger} from '../logger';
+import {DMPayload} from '.';
 
 const {notifyGroup, webhooks, notifyGroupSeries} = config.notifications.discord;
 const {pollInterval, responseTimeout, token, userId} = config.captchaHandler;
 
-let clientInstance: Discord.Client | undefined;
-let dmChannelInstance: Discord.DMChannel | undefined;
-
 function getIdAndToken(webhook: string) {
   const match = /.*\/webhooks\/(\d+)\/(.+)/.exec(webhook);
 
@@ -97,22 +95,37 @@ export function sendDiscordMessage(link: Link, store: Store) {
 }
 
 export async function sendDMAsync(
-  payload: string
+  payload: DMPayload
 ): Promise<Discord.Message | undefined> {
   if (userId && token) {
     logger.debug('↗ sending discord DM');
+    let client = undefined;
+    let dmChannel = undefined;
     try {
-      const client = await getDiscordClientAsync();
-      const dmChannel = await getDMChannelAsync(client);
+      client = await getDiscordClientAsync();
+      dmChannel = await getDMChannelAsync(client);
       if (!dmChannel) {
         logger.error('unable to get discord DM channel');
         return;
       }
-      const result = await dmChannel.send(payload);
+      let message: string | {} = payload;
+      if (payload.type === 'image') {
+        message = {
+          files: [
+            {
+              attachment: payload.content,
+              name: payload.content,
+            },
+          ],
+        };
+      }
+      const result = await dmChannel.send(message);
       logger.info('✔ discord DM sent');
       return result;
     } catch (error: unknown) {
       logger.error("✖ couldn't send discord DM", error);
+    } finally {
+      client?.destroy();
     }
   }
   return;
@@ -134,6 +147,7 @@ export async function getDMResponseAsync(
     let response = '';
     const intervalId = setInterval(async () => {
       const finish = (result: string) => {
+        client?.destroy();
         clearInterval(intervalId);
         resolve(result);
       };
@@ -153,7 +167,7 @@ export async function getDMResponseAsync(
           }
         } else {
           response = lastUserMessage.cleanContent;
-          lastUserMessage.react('✅');
+          await lastUserMessage.react('✅');
           logger.info(`✔ got captcha response: ${response}`);
           return finish(response);
         }
@@ -166,36 +180,29 @@ export async function getDMResponseAsync(
 }
 
 export async function getDiscordCaptchaInputAsync(
-  payload: string,
+  payload: DMPayload,
   timeout?: number
 ): Promise<string> {
   const message = await sendDMAsync(payload);
   const response = await getDMResponseAsync(
     message,
     timeout || responseTimeout
   );
-  closeClient();
   return response;
 }
 
-function closeClient() {
-  if (clientInstance) {
-    clientInstance.destroy();
-    clientInstance = undefined;
-    dmChannelInstance = undefined;
-  }
-}
-
 async function getDiscordClientAsync() {
-  if (!clientInstance && token) {
+  let clientInstance = undefined;
+  if (token) {
     clientInstance = new Discord.Client();
     await clientInstance.login(token);
   }
   return clientInstance;
 }
 
 async function getDMChannelAsync(client?: Discord.Client) {
-  if (!dmChannelInstance && userId && !!client) {
+  let dmChannelInstance = undefined;
+  if (userId && client) {
     const user = await new Discord.User(client, {
       id: userId,
     }).fetch();

src/messaging/index.ts

@@ -1,2 +1,9 @@
 export * from './captcha';
 export * from './notification';
+
+type DMPayloadType = 'text' | 'image';
+
+export interface DMPayload {
+  content: string; // for image type, content is local file path
+  type: DMPayloadType;
+}

src/messaging/notification.ts

@@ -2,13 +2,13 @@ import {Link, Store} from '../store/model';
 import {adjustPhilipsHueLights} from './philips-hue';
 import {playSound} from './sound';
 import {sendDesktopNotification} from './desktop';
-import {sendDiscordMessage} from './discord';
+import {sendDiscordMessage, sendDMAsync as sendDiscordDM} from './discord';
 import {sendEmail} from './email';
 import {sendMqttMessage} from './mqtt';
 import {sendPagerDutyNotification} from './pagerduty';
 import {sendPushbulletNotification} from './pushbullet';
 import {sendPushoverNotification} from './pushover';
-import {sendSlackMessage} from './slack';
+import {sendSlackMessage, sendDMAsync as sendSlackDM} from './slack';
 import {sendSms} from './sms';
 import {sendTelegramMessage} from './telegram';
 import {sendTweet} from './twitter';
@@ -19,6 +19,7 @@ import {activateSmartthingsSwitch} from './smartthings';
 import {sendStreamLabsAlert} from './streamlabs';
 import {sendFreeMobileAlert} from './freemobile';
 import {sendApns} from './apns';
+import {DMPayload} from '.';
 
 export function sendNotification(link: Link, store: Store) {
   // Priority
@@ -44,3 +45,18 @@ export function sendNotification(link: Link, store: Store) {
   sendStreamLabsAlert(link, store);
   sendFreeMobileAlert(link, store);
 }
+
+export async function sendDMAsync(service: string, payload: DMPayload) {
+  let dmFunction = undefined;
+  switch (service) {
+    case 'slack':
+      dmFunction = sendSlackDM;
+      break;
+    case 'discord':
+      dmFunction = sendDiscordDM;
+      break;
+    default:
+      dmFunction = () => void 0;
+  }
+  await dmFunction(payload);
+}