Screenshot #1194

DerHOgi · 2020-12-05T09:58:09Z

For a few tests:
Is it possible to take a screenshot everytime and not only on success?
What do I have to change in the code?

jef · 2020-12-05T13:06:54Z

This should do it for you.

diff --git a/src/store/lookup.ts b/src/store/lookup.ts
index 275557a..8082177 100644
--- a/src/store/lookup.ts
+++ b/src/store/lookup.ts
@@ -284,13 +284,13 @@ async function lookupCard(
 				inStock[link.url] = false;
 			}, 1000 * config.page.inStockWaitTime);
 		}
+	}
 
-		if (config.page.screenshot) {
-			logger.debug('ℹ saving screenshot');
+	if (config.page.screenshot) {
+		logger.debug('ℹ saving screenshot');
 
-			link.screenshot = `success-${Date.now()}.png`;
-			await page.screenshot({path: link.screenshot});
-		}
+		link.screenshot = `success-${Date.now()}.png`;
+		await page.screenshot({path: link.screenshot});
 	}
 
 	return statusCode;

Basically move this code:

	if (config.page.screenshot) {
		logger.debug('ℹ saving screenshot');

		link.screenshot = `success-${Date.now()}.png`;
		await page.screenshot({path: link.screenshot});
	}

below the if statement in lookup.ts:267-294.

DerHOgi · 2020-12-05T18:52:35Z

Thanks @jef for you reply!
Now I can see what's going on and why I allways get false negative results. Item is available but bot result is OUT OF STOCK!
What can we do against this. There is a new captcha-check that concerns all Europe amazon-*.ts sites!

jef · 2020-12-05T23:48:00Z

I actually just fixed this bug too! Not sure how it ever happened, but #1209 should make this more apparent.

gigi2006 · 2020-12-06T07:43:43Z

Thanks @jef for you reply!
Now I can see what's going on and why I allways get false negative results. Item is available but bot result is OUT OF STOCK!
What can we do against this. There is a new captcha-check that concerns all Europe amazon-*.ts sites!

hey @jef

how do you think we can solve the problem with the captcha? i don't think there will be a perfect solution, i already set and tried PAGE_BACKOFF_MIN and PAGE_BACKOFF_MAX as well as PAGE_SLEEP_MIN and PAGE_SLEEP_MAX. But it did not help much.

now a consideration, couldn't we just let the browser open when the captcha warning appears and then the real human could solve the captcha from time to time? so i would do it.

@DerHOgi
how could you solve the problem?
with me it is just the most important contact points, like amazon.de etc.

//edit:

by the way, on the same computer where the script runs i can use the amazon.de site and the playstation 5 etc... there is no captcha... so i thought of USER_AGENT.

so maybe you could do something with random USER_AGENT?

I have already set INCOGNITO and so on.

//edit2:

OK i found this from you, but I have not testet yet.

https://github.com/jef/streetmerchant/wiki/Troubleshoot:-General:-Captcha

//edit3:

OK so I tested it, unfortunately without success.
a cromium festern opens with about:blank and streetmerchant gets stuck.

[09:21:05] info :: [amazon-uk] Running linksBuilder...
[09:21:06] error :: Protocol error (Performance.enable): Target closed.
C:\streetmerchant\node_modules\puppeteer\lib\cjs\puppeteer\common\Connection.js:208
this._callbacks.set(id, { resolve, reject, error: new Error(), method });
^

Error: Protocol error (Performance.enable): Target closed.
at C:\streetmerchant\node_modules\puppeteer\lib\cjs\puppeteer\common\Connection.js:208:63
at new Promise ()
at CDPSession.send (C:\streetmerchant\node_modules\puppeteer\lib\cjs\puppeteer\common\Connection.js:207:16)
at Page._initialize (C:\streetmerchant\node_modules\puppeteer\lib\cjs\puppeteer\common\Page.js:181:26)
at Function.create (C:\streetmerchant\node_modules\puppeteer\lib\cjs\puppeteer\common\Page.js:168:20)
at C:\streetmerchant\node_modules\puppeteer\lib\cjs\puppeteer\common\Target.js:72:88
at processTicksAndRejections (node:internal/process/task_queues:93:5) {
level: '\x1B[31merror\x1B[39m',
timestamp: undefined,
metadata: {},
[Symbol(level)]: 'error',
[Symbol(message)]: '\x1B[90m[09:21:06]\x1B[39m \x1B[31merror\x1B[39m \x1B[90m::\x1B[39m Protocol error (Performance.enable): Target closed.'
}

//edit4:

ok i dont know why, but the second time the browser started normally... it was a bit too fast for me, but i still managed to get it working and could solve the captchas... but the problem came back right after that... so again captchas...

oh yeah and then again the same mistake as above and streetmerchant is interrupted...

edit5:

@jef

i have a question, would it make any difference if i log in to the chromium browser with my real data? so for all amazon, mediamarkt, saturn etc sites including google?

can it be that I as a logged in user don't have to do captchas anymore?

DerHOgi · 2020-12-06T11:51:37Z

@gigi2006 No, I have no solution for this problem until now. I tested the bot with an item that is on stock and I allways got false negative results from the bot. First I thought this is a problem with the container, but I can't find any errors. This is why I asked @jef if it is possible to make a screenshot everytime and not only on success. Then I saw, whats going on and why the bot is not working correctly. The captcha check.

I allready tried different things but with no success until now.

I run the bot in a virtual machine with ubutu. The strage thing: If I open my "normal" desktop browser in Windows or macOS, I can reload the website with the same ip without any captcha checks. Why?---- What is the difference? The operating system? The chromium brwoser? Allready tried different user agent strings, but without success. Maybe Amazon use techniques like browser fingerprinting and can notice immediately that a bot is running. Maybe because there are too many people using this bot...

gigi2006 · 2020-12-06T12:23:36Z

@DerHOgi i think, but I am Not Sure.
You have no captchas Whit Login User in your normal Browser.

DerHOgi · 2020-12-06T12:26:14Z

@gigi2006 No, I tried this with no user logged in! Maybe Amazon foud a way to identify

DerHOgi · 2020-12-06T13:37:33Z

@gigi2006 I just wrote my own little bot only for amazon.de which uses the "normal" windows browser. No user is logged in. This is running now since 1 hour and there are no captchas. If this works fine, I will add some other european amazon websites.

jef · 2020-12-06T14:06:15Z

Try using PUPPETEER_EXECUTABLE_PATH. This will use your regular browser.

This is what it looks like on my machine:

PUPPETEER_EXECUTABLE_PATH=/usr/bin/google-chrome-stable npm run start

You can add before running or add to your .env. Up to you.

DerHOgi · 2020-12-06T14:40:30Z

@jef My ubuntu is running in a virtual machine only with command line interface - no gui...

SAMLCSE · 2020-12-06T14:50:34Z

It's works on Raspberry Pi. No issues. Running with Chromium browser[not the Chromium].

…

On Sun, Dec 6, 2020, 8:37 AM DerHOgi ***@***.***> wrote: @gigi2006 <https://github.com/gigi2006> I just wrote my own little bot only for amazon.de which uses the "normal" windows browser. No user is logged in. This is running now since 1 hour and there are no captchas. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1194 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKCRMIJPUKY76SO5KE2TMQ3STOCKXANCNFSM4UON6N4Q> .

SAMLCSE · 2020-12-06T14:53:24Z

On Sun, Dec 6, 2020, 9:51 AM Madhusanka Jayawardhana < samadhusankamahesh@gmail.com> wrote:

…

On Sun, Dec 6, 2020, 9:50 AM Madhusanka Jayawardhana < ***@***.***> wrote: > It's works on Raspberry Pi. No issues. Running with Chromium browser[not > the Chromium]. > > On Sun, Dec 6, 2020, 8:37 AM DerHOgi ***@***.***> wrote: > >> @gigi2006 <https://github.com/gigi2006> I just wrote my own little bot >> only for amazon.de which uses the "normal" windows browser. No user is >> logged in. This is running now since 1 hour and there are no captchas. >> >> — >> You are receiving this because you are subscribed to this thread. >> Reply to this email directly, view it on GitHub >> <#1194 (comment)>, >> or unsubscribe >> <https://github.com/notifications/unsubscribe-auth/AKCRMIJPUKY76SO5KE2TMQ3STOCKXANCNFSM4UON6N4Q> >> . >> >

DerHOgi · 2020-12-06T14:59:48Z

@SAMLCSE Will this work on ubuntu console only version? What do i have to do?

SAMLCSE · 2020-12-06T15:44:15Z

I'm not sure I'm running HEADLESS = "true" on my .env

…

On Sun, Dec 6, 2020, 10:00 AM DerHOgi ***@***.***> wrote: @SAMLCSE <https://github.com/SAMLCSE> Will this work on ubuntu console only version? What do i have to do? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1194 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKCRMIOUHSWM3LNUZDQYZX3STOL7BANCNFSM4UON6N4Q> .

gigi2006 · 2020-12-06T18:27:03Z

Try using PUPPETEER_EXECUTABLE_PATH. This will use your regular browser.

This is what it looks like on my machine:
PUPPETEER_EXECUTABLE_PATH=/usr/bin/google-chrome-stable npm run start
You can add before running or add to your .env. Up to you.

hey @jef how can i test it on a windows machine?

i have this error here:

[19:26:39] error :: ✖ something bad happened, resetting streetmerchant in 5 seconds Tried to use PUPPETEER_EXECUTABLE_PATH env variable to launch browser but did not find any executable at: /usr/bin/google-chrome-stable npm run start {
"stack": "Error: Tried to use PUPPETEER_EXECUTABLE_PATH env variable to launch browser but did not find any executable at: /usr/bin/google-chrome-stable npm run start\n at ChromeLauncher.launch (C:\streetmerchant\node_modules\puppeteer\lib\cjs\puppeteer\node\Launcher.js:79:23)\n at async PuppeteerExtra.launch (C:\streetmerchant\node_modules\puppeteer-extra\dist\index.cjs.js:129:25)\n at async main (C:\streetmerchant\build\index.js:36:15)\n at async Timeout.loopMain [as _onTimeout] (C:\streetmerchant\build\index.js:70:9)"

SWEETDREAMSIV · 2020-12-08T20:18:02Z

Try using PUPPETEER_EXECUTABLE_PATH. This will use your regular browser.
This is what it looks like on my machine:
PUPPETEER_EXECUTABLE_PATH=/usr/bin/google-chrome-stable npm run start
You can add before running or add to your .env. Up to you.
hey @jef how can i test it on a windows machine?

i have this error here:

[19:26:39] error :: ✖ something bad happened, resetting streetmerchant in 5 seconds Tried to use PUPPETEER_EXECUTABLE_PATH env variable to launch browser but did not find any executable at: /usr/bin/google-chrome-stable npm run start {
"stack": "Error: Tried to use PUPPETEER_EXECUTABLE_PATH env variable to launch browser but did not find any executable at: /usr/bin/google-chrome-stable npm run start\n at ChromeLauncher.launch (C:\streetmerchant\node_modules\puppeteer\lib\cjs\puppeteer\node\Launcher.js:79:23)\n at async PuppeteerExtra.launch (C:\streetmerchant\node_modules\puppeteer-extra\dist\index.cjs.js:129:25)\n at async main (C:\streetmerchant\build\index.js:36:15)\n at async Timeout.loopMain [as _onTimeout] (C:\streetmerchant\build\index.js:70:9)"

you should use correct exe file location for your operating system with windows 10 is like:

C:\Program Files (x86)\Google\Chrome\Application\Chrome.exe

keep detecting captcha even with this fix I can't understand, for now the bot is unusable for 3 amazon stores.

nsar1907 · 2021-04-16T03:03:43Z

@gigi2006 I just wrote my own little bot only for amazon.de which uses the "normal" windows browser. No user is logged in. This is running now since 1 hour and there are no captchas. If this works fine, I will add some other european amazon websites.

does it works good until now ?

DerHOgi added the type: question label Dec 5, 2020

DerHOgi assigned jef Dec 5, 2020

lucarin91 mentioned this issue Dec 7, 2020

fix(store): italian store checks #1212

Merged

jef closed this as completed Dec 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Screenshot #1194

Screenshot #1194

DerHOgi commented Dec 5, 2020

jef commented Dec 5, 2020 •

edited

Loading

DerHOgi commented Dec 5, 2020 •

edited

Loading

jef commented Dec 5, 2020 •

edited

Loading

gigi2006 commented Dec 6, 2020 •

edited

Loading

DerHOgi commented Dec 6, 2020 •

edited

Loading

gigi2006 commented Dec 6, 2020

DerHOgi commented Dec 6, 2020 •

edited

Loading

DerHOgi commented Dec 6, 2020 •

edited

Loading

jef commented Dec 6, 2020

DerHOgi commented Dec 6, 2020

SAMLCSE commented Dec 6, 2020 via email

SAMLCSE commented Dec 6, 2020 via email

DerHOgi commented Dec 6, 2020

SAMLCSE commented Dec 6, 2020 via email

gigi2006 commented Dec 6, 2020

SWEETDREAMSIV commented Dec 8, 2020

nsar1907 commented Apr 16, 2021

Screenshot #1194

Screenshot #1194

Comments

DerHOgi commented Dec 5, 2020

jef commented Dec 5, 2020 • edited Loading

DerHOgi commented Dec 5, 2020 • edited Loading

jef commented Dec 5, 2020 • edited Loading

gigi2006 commented Dec 6, 2020 • edited Loading

DerHOgi commented Dec 6, 2020 • edited Loading

gigi2006 commented Dec 6, 2020

DerHOgi commented Dec 6, 2020 • edited Loading

DerHOgi commented Dec 6, 2020 • edited Loading

jef commented Dec 6, 2020

DerHOgi commented Dec 6, 2020

SAMLCSE commented Dec 6, 2020 via email

SAMLCSE commented Dec 6, 2020 via email

DerHOgi commented Dec 6, 2020

SAMLCSE commented Dec 6, 2020 via email

gigi2006 commented Dec 6, 2020

SWEETDREAMSIV commented Dec 8, 2020

nsar1907 commented Apr 16, 2021

jef commented Dec 5, 2020 •

edited

Loading

DerHOgi commented Dec 5, 2020 •

edited

Loading

jef commented Dec 5, 2020 •

edited

Loading

gigi2006 commented Dec 6, 2020 •

edited

Loading

DerHOgi commented Dec 6, 2020 •

edited

Loading

DerHOgi commented Dec 6, 2020 •

edited

Loading

DerHOgi commented Dec 6, 2020 •

edited

Loading