question: Get lots of : select: Invalid argument

[liu0fanyi]

after curl http://localhost:3000/\?_escaped_fragment_\=
get lots of select: Invalid argument, and can't stop

I do these steps:

1. meteor add jazeee:spiderable-longer-timeout
2. add Meteor.isReadyForSpiderable = true; in one of my react component's componentDidMount()

I use react-router, so .... I can't find Router.onAfterAction...
I want some suggestions or sample.....
Thanks a lot!

[dr-dimitru]

Hello @liu0fanyi ,

Could you please post full curl response please. And all errors/logs from server/phantomjs

[liu0fanyi]

Hello @dr-dimitru
I have solved my issue, there is another <meta name="fragment" content="!">, I used before for prerenderIo.
After delete it, looks ok now...
Thanks a lot...I will close this.

[dr-dimitru]

Please, support this project by:

• Star on GitHub
• Star on Atmosphere

[liu0fanyi]

@dr-dimitru Sorry...I get them back...
I try to use meteor npm start > errorlog 2>&1 to save error log, but It will curl just ok...there are no erros in errorlog...
When I try meteor npm start, then curl, I will get these errors:

: Invalid argument
W20170502-21:28:42.426(8)? (STDERR) select: Invalid argument
W20170502-21:28:42.426(8)? (STDERR) select: Invalid argument
W20170502-21:28:42.426(8)? (STDERR) select: Invalid argument
W20170502-21:28:42.426(8)? (STDERR) select: Invalid argument
W20170502-21:28:42.427(8)? (STDERR) select: Invalid argument
W20170502-21:28:42.427(8)? (STDERR) select: Invalid argument
W20170502-21:28:42.427(8)? (STDERR) select: Invalid argument
W20170502-21:28:42.427(8)? (STDERR) select: Invalid argument
W20170502-21:28:42.428(8)? (STDERR) select: Invalid argument
W20170502-21:28:42.428(8)? (STDERR) select: Invalid argument
W20170502-21:28:42.428(8)? (STDERR) select: Invalid argument
W20170502-21:28:42.432(8)? (STDERR) select: Invalid argument

and curl nothing, curl just block.
there are too many errors, I can't get the start of log......

Is there a way to get all errors from server/phantomjs?
meteor npm start, is this: meteor --settings settings-development.json
on Mac os, meteor is the latest one, 1.4.4.1

Thanks very much....I can't get what's wrong here...

[dr-dimitru]

Is phantomjs itself installed into your system?
To make sure run phantomjs -v in console.

[liu0fanyi]

@dr-dimitru Yes, I can get phantomjs -v, it's 2.1.1
and I get the error from start of log:

�[34mW20170502-22:44:19.875(8)? (STDERR) �[39m�[35m spiderable: phantomjs failed: { [Error: stderr maxBuffer exceeded]�[39m
�[34mW20170502-22:44:19.876(8)? (STDERR) �[39m�[35m  cmd: 'phantomjs  --load-images=no --ssl-protocol=TLSv1 --ignore-ssl-errors=true --web-security=false /Users/mac/learn/meteor/meteor_test/.meteor/local/build/programs/server/assets/packages/jazeee_spiderable-longer-timeout/lib/phantom_script.js "http://localhost:3000/documents/DctT76kiNsDf67oo5"' }

[liu0fanyi]

and I try to use phantomjs alone:

phantomjs  --load-images=no --ssl-protocol=TLSv1 --ignore-ssl-errors=true --web-security=false /Users/mac/learn/meteor/meteor_test/.meteor/local/build/programs/server/assets/packages/jazeee_spiderable-longer-timeout/lib/phantom_script.js "http://localhost:3000/follow"

It was ok, without any error......

[dr-dimitru]

@liu0fanyi ,

Looks like you bump into #49
Should be fixed as soon as requested changes will be fulfilled.

@nabilfreeman could you take a look on #49 ?

[dr-dimitru]

Please update to latest release.
See bufferSize option.

[dr-dimitru]

@liu0fanyi let me know if latest release solve your issue

[dr-dimitru]

Hello @liu0fanyi ,

Have you figured out how to fix this issue tuning bufferSize option?

[liu0fanyi]

Hello @dr-dimitru
sorry to replay late...
I have tried to update this package and update bufferSize to 100m...in server/main.js
but no help...
How could I get more info about this question?enable debug in this package?

[dr-dimitru]

@nabilfreeman could you please take a look on this thread?

[perak]

The same thing happens to me. After few hours of debugging: application without accounts-password doesn't produce this error. When I add accounts-password error appears.

Looks like error is produced by phantomjs stderr (but I'm not 100 % sure).

I tried with multiple versions of phantomjs, tried downgrading meteor version... still no luck.

[dr-dimitru]

Hello @perak , @liu0fanyi ,

Thank you for the update.

Could you try to execute phantomjs directly (or fork this package and edit flags) with next flags:

phantomjs --load-images=false --ssl-protocol=any --ignore-ssl-errors=true --disk-cache=true --disk-cache-path=/data/phantomjs --cookies-file=/data/phantomjs/cookies.txt --local-storage-path=/data/phantomjs --local-to-remote-url-access=true --offline-storage-path=/data/phantomjs --web-security=false

Make sure /data/phantomjs directory exists and writable.

Let's see if this will help, perhaps it's cookies, localStorage or cache issue, as it can't be stored in memory is some special circumstances.

[liu0fanyi]

Hello @dr-dimitru I have tried phantomjs directly with up flags:
phantomjs --load-images=false --ssl-protocol=any --ignore-ssl-errors=true --disk-cache=true --disk-cache-path=/data/phantomjs --cookies-file=/data/phantomjs/cookies.txt --local-storage-path=/data/phantomjs --local-to-remote-url-access=true --offline-storage-path=/data/phantomjs --web-security=false /Users/xxx/meteor_test/.meteor/local/build/programs/server/assets/packages/jazeee_spiderable-longer-timeout/lib/phantom_script.js "http://localhost:3000"

It was ok, I dont get error info

cookies.txt:

[General]
cookies="@Variant(\0\0\0\x7f\0\0\0\x16QList<QNetworkCookie>\0\0\0\0\x1\0\0\0\x6\0\0\0Qajs_user_id=null; expires=Sun, 27-May-2018 13:37:12 GMT; domain=localhost; path=/\0\0\0Rajs_group_id=null; expires=Sun, 27-May-2018 13:37:12 GMT; domain=localhost; path=/\0\0\0__ga=GA1.1.843230367.1495892233; expires=Mon, 27-May-2019 13:37:13 GMT; domain=localhost; path=/\0\0\0\x61_gid=GA1.1.1230482198.1495892233; expires=Sun, 28-May-2017 13:37:13 GMT; domain=localhost; path=/\0\0\0|ajs_anonymous_id=%22d640a636-6c70-4b8e-a0cd-2b480ad068fc%22; expires=Sun, 27-May-2018 13:37:13 GMT; domain=localhost; path=/\0\0\0G_gat=1; expires=Sat, 27-May-2017 13:38:13 GMT; domain=localhost; path=/)"

http_localhost_3000.localstorage:

�����z!11�tableItemTableItemTableCREATE TABLE ItemTable (key TEXT UNIQUE ON CONFLICT REPLACE, value BLOB NOT NULL ON CONFLICT FAIL)[!}1indexsqlite_autoindex_I����%Iajs_user_traits{}/]ajs_group_properties{}!0debugundefined
!���"Iajdebug%

[dr-dimitru]

@liu0fanyi it's a great news.
I'm going to update this package to include those flags.
Stay tuned

[dr-dimitru]

@perak @liu0fanyi ,

No update is needed, custom flags can be passed via METEOR_PKG_SPIDERABLE_PHANTOMJS_ARGS env.var.
So, I'm going to simply update the docs.

Let me know if it will work for you

[perak]

@dr-dimitru @liu0fanyi Thanks, I'll try and let you know. I hope it works... I already spent too much time with this and I'm clise to give up and stick to prerender. BTW, the same error happens with original spiderable package and also with few other forks.

[dr-dimitru]

@perak keep me updated. This should solve it. Updated docs with explanation for every flag is on the way

[perak]

👍 tnx!

[perak]

BTW, @dr-dimitru, I'm just curious, can you reproduce/do you have the same issue?

meteor create app
cd app
meteor add spiderable accounts-password
meteor

curl http://localhost:3000/?_escaped_fragment_=

and... kaboom! select: invalid argument

(didn't tried yet to set env var as you suggested in last messages, typing from mobile phone, will try as soon as I reach my computer)

[dr-dimitru]

@perak Just tried on MacOSX@10.12.5 and phantomjs@2.1.1 - no, no error.
But this one is familiar to me, either it was back in 2015, it was caused by localStorage/cache/appcache/cookies.

The one difference phantomjs I'm using is always locally compiled from sources.
See:

• PhantomJS 2.0 Segmentation Fault ariya/phantomjs#13175
• PhantomJS crashes ariya/phantomjs#11706

[perak]

The same OS and phantomjs but installed via brew install phantomjs. Also tried phantom 1.9.x.

And the same problem on Ubuntu 16.04 & phantomjs.

OK, now I'm with my computer...

[perak]

OK, thank you! I will do it when I reach my computer (most likely ~10 hours after now).

[dr-dimitru]

most likely ~10 hours after now

Oh, okay, keep me updated

[perak]

Sure. Thank you for your help again! BTW, I'll put you to the list of contributors at meteorkitchen.com once we solve this 👍

[dr-dimitru]

I'll put you to the list of contributors at meteorkitchen.com once we solve this

Thanks a lot :)

Meanwhile I've tested various older phantomjs versions including v1.x.x with no luck.
Including different combinations of flags and args, and page.settings - no luck here too :(

Not sure if it's related, but - unlike modern browsers phantomjs may not forgive developer for miswritten code. At the same time this is not applicable as babel wouldn't forgive it too, or at least will fix some mistakes on compilation. Anyways - I recommend to run ESLint against your code, perhaps some static file (which is not processed by babel) may have an error.

[vmatekole]

I have ran into the same issue. Running Meteor 1.4 using the following environment variables — SPIDERABLE_FLAGS="--load-images=no --ssl-protocol=TLSv1 --ignore-ssl-errors=true --web-security=false". Still have the same problem.

[perak]

@vmatekole did you tried latest not yet published a94a976 or you are using latest published version?

[vmatekole]

@perak I just added the package today — do I have to fork and install locally for this patch?

[dr-dimitru]

@perak yes, just drop it into app/packages directory

[perak]

@vmatekole I suggest you to try script posted here in this thread, you'll get more error messages than "select: invalid argument" abd will be easier ti debug. Copy it to file, change url to point to your site and run: phantomjs your_file.js

[macrozone]

having the exact same issue.

lots of "select: Invalid argument"

then i increased bufferSize: Spiderable.bufferSize = 500 * 1024 * 1024;

and now i got

child_process.js:275
        if (encoding)
            ^
RangeError: Invalid string length
    at Socket.<anonymous> (child_process.js:275:13)
    at emitOne (events.js:77:13)
    at Socket.emit (events.js:169:7)
    at readableAddChunk (_stream_readable.js:153:18)
    at Socket.Readable.push (_stream_readable.js:111:10)
    at Pipe.onread (net.js:540:20)

after 5-10seconds.

standalone script is running:

phantomjs --load-images=false --ssl-protocol=any --ignore-ssl-errors=true --disk-cache=true --disk-cache-path=/tmp/phantomjs --cookies-file=/tmp/phantomjs/cookies.txt --local-storage-path=/tmp/phantomjs --local-to-remote-url-access=true --offline-storage-path=/tmp/phantomjs --web-security=false --load-images=no --ssl-protocol=TLSv1 --ignore-ssl-errors=true --web-security=false /Users/path/to/myapp/app/.meteor/local/build/programs/server/assets/packages/jazeee_spiderable-longer-timeout/lib/phantom_script.js "http://localhost:3000"

but also needs roughly 10seconds

its a heavy react app

Edit: with 1.3.4 i get (after 10-15s):

Exception in callback of async function: RangeError: Invalid string length
 at exports.format (util.js:80:18)
 at Console.warn (console.js:47:34)
 at packages/jazeee:spiderable-longer-timeout/lib/server.js:238:21
 at packages/jazeee:spiderable-longer-timeout/lib/server.js:53:10
 at runWithEnvironment (packages/meteor.js:1176:24)

Edit 2: It fails at this point, because the stdout is too long there, its again the endless list of "select: Invalid argument" (RangeError might be because of this: nodejs/node-v0.x-archive#14170)

[perak]

After pause I am fighting with this again. Using remote debugging for phantomjs I finally got better error message that causes infinite loop of "select: invalid argument" which causes buffer overflow at the end. Problem is: some of js code contains ES6 (not transpiled properly to ES5) which didn't happen with Meteor 1.3. Since I upgraded to 1.4 problem appears (and the same after upgrade to 1.5). In my case modules.js contains ES6 code (arrow functions to be precise) which produces "unespected symbol >".

Now investigating in that direction (can be Meteor's bug)

[perak]

By the way, this is not directly caused by this package, but I suggest to keep this thread open as it affects SEO (abd at the end can cause people stop using phantomjs/spiderable as error messages are hidden).

[macrozone]

I switched to https://github.com/bosondata/chrome-prerender which ngnix config (node-prerender-middleware should also work), which works fine so far.

phantomjs is dead. We should finally burry it.

[dr-dimitru]

@perak could you tell me more about how you extracted select: invalid argument error?
Btw meanwhile I've run tests across all apps listed here and like top 100 here

Can't reproduce, I have to admit meteorkitchen.com is very unique project.

[dr-dimitru]

@perak anyways I believe it's not an issue for this package apps nowadays should be compatible with ES5-only browsers. Could you fix it with ecmascript package and .babelrc config?

[dr-dimitru]

@macrozone @perak speaking of rendering with headless chrome - it works.
I've spent last week trying to make it stable enough with no success, it may fail anytime.
This will change in Chrome 60 or 61, need to wait for stable release.

[perak]

@dr-dimitru finally, when I remove babel-polyfill npm module from the app - it works normally. But, I need babel-polyfill to be included in the app. Hope this helps someone.

[dr-dimitru]

@perak are you using babel-polyfill aside to babel-runtime? Just curious what for babel-polyfill may be used?

[perak]

Using it because I am instantiating global Array and Object classes which is OK in es6 but transpilation with babel doesn't work out of the box, so I added babel-plugin-transform-builtin-extend and babel-polyfill...

[dr-dimitru]

@perak Could you give a code sample? Looks like it works on my end with babel-runtime and ecmascript.

[perak]

@dr-dimitru excuse me, I gave you wrong info... I'm trying everything and my code is in mess. It's not caused by babel-pooyfill. Last what is suspicious (before I drop tired last night) was my npm module "blaze2gasoline" (plus fact it is written by me makes more chance it is a problem :D ). Will continue investigation later today...

[perak]

(if I just add "blaze2gasoline" to project it doesn't crash phantom. It crashes when I use it, so after I put that code into comment I finally got page rendered by phantom....

[dr-dimitru]

@perak thank you keep this thread updated

[perak]

@dr-dimitru OK, fixed! 😃

Actually, not sure exact solution but this is what I have done:

• npm module blaze2gasoline was ES5 with exception that I was using few arrow functions. I converted it to pure ES5.

• blaze2gasoline was using gasoline-turbo npm module, but I also had raw gasoline-turbo.js in my /lib/ dir included which was unnecessary (and versions were different). I removed gasoline-turbo.js and using only the one from npm.

• blaze2gasoline npm was using local string-utils.js file which exports one function toCamelCase() which was used by blaze2gasoline.js. I removed string-utils.js and copied it's contents into blaze2gasoline.js (and not exporting toCamelCase() function from blaze2gasoline anymore).

After that, I rebuilt everything (deleted /.meteor/local/ and /node_modules/ and started the app) and since then everything works fine.

And again: I am still not sure what exactly caused the problem but after these actions it worked and as I spent too much time on this, I simply cannot investigate more.

I believe nobody else is using gasoline-turbo and blaze2gasoline (libs specific for my project meteor-kitchen), but I guess they have similar problem - must be something with npm modules/es6/transpiling.

[perak]

@dr-dimitru By the way, I added you to the contributors list, don't know your name so I hope "Dr.Dimitru" is OK :)

https://www.meteorkitchen.com/contribute

Thanks again.

[dr-dimitru]

@perak great thanks.
I prefer all lowercase "dr.dimitru" or "Dmitriy A.".
Is there a way to contact you over email?

[perak]

@dr-dimitru Sure, I will change that. Mail: see at contribute page / donate section is my e-mail address.

[dr-dimitru]

@perak you might me interested in prerender.io alternative - prerendering it has ES6 support out of the box.

Let me know what do you think