Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Travis test gives error sometimes which gets removed by rebuild #2824

Closed
Souravirus opened this issue Jun 11, 2018 · 34 comments · Fixed by #3253
Closed

Travis test gives error sometimes which gets removed by rebuild #2824

Souravirus opened this issue Jun 11, 2018 · 34 comments · Fixed by #3253
Assignees
Labels
help wanted requires help by anyone willing to contribute Ruby testing issues are usually for adding unit tests, integration tests or any other tests for a feature

Comments

@Souravirus
Copy link
Member

Souravirus commented Jun 11, 2018

Travis test sometimes gives this error ** Error in /usr/local/bin/ruby': free(): invalid next size (fast): 0x00000000098d3950 *** which gets cleared out when tests get rebuilded.

The reported error is There was an error with Danger bot’s Junit parsing: undefined method value’ for nil:NilClass` in PRs.

@Souravirus Souravirus added testing issues are usually for adding unit tests, integration tests or any other tests for a feature help wanted requires help by anyone willing to contribute Ruby labels Jun 11, 2018
@jywarren
Copy link
Member

This is causing a lot of trouble!! It fails often. @icarito @publiclab/reviewers if you have any ideas, or could do some research, that'd be great!

@jywarren
Copy link
Member

Maybe the Debian Stretch update will help... #2865

@jywarren
Copy link
Member

Hmm, i just saw this on https://travis-ci.org/publiclab/plots2/builds/395173961 so I think it wasn't resolved... too bad!

@SidharthBansal
Copy link
Member

I usually get the following error which is resolved by rebuilding Travis

There was an error with Danger bot’s Junit parsing: undefined method `value’ for nil:NilClass

@SidharthBansal
Copy link
Member

This might help
#1852

@jywarren
Copy link
Member

OK, captured a log from this: https://gist.github.com/jywarren/6e06ae3a6647fadcf90dc7842416c5b9

@jywarren
Copy link
Member

Saw another memory-related error in https://travis-ci.org/publiclab/plots2/builds/405981763

Writing XML reports to /app/test/reports
ruby: malloc.c:2883: mremap_chunk: Assertion `((size + offset) & (GLRO (dl_pagesize) - 1)) == 0' failed.

Restarting it...

@Souravirus
Copy link
Member Author

ok lets see about it.

@jywarren
Copy link
Member

@jywarren
Copy link
Member

OK, so one plan is to try some of these Travis options. Another is to try running these in CircleCI or another service to see!

@Souravirus
Copy link
Member Author

Ok reading these options. Thanks

@Souravirus
Copy link
Member Author

Can I work on this after working on #2659?

@jywarren
Copy link
Member

jywarren commented Jul 27, 2018 via email

@jywarren
Copy link
Member

jywarren commented Jul 27, 2018 via email

@icarito
Copy link
Member

icarito commented Aug 2, 2018

Hi!
I was busy yesterday getting GitLab CI to build this project!
While it works well, after a few tries I managed to trigger this error in Gitlab too!

https://gitlab.com/publiclab/plots2/-/jobs/86207162

Has anyone found this error during local development? Thanks for your input!

@icarito icarito mentioned this issue Aug 2, 2018
5 tasks
@icarito
Copy link
Member

icarito commented Aug 4, 2018

Hi,
So I think we need to trigger this build where we can setup the kernel to dump a core and then examine the core hopefully with full debugging symbols and gdb. This is not my area of expertise. In the hope to get some guidance I'm going to try and trigger dumping a core in Jenkins.

@icarito
Copy link
Member

icarito commented Aug 4, 2018

Okay I've captured a core dump and got a backtrace from gdb!
Please review (raw)

https://gist.github.com/jywarren/6265fcc263d7cc2e369515f342957b76

@icarito
Copy link
Member

icarito commented Aug 4, 2018

I don't know what it means!

@icarito
Copy link
Member

icarito commented Aug 4, 2018

If this backtrace doesn't enlighten the situation I'd suggest changing Ruby versions!

@icarito
Copy link
Member

icarito commented Aug 4, 2018

Debian Stable is still at Ruby 2.3.3p222 while the testing branch currently runs 2.5.1.

@icarito
Copy link
Member

icarito commented Aug 7, 2018

Here's a Ruby bug report with the same error message: https://bugs.ruby-lang.org/issues/11018

That seem you used broken C extension.
Please let us know if you can reproduce it with no gem, no rvm and no extension.
-- kosaki (Motohiro KOSAKI)

@Souravirus
Copy link
Member Author

Hi @icarito, I have seen this error ** Error in /usr/local/bin/ruby': free(): invalid next size (fast): 0x00000000098d3950 ** in local development.

@icarito
Copy link
Member

icarito commented Aug 7, 2018

This is most likely a Gem with C-compiled extensions. Another way to try to catch it is with strace but we haven't found a way to reliably reproduce so I'll try to make a build with strace enabled and run it thru Jenkins until it fails and we can look at it.

@icarito
Copy link
Member

icarito commented Aug 11, 2018

Here's the relevant bit from strace 1000 lines here:

open("/usr/local/bundle/gems/concurrent-ruby-1.0.5/lib/rss/1.0.rb", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/bundle/gems/rake-12.3.1/lib/rss/1.0.rb", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/ruby/site_ruby/2.4.0/rss/1.0.rb", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/ruby/site_ruby/2.4.0/x86_64-linux/rss/1.0.rb", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/ruby/site_ruby/rss/1.0.rb", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/ruby/vendor_ruby/2.4.0/rss/1.0.rb", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/ruby/vendor_ruby/2.4.0/x86_64-linux/rss/1.0.rb", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/ruby/vendor_ruby/rss/1.0.rb", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/ruby/2.4.0/rss/1.0.rb", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = 9
fstat(9, {st_mode=S_IFREG|0644, st_size=9867, ...}) = 0
close(9)                                = 0
open("/dev/tty", O_RDWR|O_NOCTTY|O_NONBLOCK) = -1 ENXIO (No such device or address)
writev(2, [{iov_base="*** Error in `", iov_len=14}, {iov_base="/usr/local/bin/ruby", iov_len=19}, {iov_base="': ", iov_len=3}, {iov_base="free(): invalid next size (fast)", iov_len=32}, {iov_base=": 0x", iov_len=4}, {iov_base="000055f8b2d9d2e0", iov_len=16}, {iov_base=" ***\n", iov_len=5}], 7) = 93
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1a29664000
mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f1a04000000
munmap(0x7f1a08000000, 67108864)        = 0
mprotect(0x7f1a04000000, 135168, PROT_READ|PROT_WRITE) = 0
futex(0x7f1a285e5110, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7f1a2260b1a0, FUTEX_WAKE_PRIVATE, 2147483647) = 0
write(2, "======= Backtrace: =========\n", 29) = 29
writev(2, [{iov_base="/lib/x86_64-linux-gnu/libc.so.6", iov_len=31}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="70bcb", iov_len=5}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a282b8bcb", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 58
writev(2, [{iov_base="/lib/x86_64-linux-gnu/libc.so.6", iov_len=31}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="76f96", iov_len=5}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a282bef96", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 58
writev(2, [{iov_base="/lib/x86_64-linux-gnu/libc.so.6", iov_len=31}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="777de", iov_len=5}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a282bf7de", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 58
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="ruby_xfree", iov_len=10}, {iov_base="+0x", iov_len=3}, {iov_base="2d", iov_len=2}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a290043bd", iov_len=12}, {iov_base="]\n", iov_len=2}], 9) = 63
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="11e28f", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2906228f", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="1348ad", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a290788ad", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="134a70", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a29078a70", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="218adf", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2915cadf", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="rb_parser_compile_string_path", iov_len=29}, {iov_base="+0x", iov_len=3}, {iov_base="db", iov_len=2}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a29067c9b", iov_len=12}, {iov_base="]\n", iov_len=2}], 9) = 82
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="rb_iseq_compile_with_option", iov_len=27}, {iov_base="+0x", iov_len=3}, {iov_base="12a", iov_len=3}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2902b94a", iov_len=12}, {iov_base="]\n", iov_len=2}], 9) = 81
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="20bab9", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2914fab9", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="20c29b", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2915029b", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="1f9f57", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2913df57", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="2025fb", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a291465fb", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="207f7a", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2914bf7a", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="208cc4", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2914ccc4", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="rb_yield", iov_len=8}, {iov_base="+0x", iov_len=3}, {iov_base="dd", iov_len=2}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2914e32d", iov_len=12}, {iov_base="]\n", iov_len=2}], 9) = 61
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="rb_ary_each", iov_len=11}, {iov_base="+0x", iov_len=3}, {iov_base="3c", iov_len=2}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a28f7490c", iov_len=12}, {iov_base="]\n", iov_len=2}], 9) = 64
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="1f9f57", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2913df57", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="2025fb", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a291465fb", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="207f7a", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2914bf7a", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="20c954", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a29150954", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="20d316", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a29151316", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="rb_funcall", iov_len=10}, {iov_base="+0x", iov_len=3}, {iov_base="104", iov_len=3}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a29151984", iov_len=12}, {iov_base="]\n", iov_len=2}], 9) = 64
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="a7489", iov_len=5}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a28feb489", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 56
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="1f9f57", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2913df57", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="209973", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2914d973", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="2025fb", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a291465fb", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="207f7a", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2914bf7a", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="eb63f", iov_len=5}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2902f63f", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 56
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="ed381", iov_len=5}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a29031381", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 56
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="rb_require_safe", iov_len=15}, {iov_base="+0x", iov_len=3}, {iov_base="9", iov_len=1}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a290315b9", iov_len=12}, {iov_base="]\n", iov_len=2}], 9) = 67
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="1f9f57", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2913df57", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="2029a9", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a291469a9", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="207f7a", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2914bf7a", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="eb63f", iov_len=5}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2902f63f", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 56
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="ed381", iov_len=5}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a29031381", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 56
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="rb_require_safe", iov_len=15}, {iov_base="+0x", iov_len=3}, {iov_base="9", iov_len=1}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a290315b9", iov_len=12}, {iov_base="]\n", iov_len=2}], 9) = 67
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="1f9f57", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2913df57", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="2029a9", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a291469a9", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="207f7a", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2914bf7a", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="eb63f", iov_len=5}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2902f63f", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 56
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="ed381", iov_len=5}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a29031381", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 56
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="rb_require_safe", iov_len=15}, {iov_base="+0x", iov_len=3}, {iov_base="9", iov_len=1}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a290315b9", iov_len=12}, {iov_base="]\n", iov_len=2}], 9) = 67
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="1f9f57", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2913df57", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="209973", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2914d973", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="2029a9", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a291469a9", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="207f7a", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2914bf7a", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="eb63f", iov_len=5}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2902f63f", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 56
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="ed381", iov_len=5}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a29031381", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 56
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="rb_require_safe", iov_len=15}, {iov_base="+0x", iov_len=3}, {iov_base="9", iov_len=1}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a290315b9", iov_len=12}, {iov_base="]\n", iov_len=2}], 9) = 67
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="1f9f57", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2913df57", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="209973", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2914d973", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="2029a9", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a291469a9", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="207f7a", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2914bf7a", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="20c954", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a29150954", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="20d316", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a29151316", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="rb_const_missing", iov_len=16}, {iov_base="+0x", iov_len=3}, {iov_base="46", iov_len=2}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a29131f66", iov_len=12}, {iov_base="]\n", iov_len=2}], 9) = 69
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="10f240", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a29053240", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="1f9f57", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2913df57", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="2029a9", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a291469a9", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="207f7a", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2914bf7a", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
writev(2, [{iov_base="/usr/local/lib/libruby.so.2.4", iov_len=29}, {iov_base="(", iov_len=1}, {iov_base="+0x", iov_len=3}, {iov_base="208cc4", iov_len=6}, {iov_base=")", iov_len=1}, {iov_base="[0x", iov_len=3}, {iov_base="7f1a2914ccc4", iov_len=12}, {iov_base="]\n", iov_len=2}], 8) = 57
write(2, "======= Memory map: ========\n", 29) = 29
open("/proc/self/maps", O_RDONLY)       = 9
read(9, "cd358f3000-cd358f4000 r-xp 00000"..., 1024) = 1024
write(2, "cd358f3000-cd358f4000 r-xp 00000"..., 1024) = 1024
read(9, "3c54fce07000 rwxp 00000000 00:00"..., 1024) = 1024
write(2, "3c54fce07000 rwxp 00000000 00:00"..., 1024) = 1024
read(9, "0:00 0 \n7f1a10000000-7f1a1002100"..., 1024) = 1024
write(2, "0:00 0 \n7f1a10000000-7f1a1002100"..., 1024) = 1024
read(9, "6_64-linux-gnu/libffi.so.6.0.4\n7"..., 1024) = 1024
write(2, "6_64-linux-gnu/libffi.so.6.0.4\n7"..., 1024) = 1024
read(9, "0 00:38 8904                    "..., 1024) = 1024
write(2, "0 00:38 8904                    "..., 1024) = 1024
read(9, "     /lib/x86_64-linux-gnu/libns"..., 1024) = 1024
write(2, "     /lib/x86_64-linux-gnu/libns"..., 1024) = 1024
read(9, "-7f1a19b74000 ---p 00011000 00:3"..., 1024) = 1024
write(2, "-7f1a19b74000 ---p 00011000 00:3"..., 1024) = 1024
read(9, "ib/ruby/2.4.0/x86_64-linux/enc/t"..., 1024) = 1024
write(2, "ib/ruby/2.4.0/x86_64-linux/enc/t"..., 1024) = 1024
read(9, "                /usr/lib/x86_64-"..., 1024) = 1024
write(2, "                /usr/lib/x86_64-"..., 1024) = 1024
read(9, "r-xp 00000000 00:38 4158        "..., 1024) = 1024
write(2, "r-xp 00000000 00:38 4158        "..., 1024) = 1024
read(9, "kylight-2.0.2/lib/skylight_nativ"..., 1024) = 1024
write(2, "kylight-2.0.2/lib/skylight_nativ"..., 1024) = 1024
read(9, "4000 rw-p 001a5000 00:38 4042   "..., 1024) = 1024
write(2, "4000 rw-p 001a5000 00:38 4042   "..., 1024) = 1024
read(9, "0-7f1a21f7a000 rw-p 001b8000 00:"..., 1024) = 1024
write(2, "0-7f1a21f7a000 rw-p 001b8000 00:"..., 1024) = 1024
read(9, "le/gems/rdiscount-2.2.0.1/lib/rd"..., 1024) = 1024
write(2, "le/gems/rdiscount-2.2.0.1/lib/rd"..., 1024) = 1024
read(9, "                /usr/lib/x86_64-"..., 1024) = 1024
write(2, "                /usr/lib/x86_64-"..., 1024) = 1024
read(9, ".0/x86_64-linux/syslog.so\n7f1a23"..., 1024) = 1024
write(2, ".0/x86_64-linux/syslog.so\n7f1a23"..., 1024) = 1024
read(9, "000 00:38 2507                  "..., 1024) = 1024
write(2, "000 00:38 2507                  "..., 1024) = 1024
read(9, "40a1000-7f1a242a0000 ---p 000010"..., 1024) = 1024
write(2, "40a1000-7f1a242a0000 ---p 000010"..., 1024) = 1024
read(9, "246ba000 ---p 00014000 00:38 137"..., 1024) = 1024
write(2, "246ba000 ---p 00014000 00:38 137"..., 1024) = 1024
read(9, "00001000 00:38 1380             "..., 1024) = 1024
write(2, "00001000 00:38 1380             "..., 1024) = 1024
read(9, "ed000-7f1a24eee000 r--p 00011000"..., 1024) = 1024
write(2, "ed000-7f1a24eee000 r--p 00011000"..., 1024) = 1024
read(9, "                  /usr/local/lib"..., 1024) = 1024
write(2, "                  /usr/local/lib"..., 1024) = 1024
read(9, "38 2025                       /u"..., 1024) = 1024
write(2, "38 2025                       /u"..., 1024) = 1024
read(9, "000 r--p 0001d000 00:38 1702    "..., 1024) = 1024
write(2, "000 r--p 0001d000 00:38 1702    "..., 1024) = 1024
read(9, "cal/lib/ruby/2.4.0/x86_64-linux/"..., 1024) = 1024
write(2, "cal/lib/ruby/2.4.0/x86_64-linux/"..., 1024) = 1024
read(9, "l/lib/ruby/2.4.0/x86_64-linux/en"..., 1024) = 1024
write(2, "l/lib/ruby/2.4.0/x86_64-linux/en"..., 1024) = 1024
read(9, ".0/x86_64-linux/io/nonblock.so\n7"..., 1024) = 1024
write(2, ".0/x86_64-linux/io/nonblock.so\n7"..., 1024) = 1024
read(9, "nux-gnu/libssl.so.1.1\n7f1a26d830"..., 1024) = 1024
write(2, "nux-gnu/libssl.so.1.1\n7f1a26d830"..., 1024) = 1024
read(9, "1.so\n7f1a2741d000-7f1a2741e000 r"..., 1024) = 1024
write(2, "1.so\n7f1a2741d000-7f1a2741e000 r"..., 1024) = 1024
read(9, "63                        /usr/l"..., 1024) = 1024
write(2, "63                        /usr/l"..., 1024) = 1024
read(9, "r/local/lib/ruby/2.4.0/x86_64-li"..., 1024) = 1024
write(2, "r/local/lib/ruby/2.4.0/x86_64-li"..., 1024) = 1024
read(9, "al/lib/ruby/2.4.0/x86_64-linux/e"..., 1024) = 1024
write(2, "al/lib/ruby/2.4.0/x86_64-linux/e"..., 1024) = 1024
read(9, "00:00 0 \n7f1a285e7000-7f1a286ea0"..., 1024) = 1024
write(2, "00:00 0 \n7f1a285e7000-7f1a286ea0"..., 1024) = 1024
read(9, "inux-gnu/libdl-2.24.so\n7f1a28b26"..., 1024) = 1024
write(2, "inux-gnu/libdl-2.24.so\n7f1a28b26"..., 1024) = 1024
read(9, "         /usr/local/lib/libruby."..., 1024) = 1024
write(2, "         /usr/local/lib/libruby."..., 1024) = 1024
read(9, "lib/x86_64-linux-gnu/ld-2.24.so\n"..., 1024) = 512
write(2, "lib/x86_64-linux-gnu/ld-2.24.so\n"..., 512) = 512
read(9, "", 1024)                       = 0
close(9)                                = 0
rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0
getpid()                                = 315
gettid()                                = 315
tgkill(315, 315, SIGABRT)               = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=315, si_uid=0} ---
+++ killed by SIGABRT (core dumped) +++

Full trace (88MB) at https://gitlab.com/publiclab/plots2/-/jobs/88571931/artifacts/browse

@icarito
Copy link
Member

icarito commented Aug 11, 2018

Still don't know what it means. Could it be related to #3086 ? :-/

@icarito
Copy link
Member

icarito commented Aug 11, 2018

I left a comment with the question at https://bugs.ruby-lang.org/issues/11018 - hopefully we'll get a response!

@icarito
Copy link
Member

icarito commented Aug 15, 2018

Okay I've setup Jenkins to build/test every 10 minutes. We're seeing this error consistently. I've set it up to get us strace logs for each build and set string capture to 256 bytes.

We can move things around and try to get it to stop failing / we can measure results now.

@icarito
Copy link
Member

icarito commented Aug 15, 2018

The last strace had truncated strings. Here's the last 3000 lines.

I think here's the more interesting bits:

open("/usr/local/lib/ruby/2.4.0/rss/dublincore/atom.rb", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = 9
fcntl(9, F_SETFL, O_RDONLY)             = 0
fstat(9, {st_mode=S_IFREG|0644, st_size=282, ...}) = 0
ioctl(9, TCGETS, 0x7ffc74c55810)        = -1 ENOTTY (Inappropriate ioctl for device)
read(9, "# frozen_string_literal: false\nrequire \"rss/atom\"\n\nmodule RSS\n  module Atom\n    Feed.install_ns(DC_PREFIX, DC_URI)\n\n    class Feed\n      include DublinCoreModel\n      class Entry; include DublinCoreModel; end\n    end\n\n    class Entry\n      include DublinCo"..., 8192) = 282
read(9, "", 8192)                       = 0
close(9)                                = 0
lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/usr/local", {st_mode=S_IFDIR|S_ISGID|0775, st_size=4096, ...}) = 0
lstat("/usr/local/lib", {st_mode=S_IFDIR|S_ISGID|0775, st_size=4096, ...}) = 0
lstat("/usr/local/lib/ruby", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/usr/local/lib/ruby/2.4.0", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/usr/local/lib/ruby/2.4.0/rss", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/usr/local/lib/ruby/2.4.0/rss/dublincore", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/usr/local/lib/ruby/2.4.0/rss/dublincore/atom.rb", {st_mode=S_IFREG|0644, st_size=282, ...}) = 0
open("/dev/tty", O_RDWR|O_NOCTTY|O_NONBLOCK) = -1 ENXIO (No such device or address)
writev(2, [{iov_base="*** Error in `", iov_len=14}, {iov_base="/usr/local/bin/ruby", iov_len=19}, {iov_base="': ", iov_len=3}, {iov_base="free(): invalid pointer", iov_len=23}, {iov_base=": 0x", iov_len=4}, {iov_base="00005614dcc19f00", iov_len=16}, {iov_base=" ***\n", iov_len=5}], 7) = 84
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fef0abc6000
futex(0x7fef09b47110, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7fef03b6d1a0, FUTEX_WAKE_PRIVATE, 2147483647) = 0
write(2, "======= Backtrace: =========\n", 29) = 29

@icarito
Copy link
Member

icarito commented Aug 15, 2018

Having set to trace and test every 10 minutes, I have a lot of traces for this failure.
It consistently mentions dublincore and rss/maker Gem.
gem list did not reveal where this gem is coming from.

@siaw23-retired
Copy link
Collaborator

There's no such gem as dublincore on RubyGems and rss/maker looks strange for a gem name @icarito. Maybe they are not gems after all?

@jywarren
Copy link
Member

jywarren commented Aug 16, 2018 via email

@icarito
Copy link
Member

icarito commented Aug 16, 2018

Yes this component is part of Ruby proper. I've made the experiment of upgrading to Ruby 2.4.4 (we were at 2.4.1) - didn't fail all night testing every 10 minutes. This morning it began building master because it was pushed but I've restored testing of jenkins_strace branch if it doesn't fail then it's likely this was a Ruby bug that hopefully was fixed in 2.4.x.

@ghost ghost assigned icarito Aug 22, 2018
@ghost ghost added in progress and removed high-priority labels Aug 22, 2018
@icarito
Copy link
Member

icarito commented Aug 22, 2018

Okay now It's building every 15 minutes hopefully will stay green, fixes in PR #3253

@ghost ghost removed the in progress label Aug 27, 2018
jywarren pushed a commit that referenced this issue Aug 27, 2018
* Added strace option and artifact

* Restore Gitlab and add strace to Makefile

* Use strace within container

* Give more trace output

* Switch invocation of passenger to fork 9 processes in testing

* 256 string length. No op for triggering builds.

* Try to speed up by not running bower / bundler

* Build, then start.

* Built before

* Ignore logs in container

* Try with newer docker ruby image

* Set ruby version in Gemfile to 2.4.4

* Increase trace log to 512

* Don't export Redis port

* Don't export web app port for testing

* Set explicit Ruby version in Dockerfile

* Avoid running bundle in Travis

* Move strace to Gitlab to save disk space.

* Add strace and artifacts to tests.

* Remove redundant bundle and bower calls.

* Add capability to be able to strace on Gitlab.

* Run bower in Gitlab

* Remove bash

* Ignore tmp dir from docker container
SrinandanPai pushed a commit to SrinandanPai/plots2 that referenced this issue May 5, 2019
* Added strace option and artifact

* Restore Gitlab and add strace to Makefile

* Use strace within container

* Give more trace output

* Switch invocation of passenger to fork 9 processes in testing

* 256 string length. No op for triggering builds.

* Try to speed up by not running bower / bundler

* Build, then start.

* Built before

* Ignore logs in container

* Try with newer docker ruby image

* Set ruby version in Gemfile to 2.4.4

* Increase trace log to 512

* Don't export Redis port

* Don't export web app port for testing

* Set explicit Ruby version in Dockerfile

* Avoid running bundle in Travis

* Move strace to Gitlab to save disk space.

* Add strace and artifacts to tests.

* Remove redundant bundle and bower calls.

* Add capability to be able to strace on Gitlab.

* Run bower in Gitlab

* Remove bash

* Ignore tmp dir from docker container
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted requires help by anyone willing to contribute Ruby testing issues are usually for adding unit tests, integration tests or any other tests for a feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants