-
-
Notifications
You must be signed in to change notification settings - Fork 358
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pharo CI stopped working - libgit errors #11481
Comments
I am bitten by this in an other project (pillar-markup/Microdown). I believe it is related to #11222. |
Yes, It kills all my projects and Moose projects |
Is this happening for GitHub actions only? or others too? |
@guillep I use only GitHub actions, unfortunately, but @badetitou said in hpi-swa/smalltalkCI#562 (comment) that GitLab works fine for him |
Most of my projects use Moose and all these fail consistently at the time it usually fetches the Moose, but that is probably because the Moose is large. My small projects with few simple dependencies (and no Moose) seem to mostly work, but some of them not (and these fail always the same time as well). |
I get it not when building, but this piece of code in a fresh image (before I apply my github credentials) IceGitHubAPI new in: [ :api | 50 timesRepeat: [ api get: 'repos/pillar-markup/Microdown'] ] However, I am not sure it is related. |
Does it work after setting up credentials? |
After credentials I can do around 5000 api calls per hour, which so far has been enough. It is in the iceberg wiki. But how to make sure the credentials are set in the CI I have no idea. Also, I have no idea why the CI is suddenly broken. |
After doing this: #(
'https://github.com/SeasideSt/Seaside-Legacy30'
'https://github.com/pharo-project/pharo-zeroconf.git'
'https://github.com/pharo-project/pharo.git'
'https://github.com/pharo-project/pharo-launcher.git'
'https://github.com/pharo-project/pharo-vm.git'
'https://github.com/pharo-project/pharo-site.git'
'https://github.com/pharo-project/pharo-changelogs.git'
'https://github.com/pharo-project/pharo-core.git'
) do: [ :url |
Transcript show: url; cr.
[
IceRepositoryCreator new
url: url;
location: FileLocator temp / ('IceRepositoryCreator Test ' , Time microsecondClockValue asString);
createRepository
] on: Error do: [ :error |
Transcript show: error asString; cr ]
] separatedBy: [ Transcript cr ] My Transcript shows:
So: 3 OK, 2 times ‘bad packet length’ and 3 times ‘error reading from the zlib stream’. |
I try setting up the token as described in the Iceberg wiki and proposed by @kasperosterbye in moosetechnology/Moose@30d5dc6
|
GitHub action virtual environment has changed the 20220724. Maybe it created the problem for us (updating of one lib)
|
And also, we have this issue now
|
I suspect the problem has to do with the following point in the changes listed for libgit2 v1.0.1:
The following gives v1.0.0 as the version of libgit2 used in Pharo: LGitLibrary uniqueInstance version "=> #(1 0 0)" Using that version of libgit2, the C program given below reproduces the error:
Upgrading to v1.0.1 fixes the error:
I used MacPorts to install the different versions of libgit2, making use of the instructions on the page “How to install an older version of a port”. The ‘Portfile’ for v1.0.1 can be found in: macports/macports-ports@abe3087564d83e7e. A diff for v1.0.0 is given below. test.c: #include <unistd.h>
#include <stdio.h>
#include <git2/global.h>
#include <git2/clone.h>
#include <git2/errors.h>
int main() {
int major, minor, rev;
git_libgit2_init();
if (git_libgit2_version(&major, &minor, &rev) != 0) {
printf("git_libgit2_version failed\n");
return 1;
}
printf("libgit2 version: %i.%i.%i\n", major, minor, rev);
char dir[] = "/tmp/libgit2_clone_test.XXXXXX";
if (mkdtemp(dir) == NULL) {
printf("mkdtemp failed\n");
return 1;
}
printf ("cloning to: %s\n", dir);
git_repository *repo;
int value = git_clone(&repo, "https://github.com/SeasideSt/Seaside-Legacy30", dir, NULL);
printf("git_clone returned: %i\n", value);
const git_error *error = git_error_last();
if (error != NULL)
printf("error message: %s\n", error->message);
return (value == 0) ? 0 : 1;
} Makefile: test: test.c
cc -I/opt/local/include -L/opt/local/lib test.c -o test -lgit2 Portfile diff: diff --git a/devel/libgit2/Portfile b/devel/libgit2/Portfile
index 7c7d6c532e0..e8120559c40 100644
--- a/devel/libgit2/Portfile
+++ b/devel/libgit2/Portfile
@@ -9 +9 @@ PortGroup legacysupport 1.0
-github.setup libgit2 libgit2 1.0.1 v
+github.setup libgit2 libgit2 1.0.0 v
@@ -27,3 +27,3 @@ homepage https://libgit2.org/
-checksums rmd160 3f9ced0e0dff170a8156e4aa8fcb0abce66c8f60 \
- sha256 5dae7cb32b6977cd95ed849d24f3692f0b7e9eb9b0ee9ffaa14baebb9cac76e1 \
- size 5304918
+checksums rmd160 f46ca0500f159e058d854ed6aeb8d4418420b419 \
+ sha256 1d3135077f7b0401c1172e41f561cadd06fd159e75aa24d710de1bd3a24b1440 \
+ size 5302852 |
I can confirm that version 1.0.0 is used in the CI
|
Was not the libgit 1.0.0 there the whole time? I wonder why is it problem now. Maybe GitHub changed something about data transfers (since GitLab does not seem affected)? |
The description of the change in libgit2 I referred to says “fetching from servers like Gerrit, that sent large data packets, would error.” I assume GitHub just didn’t send large data packets before but now does. |
Sadly, the error seems to be a bit random. I've run again the failing builds and sometimes fail but other times don't. And for jobs having a build matrix, they failed for some configurations but not for others, even when the only difference is the packages loaded. |
I was able to reproduce the problem without using GitHub actions at all, but by doing a docker build in my notebook. The first 2 times it failed with |
As mentioned in #11481 (comment), libgit2 needs an update in the Pharo vm... where does this need to be reported to get it done? |
Here but the team is on vacation :( |
It is a bit crazy that such changes are not backward compatible. :( and impacts everybody. |
@Ducasse thanks for the reply Stef. I suspected that France is in holiday mode right now ;-). Just wanted to make sure it reaches the right people. |
It does :) we got impacted too. I cannot build some of my projects and not even load the code :( |
Yes, the OSX problem is separate, check the issue here: #11561 |
@guillep can we get a new Pharo 10 release once all the related issues are merged? I want to update our docker images for Pharo but prefer to base it on a tagged version (like v10.1.0) and not a commit hash. |
@gcotelli The version v10.0.1 is ready |
@guillep @tesonep I just saw there was another ‘IceGenericError: bad packet length’ on our Jenkins server last night, so I was wondering what the status of this issue is with respect to Pharo 9? Is an update still coming? Info from PharoDebug.log:
|
Actually, I still randomly see errors as well even for Pharo 10 on Mac. At least it is "sometimes randomly", and not every time like before. |
Hi @Rinzwind , the issue is still not backported to Pharo9. There is PR #11596 on hold, but it requires some more work. There is probably a mismatch between Pharo9 and the version of NewTools that is trying to be installed. @JanBliznicenko Can you tell us if the problem persists? having connection errors is something that happens... |
@guillep Unfortunately it seems so. It is random, so MUCH less often than before, but it used to be completely ok before all these GitHub-related problems started.
|
Hi @JanBliznicenko, to minimize the noise maybe a good alternative is to put as preInstall script:
In this way it will just try to use HTTPS and don't try to use SSH. Now, it tries with SSH and if it fails retries with HTTPS. |
Might be useful to someone else: to avoid the problem on our Jenkins server, which uses Debian, I now extended our build scripts to install the package ‘libgit2-1.1’, and to apply a patch to smalltalkCI like the one given below. The output then shows diff --git a/pharo/run.sh b/pharo/run.sh
index c35c456..6dde247 100644
--- a/pharo/run.sh
+++ b/pharo/run.sh
@@ -334,6 +334,25 @@ pharo::run_script() {
# Load project into Pharo image.
################################################################################
pharo::load_project() {
+ pharo::run_script "
+ LGitLibrary compile: 'unix64LibraryName
+
+ \"Patched to try libgit2.so.1.1 first, see: https://github.com/pharo-project/pharo/issues/11481\"
+
+ ^ FFIUnix64LibraryFinder findAnyLibrary: #(
+ ''libgit2.so.1.1''
+ \"This name is wrong, but some versions of the VM has this library shipped with the bad name\"
+ ''libgit2.1.0.0.so''
+ ''libgit2.so.1.0.0''
+ ''libgit2.so.1.0''
+ ''libgit2.so.1.2''
+ ''libgit2.so.0.25.1'')'.
+ Smalltalk snapshot: true andQuit: true
+ "
+ pharo::run_script "
+ Transcript show: 'LGitLibrary uniqueInstance version = ' , LGitLibrary uniqueInstance version asString; cr.
+ Smalltalk snapshot: true andQuit: true
+ "
pharo::run_script "
| smalltalkCI |
$(conditional_debug_halt) |
Yes, that looks much better now, I have been postponing doing something with those warnings for years and it is actually simpler than I thought :) Unfortunately, it does not solve the problem I have. It seems really Mac-only now. It happens for me about 50 % of the time. |
@guillep I’m not sure you linked to the right issue (1612 in this repository is a pull request, ‘add window tiling shortcuts’). Edit: I hadn’t noticed the right issue is actually mentioned right above your message (so: pharo-vcs/iceberg#1612). One question I had here still: in |
Oups, yes, different repositories :) I'll fix the link
Yes, I think so! |
I have integrated a fix for the OSX problem with the connection, it is integrated into P11, later we are going to do a release of Iceberg and integrated it into P10. This version should improve the problem with OSX, as it will retry if there is a network issue. |
Hi all, is this problem finally fixed? |
I still havbe problem with Pharo 9. |
Hmmm.. I'll check again and send you the trace if one |
@guillep All my builds use Pharo 10 and are on Win, Linux and Mac and all are completely fine lately, thank you. |
I have not seen either of the two errors anymore (and the workaround for our build scripts has been removed from the scripts). |
Thank you all! @badetitou yes, a more concrete case would help, because
I'll close it. We can reopen a new case if needed. |
So, it seems sometimes I still get the error for my biggest projects. Like this: https://github.com/OpenPonk/class-editor/actions/runs/3321581366/jobs/5489424282
|
I am not exactly sure what is the source of the problem, but all Pharo CI/CD on Linux and Mac stopped working in last 24 hours. Win builds seem unaffected.
I have seen different errors. Mostly
IceGenericError: error reading from the zlib stream
, but few timesIceGenericError: bad packet length
.Source of the log: https://github.com/OpenPonk/class-editor/runs/7543017828?check_suite_focus=true
Probably related to pharo-vcs/iceberg#1600 and hpi-swa/smalltalkCI#562
The text was updated successfully, but these errors were encountered: