Bug: sam init error: "could not clone repo" (non-zero exit status 128)

[mcalligator]

Description:

When I run sam init, it fails to clone the aws/aws-sam-cli-app-templates repo, and thus does not properly initialise the project directory. This happens regardless of which Quick Start Application Template I select.

Steps to reproduce:

1. Install latest version of SAM CLI (1.123.0).
2. Run sam init:

Which template source would you like to use?
        1 - AWS Quick Start Templates
        2 - Custom Template Location
Choice: 1

Choose an AWS Quick Start application template
        1 - Hello World Example
        2 - Data processing
...
        15 - DynamoDB Example
        16 - Machine Learning
Template: 1

Use the most popular runtime and package type? (Python and zip) [y/N]: y

Would you like to enable X-Ray tracing on the function(s) in your application?  [y/N]: n

Would you like to enable monitoring using CloudWatch Application Insights? [y/N]: n

Would you like to set Structured Logging in JSON format on your Lambda functions?  [y/N]: n

Project name [sam-app]: sam-app

I did a thorough search of existing issues (closed and open), and found none closely matching this one. I might be doing something wrong, but for now, I've designated it as a bug.

Observed result:

2024-08-26 21:05:54,743 | Cache directory does not yet exist, creating one.
2024-08-26 21:05:54,749 |
Cloning from https://github.com/aws/aws-sam-cli-app-templates (process may take a moment)
2024-08-26 21:05:54,760 | WARN: Could not clone repo https://github.com/aws/aws-sam-cli-app-templates
Traceback (most recent call last):
  File "samcli/lib/utils/git_repo.py", line 149, in clone
  File "subprocess.py", line 466, in check_output
  File "subprocess.py", line 571, in run
subprocess.CalledProcessError: Command '['git', 'clone', 'https://github.com/aws/aws-sam-cli-app-templates',
'aws-sam-cli-app-templates']' returned non-zero exit status 128.
2024-08-26 21:05:54,765 | Clone error, attempting to use an old clone from a previous run

    -----------------------
    Generating application:
    -----------------------
    Name: sam-son
    Runtime: python3.9
    Architectures: x86_64
    Dependency Manager: pip
    Application Template: hello-world
    Output Directory: .
    Configuration file: sam-son/samconfig.toml

    Next steps can be found in the README file at sam-son/README.md

Commands you can use next
=========================
[*] Create pipeline: cd sam-son && sam pipeline init --bootstrap
[*] Validate SAM template: cd sam-son && sam validate
[*] Test Function in the Cloud: cd sam-son && sam sync --stack-name {stack-name} --watch

2024-08-26 21:05:54,766 | Parameters dict created with input given
2024-08-26 21:05:54,767 | {'template':
'/usr/local/aws-sam-cli/1.122.0/dist/_internal/samcli/lib/init/templates/cookiecutter-aws-sam-hello-python',
'output_dir': '.', 'no_input': True, 'extra_context': {'project_name': 'sam-son', 'runtime': 'python3.9',
'architectures': {'value': ['x86_64']}}}
2024-08-26 21:05:54,768 | Baking a new template with cookiecutter with all parameters
2024-08-26 21:05:54,812 | Moving samconfig.toml into cookie cutter template
2024-08-26 21:05:54,813 | No config file found in this directory.
2024-08-26 21:05:54,814 | OSError occurred while reading TOML file: [Errno 2] No such file or directory:
'sam-son/samconfig.toml'

Other issues do report Clone error, attempting to use an old clone from a previous run, but these all seem to be from 2022, and relate to long Windows filepaths, which doesn't apply here.

I am able to clone the /aws/aws-sam-cli-app-templates repo manually in the project directory successfully, so I don't think there's a problem with my GitHub credentials.

Expected result:

The project initialises as expected, creating a project sub-directory appropriately populated with the correct files.

Additional environment details

1. OS: Arch Linux on WSL Windows 11 Home (10.0.22631 N/A Build 22631)
2. sam --version: SAM CLI, version 1.122.0
3. AWS region: us-east-1 (not that this is relevant to the issue, since I can't even get it to initialise locally).
4. Output of sam --info:

{
  "version": "1.123.0",
  "system": {
    "python": "3.11.8",
    "os": "Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.40"
  },
  "additional_dependencies": {
    "docker_engine": "Not available",
    "aws_cdk": "Not available",
    "terraform": "1.9.5"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

Add --debug flag to command you are running

[mcalligator]

Just to shed a bit more light on this, I compared the installations between Windows and WSL, and they differ substantially:

Windows

 Directory of c:\Program Files\Amazon\AWSSAMCLI\runtime\Lib\site-packages\samcli\lib

27/08/2024  09:52    <DIR>          .
27/08/2024  09:52    <DIR>          ..
27/08/2024  09:52    <DIR>          bootstrap
27/08/2024  09:52    <DIR>          build
27/08/2024  09:52    <DIR>          cli_validation
27/08/2024  09:52    <DIR>          config
20/08/2024  00:23                32 constants.py
27/08/2024  09:52    <DIR>          cookiecutter
27/08/2024  09:52    <DIR>          delete
27/08/2024  09:52    <DIR>          deploy
27/08/2024  09:52    <DIR>          docker
27/08/2024  09:52    <DIR>          docs
27/08/2024  09:52    <DIR>          generated_sample_events
27/08/2024  09:52    <DIR>          hook
27/08/2024  09:52    <DIR>          iac
27/08/2024  09:52    <DIR>          init
27/08/2024  09:52    <DIR>          intrinsic_resolver
27/08/2024  09:52    <DIR>          list
27/08/2024  09:52    <DIR>          observability
27/08/2024  09:52    <DIR>          package
27/08/2024  09:52    <DIR>          pipeline
27/08/2024  09:52    <DIR>          providers
27/08/2024  09:52    <DIR>          remote_invoke
27/08/2024  09:52    <DIR>          samlib
27/08/2024  09:52    <DIR>          schemas
27/08/2024  09:52    <DIR>          shared_test_events
27/08/2024  09:52    <DIR>          sync
27/08/2024  09:52    <DIR>          telemetry
27/08/2024  09:52    <DIR>          translate
27/08/2024  09:52    <DIR>          utils			<--- Missing from WSL installation ---
27/08/2024  09:52    <DIR>          warnings
20/08/2024  00:23                 0 __init__.py
27/08/2024  09:52    <DIR>          __pycache__

WSL

[me@pc4 lib]$ /usr/local/aws-sam-cli/1.123.0/dist/_internal/samcli/lib
[me@pc4 lib]$  ls -al
drwxr-xr-x 2 root root 4096 Aug 27 17:59 docs
drwxr-xr-x 3 root root 4096 Aug 27 17:59 generated_sample_events
drwxr-xr-x 2 root root 4096 Aug 27 17:59 hook
drwxr-xr-x 3 root root 4096 Aug 27 17:59 init
drwxr-xr-x 3 root root 4096 Aug 27 17:59 pipeline

The utils directory (which contains git_repo.py) only appears in the Windows installation. All but 5 of the directories in the Windows installation are absent from the WSL one.

What confuses me is that it still seems to be executing (otherwise I don't see how the error itself would even occur) - but I can't identify where from.

[dkphm]

Hi @mcalligator , thank you for reporting the issue.

For sam init, it will call git command running on your host, so looks like there was something wrong with git installed on your WSL box. Could you please try to run these commands and give us the output?

cd /tmp
git clone https://github.com/aws/aws-sam-cli-app-templates

Also it would be useful to confirm where git was installed by running which git.

[mcalligator]

Hi @dkphm, thanks for your reply. Here are the outputs as requested.

git clone https://github.com/aws/aws-sam-cli-app-templates:

Cloning into 'aws-sam-cli-app-templates'...
remote: Enumerating objects: 10655, done.
remote: Counting objects: 100% (2443/2443), done.
remote: Compressing objects: 100% (629/629), done.
remote: Total 10655 (delta 1883), reused 2080 (delta 1755), pack-reused 8212 (from 1)
Receiving objects: 100% (10655/10655), 21.99 MiB | 1.32 MiB/s, done.
Resolving deltas: 100% (7491/7491), done.

And which git output: /usr/sbin/git. But is that a symlink?
ls -l /usr/sbin output: lrwxrwxrwx 1 root root 3 Apr 7 19:02 /usr/sbin -> bin - yes it is, so let's follow the trail:
ls -l /bin output: lrwxrwxrwx 1 root root 7 Apr 7 19:02 /bin -> usr/bin - also a symlink, so let's keep following:
ls -l /usr output:

drwxr-xr-x   6 root root  61440 Aug 27 16:07 bin
drwxr-xr-x   3 root root   4096 Jul  8 17:33 etc
drwxr-xr-x 225 root root  20480 Aug 26 16:35 include
drwxr-xr-x 109 root root 118784 Aug 26 16:35 lib
drwxr-xr-x   2 root root   4096 Aug 26 16:35 lib32
lrwxrwxrwx   1 root root      3 Apr  7 19:02 lib64 -> lib
drwxr-xr-x  13 root root   4096 Aug 28 17:51 local
lrwxrwxrwx   1 root root      3 Apr  7 19:02 sbin -> bin
drwxr-xr-x 121 root root   4096 Aug 26 16:35 share
drwxr-xr-x   5 root root   4096 May 25 21:04 src

No more symlinks, so /usr/bin is where git binary is stored. Confirming with ls -l /usr/bin/ | grep -w git:

-rwxr-xr-x 1 root root    4333968 Jul 29 21:39 git
-rwxr-xr-x 1 root root     163763 Jul 29 21:39 git-cvsserver
lrwxrwxrwx 1 root root          3 Jul 29 21:39 git-receive-pack -> git
-rwxr-xr-x 1 root root     775776 Jul 29 21:39 git-shell
lrwxrwxrwx 1 root root          3 Jul 29 21:39 git-upload-archive -> git
lrwxrwxrwx 1 root root          3 Jul 29 21:39 git-upload-pack -> git

That all looks OK to me; I can't see any WSL-specific gotchas there. But happy to keep digging as advised!

[dkphm]

Hi @mcalligator thanks for the info. I was able to reproduce the issue. Looks like there is something wrong with calling git under ArchLinux WSL. We will investigate more and let you know once this is fixed.

[IrishPrime]

Looks like there is something wrong with calling git under ArchLinux WSL.

Not just WSL. I'm encountering the same issue on my ArchLinux hosts with SAM CLI, version 1.123.0.

[dkphm]

I see, thanks for the info @IrishPrime

[mcalligator]

I was able to reproduce the issue. Looks like there is something wrong with calling git under ArchLinux WSL. We will investigate more and let you know once this is fixed.

Thanks @dkphm - looking forward to hearing what you discover about SAM CLI deployment to Arch Linux, and how to fix this.

[mcalligator]

Are there any updates you can share @dkphm?

[hnnasit]

Hi @mcalligator, thanks for reporting the bug and providing the details. The following issue has been marked as a bug and added to our backlog. The team will be tackling the bug based on the priority we follow.

[dkphm]

Hi @mcalligator, sorry for the late reply.

From my testing (on WSL), looks like even though the CLI throws an ERROR, but everything seems working fine for me (I still have the template on my machine and was able to build and run basic commands). Just wondering is that the same for you, or it is totally not working on your environment at all?

[adds68]

Also seeing this on Fedora 40 and sam v1.125.0

Downgrading to sam v1.121.0 stops this issue, so it must be something introduced after this release?

[dkphm]

Thanks for the info @adds68 , looks like this is the known issue with OpenSSL 3 and Python subprocess.check_output. We are working on the workaround and will release in the next version.

In the meantime, you can run this command prior to sam init to mitigate the issue

rm -rf ~/.aws-sam/aws-sam-cli-app-templates && git clone https://github.com/aws/aws-sam-cli-app-templates ~/.aws-sam/aws-sam-cli-app-templates

[github-actions]

Patch is released in v1.127.0. Closing