creating new sibling ria+ssh : [ERROR ] 'SSHRemoteIO' object has no attribute 'url2transport_path'

[kevinCassou]

Description

Dear dataladers,

We are trying to set an RIA to test a new sibling on a remote SSH server any idea about the following error ? We cannot install datalad or anything that require admin grant on the remote server :

datalad --version
datalad 1.1.3
datalad create-sibling-ria -s silverTemp --new-store-ok ria+ssh://user@server/silver/data_store/
[INFO ] create siblings 'silverTemp' and 'silverTemp-storage' ...
[ERROR ] 'SSHRemoteIO' object has no attribute 'url2transport_path'
0.00 [00:02, ?/s]%

Many thanks for help.

[adswa]

Hi, thanks for the issue! Could you let me know which version of datalad-next is installed (e.g. with pip freeze | grep next)?

[kevinCassou]

Hi
thanks for quick comment !

pip freeze | grep next

datalad-next==1.5.0
Yes, in fact the datalad extension is installed.

[kevinCassou]

so I try
pip uninstall datalad-next
then
datalad create-sibling-ria -s silverTemp --new-store-ok ria+ssh://user@server/silver/data_store/

[INFO ] create siblings 'silverTemp' and 'silverTemp-storage' ...
[INFO ] Fetching updates for Dataset(/temp/sandbox/mydataset)
update(ok): . (dataset)
update(ok): . (dataset)
[INFO ] Configure additional publication dependency on "silverTemp-storage"
configure-sibling(ok): . (sibling)
create-sibling-ria(ok): /sandbox/mydataset (dataset)
action summary:
configure-sibling (ok: 1)
create-sibling-ria (ok: 1)
update (ok: 1)
0.00 [00:04, ?/s]%

checking

datalad siblings

.: here(+) [git]
.: silverTemp(-) [ssh://user@server/silver/data_store/b1d/10d7e-64b0-4d60-a4f2-9a14d366dfb0 (git)]
.: silverTemp-storage(+) [ora]

👍
Thanks ! So datalad-next is not compatible with the RIA store feature ?

[adswa]

Thanks ! So datalad-next is not compatible with the RIA store feature ?

Actually, it should be. Quite the opposite, it should improve RIA functionality - so what you are seeing is certainly a bug. datalad-next patches the RIA/ORA code in datalad with the url2transport_path method to provide support for Windows. I will transfer this issue to datalad-next for for further exploration.

[adswa]

@christian-monch do you maybe have an immediate intuition what is going on here (given that you wrote #669)? It looks like SSHRemoteIO somehow has not (yet?) gotten patched when next-patched ORA code calls url2transport_path? I haven't found a way to reproduce this, though.

[kevinCassou]

Ah ok, I was testing from a Mac arm64 with python=Python 3.10.6
Thanks for all

[christian-monch]

Hi @kevinCassou thanks for the report. I tried to reproduce the error locally but did not yet succeed. The commands you provided worked for me.

Would it be possible for you to set the environment variable DATALAD_LOG_LEVEL to debug, run the same command again, and post the output here?

Something like:

> export DATALAD_LOG_LEVEL=debug
> datalad create-sibling-ria -s silverTemp --new-store-ok ria+ssh://user@server/silver/data_store/

[kevinCassou]

Hi,
So try to reproduce it:

 conda create -n dltest python=3.10
 conda activate dltest
 conda install -c conda-forge datalad
 python -m pip install datalad-next
 datalad --version
     datalad 1.1.3

then creating a dataset for test :

datalad create -c text2git mydataset01

.. adding some files

datalad save -m'first files add'

add(ok): README.md (file)
add(ok): data/spec1-test1.pickle (file)
add(ok): test.txt (file)
save(ok): . (dataset)
action summary:
  add (ok: 3)
  save (ok: 1)

then

export  DATA_LOG_LEVEL=debug
datalad create-sibling-ria -s silverTemp --new-store-ok ria+ssh://user@server/silver/temp/data_store/

and I got

[DEBUG  ] Not retro-fitting GitRepo with deprecated symbols, datalad-deprecated package not found
[DEBUG  ] Command line args 1st pass for DataLad 1.1.3. Parsed: Namespace() Unparsed: ['create-sibling-ria', '-s', 'silverTemp', '--new-store-ok', 'ria+ssh://user@server/silver/temp/data_store/']
[DEBUG  ] Building doc for <class 'datalad.core.distributed.clone.Clone'>
[DEBUG  ] Building doc for <class 'datalad.distributed.create_sibling_ria.CreateSiblingRia'>
[DEBUG  ] Parsing known args among ['/opt/miniconda3/envs/py39/dltest/bin/datalad', 'create-sibling-ria', '-s', 'silverTemp', '--new-store-ok', 'ria+ssh://user@server/silver/temp/data_store/']
[DEBUG  ] Determined class of decorated function: <class 'datalad.distributed.create_sibling_ria.CreateSiblingRia'>
[DEBUG  ] Resolved dataset to create RIA sibling(s): /Users/user/temp/sandbox/mydataset01
[DEBUG  ] Run ['git', 'version'] (protocol_class=StdOutErrCapture) (cwd=None)
[DEBUG  ] Finished ['git', 'version'] with status 0
[DEBUG  ] Run ['git', 'config', '-z', '-l', '--show-origin'] (protocol_class=StdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[DEBUG  ] Finished ['git', 'config', '-z', '-l', '--show-origin'] with status 0
[DEBUG  ] Run ['git', 'config', '-z', '-l', '--show-origin', '--file', '/Users/user/temp/sandbox/mydataset01/.datalad/config'] (protocol_class=StdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[DEBUG  ] Finished ['git', 'config', '-z', '-l', '--show-origin', '--file', '/Users/user/temp/sandbox/mydataset01/.datalad/config'] with status 0
[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'rev-parse', '--quiet', '--verify', 'HEAD^{commit}'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'cat-file', 'blob', 'git-annex:remote.log'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[Level 11] CommandError: 'git -c diff.ignoreSubmodules=none -c core.quotepath=false cat-file blob git-annex:remote.log' failed with exitcode 128 [err: 'fatal: path 'remote.log' does not exist in 'git-annex'']
[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'cat-file', 'blob', 'git-annex:trust.log'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[Level 11] CommandError: 'git -c diff.ignoreSubmodules=none -c core.quotepath=false cat-file blob git-annex:trust.log' failed with exitcode 128 [err: 'fatal: path 'trust.log' does not exist in 'git-annex'']
[DEBUG  ] Opening MultiplexSSHConnection(ctrl_path=<<PosixPath('/Us++48 chars++4e')>>, sshri=<<SSHRI(hostname++46 chars++ou')>>) by calling ['ssh', '-fN', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=15m', '-o', 'ControlPath=/Users/user/Library/Caches/datalad/sockets/64fec64e', 'user@server']
[DEBUG  ] Importing datalad.api to possibly discover possibly not yet bound method 'siblings'
[DEBUG  ] Building doc for <class 'datalad.core.local.create.Create'>
[DEBUG  ] Building doc for <class 'datalad.core.local.status.Status'>
[DEBUG  ] Building doc for <class 'datalad.core.local.save.Save'>
[DEBUG  ] Building doc for <class 'datalad.local.subdatasets.Subdatasets'>
[DEBUG  ] Building doc for <class 'datalad.distribution.get.Get'>
[DEBUG  ] Building doc for <class 'datalad.core.local.diff.Diff'>
[DEBUG  ] Building doc for <class 'datalad.core.distributed.push.Push'>
[DEBUG  ] Building doc for <class 'datalad.distribution.install.Install'>
[DEBUG  ] Building doc for <class 'datalad.local.unlock.Unlock'>
[DEBUG  ] Building doc for <class 'datalad.core.local.run.Run'>
[DEBUG  ] Building doc for <class 'datalad.distributed.create_sibling_github.CreateSiblingGithub'>
[DEBUG  ] Building doc for <class 'datalad.distribution.update.Update'>
[DEBUG  ] Building doc for <class 'datalad.distribution.siblings.Siblings'>
[DEBUG  ] Building doc for <class 'datalad.distributed.create_sibling_gitlab.CreateSiblingGitlab'>
[DEBUG  ] Building doc for <class 'datalad.distributed.create_sibling_gogs.CreateSiblingGogs'>
[DEBUG  ] Building doc for <class 'datalad.distributed.create_sibling_gin.CreateSiblingGin'>
[DEBUG  ] Building doc for <class 'datalad.distributed.create_sibling_gitea.CreateSiblingGitea'>
[DEBUG  ] Building doc for <class 'datalad.distribution.create_sibling.CreateSibling'>
[DEBUG  ] Building doc for <class 'datalad.distributed.drop.Drop'>
[DEBUG  ] Building doc for <class 'datalad.local.remove.Remove'>
[DEBUG  ] Building doc for <class 'datalad.local.addurls.Addurls'>
[DEBUG  ] Building doc for <class 'datalad.local.copy_file.CopyFile'>
[DEBUG  ] Building doc for <class 'datalad.local.download_url.DownloadURL'>
[DEBUG  ] Building doc for <class 'datalad.local.foreach_dataset.ForEachDataset'>
[DEBUG  ] Building doc for <class 'datalad.local.rerun.Rerun'>
[DEBUG  ] Building doc for <class 'datalad.local.run_procedure.RunProcedure'>
[DEBUG  ] Building doc for <class 'datalad.local.configuration.Configuration'>
[DEBUG  ] Building doc for <class 'datalad.local.wtf.WTF'>
[DEBUG  ] Building doc for <class 'datalad.local.clean.Clean'>
[DEBUG  ] Building doc for <class 'datalad.local.add_archive_content.AddArchiveContent'>
[DEBUG  ] Building doc for <class 'datalad.local.add_readme.AddReadme'>
[DEBUG  ] Building doc for <class 'datalad.local.export_archive.ExportArchive'>
[DEBUG  ] Building doc for <class 'datalad.distributed.export_archive_ora.ExportArchiveORA'>
[DEBUG  ] Building doc for <class 'datalad.distributed.export_to_figshare.ExportToFigshare'>
[DEBUG  ] Building doc for <class 'datalad.local.no_annex.NoAnnex'>
[DEBUG  ] Building doc for <class 'datalad.local.check_dates.CheckDates'>
[DEBUG  ] Building doc for <class 'datalad.distribution.uninstall.Uninstall'>
[DEBUG  ] Building doc for <class 'datalad.distribution.create_test_dataset.CreateTestDataset'>
[DEBUG  ] Building doc for <class 'datalad.support.sshrun.SSHRun'>
[DEBUG  ] Building doc for <class 'datalad.interface.shell_completion.ShellCompletion'>
[DEBUG  ] Processing entrypoints
[DEBUG  ] Loading entrypoint next from datalad.extensions
[DEBUG  ] Enable posting DataLad config overrides CLI/ENV as GIT_CONFIG items in process ENV
[DEBUG  ] Apply datalad-next patch to annexrepo.py:AnnexRepo.enable_remote
[DEBUG  ] Building doc for <class 'datalad_next.patches.configuration.Configuration'>
[DEBUG  ] Building doc for <class 'datalad.local.configuration.Configuration'>
[DEBUG  ] Apply datalad-next patch to create_sibling_ghlike.py:_GitHubLike._set_request_headers
[DEBUG  ] Apply datalad-next patch to interface.(utils|base).py:_execute_command_
[DEBUG  ] Apply patch to datalad.core.distributed.push._transfer_data
[DEBUG  ] Patching datalad.core.distributed.push.Push docstring and parameters
[DEBUG  ] Building doc for <class 'datalad.core.distributed.push.Push'>
[DEBUG  ] Patching datalad.support.AnnexRepo.get_export_records (new method)
[DEBUG  ] Apply patch to datalad.core.distributed.push._push
[DEBUG  ] Apply patch to datalad.distribution.siblings._enable_remote
[DEBUG  ] Retrofit `SpecialRemote` with a `close()` handler
[DEBUG  ] Replace special remote _main() with datalad-next's progress logging enabled variant
[DEBUG  ] Apply patch to datalad.distributed.create_sibling_gitlab._proc_dataset
[DEBUG  ] Stop advertising discontinued "hierarchy" layout for `create_siblign_gitlab()`
[DEBUG  ] Building doc for <class 'datalad.distributed.create_sibling_gitlab.CreateSiblingGitlab'>
[DEBUG  ] Apply patch to datalad.distributed.ora_remote.url2transport_path
[DEBUG  ] Apply patch to datalad.distributed.ora_remote.url2transport_path
[DEBUG  ] Apply patch to datalad.distributed.ora_remote.SSHRemoteIO
[DEBUG  ] Apply patch to datalad.distributed.ora_remote.close
[DEBUG  ] Apply patch to datalad.customremotes.ria_utils.create_store
[DEBUG  ] Apply patch to datalad.customremotes.ria_utils.create_ds_in_store
[DEBUG  ] Apply patch to datalad.customremotes.ria_utils._ensure_version
[DEBUG  ] Apply patch to datalad.distributed.ora_remote.ORARemote
[DEBUG  ] Building doc for <class 'datalad_next.patches.replace_create_sibling_ria.CreateSiblingRia'>
[DEBUG  ] Apply patch to datalad.distributed.create_sibling_ria.CreateSiblingRia
[DEBUG  ] Apply patch to datalad.distribution.dataset.create_sibling_ria
[DEBUG  ] Apply patch to datalad.core.local.run.format_command
[DEBUG  ] Apply patch to datalad.distribution.update._choose_update_target
[DEBUG  ] Apply patch to datalad.support.sshconnector._exec_ssh
[DEBUG  ] Apply patch to datalad.support.sshconnector.get
[DEBUG  ] Apply patch to datalad.support.sshconnector.put
[DEBUG  ] Building doc for <class 'datalad_next.commands.credentials.Credentials'>
[DEBUG  ] Building doc for <class 'datalad_next.commands.create_sibling_webdav.CreateSiblingWebDAV'>
[DEBUG  ] Building doc for <class 'datalad_next.commands.tree.TreeCommand'>
[DEBUG  ] Building doc for <class 'datalad_next.commands.download.Download'>
[DEBUG  ] Building doc for <class 'datalad_next.commands.ls_file_collection.LsFileCollection'>
[DEBUG  ] Building doc for <class 'datalad_next.commands.status.Status'>
[DEBUG  ] Loaded entrypoint next from datalad.extensions
[DEBUG  ] Done processing entrypoints
[DEBUG  ] Determined class of decorated function: <class 'datalad.distribution.siblings.Siblings'>
[DEBUG  ] Command parameter validation skipped. <class 'datalad.distribution.siblings.Siblings'> declares no validator
[DEBUG  ] Resolved dataset to configure sibling: /Users/user/temp/sandbox/mydataset01
[DEBUG  ] Run ['git', 'annex', 'version', '--raw'] (protocol_class=StdOutErrCapture) (cwd=None)
[DEBUG  ] Finished ['git', 'annex', 'version', '--raw'] with status 0
[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'info', '--bytes', '--fast', '--json', '--json-error-messages', '-c', 'annex.dotfiles=true'] (protocol_class=AnnexJsonProtocol) (cwd=/Users/user/temp/sandbox/mydataset01)
[DEBUG  ] Finished ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'info', '--bytes', '--fast', '--json', '--json-error-messages', '-c', 'annex.dotfiles=true'] with status 0
[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'cat-file', 'blob', 'git-annex:remote.log'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[Level 11] CommandError: 'git -c diff.ignoreSubmodules=none -c core.quotepath=false cat-file blob git-annex:remote.log' failed with exitcode 128 [err: 'fatal: path 'remote.log' does not exist in 'git-annex'']
[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'cat-file', 'blob', 'git-annex:trust.log'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[Level 11] CommandError: 'git -c diff.ignoreSubmodules=none -c core.quotepath=false cat-file blob git-annex:trust.log' failed with exitcode 128 [err: 'fatal: path 'trust.log' does not exist in 'git-annex'']
[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'wanted', '.', '-c', 'annex.dotfiles=true'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'required', '.', '-c', 'annex.dotfiles=true'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'group', '.', '-c', 'annex.dotfiles=true'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'groupwanted', 'here', '-c', 'annex.dotfiles=true'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[DEBUG  ] MultiplexSSHConnection(ctrl_path=<<PosixPath('/Us++48 chars++4e')>>, sshri=<<SSHRI(hostname++46 chars++ou')>>) is used to run ['ssh', '-o', 'ControlPath=/Users/user/Library/Caches/datalad/sockets/64fec64e', 'user@server', '[ -e /silver/temp/data_store/702/811de-9f93-48f3-9e59-1487a32dfbc5/config ]']
[DEBUG  ] Run ['ssh', '-o', 'ControlPath=/Users/user/Library/Caches/datalad/sockets/64fec64e', 'user@server', '[ -e /silver/temp/data_store/702/811de-9f93-48f3-9e59-1487a32dfbc5/config ]'] (protocol_class=StdOutErrCapture) (cwd=None)
[DEBUG  ] Finished ['ssh', '-o', 'ControlPath=/Users/user/Library/Caches/datalad/sockets/64fec64e', 'user@server', '[ -e /silver/temp/data_store/702/811de-9f93-48f3-9e59-1487a32dfbc5/config ]'] with status 1
[INFO   ] create siblings 'silverTemp' and 'silverTemp-storage' ...
[ERROR  ] 'SSHRemoteIO' object has no attribute 'url2transport_path'
[DEBUG  ] Closing 1 SSH connections...
[DEBUG  ] Closing MultiplexSSHConnection(ctrl_path=<<PosixPath('/Us++48 chars++4e')>>, sshri=<<SSHRI(hostname++46 chars++ou')>>) by calling ['ssh', '-O', 'stop', '-o', 'ControlPath=/Users/user/Library/Caches/datalad/sockets/64fec64e', 'user@server']
[DEBUG  ] Run ['ssh', '-O', 'stop', '-o', 'ControlPath=/Users/user/Library/Caches/datalad/sockets/64fec64e', 'user@server'] (protocol_class=StdOutErrCapture) (cwd=None)
[DEBUG  ] Finished ['ssh', '-O', 'stop', '-o', 'ControlPath=/Users/user/Library/Caches/datalad/sockets/64fec64e', 'user@server'] with status 0
0.00 [00:01, ?/s]%

[christian-monch]

Thanks @kevinCassou for the info. It seems like the problem is that the "Building doc"-steps are executed too early, i.e. before SSHRemoteIO is patched. I am not sure why this is the case yet (I was unable to reproduce the error in a Conda environment on Linux-x86 or an Intel Mac). The problem might be related to an extension that is installed on your site.

I will have to go through the starting process. It would be helpful if you could provide a Traceback of the error by executing datalad with --dbg. I know this "remote debugging" is tedious, but it is quite helpful until I can recreate the error locally.

Another useful source of information would be the output of datalad wtf (should tell us about installed extensions) and git config -l|grep datalad|grep -v credential|grep -v identityfile in the dataset mydataset01.

[kevinCassou]

Hi @christian-monch

Here the result of datalad wtf

datalad wtf
# WTF
## configuration <SENSITIVE, report disabled by configuration>
## credentials
  - keyring:
    - active_backends:
      - macOS Keyring
      - PlaintextKeyring with no encyption v.1.0 at /Users/user/.local/share/python_keyring/keyring_pass.cfg
    - config_file: /Users/user/.config/python_keyring/keyringrc.cfg
    - data_root: /Users/user/.local/share/python_keyring
## datalad
  - version: 1.1.3
## dataset
  - branches:
    - git-annex@0b4815d
    - master@170d1e3
  - id: 702811de-9f93-48f3-9e59-1487a32dfbc5
  - path: /Users/user/temp/sandbox/mydataset01
  - repo: AnnexRepo
## dependencies
  - annexremote: 1.2.1
  - boto3: 1.35.13
  - cmd:7z: 16.02
  - cmd:annex: 10.20240831
  - cmd:bundled-git: UNKNOWN
  - cmd:git: 2.46.0
  - cmd:ssh: 9.0p1
  - cmd:system-git: 2.46.0
  - cmd:system-ssh: 9.0p1
  - humanize: 4.10.0
  - iso8601: 2.1.0
  - keyring: 25.3.0
  - keyrings.alt: 5.0.1
  - msgpack: 1.0.8
  - platformdirs: 4.2.2
  - requests: 2.32.3
## environment
  - LANG: en_US.UTF-8
  - LANGUAGE: en_US
  - LC_ALL: en_US.UTF-8
  - LC_CTYPE: UTF-8
  - PATH: /opt/homebrew/opt/imagemagick@6/bin:/opt/miniconda3/envs/py39/dltest/bin:/Users/user/mambaforge/condabin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/Library/TeX/texbin:/opt/X11/bin:/Applications/quarto/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin
  - PYTHONEXE: python3
## extensions
## git-annex
  - build flags:
    - Assistant
    - Webapp
    - Pairing
    - FsEvents
    - TorrentParser
    - MagicMime
    - Servant
    - Benchmark
    - Feeds
    - Testsuite
    - S3
    - WebDAV
  - dependency versions:
    - aws-0.24.2
    - bloomfilter-2.0.1.2
    - crypton-1.0.0
    - DAV-1.3.4
    - feed-1.3.2.1
    - ghc-9.8.2
    - http-client-0.7.17
    - persistent-sqlite-2.13.3.0
    - torrent-10000.1.3
    - uuid-1.3.16
    - yesod-1.6.2.1
  - key/value backends:
    - SHA256E
    - SHA256
    - SHA512E
    - SHA512
    - SHA224E
    - SHA224
    - SHA384E
    - SHA384
    - SHA3_256E
    - SHA3_256
    - SHA3_512E
    - SHA3_512
    - SHA3_224E
    - SHA3_224
    - SHA3_384E
    - SHA3_384
    - SKEIN256E
    - SKEIN256
    - SKEIN512E
    - SKEIN512
    - BLAKE2B256E
    - BLAKE2B256
    - BLAKE2B512E
    - BLAKE2B512
    - BLAKE2B160E
    - BLAKE2B160
    - BLAKE2B224E
    - BLAKE2B224
    - BLAKE2B384E
    - BLAKE2B384
    - BLAKE2BP512E
    - BLAKE2BP512
    - BLAKE2S256E
    - BLAKE2S256
    - BLAKE2S160E
    - BLAKE2S160
    - BLAKE2S224E
    - BLAKE2S224
    - BLAKE2SP256E
    - BLAKE2SP256
    - BLAKE2SP224E
    - BLAKE2SP224
    - SHA1E
    - SHA1
    - MD5E
    - MD5
    - WORM
    - URL
    - GITBUNDLE
    - GITMANIFEST
    - VURL
    - X*
  - local repository version: 10
  - operating system: darwin aarch64
  - remote types:
    - git
    - gcrypt
    - p2p
    - S3
    - bup
    - directory
    - rsync
    - web
    - bittorrent
    - webdav
    - adb
    - tahoe
    - glacier
    - ddar
    - git-lfs
    - httpalso
    - borg
    - rclone
    - hook
    - external
  - supported repository versions:
    - 8
    - 9
    - 10
  - upgrade supported from repository versions:
    - 0
    - 1
    - 2
    - 3
    - 4
    - 5
    - 6
    - 7
    - 8
    - 9
    - 10
  - version: 10.20240831
## location
  - path: /Users/user/temp/sandbox/mydataset01
  - type: dataset
## metadata.extractors
## metadata.filters
## metadata.indexers
## python
  - implementation: CPython
  - version: 3.10.14
## system
  - distribution: darwin/22.6.0 13.6.9/arm64
  - encoding:
    - default: utf-8
    - filesystem: utf-8
    - locale.prefered: UTF-8
  - filesystem:
    - CWD:
      - mount_opts: ro,local,rootfs,dovolfs,journaled,multilabel
      - path: /Users/user/temp/sandbox/mydataset01
      - type: apfs
    - HOME:
      - mount_opts: ro,local,rootfs,dovolfs,journaled,multilabel
      - path: /Users/user
      - type: apfs
    - TMP:
      - mount_opts: ro,local,rootfs,dovolfs,journaled,multilabel
      - path: /var/folders/6r/wvdg2x2s6szd3tqk7kx441f80000gn/T
      - type: apfs
  - max_path_length: 294
  - name: Darwin
  - release: 22.6.0
  - type: posix
  - version: Darwin Kernel Version 22.6.0: Mon Jun 24 01:22:14 PDT 2024; root:xnu-8796.141.3.706.2~1/RELEASE_ARM64_T6000

and git config -l|grep datalad|grep -v credential|grep -v identityfile is giving

remote.silverTemp.datalad-publish-depends=silverTemp-storage
remote.origin.url=git@gitlab.in2p3.fr:user/dataladtest01.git

[kevinCassou]

then I did

datalad --dbg create-sibling-ria -s silverTemp --new-store-ok ria+ssh://user@server/silver/temp/data_store/

create-sibling-ria(error): /Users/user/temp/sandbox/mydataset01 (sibling) [a sibling 'silverTemp' is already configured in dataset '/Users/user/temp/sandbox/mydataset01']
create-sibling-ria(error): /Users/user/temp/sandbox/mydataset01 (sibling) [a sibling 'silverTemp-storage' is already configured in dataset '/Users/user/temp/sandbox/mydataset01']
action summary:
  create-sibling-ria (error: 2)
Traceback (most recent call last):
  File "/opt/miniconda3/envs/py39/dltest/bin/datalad", line 11, in <module>
    sys.exit(main())
  File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/cli/main.py", line 143, in main
    _run(cmdlineargs)
  File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/cli/main.py", line 167, in _run
    ret = _run_with_debugger(namespace) \
  File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/cli/main.py", line 181, in _run_with_debugger
    return cmdlineargs.func(cmdlineargs)
  File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/cli/exec.py", line 107, in call_from_parser
    ret = list(ret)
  File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/interface/base.py", line 940, in _execute_command_
    raise IncompleteResultsError(
datalad.support.exceptions.IncompleteResultsError: Command did not complete successfully. 2 failed:
[{'action': 'create-sibling-ria',
  'message': ('a sibling %r is already configured in dataset %r',
              'silverTemp',
              '/Users/user/temp/sandbox/mydataset01'),
  'name': 'silverTemp',
  'path': '/Users/user/temp/sandbox/mydataset01',
  'status': 'error',
  'type': 'sibling'},
 {'action': 'create-sibling-ria',
  'message': ('a sibling %r is already configured in dataset %r',
              'silverTemp-storage',
              '/Users/user/temp/sandbox/mydataset01'),
  'name': 'silverTemp-storage',
  'path': '/Users/user/temp/sandbox/mydataset01',
  'status': 'error',
  'type': 'sibling'}]

> /opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/interface/base.py(940)_execute_command_()
-> raise IncompleteResultsError(
(Pdb)

terminal scratch. I guess it's because siblings is already set ... ?

[christian-monch]

[...]
terminal scratch. I guess it's because siblings is already set ... ?

Yes, it would be good to call datalad create-sibling-ria with a different name, e.g. -s silverTemp2.

[christian-monch]

[...]

datalad wtf
...
## extensions
## git-annex
...

...

Thanks for the info. A quick first question: I am wondering why datalad-next does not show up in "extensions"? Does a pip install --upgrade datalad-next make a difference there?

[kevinCassou]

ok, I did pip install --upgrade datalad-next , and in fact, now we have the following:

datalad wtf 
...
## extensions
  - next:
    - description: What is next in DataLad
    - entrypoints:
      - datalad_next.commands.create_sibling_webdav.CreateSiblingWebDAV:
        - class: CreateSiblingWebDAV
        - module: datalad_next.commands.create_sibling_webdav
        - names:
          - create-sibling-webdav
      - datalad_next.commands.credentials.Credentials:
        - class: Credentials
        - module: datalad_next.commands.credentials
        - names:
      - datalad_next.commands.download.Download:
        - class: Download
        - module: datalad_next.commands.download
        - names:
          - download
      - datalad_next.commands.ls_file_collection.LsFileCollection:
        - class: LsFileCollection
        - module: datalad_next.commands.ls_file_collection
        - names:
          - ls-file-collection
      - datalad_next.commands.status.Status:
        - class: Status
        - module: datalad_next.commands.status
        - names:
          - next-status
          - next_status
      - datalad_next.commands.tree.TreeCommand:
        - class: TreeCommand
        - module: datalad_next.commands.tree
        - names:
          - tree
    - module: datalad_next
    - version: 1.5.0
## git-annex
...

but then

datalad --dbg create-sibling-ria -s silverTemp4 --new-store-ok ria+ssh://user@ssh-ubuntu3.ijclab.in2p3.fr/silver/temp/data_kev/data_store04/
[INFO   ] create siblings 'silverTemp4' and 'silverTemp4-storage' ...
0.00 [00:01, ?/s]Traceback (most recent call last):
  File "/opt/miniconda3/envs/py39/dltest/bin/datalad", line 11, in <module>
    sys.exit(main())
  File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/cli/main.py", line 143, in main
    _run(cmdlineargs)
  File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/cli/main.py", line 167, in _run
    ret = _run_with_debugger(namespace) \
  File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/cli/main.py", line 181, in _run_with_debugger
    return cmdlineargs.func(cmdlineargs)
  File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/cli/exec.py", line 107, in call_from_parser
    ret = list(ret)
  File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/interface/base.py", line 873, in _execute_command_
    for r in _process_results(
  File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/interface/utils.py", line 319, in _process_results
    for res in results:
  File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/distributed/create_sibling_ria.py", line 430, in __call__
    yield from _create_sibling_ria(
  File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/distributed/create_sibling_ria.py", line 601, in _create_sibling_ria
    create_ds_in_store(SSHRemoteIO(ssh_host) if ssh_host else LocalIO(),
  File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/customremotes/ria_utils.py", line 227, in create_ds_in_store
    _ensure_version(io, dsgit_dir, obj_version)
  File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad_next/patches/ria_utils.py", line 52, in ria_utils__ensure_version
    version_file = io.url2transport_path(base_path / 'ria-layout-version')
AttributeError: 'SSHRemoteIO' object has no attribute 'url2transport_path'

> /opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad_next/patches/ria_utils.py(52)ria_utils__ensure_version()
-> version_file = io.url2transport_path(base_path / 'ria-layout-version')
(Pdb)

terminal scratch again. On the remote storage, the directory error-logs and the file ria-layout-version are created...

Hoping it helps

[christian-monch]

Thanks for all the info @kevinCassou. I found it, and I should have seen the problem earlier.

Please set the git config variable datalad.extensions.load to next, for example with the command:

git config --global --add datalad.extensions.load next

That should fix it.

[kevinCassou]

Yes I confirm ! I'm still a newbie in datalad and testing ! 🔰

Thanks for help 🙏🏼

[christian-monch]

Yes I confirm ! I'm still a newbie in datalad and testing ! 🔰

Thanks for help 🙏🏼

you are very welcome.