Testing of PR #2944 - Ensure no mtime database corruption regression due to database performance tweaks

[abraunegg]

Hi @IsmaelSF and @phlibi,

Since you’ve both encountered the database corruption issue with mtime values, I wanted to reach out to see if either of you would be able to test PR #2944 given that I am 100% unable to reproduce this issue despite extensive attempts.

This PR aims to address some application performance issues by adding more frequent database checkpoints in a way that minimises impact on system I/O performance. The goal is to make database management more resilient while maintaining efficiency.

If either (or both) of you, or anyone else who is interested, could try out the changes in this PR, it would be incredibly helpful to either validate no 'mtime' corruption regression and/or confirm the performance boost this PR provides. Your feedback and any insights from testing would be greatly appreciated!

To test this PR, please look at the steps below:

Install dependencies

First install all the required platform dependencies to build the client on your respective platforms. Please read https://github.com/abraunegg/onedrive/blob/master/docs/install.md#building-from-source---high-level-requirements and then follow correctly for your platform.

Script to build PR

Once this is done, to clone the PR to resolve your issue, you can use a script like the following:

#!/bin/bash

PR=2944

rm -rf ./onedrive-pr${PR}
git clone https://github.com/abraunegg/onedrive.git onedrive-pr${PR}
cd onedrive-pr${PR}
git fetch origin pull/${PR}/head:pr${PR}
git checkout pr${PR}

# Configure and Build
./configure --enable-debug --enable-notifications; make clean; make;
./onedrive --version

This script will create a local folder called onedrive-pr2943 with the PR version. You may need to augment the script to activate/deactivate the DMD|LDC compiler.

Testing the PR

To run the PR, you need to run the client from the PR build directory:

./onedrive <any other options needed>

To install the PR, you will need to perform sudo make install to install the PR version to your system.

When running the PR, your version should be: onedrive v2.5.2-43-gc8d26e7 or greater.

Testing Evidence

• Lenovo S130 Laptop running Fedora 40
  • Intel(R) Celeron(R) N4000
  • 4 GB LPDDR4 RAM
• Microsoft OneDrive Business Account with 100K + files and folders

Current Release v2.5.2

real    22m33.208s
user    12m45.654s
sys     2m8.464s

Current 'master'

real    19m44.729s
user    10m31.648s
sys     1m37.970s

PR #2944

real    13m51.193s
user    8m56.121s
sys     0m49.736s

Thank you so much in advance for considering this request, and please let me know if you have any questions or need assistance in setting up the test.

[IsmaelSF]

No problem, I'll setup the testbed today and let you know my findings.

[phlibi]

@abraunegg Did some tests.

First to ensure that I can still reproduce the original issue with an older version:

$ git checkout release-2.5.0
... # Build
$ ./onedrive --version
onedrive v2.5.0-rc3-74-ga9c5939
$ time ./onedrive --sync --confdir=~/.config/onedrive/pruefprotokolle
...
core.time.TimeException@/home/phip/dlang/dmd-2.109.1/linux/bin64/../../src/phobos/std/datetime/systime.d(9112): Invalid ISO Extended String: ��o}�
----------------
/home/phip/dlang/dmd-2.109.1/linux/bin64/../../src/phobos/std/exception.d:460 pure @safe bool std.exception.enforce!(bool).enforce(bool, lazy object.Throwable) [0x5571b6b4d684]
/home/phip/dlang/dmd-2.109.1/linux/bin64/../../src/phobos/std/datetime/systime.d:9112 @safe std.datetime.systime.SysTime std.datetime.systime.SysTime.fromISOExtString!(const(char)[]).fromISOExtString(scope const(char[]), immutable(std.datetime.timezone.TimeZone)) [0x5571b6c02926]
src/itemdb.d:694 itemdb.Item itemdb.ItemDatabase.buildItem(sqlite.Statement.Result) [0x5571b6cb403a]
src/itemdb.d:470 itemdb.Item[] itemdb.ItemDatabase.selectChildren(const(char)[], const(char)[]) [0x5571b6cb21a7]
src/sync.d:3370 void syncEngine.SyncEngine.checkDirectoryDatabaseItemForConsistency(itemdb.Item, immutable(char)[]) [0x5571b6c8abdf]
src/sync.d:3216 void syncEngine.SyncEngine.checkDatabaseItemForConsistency(itemdb.Item) [0x5571b6c89b45]
src/sync.d:3372 void syncEngine.SyncEngine.checkDirectoryDatabaseItemForConsistency(itemdb.Item, immutable(char)[]) [0x5571b6c8ac62]
src/sync.d:3216 void syncEngine.SyncEngine.checkDatabaseItemForConsistency(itemdb.Item) [0x5571b6c89b45]
src/sync.d:3131 void syncEngine.SyncEngine.performDatabaseConsistencyAndIntegrityCheck() [0x5571b6c89656]
src/main.d:1228 void main.performStandardSyncProcess(immutable(char)[], monitor.Monitor) [0x5571b6b343de]
src/main.d:777 _Dmain [0x5571b6b3146d]

real    14m9.470s
user    10m41.258s
sys     0m26.051s

Then with the current software for reference:

$ git checkout release-2.5.2
... # Build
$ ./onedrive --version
onedrive v2.5.1-5-g686c2a3
$ time ./onedrive --sync --confdir=~/.config/onedrive/pruefprotokolle
...
Sync with Microsoft OneDrive is complete
Attempting to perform a database vacuum to optimise database
Database vacuum is complete

real    14m3.985s
user    11m56.411s
sys     0m29.541s

And finally with the PR:

$ git checkout pr2944
... # Build
$ ./onedrive --version
onedrive v2.5.2-43-gc8d26e7
$ time ./onedrive --sync --confdir=~/.config/onedrive/pruefprotokolle
...
Sync with Microsoft OneDrive is complete
Attempting to perform a database vacuum to optimise database
Database vacuum is complete

real    14m21.137s
user    9m36.088s
sys     0m27.100s

And again, just to be sure:

$ ./onedrive --version
onedrive v2.5.2-43-gc8d26e7
$ time ./onedrive --sync --confdir=~/.config/onedrive/pruefprotokolle
...
Sync with Microsoft OneDrive is complete
Attempting to perform a database vacuum to optimise database
Database vacuum is complete

real    15m53.996s
user    10m7.452s
sys     0m24.879s

PR seems to be fine, at least in regards to any regressions of this issue :)
When it comes to the performance, I don't see a substantial change; however, most of the time is spent waiting after throttling effects of the server, so I doubt that there is very much to gain here for us.

[IsmaelSF]

Config used in the test environment:

sync_dir="/OneDrive/LinuxServerBackup/"
upload_only = "true"
debug_https = "false"
enable_logging = "true"
ip_protocol_version = "1"
force_http_11 = "true"

[root@anakin onedrive-pr2944]# ./onedrive --version
onedrive v2.5.2-43-gc8d26e7

[root@anakin onedrive-pr2944]# time ./onedrive --sync

[2024-Nov-04 11:41:23.3108378] #######################################################################################################################################
[2024-Nov-04 12:59:14.5102526] Deleting the saved application sync status ...
[2024-Nov-04 12:59:14.5103345] Forcing client to use IPv4 connections only
[2024-Nov-04 12:59:14.5199694] Attempting to contact Microsoft OneDrive Login Service
[2024-Nov-04 12:59:14.8415583] Successfully reached Microsoft OneDrive Login Service
[2024-Nov-04 12:59:15.1918650] Configuring Global Azure AD Endpoints
[2024-Nov-04 12:59:18.1970711] Performing a database consistency and integrity check on locally stored data
[2024-Nov-04 12:59:18.5916839] Scanning the local file system '/OneDrive/LinuxServerBackup' for new data to upload
[2024-Nov-04 13:01:34.3244167] New directories to create on Microsoft OneDrive: 33196
[2024-Nov-04 16:55:21.9573845] New items to upload to Microsoft OneDrive: 378240


Due to a termination signal, internal processing stopped abruptly. The application will now exit in a unclean manner.
Due to a termination signal, internal processing stopped abruptly. The application will now exit in a unclean manner.

real    262m45.661s
user    37m16.244s
sys     1m55.333s

After resyncing more than 140.000 items process exits abruptly (it seems related to curl signal managing that your are talking in other topics) but no corruption in mtime field database where found.

My thoughts are the same, this PR works really well nailing down mtime corruption issue. On the performance side, same topic, too much throttling from Graph API call to get some real measures.

[IsmaelSF]

What is the curl version you are using when you hit this ?

curl 8.9.1 (x86_64-pc-linux-gnu) libcurl/8.9.1 OpenSSL/1.0.2k-fips zlib/1.2.7 libpsl/0.7.0 (+libicu/50.1.2)
Release-Date: 2024-07-31
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS HSTS HTTPS-proxy IPv6 Largefile libz NTLM PSL SSL threadsafe UnixSockets

What is the exit code when this occurs ? Please type echo $? right after it exits like this

* Connection #0 to host graph.microsoft.com left intact
Due to a termination signal, internal processing stopped abruptly. The application will now exit in a unclean manner.
[root@anakin onedrive-pr2944]#
[root@anakin onedrive-pr2944]# echo $?
130

What was your test platform ?

[root@anakin onedrive-pr2944]# cat /etc/redhat-release
Rocky Linux release 8.10 (Green Obsidian)

I'm afraid that last lines of debug-https doesnt show any relevant information:

* Request completely sent off
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 / [blank] / UNDEF
* ALPN: server accepted http/1.1                                                                                                                                      
* Server certificate:                                                                                                                                                 
*  subject: C=US; ST=Washington; L=Redmond; O=Microsoft Corporation; CN=graph.microsoft.com                                                                           *  start date: Oct 19 00:00:00 2024 GMT                                                                                                                               
*  expire date: Apr 19 23:59:59 2025 GMT                                                                                                                              
*  subjectAltName: host "graph.microsoft.com" matched cert's "graph.microsoft.com"                                                                                    *  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA                                                                                                   
*  *  SSL certificate verify ok.                                                                                                                                         
* using HTTP/1.x                                                                                                                                                      
* > GET /v1.0/drives/4045D2C97D06D165/root:/.%2Fanakin%2Fhome%2Fdocumentos%2FEmpresa%20MODELOS%20y%20PLANTILLAS%2FEquipos%20OFICINA%20ordenadores%2C%20impresoras%2FProveedores%2FPC%20COMPONENTES%20Murcia%2FPC%20COMPONENTES%20Factura_impresora_20160115.pdf:?select=id,name,eTag,cTag,deleted,file,folder,root,fileSystemInfo,remoteItem,parentReference,size HTTP/1.1
Host: graph.microsoft.com
User-Agent: ISV|abraunegg|OneDrive Client for Linux/v2.5.2-43-gc8d26e7
Accept: */*
Authorization: bearer XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

* Request completely sent off
Due to a termination signal, internal processing stopped abruptly. The application will now exit in a unclean manner.
< HTTP/1.1 200 OK
< Cache-Control: no-cache
< Transfer-Encoding: chunked
< Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
< ETag: "{D175EE89-9976-45B6-9398-384B551D76FD},2"
< Strict-Transport-Security: max-age=31536000
< request-id: 6418a946-e6ae-4ecc-95b0-310b442c6338
< client-request-id: 6418a946-e6ae-4ecc-95b0-310b442c6338
< x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"France Central","Slice":"E","Ring":"5","ScaleUnit":"000","RoleInstance":"PA1PEPF0006B21C"}}
< OData-Version: 4.0
< Date: Tue, 05 Nov 2024 20:33:07 GMT
<
* Connection #0 to host graph.microsoft.com left intact
Due to a termination signal, internal processing stopped abruptly. The application will now exit in a unclean manner.
< HTTP/1.1 200 OK
< Cache-Control: no-cache
< Transfer-Encoding: chunked
< Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
< ETag: "{EC783F68-0030-4F2A-B388-F33335A02FFD},2"
< Strict-Transport-Security: max-age=31536000
< request-id: 0ffa5eee-6280-4119-a1a0-111341601b57
< client-request-id: 0ffa5eee-6280-4119-a1a0-111341601b57
< x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"France Central","Slice":"E","Ring":"5","ScaleUnit":"000","RoleInstance":"PA1PEPF000A962B"}}
< OData-Version: 4.0
< Date: Tue, 05 Nov 2024 20:33:07 GMT
<
* Connection #0 to host graph.microsoft.com left intact                        
Due to a termination signal, internal processing stopped abruptly. The application will now exit in a unclean manner.

[abraunegg]

OK .. so its the same issue rearing its ugly head where an upstream device is killing the active TCP connection despite keep alives being set and then curl issues a SIGPIPE due to it being closed upstream in an unclean manner (curl bug)

I will have to dig into this some more when I have some free time on Friday

[abraunegg]

@IsmaelSF
To assist with debugging this issue further, please can you run the following python script on your system:

import socket
import time
import sys

def keep_connection_open(host, port):
    # Create a TCP socket
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    try:
        # Connect to the server
        print(f"Connecting to {host}:{port}...")
        sock.connect((host, port))
        print(f"Connected to {host}:{port}")

        # Record the start time
        start_time = time.time()

        # Keep the connection open and idle
        print("Keeping the connection open and idle...")
        while True:
            try:
                # Check if the connection is still open by sending a zero-length packet
                sock.send(b'')
            except socket.error:
                # Connection was closed
                break

            # Sleep for a bit to avoid busy-looping
            time.sleep(1)

        # Record the end time when the connection is closed
        end_time = time.time()
        duration = end_time - start_time

        print(f"Connection closed. It was open for {duration:.2f} seconds.")

    except socket.error as e:
        print(f"Error: {e}")
    finally:
        sock.close()

if __name__ == "__main__":
    if len(sys.argv) != 3:
        print(f"Usage: {sys.argv[0]} <host> <port>")
        sys.exit(1)

    host = sys.argv[1]
    port = int(sys.argv[2])

    keep_connection_open(host, port)

Please save this file as whatever you wish, or for example: test_connection_age.py

When running this python script, please do the following:

./test_connection_age.py graph.microsoft.com 443

The output, once complete will be similar to the following:

[alex@onedrive-client-dev ~]$ python3 ./test_connection_age.py graph.microsoft.com 443    
Connecting to graph.microsoft.com:443...
Connected to graph.microsoft.com:443
Keeping the connection open and idle...
Connection closed. It was open for 123.12 seconds.
[alex@onedrive-client-dev ~]$ 

Please can you post your results.

[IsmaelSF]

Output result:

[root@anakin ~]# ./test_connection_age.py graph.microsoft.com 443
Connecting to graph.microsoft.com:443...
Connected to graph.microsoft.com:443
Keeping the connection open and idle...
Connection closed. It was open for 127.13 seconds.
[root@anakin ~]# ./test_connection_age.py graph.microsoft.com 443
Connecting to graph.microsoft.com:443...
Connected to graph.microsoft.com:443
Keeping the connection open and idle...
Connection closed. It was open for 125.12 seconds.
[root@anakin ~]# ./test_connection_age.py graph.microsoft.com 443
Connecting to graph.microsoft.com:443...
Connected to graph.microsoft.com:443
Keeping the connection open and idle...
Connection closed. It was open for 124.12 seconds.
[root@anakin ~]# ./test_connection_age.py graph.microsoft.com 443
Connecting to graph.microsoft.com:443...
Connected to graph.microsoft.com:443
Keeping the connection open and idle...
Connection closed. It was open for 128.12 seconds.

[abraunegg]

OK .. so similar results to what I am getting as to curl / tcp connection handling and closing.

So .. something is signalling a SIGINT (CTRL+C) in your situation (given the 130 exit code) .... if you are not issuing this from your keyboard - something is making curl issue this despite handling for SIGINT signal handlers ....

So either this is a 'curl' bug or something else.

Given you are using HTTP/1.1 and IPv4 ... that should be well bedded down ... and there should not be a bug in curl with that combination .......

Now .. Rocky Linux 8.10 has the following 'curl' versions:

and you are running curl 'curl 8.9.1' .... did you self compile this?

If you did self compile, can you either:

• downgrade to the official Rocky Linux curl version
• upgrade to curl version 8.10.1 (this has been tested by other) or to curl version 8.11.0 .. however this has not been tested and was released today ...

Something is triggering a SIGINT ... and you are not getting this log message:

CurlEngine idle for > %d seconds .... destroying and returning a new curl engine instance

which means the 'idle' time trigger is not being hit or triggered ... so this is unlikely to be the cause of the SIGINT being triggered .....

How much system memory do you have?

Is the system process being killed by an over-aggressive OOM killer ?

An OOM killer is a process that the linux kernel employs when the system is critically low on memory. This situation occurs because the linux kernel has over allocated memory to its processes. When a process starts it requests a block of memory from the kernel. It could issue a SIGINT (should issue a SIGKILL however ...) .....

[abraunegg]

@IsmaelSF
Any update here ?

[IsmaelSF]

I have discarded memory shortage or OOM, but I would like to identify first which process is sending that signal (running process with strace -p PID -e 'trace=!all) before trying to update or downgrade curl library (I reviewed all libcurl issues and nothing point to this kind of problems). I will come back to you in the coming hours with any finding.

[IsmaelSF]

This is the end of the log of strace signals when I reproduce the issue. Main PID is 18109:

18130 --- SIGUSR2 {si_signo=SIGUSR2, si_code=SI_TKILL, si_pid=18109, si_uid=0} ---
18126 --- SIGUSR2 {si_signo=SIGUSR2, si_code=SI_TKILL, si_pid=18109, si_uid=0} ---
18129 --- SIGUSR2 {si_signo=SIGUSR2, si_code=SI_TKILL, si_pid=18109, si_uid=0} ---
18128 --- SIGUSR2 {si_signo=SIGUSR2, si_code=SI_TKILL, si_pid=18109, si_uid=0} ---
18125 --- SIGUSR2 {si_signo=SIGUSR2, si_code=SI_TKILL, si_pid=18109, si_uid=0} ---
18110 --- SIGUSR2 {si_signo=SIGUSR2, si_code=SI_TKILL, si_pid=18109, si_uid=0} ---
18109 --- SIGUSR2 {si_signo=SIGUSR2, si_code=SI_TKILL, si_pid=18109, si_uid=0} ---
18109 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_ACCERR, si_addr=0x7f28caa8a000} ---
18132 +++ exited with 0 +++
18127 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} ---
18126 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} ---
18129 +++ exited with 130 +++
18130 +++ exited with 130 +++
18126 +++ exited with 130 +++
18125 +++ exited with 130 +++
18122 +++ exited with 130 +++

My point of view, signal is generated in the process because of thread accesing mmap protected memory. Has to be a corner case it took me 3 hours of execution to be able to reproduce it.

[abraunegg]

Was there a core dump created for PID 18109 ?

If yes, can you run that through gdb - that might provide some further insights as where to look in the code for a race condition or similar.

Analysing the changelog for curl versions 8.9.1 through 8.11.0, I can identify several memory-related issues that were addressed during this period. These fixes may be relevant to the segmentation faults observed in the strace output:

Identified Memory-Related Fixes:

1. Memory Leak in Curl_close:

   • Description: A memory leak was identified in the Curl_close function, where Curl_resolver_cancel was not called, leading to memory not being freed properly.
   • Fixed in Version: curl 8.10.0.
   • Details: This fix ensures that memory is properly released when closing connections.

2. Memory Leak in Curl_http2_done:

   • Description: A memory leak was identified in the Curl_http2_done function, where the drain buffer was not properly freed.
   • Fixed in Version: curl 8.10.0.
   • Details: This fix ensures that the drain buffer is properly freed when HTTP/2 transfers are completed.

3. Memory Leak in Curl_http_done:

   • Description: A memory leak was identified in the Curl_http_done function, where the conn->proto.httpc.h2 pointer was not properly freed.
   • Fixed in Version: curl 8.10.0.
   • Details: This fix ensures that the conn->proto.httpc.h2 pointer is properly freed when HTTP transfers are completed.

Now ... 2 of the 3 are HTTP/2 related , and 8.10.0 also had other issues fixed by 8.10.1 ... but these are the memory leaks and potentially the first one is a likely suspect given it covers HTTP/1.1 and HTTP/2 operations (and your forcing HTTP/1.1)

I would also see if upgrading to a newer curl version solves this given your 8.9.1 curl version.

[IsmaelSF]

No dump is created. I've tried with libcurl 8.11.0 and same behavior. I'm running with gdb attached and catching SIGSEGV to be able to inspect stack to see if we'll have more information. Let you know my findings.

[abraunegg]

@IsmaelSF
What compiler are you using?

It is crashing / exiting at the same point as #2951 - where the Alpine 3.20 version of LDC creates the same situation - but on Alpine 3.19 .. the exact same compiler version (just the Alpine 3.19 one) works without issue.

What is the default compiler for Rocky Linux 8.10 ?

I have a busy weekend - so I will be unable to create a test / debug environment .. but will start looking into this as potentially a compiler version issue.

[abraunegg]

@IsmaelSF
I have found a few hours this morning to build a VM of Rocky Linux 8.10. Post install, this is all I have done:

as root

dnf -y update; reboot;

as normal user

1. Followed RHEL8 | CentOS 8 Dependency Installation details
2. Created a file to clone & compile

#!/bin/bash
rm -rf ./onedrive-master/
git clone https://github.com/abraunegg/onedrive.git onedrive-master
cd onedrive-master
source ~/dlang/dmd-2.109.1/activate
./configure --enable-debug --enable-notifications; make clean; make;
deactivate
./onedrive --version

3. Authenticated the application to a Business Account
4. Configured the client to use my 100K item online data set
5. Started the application from the cloned directory using ./onedrive -s -v --resync and the client is currently processing 100K of online JSON items for applicability for download

The default curl version curl 7.61.1 (x86_64-redhat-linux-gnu) is being used:

curl 7.61.1 (x86_64-redhat-linux-gnu) libcurl/7.61.1 OpenSSL/1.1.1k zlib/1.2.11 brotli/1.0.6 libidn2/2.2.0 libpsl/0.20.2 (+libidn2/2.2.0) libssh/0.9.6/openssl/zlib nghttp2/1.33.0
Release-Date: 2018-09-05
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz brotli TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL 

At this point, if this is all operating OK, I will go about using mock to build a newer version of curl

If that is all operating correctly, potentially you might need to run a memory test against your system|platform if this is a physical system - you could have bad system memory ....

[abraunegg]

@IsmaelSF
And no issue .. straight into creating the local directory structure and downloading files:

Now to do the reverse and have 100K files needed for upload .. which is your failure:

[2024-Nov-04 12:59:14.5199694] Attempting to contact Microsoft OneDrive Login Service
[2024-Nov-04 12:59:14.8415583] Successfully reached Microsoft OneDrive Login Service
[2024-Nov-04 12:59:15.1918650] Configuring Global Azure AD Endpoints
[2024-Nov-04 12:59:18.1970711] Performing a database consistency and integrity check on locally stored data
[2024-Nov-04 12:59:18.5916839] Scanning the local file system '/OneDrive/LinuxServerBackup' for new data to upload
[2024-Nov-04 13:01:34.3244167] New directories to create on Microsoft OneDrive: 33196
[2024-Nov-04 16:55:21.9573845] New items to upload to Microsoft OneDrive: 378240


Due to a termination signal, internal processing stopped abruptly. The application will now exit in a unclean manner.
Due to a termination signal, internal processing stopped abruptly. The application will now exit in a unclean manner.

[abraunegg]

@IsmaelSF
And in reverse:

after creating the directory structure , uploading the files without issue ...

In your upload scenario, you have >300K files to upload ... so I also have tried that:

Zero issue ...

So at this point, I have think your issue is something local environment related.

I can go and install Rocky Linux on a physical laptop that I have (thats going to take some time) ..

If your system is a physical one, can you please download the 64Bit Memtest86 ISO from here and run it - https://www.memtest.org/ ... ensure that the system runs through all tests/patterns.

If your system is not physical, you may need to redeploy / rebuild your system as you might some some sort of corruption somewhere.

[abraunegg]

@IsmaelSF
Any further update here?

[IsmaelSF]

Quick update from mobile, I think that I get it. Gdb debugging activity showed results yesterday and in back trace is segfaulting in OpenSSL function. Let me recap all the information and will post when recover Pc desktop access. El sáb, 9 nov 2024, 21:50, abraunegg ***@***.***> escribió:

…

@IsmaelSF <https://github.com/IsmaelSF> Any further update here? — Reply to this email directly, view it on GitHub <#2950 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE4VQZYMUUTEYIHIYNQNY23Z7ZYRPAVCNFSM6AAAAABRDLLT66VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMJZHE4TANQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[abraunegg]

Quick update from mobile, I think that I get it. Gdb debugging activity
showed results yesterday and in back trace is segfaulting in OpenSSL
function. Let me recap all the information and will post when recover Pc
desktop access.

Very interesting ... in the very early days of v2.5.x development (pre alpha 0) I came across an OpenSSL issue where the CA certificate could not be read.

What are your ulimits ? These are the defaults on Rocky Linux 8.10 on my test system:

[alex@rocky-linux-8 onedrive-master]$ ulimit
unlimited
[alex@rocky-linux-8 onedrive-master]$ ulimit -Sa
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 14437
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 14437
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[alex@rocky-linux-8 onedrive-master]$ 
[alex@rocky-linux-8 onedrive-master]$ ulimit -Ha
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 14437
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 262144
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 14437
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[alex@rocky-linux-8 onedrive-master]$ 

[IsmaelSF]

@abraunegg this is the information I got from gdb:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffea37c700 (LWP 442)]
0x00007ffff5366114 in sha1_block_data_order_avx () from /lib64/libcrypto.so.10
(gdb) bt
#0  0x00007ffff5366114 in sha1_block_data_order_avx () at /lib64/libcrypto.so.10
Python Exception <type 'exceptions.OverflowError'> long too big to convert:
Python Exception <type 'exceptions.OverflowError'> long too big to convert:
Python Exception <type 'exceptions.OverflowError'> long too big to convert:
Python Exception <type 'exceptions.OverflowError'> long too big to convert:
Python Exception <type 'exceptions.OverflowError'> long too big to convert:
Python Exception <type 'exceptions.OverflowError'> long too big to convert:
Python Exception <type 'exceptions.OverflowError'> long too big to convert:
Python Exception <type 'exceptions.OverflowError'> long too big to convert:
#1  0xffffffffffffffff in  ()#2  0xffffffffffffffff in  ()#3  0xffffffffffffffff in  ()#4  0xffffffffffffffff in  ()#5  0xffffffffffffffff in  ()#6  0xffffffffffffffff in  ()#7  0xffffffffffffffff in  ()#8  0xffffffffffffffff in  ()#9  0x00007fffd449c60c in  ()
Python Exception <type 'exceptions.OverflowError'> long too big to convert:
#10 0xffffffffffffffff in  ()#11 0x00007ffff574abac in  () at /lib64/libcrypto.so.10
#12 0x00007fffd449c5f0 in  ()
#13 0x0000000000000013 in  ()
#14 0x00007ffff536250f in SHA1_Update () at /lib64/libcrypto.so.10
#15 0x00007ffff5413537 in ssleay_rand_add.part.0 () at /lib64/libcrypto.so.10
#16 0x00007ffff5774afe in ssl3_connect () at /lib64/libssl.so.10
#17 0x00007ffff7ba8652 in ossl_connect_step2 () at /usr/local/lib/libcurl.so.4
#18 0x00007ffff7bac014 in ossl_connect_common () at /usr/local/lib/libcurl.so.4
#19 0x00007ffff7baebd3 in ssl_cf_connect () at /usr/local/lib/libcurl.so.4
#20 0x00007ffff7b4f4a7 in cf_setup_connect () at /usr/local/lib/libcurl.so.4
#21 0x00007ffff7b476dd in cf_hc_connect () at /usr/local/lib/libcurl.so.4
#22 0x00007ffff7b4c131 in Curl_conn_connect () at /usr/local/lib/libcurl.so.4
#23 0x00007ffff7b7f781 in multi_runsingle () at /usr/local/lib/libcurl.so.4
#24 0x00007ffff7b8054a in curl_multi_perform () at /usr/local/lib/libcurl.so.4
#25 0x00007ffff7b5a80b in curl_easy_perform () at /usr/local/lib/libcurl.so.4
#26 0x00000000007f71da in std.net.curl.Curl.perform() ()
#27 0x00000000007f5d28 in std.net.curl.HTTP.perform() ()
#28 0x0000000000717a4e in curlEngine.CurlEngine.execute() (this=0x7fffc2917780) at src/curlEngine.d:423
#29 0x00000000007213ae in onedrive.OneDriveApi.get() (__capture=0x7fffc2cedd50, response=0x7fffc28ab580) at src/onedrive.d:1120
#30 0x00000000007219d9 in onedrive.OneDriveApi.oneDriveErrorHandlerWrapper() (this=0x7fffc2336a10, result=0x7fffea37a8a0, lineno=572, callingFunction=..., validateJSONResponse=true, executer=...) at src/onedrive.d:1185
#31 0x000000000072134e in onedrive.OneDriveApi.get() (this=0x7fffc2336a10, __HID208=0x7fffea37a8a0, lineno=572, callingFunction=..., requestHeaders=..., skipToken=false, url=...) at src/onedrive.d:1121
#32 0x000000000071d17c in onedrive.OneDriveApi.getPathDetailsByDriveId() (this=0x7fffc2336a10, __HID193=0x7fffea37a8a0, path=..., driveId=...) at src/onedrive.d:572
#33 0x000000000074704e in syncEngine.SyncEngine.uploadNewFile() (this=0x7ffff7ef0000, fileToUpload=...) at src/sync.d:5830
#34 0x0000000000746615 in syncEngine.SyncEngine.uploadNewLocalFileItemsInParallel() (__capture=0x7fffffffca10, __applyArg1=..., i=2) at src/sync.d:5661
#35 0x00000000006a4879 in std.parallelism() (__capture=0x7fffffffc990) at /root/dlang/dmd-2.099.0/linux/bin64/../../src/phobos/std/parallelism.d-mixin-4100:4142
#36 0x00000000007d1ac8 in std.parallelism() ()
#37 0x00000000007d15a8 in std.parallelism() ()
#38 0x00000000008200ff in std.parallelism.AbstractTask.job() ()
#39 0x00000000007d0184 in std.parallelism.TaskPool.doJob() ()
#40 0x00000000007d02f3 in std.parallelism.TaskPool.executeWorkLoop() ()
#41 0x00000000007d029c in std.parallelism.TaskPool.startWorkLoop() ()
#42 0x00000000007acfb1 in core.thread.context.Callable.opCall() ()
#43 0x00000000007aca8f in thread_entryPoint ()
#44 0x00007ffff672eea5 in start_thread () at /lib64/libpthread.so.0
#45 0x00007ffff5d37b0d in clone () at /lib64/libc.so.6

I also discover that I was running onedrive client under CentoOS 7.9.2009 and not Rocky 8.x, sorry for the confusion it was my fault, my apologies, too many different servers.

I'm going to upgrade openssl from default v1.0.2k to openssl-1.1.1w (lastest in 1.1.1 serie) to see if helps. Let you know my findings.

[abraunegg]

Given that CentOS 7 is End of Life on June 30 2024 ... unless you updated your entire platform from the repositories or have your own cached update repository locally, it is going to be difficult to update and/or maintain. Your are better backing up your data and re-installing with a newer version of CentOS or similar.

[abraunegg]

@IsmaelSF
If you are going to rebuild, I suggest you use 1.1.1q and use mock to rebuild the SRPM from Koji from F35 - https://koji.fedoraproject.org/koji/buildinfo?buildID=1997398

Whilst not 1.1.1s , 1.1.1q is the last 1.1.1 series that the RedHat team put together.

[IsmaelSF]

I've recompiled curl latest version (v8.11.0) with openssl 3.4.0, --resync for a complete synchronization and everything works!

curl 8.11.0 (x86_64-pc-linux-gnu) libcurl/8.11.0 OpenSSL/3.4.0 zlib/1.2.7 libpsl/0.7.0 (+libicu/50.1.2)
Release-Date: 2024-11-06
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS HSTS HTTPS-proxy IPv6 Largefile libz NTLM PSL SSL threadsafe TLS-SRP UnixSockets

No issues with abnormal termination or mtime corruption. Instead of changing default openssl version, I'm going to keep them in a separate environment /usr/local/ only for onedrive client and meanwhile spend much of the effort to migrate to Rocky 8.x entire server.

Thank you for your incredible support. Don't hesitate to contact me to test any PRs or releases in my environment. Good work!