NuGET deltalake.net.0.2.0 is missing libdelta_kernel_ffi

[dmunch]

I was trying to use the NuGet by adopting the example and instead of referencing the csproj and native binaries directly, I used the NuGet package.

I.e. my .csproj looks like this

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="DeltaLake.Net" Version="0.2.0" />
    <!-- Entra ID interaction with Azure Data Lake Storage -->
    <PackageReference Include="Azure.Identity" Version="1.12.1" />
  </ItemGroup>

</Project>

Project compiles fine, however when running, I get

python-env) ➜  cli git:(main) ✗ dotnet run $PWD/demo-table 20
Unhandled exception. System.DllNotFoundException: Unable to load shared library 'delta_kernel_ffi' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: 
[...(truncatd for brevity)...]

So I've downloaded the package manually and unzipped it, verifying its contents, and indeed, that file is missing.

(python-env) ➜  deltalake.net.0.2.0 git:(main) ✗ tree
.
├── DeltaLake.Net.nuspec
├── README.md
├── [Content_Types].xml
├── _rels
├── lib
│   ├── net6.0
│   │   ├── DeltaLake.dll
│   │   └── DeltaLake.xml
│   └── net8.0
│       ├── DeltaLake.dll
│       └── DeltaLake.xml
├── package
│   └── services
│       └── metadata
│           └── core-properties
│               └── 30cc255eb0124b6a8ddc995f282b59f8.psmdcp
└── runtimes
    ├── linux-arm64
    │   └── native
    │       └── libdelta_rs_bridge.so
    ├── linux-x64
    │   └── native
    │       └── libdelta_rs_bridge.so
    ├── osx-arm64
    │   └── native
    │       └── libdelta_rs_bridge.dylib
    ├── osx-x64
    │   └── native
    │       └── libdelta_rs_bridge.dylib
    └── win-x64
        └── native
            └── delta_rs_bridge.dll

20 directories, 13 files

Having had a look at DeltaLake.csproj, it seems like its currently referencing the debug binary of the missing file, even in release mode (due to reasons well explained in the file). So maybe it's a case of this file not being explicitly built in the Github Action package.yml ?

[mightyshazam]

@dmunch We weren't setting the KernelLibraryRoot parameter in the dotnet pack command. I removed the listing for the 0.2.0 package, and I will get the fixed version out there soon.

[dmunch]

Awesome, thanks a lot for the quick fix!

[dmunch]

Unfortunately publishing the NuGet is now failing since the package size (305mb) exceeds the package size limit of 250mb. NuGet/NuGetGallery#9473

[mightyshazam]

@mdrakiburrahman We have this comment in the code about release builds of delta_kernel. I have tried testing locally with a release build and it seems to work. Are you still experiencing this issue? This is working on a branch with the latest delta-rs.

[mdrakiburrahman]

The CI test was failing when I wrote that. If the tests are passing now on the Release build*, then it's probably some Rust thing that I do not understand, and no concerns on my end if we want to start using the Release build

[mdrakiburrahman]

@mightyshazam - so basically, if the branch above passes the new read test I added with the Kernel-based read on release build, we're good to go.

If not, then let me know, if this Nuget size is caused due to the debug build, I need to sort that out since I introduced it.

[mightyshazam]

The nuget size is probably due to the debug build. It adds 50MB on osx-arm, so it is safe to assume the other targets are responsible for at least 50 more, putting us over the limit. I have a PR to test the build #108 with the update to delta rs.

[mdrakiburrahman]

Sounds good, please let me know if you need me for anything.

[dmunch]

Thanks again for the quick turnaround: NuGet with version 0.22.3 now works fine on my local machine