hsa_amd_pointer_info doesn't work when calling from a function?

[ye-luo]

On my machine, I got hsa_amd_pointer_info failed.
If I change if(0) to if(1), the call succeeds.

reproducer.

#include <hsa/hsa.h>
#include <hsa/hsa_ext_amd.h>
#include <stdio.h>

#define N 100293

int checkLocked(void *ptr) {
  hsa_amd_pointer_info_t info;
  hsa_status_t herr = HSA_STATUS_SUCCESS;

  herr = hsa_amd_pointer_info(ptr, &info, NULL, NULL, NULL);
  if (herr != HSA_STATUS_SUCCESS) {
    printf("  hsa_amd_pointer_info failed %d\n", herr);
    return 1;
  }

  if (info.type != HSA_EXT_POINTER_TYPE_LOCKED) {
    printf("  pointer is noooooooooooot locked\n");
    return 1;
  } else
    printf("  pointer is locked func\n");

  return 0;
}

int main() {
  hsa_status_t herr = hsa_init();
  if (herr != HSA_STATUS_SUCCESS) {
    printf("hsa_init failed\n");
    return 1;
  }

  const int n = N;
  int *a = new int[n];
  for (int i = 0; i < n; i++)
    a[i] = 0;

  checkLocked(a);

  int *a_locked = nullptr;
  herr = hsa_amd_memory_lock(a, n * sizeof(int), nullptr, 0, (void **)&a_locked);
  if (herr != HSA_STATUS_SUCCESS) {
    printf("Locking failed\n");
    return 1;
  }

  if(0)
  {
    hsa_amd_pointer_info_t info;
    herr = hsa_amd_pointer_info(a, &info, NULL, NULL, NULL);
    if (herr != HSA_STATUS_SUCCESS) {
      printf("  hsa_amd_pointer_info failed\n");
      return 1;
    }

    if (info.type != HSA_EXT_POINTER_TYPE_LOCKED) {
      printf("  pointer is noooooooooooot locked\n");
      return 1;
    } else
      printf("  pointer is locked main\n");
  }
  else
    checkLocked(a);

  hsa_shut_down();
  return herr;
}

[JonChesterfield]

That's weird. I see the void* vs int* type on the pointer, and the indirection through a function call, but hsa_amd_pointer_info is in a shared library (so abi is fixed), can't imagine why this would matter. Will see if it reproduces locally.

Don't think it does. I appended 'func' and 'main' to the print statements to distinguish them,

 export LLVM=$HOME/llvm-install ; clang++ -O1 pointer_info.cpp -I$LLVM/include/ -Wl,--rpath=$LLVM/lib/ $LLVM/lib/libhsa-runtime64.so.1 && ./a.out 
  pointer is noooooooooooot locked func
  pointer is locked func

export LLVM=$HOME/llvm-install ; clang++ -O1 pointer_info.cpp -I$LLVM/include/ -Wl,--rpath=$LLVM/lib/ $LLVM/lib/libhsa-runtime64.so.1 && ./a.out 
  pointer is noooooooooooot locked func
  pointer is locked main

(my local machine has a gfx906, rocm 4.5, linux 5.4, driver from rocm as opposed to upstream)

[ye-luo]

I updated a bit the above code

yeluo@epyc-server:~$ clang++ -fsanitize=address -fopenmp -O0 test_hsa.cpp -I/opt/rocm/include -L/opt/rocm/lib -lhsa-runtime64 && ./a.out
  pointer is noooooooooooot locked
  pointer is locked func
yeluo@epyc-server:~$ clang++ -O0 -I/opt/rocm-4.5.2/include test_hsa.cpp -L/opt/rocm-4.5.2/lib -lhsa-runtime64 && ./a.out 
  pointer is noooooooooooot locked
  hsa_amd_pointer_info failed 4097
yeluo@epyc-server:~$ clang++ -O1 -I/opt/rocm-4.5.2/include test_hsa.cpp -L/opt/rocm-4.5.2/lib -lhsa-runtime64 && ./a.out 
  pointer is noooooooooooot locked
  pointer is locked func
yeluo@epyc-server:~$ clang++ -O3 -I/opt/rocm-4.5.2/include test_hsa.cpp -L/opt/rocm-4.5.2/lib -lhsa-runtime64 && ./a.out 
  pointer is noooooooooooot locked
  pointer is locked func

So It seems to be a real issue at "-O0". So either I misused the library or a bug.

[JonChesterfield]

What's clang++ in this context? Different behaviour on O0 and O1 is a bug and that's relatively unusual on x64

[ye-luo]

upstream clang and AOMP clang don't matter. g++ constantly failed.

yeluo@epyc-server:~$ g++ -O0 -I/opt/rocm-4.5.2/include test_hsa.cpp -L/opt/rocm-4.5.2/lib -lhsa-runtime64 && ./a.out 
  pointer is noooooooooooot locked
  hsa_amd_pointer_info failed 4097
yeluo@epyc-server:~$ g++ -O1 -I/opt/rocm-4.5.2/include test_hsa.cpp -L/opt/rocm-4.5.2/lib -lhsa-runtime64 && ./a.out 
  pointer is noooooooooooot locked
  hsa_amd_pointer_info failed 4097
yeluo@epyc-server:~$ g++ -O3 -I/opt/rocm-4.5.2/include test_hsa.cpp -L/opt/rocm-4.5.2/lib -lhsa-runtime64 && ./a.out 
  pointer is noooooooooooot locked
  hsa_amd_pointer_info failed 4097

[fxkamd]

What if you change if(0) to if(!a_locked)? That way the compiler would be forced to run hsa_amd_memory_lock before checkLocked(a). Without that, it doesn't know that the checkLocked call depends on hsa_amd_memory_lock.

[ye-luo]

What if you change if(0) to if(!a_locked)? That way the compiler would be forced to run hsa_amd_memory_lock before checkLocked(a). Without that, it doesn't know that the checkLocked call depends on hsa_amd_memory_lock.

It doesn't change the above behavior.

[JonChesterfield]

hsa_amd_memory_lock and hsa_amd_pointer_info are both calls to unknown functions in a shared library - the C++ compiler has to pessimistically assume they might mutate the same global state so can't reorder them. Though different behaviour on O0 and O1 does support the x64 compiler bug theory. Sadly this still doesn't reproduce for me, not sure where the config difference lies

[schung-amd]

Hi @ye-luo, are you still having this issue? I was able to reproduce the discrepancy between the function and inline versions as well as the discrepancy between -O0 and-O1. The documentation at

ROCR-Runtime/src/inc/hsa_ext_amd.h

Line 2020 in d614ea8

revision. Set to sizeof(hsa_amd_pointer_t) prior to calling

states that info.size must be "Set to sizeof(hsa_amd_pointer_t) prior to calling
hsa_amd_pointer_info". Without doing this, the size field is uninitialized, leading to UB. On my repro, with -O0, this consistently results in a size of 0 when the second call is inside the function, and a size of 2 when it is inlined.

ROCR-Runtime/src/core/runtime/runtime.cpp

Line 870 in d614ea8

if (info->size == 0) return HSA_STATUS_ERROR_INVALID_ARGUMENT;

checks for a size of 0 to test if the size is uninitialized, so this fails only for the specific in-function case even though all of these values are uninitialized and should fail. Notably, when -O1 is passed, the uninitialized size fields contain random values, so this check is not tripped, which explains your observations on -O1 and higher.

Therefore, to address your issue, set info.size to sizeof(hsa_amd_pointer_t) before calling hsa_amd_pointer_info. On my local repro this code actually does not work properly even with the size field set, as it appears to be failing to lock the pointer, but I suspect this failure is unrelated.

[ye-luo]

@schung-amd thank you for pointing out the issue in my code. After setting the info.size, I got consistent behaviors.