Moving a pointer up in pointerscope stack

[HGuillemet]

Say I have a method doing some complex computation in a PointerScope and producing a result as a Pointer. We can use PointerScope.detach(Pointer) to prevent the result from being deallocated when we exit the scope:

SomePointer compute() {
  try (PointerScope ps = new PointerScope()) {
    /* Computation, allocating pointers */
    SomePointer result = new SomePointer();
    ps.detach(result);
    /* ... */
    return result;
  }
}

But what if the call to compute is itself within a PointerScope ?
We would like the result to be deallocated when we exit this outper pointer scope.
So what about adding some method PointerScope.detachUp(Pointer) to move a pointer from an inner scope to the enclosing scope (if it exists, else simply detach) ?
Or is there already a way to do it ?

[saudet]

There's no way to do that in C++ so it's not something that we should need, but there's a couple of ways we can do something that like that. Either pass the outer scope to compute() and attach it that way before return, or call innerScope.extend() before return, and outerScope.attach() after return.

[HGuillemet]

There's no way to do that in C++ so it's not something that we should need,

Why do you mean by that ? There is no C++ equivalent to PointerScope, is it ?
And I'd say the need is driven by the use case. And I believe there is a use case. I'm even close to think that this behaviour could be the default one of detach, but changing detach could break existing code, so I suggested detachUp.

Either pass the outer scope to compute() and attach it that way before return

That would work, but compute may lie in a library and the client code may not know anything about JavaCPP and not depend on it.

or call innerScope.extend() before return, and outerScope.attach() after return.

extend doesn't take a pointer arg, it will give "ownership" of all pointers allocated in inner scope to the outer scope, right ?

The workaround I use in such case is to allocate my result object before entering the inner scope, but that's not always possible or easy if I need to do some computations involving pointers to determine arguments needed for allocating the result object.

So nothing essential, but I think it would be a nice addition to JavaCPP that would only add a small number of lines of code.

[saudet]

It might be nice for your use case, but what if someone else wants to attach it to outerScope2 instead of outerScope1, or even someCompletelyUnrelatedScope? Your idea doesn't support an important or general enough purpose. If you want to do that, you can use your own extended version of PointerScope tough.

[HGuillemet]

The idea is that there might be pointer scopes in the client code, and pointer scopes in the library scope. And the library code shouldn't mess up or assume anything about client code.
The client code must be able to do:

SomePointer sp = compute();

Just like it does:

BytePointer bp = new BytePointer(10);

and the resulting pointer should have the same lifespan on both case, depending if the client uses pointer scopes or not.
I find the purpose general enough.

If compute uses pointers scopes, we want to detach the returned object from THESE pointer scopes, not to attach to a specific one. But I agree that if compute uses nested scopes itself, it could be more general to target a specific scope. Maybe could we directly use detachUp from the outer scope even if is pointer is currently attached to the inner one :

SomePointer compute() {
  SomePointer result;
  try (PointerScope ps1 = new PointerScope()) {
    /* ... */
    try (PointerScope ps2 = new PointerScope()) {
      /* ... */
      result = new SomePointer();
      ps1.detachUp(result);
      /* ... */
    }
    /* ... */
  }
  return result;
}

Else we would need to detachUp twice:

SomePointer compute() {
  SomePointer result;
  try (PointerScope ps1 = new PointerScope()) {
    /* ... */
    try (PointerScope ps2 = new PointerScope()) {
      /* ... */
      result = new SomePointer();
      ps2.detachUp(result);
      /* ... */
    }
    /* ... */
    ps1.detachUp(result);
  }
  return result;
}

[saudet]

Let's try to figure out something more general

[HGuillemet]

Back to this idea.
Struggling with limited GPU memory, and against the non-ref-counting Java GC, I find this feature quite essential.

I have written this function, which doesn't need to patch anything in JavaCPP:

public static <P extends Pointer> P scopeEscape(P p) {
    if (p == null) return null;
    Iterator<PointerScope> it = PointerScope.getScopeIterator();
    if (it == null) throw new IllegalStateException("Not in a pointer scope");
    PointerScope inner = it.next();
    if (it.hasNext()) {
      it.next().attach(p);
    }
    inner.detach(p);
    return p;
  }

And I'm heavily using it like this:

Tensor forward(Tensor input) {
  try (PointerScope ignored = new PointerScope()) {
     /* Computations creating lots of Pointer subclasses */
    return scopeEscape(result);
  }
}

Maybe can you think of something smarter ?

[saudet]

We can already do that with ps1.attach(result), we don't need anything new for that case.

Actually I think adding a method to Pointer is what makes most sense. It's like "reiniting" the Pointer

[saudet]

    return result.escapeScope();

How about we name it Pointer.exitInnerScope() ?

[HGuillemet]

I think "escape" is the coined term for this concept, like in Escape Analysys.
escapeInnerScope ?

However, there is an issue about my suggested implementation above: if there is no outer scope, then we will detach from the inner scope without attaching to another, and since detach calls releaseReference and the ref counter is 0 afterwards in this case, it will deallocate the pointer. What we need is to remove the pointer from the list of pointers to deallocate when we exit the scope, but not call releaseReference. I don't think this is currently possible with the public API.

EDIT: or, rather, decrease the ref counter, since it has been increased when entering the scope, but not call the deallocator if it's 0. BTW I don't understand why PointerScope needs to modify the ref counter. Couldl't we simply call the deallocator eagerly at exit if ref counter is null ?

[saudet]

Sure, escapeInnerScope() sounds good too.

If there is no outer scope, we can just call retainReference() instead

[saudet]

EDIT: or, rather, decrease the ref counter, since it has been increased when entering the scope, but not call the deallocator if it's 0. BTW I don't understand why PointerScope needs to modify the ref counter. Couldl't we simply call the deallocator eagerly at exit if ref counter is null ?

How do you propose to do reference counting without a reference counter??

[HGuillemet]

We can already do that with ps1.attach(result), we don't need anything new for that case.

ps.escape(result) attachs result to the scope enclosing ps (if any) and detachs result from ps (or from any enclosing scope). What do you mean ?

[HGuillemet]

Another advantage of binding the method to PointerScope and not Pointer is that there is no way to escape a scope we don't "own":
Client code:

try (PointerScope ps = new PointerScope()) {
  // ...
  p = new SomePointer();
  // ...
  compute(p);
  // ...
}

Library code:

void compute(SomePointer p) {
  // ...
  p.escapeInnerScope();
  // ...
}

Here the library code messes up with client code by escaping the client scope, while the client code legitimately expects that all the pointers allocated in its scope will be deallocated when scope exits.

With ps.escape(p) we must have access to scope ps to be able to escape it.

[saudet]

Like I keep telling you, we can already to that, and the name of that method is PointerScope.attach(Pointer). What is it that you don't understand?

[saudet]

Or are you talking about some method that performs both attach and detach on some random scopes? Like if the user calls someRandomScope.escape(Pointer) on what scope would it call attach()? I don't understand what you're trying to tell me here. You'll need to be more specific

[HGuillemet]

So how would we perform this:

Tensor forward(Tensor input) {
  try (PointerScope ps1 = new PointerScope()) {
    /* ... */
    try (PointerScope ps2 = new PointerScope()) {
      /* ... */
      return ps1.escape(result);
    }
  }
}

with PointerScope.attach(Pointer) ?
If we do this:

Tensor forward(Tensor input) {
  try (PointerScope ps1 = new PointerScope()) {
    /* ... */
    try (PointerScope ps2 = new PointerScope()) {
      /* ... */
      ps1.attach(result);
      return result;
    }
  }
}

Then result will survive to the closing of ps2 but still will be deallocated when ps1 exits, right ?

[saudet]

Yes, so what's your point? I still don't understand how you want to make ps1.escape(result) work.

[HGuillemet]

If there is no outer scope, we can just call retainReference() instead

We don't want the pointer to live forever either. Just to escape the current scope (avoid it for being deallocated when the scope exits).

How do you propose to do reference counting without a reference counter??

I must be missing something. What will not work if we just register the pointer in the pointer stack of the inner scope when the pointer is created, and call deallocate on pointers of the pointer stack when we exit the scope, without doing retain/release reference ?

[saudet]

That doesn't allow us to attach the pointer to multiple scopes

[HGuillemet]

What will not work if we just register the pointer in the pointer stack of the inner scope when the pointer is created, and call deallocate on pointers of the pointer stack when we exit the scope, without doing retain/release reference ?

I see, attaching to multiple scopes will break. And we are back to your remark about "we can already to that, and the name of that method is PointerScope.attach(Pointer)." So is ref counting in pointer scopes only useful when the user manually attaches a pointer to some arbitrary scope ?

[saudet]

Yeah, that's what reference counting is for

[HGuillemet]

Like if the user calls PointerScope.escape(someRandomScope) on what scope would it call attach()?

Such method would rather be someRandomScope.escape(somePointer)) and it would attach the pointer to the scope enclosing someRandomScope, like I do in the proposed implementation of scopeEscape above, but for an arbitrary scope, not the innermost one.

[saudet]

Your understanding of the relationship between the reference counter and pointer scopes is accurate, so I don't really see a nice way to make everything that you want it do work. We can add some other version of detach(), sure, but we still need to know which scope it has been attached to. If the user doesn't know, then I don't see a clean solution.

[HGuillemet]

The user knows the scope he wants to detach from.

To sum it up, here is my suggestion:

• Create Pointer.releaseReference(boolean deallocate) and DeallocatorReference.release(boolean deallocate) that do the same thing that current versions of these methods, but trigger deallocation only if ref count drops to 0 AND deallocate is true. Add no-arg shortcuts for backward compatibility.
• Then 3 options:

1. Add an instance method PointerScope.escape(Pointer) that removes the pointer from the pointerStack of this scope, calls releaseReference(false) on the pointer, and attaches the pointer to the scope enclosing this scope, in the current thread, IF ANY. If the pointer is not in the pointerStack of this scope, throws an exception.
2. Add an instance method PointerScope.escape(Pointer) that does the same thing, but this scope can be any scope of the current thread. The pointer is removed from the pointerStack of any scope from the current innermost scope of this thread up to this scope. This allows to escape from 2 scopes with a single call for convenience. Throws an exception if the pointer is not found in any of these pointerStack.
3. Add a static method escapeScope(Pointer) that does the same thing than 1) but on getInnerScope().

My preference is for 2).

[saudet]

What if instead we could set Pointer objects into some "return mode" by calling returnReference() or something like that in which when the counter reaches 0, instead of deallocating, it would first try to attach itself to the new inner scope, or failing that, remain in an alive state at 0? Then we could achieve what you want by calling something like somePointerScope.attach(somePointer.returnReference()) or through some special version of attach() like somePointerScope.attachAndReturn(somePointer) or maybe the other way around somePointer.returnReference(somePointerScope)? Doesn't feel exactly right, but still user-friendler than than a new concept of "escape".

[HGuillemet]

What is the role of somePointerScope in your proposal ? How would that be used in the typical use case ?

[HGuillemet]

If somePointerScope is the scope we want to "escape" from then this attach looks pretty well like the PointerScope.escape I'm suggesting, but why calling this attach ? The pointer has already been attached automatically to the scope when created.