If Activity stack is broken, End_Request hangs forever

[lmolkova]

Example:

            Activity activity = new Activity("request").Start();

            for (int i = 0; i < 2; i++)
            {
                Task.Run(() =>
                {
                    Console.WriteLine($"start id={Activity.Current?.Id} name={Activity.Current?.OperationName}");
                    Activity.Current?.Stop();
                    Console.WriteLine($"stop id={Activity.Current?.Id} name={Activity.Current?.OperationName}");
                }).Wait();
            }

            int iteration = 0; 
            while (Activity.Current != null)
            {
                Activity.Current.Stop();
                if (++iteration >= 1000) break;
            }

            Console.WriteLine($"{Activity.Current?.Id}, {iteration}");

Output:

start id=|caba72ef-4be0e60ccb5e46b9. name=request
stop id= name=
start id=|caba72ef-4be0e60ccb5e46b9. name=request
stop id=|caba72ef-4be0e60ccb5e46b9. name=request

|caba72ef-4be0e60ccb5e46b9., 1000

What happens here is:

1. The first time we try to stop Activity:

start id=|caba72ef-4be0e60ccb5e46b9. name=request
stop id= name=

It is stopped successfully but only within this Task.Run 'child' context.

2. The second Task.Run is called from the 'parent' context, where Current is still set to "request" activity.
   However the activity instance is shared between contexts, so activity is considered as stopped, and second stop does not attempt to change Current

Here is the Activity.Stop code

        public void Stop()
        {
            if (Id == null)
            {
                NotifyError(new InvalidOperationException("Trying to stop an Activity that was not started"));
                return;
            }

            if (!isFinished) // it is finished second time!
            {
                isFinished = true;

                if (Duration == TimeSpan.Zero)
                {
                    SetEndTime(GetUtcNow());
                }

                Current = Parent;
            }
        }

3. When we attempt to stop all Activities on the way to parent here

while (Activity.Current != activity && Activity.Current != null)
{
    Activity.Current.Stop();
}

we never alter Activity.Current and the loop never ends.

While it's invalid to expect correct behavior what AsyncLocal is modified in child context, it's totally possible to implement such behavior and we should not hang on it.