Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NullReferenceException thrown from AmazonS3Client.Begin/EndGetObject if no internet connection #274

Closed
callorico opened this issue Nov 18, 2015 · 5 comments
Labels
bug This issue is a bug.

Comments

@callorico
Copy link

Using 3.1.3.5 with .NET 4.0 Client Profile

The following works fine if I have an active internet connection:

using (IAmazonS3 s3Client = new AmazonS3Client())
{
    // Setup request for putting an object in S3.                 
    GetObjectRequest request = new GetObjectRequest
    {
        BucketName = "<bucket>",
        Key = "<key>",
    };

    IAsyncResult result = s3Client.BeginGetObject(request, null, null);

    GetObjectResponse response = s3Client.EndGetObject(result);
    response.WriteResponseStreamToFile("test.out");
}

However, if I disable my network adapter prior to running the above, a NullReferenceException is thrown out of the SDK with the following stack trace:

   at Amazon.Runtime.Internal.RedirectHandler.HandleRedirect(IExecutionContext executionContext) in d:\Jenkins\jobs\v3-stage-release\workspace\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\RedirectHandler.cs:line 101
   at Amazon.Runtime.Internal.RedirectHandler.InvokeAsyncCallback(IAsyncExecutionContext executionContext) in d:\Jenkins\jobs\v3-stage-release\workspace\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\RedirectHandler.cs:line 75
   at Amazon.Runtime.Internal.PipelineHandler.AsyncCallback(IAsyncExecutionContext executionContext) in d:\Jenkins\jobs\v3-stage-release\workspace\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\PipelineHandler.cs:line 95
   at Amazon.Runtime.AmazonServiceClient.EndInvoke[TResponse](IAsyncResult result) in d:\Jenkins\jobs\v3-stage-release\workspace\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\AmazonServiceClient.cs:line 212
   at Amazon.S3.AmazonS3Client.EndGetObject(IAsyncResult asyncResult) in d:\Jenkins\jobs\v3-stage-release\workspace\AWSDotNetPublic\sdk\src\Services\S3\Generated\_bcl35\AmazonS3Client.cs:line 1956
   at S3Test.Program.Main(String[] args) in c:\Users\callorico\Desktop\S3Test\S3Test\Program.cs:line 28

In this scenario, I would expect to see a network exception of some type. If I replace the APM code with a synchronous s3Client.GetObject call then I do see an AmazonServiceException with a nested WebException.

@jimfl jimfl added the bug This issue is a bug. label Nov 18, 2015
@jimfl
Copy link
Contributor

jimfl commented Nov 18, 2015

This is an obvious bug in the request handling logic in the 3.5 async client. Thanks for pointing it out.

PavelSafronov pushed a commit that referenced this issue Nov 21, 2015
… for .NET 3.5 async, when an exception is thrown in the pipeline.
@gokarnm
Copy link
Contributor

gokarnm commented Nov 21, 2015

The fix for this issue is available in the latest version of the AWS S3 SDK (version 3.1.3.7). The APM API should now throw the same exception as the synchronous version.

@callorico
Copy link
Author

OK, I tried testing with 3.1.3.8 and this doesn't seem to be working correctly:

I tried running the following with no Internet connection:

class Program
{
    static IAmazonS3 s3Client;

    static void GetObjectCallback(IAsyncResult result)
    {
        Console.WriteLine("{0} GetObjectCallback was triggered", DateTime.UtcNow);
        GetObjectResponse response = s3Client.EndGetObject(result);
        Console.WriteLine("{0} EndGetObject returned", DateTime.UtcNow);
    }

    static void Main(string[] args)
    {
        s3Client = new AmazonS3Client();

        GetObjectRequest request = new GetObjectRequest
        {
            BucketName = "<bucket-name>",
            Key = "<key-name>",
        };

        Console.WriteLine("{0} About to call BeginGetObject", DateTime.UtcNow);
        try
        {
            s3Client.BeginGetObject(request, new AsyncCallback(GetObjectCallback), null);
        }
        catch (Exception e)
        {
            Console.WriteLine("{0} BeginGetObject threw {1}", DateTime.UtcNow, e);
        }
        finally
        {
            Console.WriteLine("{0} BeginGetObject Elapsed time", DateTime.UtcNow);
        }

        Console.WriteLine("Hit return to exit");
        Console.ReadLine();
    }
}

Here is the output from my machine:

PS C:\Users\Callorico\Desktop\S3Test\S3Test\bin\Debug> .\S3Test.exe
11/24/2015 5:36:33 AM About to call BeginGetObject
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM GetObjectCallback was triggered: IsCompleted? True
11/24/2015 5:37:07 AM BeginGetObject threw System.NullReferenceException: Object reference not set to an instance of an
object.
   at Amazon.Runtime.Internal.PipelineHandler.AsyncCallback(IAsyncExecutionContext executionContext) in d:\Jenkins\jobs\
v3-stage-release\workspace\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\PipelineHandler.cs:line 111
   at Amazon.Runtime.Internal.HttpHandler`1.GetResponseCallback(IAsyncResult result) in d:\Jenkins\jobs\v3-stage-release
\workspace\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HttpHandler\HttpHandler.cs:line 302
   at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
   at System.Net.ContextAwareResult.CaptureOrComplete(ExecutionContext& cachedContext, Boolean returnContext)
   at System.Net.HttpWebRequest.BeginGetResponse(AsyncCallback callback, Object state)
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync(IAsyncExecutionContext executionContext) in d:\Jenkins\jobs\v3-s
tage-release\workspace\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HttpHandler\HttpHandler.cs:line 253
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync(IAsyncExecutionContext executionContext) in d:\Jenkins\jobs\v3-
stage-release\workspace\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\MetricsHandler.cs:line 87
   at Amazon.Runtime.AmazonServiceClient.BeginInvoke[TRequest](TRequest request, IMarshaller`2 marshaller, ResponseUnmar
shaller unmarshaller, AsyncCallback callback, Object state) in d:\Jenkins\jobs\v3-stage-release\workspace\AWSDotNetPubli
c\sdk\src\Core\Amazon.Runtime\AmazonServiceClient.cs:line 169
   at Amazon.S3.AmazonS3Client.BeginGetObject(GetObjectRequest request, AsyncCallback callback, Object state) in d:\Jenk
ins\jobs\v3-stage-release\workspace\AWSDotNetPublic\sdk\src\Services\S3\Generated\_bcl35\AmazonS3Client.cs:line 1943
   at S3Test.Program.Main(String[] args) in c:\Users\Ryan\Desktop\S3Test\S3Test\Program.cs:line 35
11/24/2015 5:37:07 AM BeginGetObject Elapsed time
Hit return to exit
  1. First, it looks like there is still a NullReferenceException being thrown in this scenario.
  2. The thread that calls BeginGetObject is blocked for ~30 seconds (presumably while going through the retry loop). In the APM model, I believe the Begin* methods should return immediately.
  3. Why is the GetObjectCallback method called so many times? I would have expected it to just be called once.

Is there anyway to use the *Async methods with .NET 4.0 instead?

@gokarnm
Copy link
Contributor

gokarnm commented Nov 25, 2015

Hi,

1 & 3. You need to have a try catch block around the EndGetObject call. In the current code, the EndGetObject throwns an exception as expected, which is not handled by GetObjectCallback method, causing the callback method to be called again. The callback should not be called multiple time by the SDK, this is a bug and we are working on a fix for this. If you catch the appropriate exception in the callback, it won't be called again.

try
{
    GetObjectResponse response = s3Client.EndGetObject(result);
}
catch (AmazonServiceException exception)
{
    Console.WriteLine("{0} EndGetObject threw {1}", DateTime.UtcNow, exception);
    Console.WriteLine("{0} EndGetObject threw inner exception {1}", DateTime.UtcNow, exception.InnerException);
}

2 . You are correct, the BeginGetObject should not block for the retries, I'll let you know when a fix is available for this. The work around for this may be to reduce the number of retries, the default value for max retries is 4.

s3Client = new AmazonS3Client(new AmazonS3Config
    {
        MaxErrorRetry = 2
    });

You cannot use the *Async methods on .NET 4.0, they are only available .NET 4.5 onward.

PavelSafronov pushed a commit that referenced this issue Jan 6, 2016
- If the user callback method throws an unhandled exception into the SDK code the user callback is called again. Added code to catch unhandled exception from user callback and log it.
- In certain cases (network unplugged), the callback methods in HttpHandler are called on the same thread (synchronously). Added code to detect this case using AsyncResult.CompletedSynchronously, if callback is invoked synchronously, we call the subsequent logic on a background thread.
@gokarnm
Copy link
Contributor

gokarnm commented Jan 8, 2016

A fix for this issue is available in version 3.1.3.10 of AWS S3 SDK . This fix catches unhandled exceptions being thrown from the user code in the async callback, and does not block the BeginXyz call under certain conditions when retries are performed.

@gokarnm gokarnm closed this as completed Jan 12, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug.
Projects
None yet
Development

No branches or pull requests

3 participants