Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integration test failure due to SQLite database locking #48262

Closed
sharwell opened this issue Oct 2, 2020 · 3 comments
Closed

Integration test failure due to SQLite database locking #48262

sharwell opened this issue Oct 2, 2020 · 3 comments
Assignees
@CyrusNajmabadi
Labels
Area-IDE Integration-Test Test failures in roslyn-integration-CI
Milestone
16.9

Comments

@sharwell
Copy link
Member

sharwell commented Oct 2, 2020

Version Used: #48250

🔗 https://dev.azure.com/dnceng/public/_build/results?buildId=840074

Steps to Reproduce:

Integration tests

Actual Behavior:

10/02/2020 18:43:52 Coordinated Universal Time: Error : 1 :[devenv:1868] Unexpected exception: StreamJsonRpc.RemoteInvocationException: database table is locked: DocumentData3
database table is locked
   at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__123`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at Microsoft.CodeAnalysis.Remote.BrokeredServiceConnection`1.<TryInvokeAsync>d__10`1.MoveNext() in /_/src/Workspaces/Remote/Core/BrokeredServiceConnection.cs:line 114
RPC server exception:
Microsoft.CodeAnalysis.SQLite.Interop.SqlException: database table is locked: DocumentData3
database table is locked
      at Microsoft.CodeAnalysis.SQLite.v2.Interop.SqlConnection.Throw(SafeSqliteHandle handle, Result result) in /_/src/Workspaces/Core/Portable/Storage/SQLite/v2/Interop/SqlConnection.cs:line 326
      at Microsoft.CodeAnalysis.SQLite.v2.SQLitePersistentStorage.Accessor`3.InsertOrReplaceBlobIntoWriteCache(SqlConnection connection, TDatabaseId dataId, Byte[] checksumBytes, Int32 checksumLength, Byte[] dataBytes, Int32 dataLength) in /_/src/Workspaces/Core/Portable/Storage/SQLite/v2/SQLitePersistentStorage.Accessor.cs:line 288
      at Microsoft.CodeAnalysis.SQLite.v2.SQLitePersistentStorage.Accessor`3.WriteStream(TKey key, Stream stream, Checksum checksumOpt, CancellationToken cancellationToken) in /_/src/Workspaces/Core/Portable/Storage/SQLite/v2/SQLitePersistentStorage.Accessor.cs:line 140
      at Microsoft.CodeAnalysis.SQLite.v2.SQLitePersistentStorage.Accessor`3.<>c.<WriteStreamAsync>b__17_0(ValueTuple`5 t) in /_/src/Workspaces/Core/Portable/Storage/SQLite/v2/SQLitePersistentStorage.Accessor.cs:line 126
      at Microsoft.CodeAnalysis.PooledObjects.PooledDelegates.FuncWithBoundArgument`2.<Bind>b__0_0() in /_/src/Dependencies/PooledObjects/PooledDelegates.cs:line 399
      at System.Threading.Tasks.Task`1.InnerInvoke()
      at System.Threading.Tasks.Task.Execute()
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.SQLite.v2.SQLitePersistentStorage.<PerformTaskAsync>d__80`2.MoveNext() in /_/src/Workspaces/Core/Portable/Storage/SQLite/v2/SQLitePersistentStorage_Threading.cs:line 51
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.SyntaxTreeIndex.<SaveAsync>d__69.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/SyntaxTree/SyntaxTreeIndex_Persistence.cs:line 87
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.SyntaxTreeIndex.<GetIndexWorkerAsync>d__12.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/SyntaxTree/SyntaxTreeIndex.cs:line 124
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.SyntaxTreeIndex.<GetIndexAsync>d__11.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/SyntaxTree/SyntaxTreeIndex.cs:line 79
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at Microsoft.CodeAnalysis.FindSymbols.DependentTypeFinder.ProjectIndex.<CreateIndexAsync>d__8.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/FindReferences/DependentTypeFinder_ProjectIndex.cs:line 61
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.DependentTypeFinder.<AddDescendantSourceTypesInProjectAsync>d__20.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/FindReferences/DependentTypeFinder.cs:line 475
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.DependentTypeFinder.<DescendInheritanceTreeInProjectAsync>d__7.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/FindReferences/DependentTypeFinder.cs:line 199
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at Microsoft.CodeAnalysis.FindSymbols.DependentTypeFinder.<DescendInheritanceTreeAsync>d__6.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/FindReferences/DependentTypeFinder.cs:line 122
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.DependentTypeFinder.<FindTypesInCurrentProcessAsync>d__30.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/FindReferences/DependentTypeFinder_Remote.cs:line 80
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   --- End of stack trace from previous location where exception was thrown ---
      at Microsoft.CodeAnalysis.FindSymbols.SymbolFinder.<FindDerivedClassesArrayAsync>d__57.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/SymbolFinder_Hierarchy.cs:line 217
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.DependentTypeFinder.<FindTypesAsync>d__29.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/FindReferences/DependentTypeFinder_Remote.cs:line 51
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.SymbolFinder.<FindDerivedClassesArrayAsync>d__57.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/SymbolFinder_Hierarchy.cs:line 217
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.SymbolFinder.<FindDerivedClassesAsync>d__56.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/SymbolFinder_Hierarchy.cs:line 209
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.SymbolFinder.<FindImplementedInterfaceMembersArrayAsync>d__54.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/SymbolFinder_Hierarchy.cs:line 124
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.Finders.AbstractMethodOrPropertyOrEventSymbolReferenceFinder`1.<DetermineCascadedSymbolsAsync>d__1.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/FindReferences/Finders/AbstractMethodOrPropertyOrEventSymbolReferenceFinder.cs:line 41
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.Finders.OrdinaryMethodReferenceFinder.<DetermineCascadedSymbolsAsync>d__1.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/FindReferences/Finders/OrdinaryMethodReferenceFinder.cs:line 39
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.FindReferencesSearchEngine.<>c__DisplayClass20_1.<<DetermineAllSymbolsCoreAsync>b__0>d.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/FindReferences/FindReferencesSearchEngine_MapCreation.cs:line 148
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.FindReferencesSearchEngine.<DetermineAllSymbolsCoreAsync>d__20.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/FindReferences/FindReferencesSearchEngine_MapCreation.cs:line 168
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.FindReferencesSearchEngine.<DetermineAllSymbolsAsync>d__19.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/FindReferences/FindReferencesSearchEngine_MapCreation.cs:line 113
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.SymbolFinder.<FindDerivedClassesAsync>d__56.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/SymbolFinder_Hierarchy.cs:line 209
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.FindReferencesSearchEngine.<FindReferencesAsync>d__8.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/FindReferences/FindReferencesSearchEngine.cs:line 57
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.SymbolFinder.<FindImplementedInterfaceMembersArrayAsync>d__54.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/SymbolFinder_Hierarchy.cs:line 124
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at Microsoft.CodeAnalysis.FindSymbols.FindReferencesSearchEngine.<FindReferencesAsync>d__8.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/FindReferences/FindReferencesSearchEngine.cs:line 63
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.Finders.AbstractMethodOrPropertyOrEventSymbolReferenceFinder`1.<DetermineCascadedSymbolsAsync>d__1.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/FindReferences/Finders/AbstractMethodOrPropertyOrEventSymbolReferenceFinder.cs:line 41
      at Microsoft.CodeAnalysis.FindSymbols.FindReferencesSearchEngine.<FindReferencesAsync>d__8.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/FindReferences/FindReferencesSearchEngine.cs:line 67
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.Finders.OrdinaryMethodReferenceFinder.<DetermineCascadedSymbolsAsync>d__1.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/FindReferences/Finders/OrdinaryMethodReferenceFinder.cs:line 39
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.SymbolFinder.<FindReferencesAsync>d__34.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/SymbolFinder_FindReferences_Current.cs:line 56
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.FindReferencesSearchEngine.<DetermineAllSymbolsCoreAsync>d__20.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/FindReferences/FindReferencesSearchEngine_MapCreation.cs:line 168
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.SymbolFinder.<FindReferencesAsync>d__39.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/SymbolFinder_FindReferences_Legacy.cs:line 80
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.FindReferencesSearchEngine.<DetermineAllSymbolsAsync>d__19.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/FindReferences/FindReferencesSearchEngine_MapCreation.cs:line 113
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.CodeLens.CodeLensReferencesService.<FindAsync>d__1`1.MoveNext() in /_/src/Features/Core/Portable/CodeLens/CodeLensReferencesService.cs:line 52
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.FindReferencesSearchEngine.<FindReferencesAsync>d__8.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/FindReferences/FindReferencesSearchEngine.cs:line 57
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at Microsoft.CodeAnalysis.FindSymbols.FindReferencesSearchEngine.<FindReferencesAsync>d__8.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/FindReferences/FindReferencesSearchEngine.cs:line 63
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.Remote.RemoteCodeLensReferencesService.<>c__DisplayClass5_0.<<FindReferenceMethodsAsync>b__0>d.MoveNext() in /_/src/Workspaces/Remote/ServiceHub/Services/CodeLensReferences/RemoteCodeLensReferencesService.cs:line 95
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at Microsoft.CodeAnalysis.FindSymbols.FindReferencesSearchEngine.<FindReferencesAsync>d__8.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/FindReferences/FindReferencesSearchEngine.cs:line 67
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.Remote.BrokeredServiceBase.<RunServiceAsync>d__11`1.MoveNext() in /_/src/Workspaces/Remote/ServiceHub/Services/BrokeredServiceBase.cs:line 69
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.SymbolFinder.<FindReferencesAsync>d__40.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/SymbolFinder_FindReferences_Legacy.cs:line 99
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.FindSymbols.SymbolFinder.<FindReferencesAsync>d__39.MoveNext() in /_/src/Workspaces/Core/Portable/FindSymbols/SymbolFinder_FindReferences_Legacy.cs:line 80
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.CodeLens.CodeLensReferencesService.<FindAsync>d__1`1.MoveNext() in /_/src/Features/Core/Portable/CodeLens/CodeLensReferencesService.cs:line 52
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.CodeAnalysis.Remote.RemoteCodeLensReferencesService.<>c__DisplayClass5_0.<<FindReferenceMethodsAsync>b__0>d.MoveNext() in /_/src/Workspaces/Remote/ServiceHub/Services/CodeLensReferences/RemoteCodeLensReferencesService.cs:line 95
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at System.Threading.Tasks.ValueTask`1.get_Result()
      at Microsoft.CodeAnalysis.Remote.BrokeredServiceBase.<RunServiceAsync>d__11`1.MoveNext() in /_/src/Workspaces/Remote/ServiceHub/Services/BrokeredServiceBase.cs:line 69
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at Microsoft.VisualStudio.Telemetry.WindowsErrorReporting.WatsonReport.GetClrWatsonExceptionInfo(Exception exceptionObject)
@sharwell sharwell added Area-IDE Integration-Test Test failures in roslyn-integration-CI labels Oct 2, 2020
@CyrusNajmabadi
Copy link
Member

This is bad, and should be impossible by design. Here's why.

  1. We're getting a SQLITE_LOCKED error being thrown in InsertOrReplaceBlobIntoWriteCache.
  2. SQLITE_LOCKED is documented at: https://www.sqlite.org/rescode.html#locked. There are two potential cases it lists.
    2.1 "The SQLITE_LOCKED result code indicates that a write operation could not continue because of a conflict within the same database connection ..."
    2.2 "The SQLITE_LOCKED result code indicates that a write operation could not continue because of ... a conflict with a different database connection that uses a shared cache. "
  3. 2.1 should not happen as we're only doing one thing with the DB connection. Connections are handed out for single purpose tasks, so we should not be able to conflict with anything there unless our pooling logic is bad and somehow a connection is shared among two tasks. Unlikely, but included for completeness.
  4. 2.2 We are using a shared-cache here. Indeed, that's what the "write cache" is. we explicitly attach is using attach database 'file::memory:?cache=shared' so that all our connections can use it. So this seems to be the likely case that is throwing. However:
  5. There are only two operations that mutate the write cache, and several that read from it. The two writing operations are InsertOrReplaceBlobIntoWriteCache and FlushInMemoryDataToDisk. However, both of these are supposed to only run on the exclusive portion of the ExclusiveConcurentPairScheduler. So they should never be able to conflict with other mutating calls. They shoudl also not conflict with any reading calls as they go on the Cuncurrent scheduler which should never run when the exclusive part runs.

This is extremely concerning, but i also can't explain it at all.

@CyrusNajmabadi CyrusNajmabadi mentioned this issue Oct 3, 2020
Merged
@CyrusNajmabadi
Copy link
Member

Have spent a couple of hours looking into this and i cannot rationalize how this is happening. All our DB accesses seem safe and are written as per the documented API. Have asked for the A/B test to be disabled in the meantime.

@sharwell sharwell mentioned this issue Oct 13, 2020
Merged
@jinujoseph jinujoseph added this to the 16.9 milestone Oct 15, 2020
@CyrusNajmabadi
Copy link
Member

Fixed with #48428

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@CyrusNajmabadi CyrusNajmabadi

Labels
Area-IDE Integration-Test Test failures in roslyn-integration-CI
Projects
None yet
Milestone
16.9
Development

No branches or pull requests
3 participants
@sharwell @CyrusNajmabadi @jinujoseph