Extension enters duplicate data into local globalStorage sqlite db

[mattseddon]

Environment data

• VS Code version: 1.60.1
• Extension version (available under the Extensions sidebar): v2021.9.1246542782
• OS and version: Darwin x64 20.6.0
• Python version (& distribution if applicable, e.g. Anaconda): Multiple (pyenv)
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): Multiple (all three listed)
• Relevant/affected Python packages and their versions: N/A
• Relevant/affected Python-related VS Code extensions and their versions: ms-python.python
• Value of the python.languageServer setting: Pylance

Expected behaviour

Extension does not fill the local globalStorage sqlite db (Memento) with erroneous data and maintains a list of available python interpreters.

When left unchecked this can lead to the extension host behaving in unexpected ways and it was difficult to diagnose without significant effort (see #17432 for original issue).

Actual behaviour

Each reload of a window causes PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS under the ms-python.python key to grow. Under most circumstances the list of environments underneath is not refreshed, i.e environments that are no longer on the machine are still shown.

This is a copy of the ms-python.python key before I managed to clear the cache by deleting the extension and going through the steps listed here:

globalStorage.txt

Note: There are 712 matches to INTERPRETERS_CACHE in that file.

Steps to reproduce:

Unsure how to recreate the stale environments issue (seems cache related) but for the ever growing list of PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS:

1. Completely remove extension from machine.
2. delete from ItemTable where key = 'ms-python.python'; in ~/Library/Application\ Support/Code/User/globalStorage/state.vscdb (mac).
3. Confirm key is empty in db.
4. Reinstall extension.
5. Reload window.
6. Note entry under ms-python.python in the db.
7. Reload window.
8. Note entry under ms-python.python in the db (PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS will have grown in size and contain duplicate entries).

Logs

Nothing of note is captured in the logs.

[karrtikr]

Thanks for the bug report! We just wanted to quickly acknowledge we received it and we will triage this as soon as we can.

[karrtikr]

Thank you @mattseddon for investigating the cause, that must not have been easy so I appreciate it very much! The issue should be fixed once #17598 is merged, you're welcome to have a look.

[karrtikr]

Can you verify if the following VSIX fixes the database issue for you: https://github.com/microsoft/vscode-python/suites/3936300663/artifacts/98621813? Use Extensions: Install from VSIX command to install this VSIX.

[mattseddon]

@karrtikr I have just tested this and it does appear to have made things better. I can see that PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS has been deprecated:

"PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS": []

I am still seeing duplicates being created in the PYTHON_GLOBAL_STORAGE_KEYS array though:

	"PYTHON_GLOBAL_STORAGE_KEYS": [{
		"key": "PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS",
		"defaultValue": []
	}, {
		"key": "PYTHON_ENV_INFO_CACHE"
	}, {
		"key": "PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS",
		"defaultValue": []
	}, {
		"key": "PYTHON_ENV_INFO_CACHE"
	}, {
		"key": "PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS",
		"defaultValue": []
	}, {
		"key": "PYTHON_ENV_INFO_CACHE"
	}, {
		"key": "SWITCH_LS"
	}, {
		"key": "isGlobalSettingCopiedKey",
		"defaultValue": false
	}, {
		"key": "workspaceFolderKeysForWhichTheCopyIsDone_Key",
		"defaultValue": []
	}, {
		"key": "PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS",
		"defaultValue": []
	}, {
		"key": "ShowBanner",
		"defaultValue": true
	}, {
		"key": "JoinMailingListPrompt",
		"defaultValue": false
	}, {
		"key": "PythonTensorBoardWebviewPreferredViewGroup",
		"defaultValue": -1
	}, {
		"key": "INSIDERS_PROMPT_STATE_KEY",
		"defaultValue": false
	}, {
		"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/example-get-started"
	}, {
		"key": "preferredGlobalPyInterpreter"
	}, {
		"key": "PYTHON_ENV_INFO_CACHE"
	}]

I don't think that you can close the ticket using the current fix.

[karrtikr]

Thanks for reporting back. The VSIX I provided initially is actually outdated as PR got updated further, can you try https://github.com/microsoft/vscode-python/suites/3954255787/artifacts/99265047 which appears to be the latest VSIX on the PR. Alternatively, you can try Python: Switch to Insiders command and wait a bit to install the latest insiders version of the extension.

[karrtikr]

FYI I can verify there's still an issue with PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS, however I expect the key PYTHON_ENV_INFO_CACHE to not appear multiple times.

[mattseddon]

I am also seeing the issue with PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS. Maybe the other issue is coming from the first vsix that I installed. I have 4 entries now but it appears to be stable:

[
  {
    "key": "INSIDERS_PROMPT_STATE_KEY",
    "defaultValue": false
  },
  {
    "key": "JoinMailingListPrompt",
    "defaultValue": false
  },
  {
    "key": "PYTHON_ENV_INFO_CACHE"
  },
  {
    "key": "PYTHON_ENV_INFO_CACHE"
  },
  {
    "key": "PYTHON_ENV_INFO_CACHE"
  },
  {
    "key": "PYTHON_ENV_INFO_CACHE"
  },
  {
    "key": "PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS",
    "defaultValue": []
  },
  {
    "key": "PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS",
    "defaultValue": []
  },
  {
    "key": "PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS",
    "defaultValue": []
  },
  {
    "key": "PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS",
    "defaultValue": []
  },
  {
    "key": "PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS",
    "defaultValue": []
  },
  {
    "key": "PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS",
    "defaultValue": []
  },
  {
    "key": "PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS",
    "defaultValue": []
  },
  {
    "key": "PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS",
    "defaultValue": []
  },
  {
    "key": "PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS",
    "defaultValue": []
  },
  {
    "key": "PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS",
    "defaultValue": []
  },
  {
    "key": "PythonTensorBoardWebviewPreferredViewGroup",
    "defaultValue": -1
  },
  {
    "key": "SWITCH_LS"
  },
  {
    "key": "ShowBanner",
    "defaultValue": true
  },
  {
    "key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/example-get-started"
  },
  {
    "key": "isGlobalSettingCopiedKey",
    "defaultValue": false
  },
  {
    "key": "preferredGlobalPyInterpreter"
  },
  {
    "key": "workspaceFolderKeysForWhichTheCopyIsDone_Key",
    "defaultValue": []
  }
]

The list of PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS seems to be ever growing.

[karrtikr]

I see, thanks for confirming. You can use the command Python: Clear Internal Extension cache to clear this storage anytime as well.

[karrtikr]

@mattseddon Can you please try https://github.com/microsoft/vscode-python/suites/3965062360/artifacts/99642505, thanks!

[mattseddon]

@karrtikr that seems to have fixed the issue.