Semantic tokens full failed after vs code update

[RaptorX]

Type: Bug

After updating to the latest version of VSCode I am getting the following warning:

---

---

The output looks like this:

Output log

[Info  - 9:33:27 AM] file:///s%3A/lib/joe/Gdip_All.ahk This might be a v1 script, and the lexer stops parsing. Try switching to AutoHotkey v1.
[Error - 9:33:30 AM] Request textDocument/semanticTokens/range failed.
  Message: Request textDocument/semanticTokens/range failed with message: Cannot read properties of undefined (reading 'previous_token')
  Code: -32603 
[Error - 9:33:30 AM] Request textDocument/semanticTokens/full failed.
  Message: Request textDocument/semanticTokens/full failed with message: Cannot read properties of undefined (reading 'previous_token')
  Code: -32603 
[Error - 9:33:32 AM] Request textDocument/documentSymbol failed.
  Message: Request textDocument/documentSymbol failed with message: Cannot read properties of undefined (reading 'previous_token')
  Code: -32603 
[Info  - 9:39:43 AM] file:///s%3A/lib/joe/Gdip_All.ahk This might be a v1 script, and the lexer stops parsing. Try switching to AutoHotkey v1.
[Error - 10:51:14 AM] Request textDocument/semanticTokens/full failed.
  Message: Request textDocument/semanticTokens/full failed with message: Cannot read properties of undefined (reading 'previous_token')
  Code: -32603 
[Error - 10:51:15 AM] Request textDocument/semanticTokens/full failed.
  Message: Request textDocument/semanticTokens/full failed with message: Cannot read properties of undefined (reading 'previous_token')
  Code: -32603 
[Error - 10:51:31 AM] Request textDocument/semanticTokens/full failed.
  Message: Request textDocument/semanticTokens/full failed with message: Cannot read properties of undefined (reading 'previous_token')
  Code: -32603 
[Error - 10:52:02 AM] Request textDocument/semanticTokens/full failed.
  Message: Request textDocument/semanticTokens/full failed with message: Cannot read properties of undefined (reading 'previous_token')
  Code: -32603 
[Error - 10:52:05 AM] Request textDocument/semanticTokens/full failed.
  Message: Request textDocument/semanticTokens/full failed with message: Cannot read properties of undefined (reading 'previous_token')
  Code: -32603 

---

The file im working with is a v2 script but the v1 file is opened in a different tab as seen here:

Extension version: 2.1.9
VS Code version: Code 1.83.0 (e7e037083ff4455cf320e344325dacb480062c3c, 2023-10-03T16:12:16.321Z)
OS version: Windows_NT x64 10.0.22621
Modes:

System Info

Item	Value
CPUs	Intel(R) Core(TM) i5-4460 CPU @ 3.20GHz (4 x 3198)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	undefined
Memory (System)	31.88GB (19.65GB free)
Process Argv	--crash-reporter-id 3e042e59-cb22-4371-968a-6f3e35abe666
Screen Reader	no
VM	0%

A/B Experiments

vsliv368cf:30146710
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscorecescf:30445987
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593cf:30376535
pythonvs932:30410667
vsclangdc:30486549
c4g48928:30535728
dsvsc012:30540252
pynewext54:30695312
azure-dev_surveyone:30548225
3biah626:30602489
f6dab269:30613381
showlangstatbar:30737416
0bi6i642:30841073
a2ce3375:30757347
pythonfmttext:30731395
fixshowwlkth:30771522
showindicator:30805244
pythongtdpath:30769146
i26e3531:30792625
pythonnosmt12:30797651
pythonidxpt:30805730
pythonnoceb:30805159
copilotsettingc:30839828
dsvsc013:30795093
dsvsc014:30804076
diffeditorv2:30821572
dsvsc015:30845448

[thqby]

Is there code to reproduce this issue.

[RaptorX]

It was happening with all files. It is only happening with 1 file.
I'll reboot and try again.

If the problem persists I'll send one of the files.

[RaptorX]

UPDATE:

The file that is having issues is a very complex one.
I am converting a v1 script to v2.

So if there's a mix of v1 and v2 code the lexer gets confused and crashes.
Im not sure if theres anything you can do about that.

Here is a sample file that reproduces the error on my end and on another developers computer.

Gdip(devv2).zip