Introduce API to locate evaluatable expressions in debugged source

[weinand]

Today VS Code's debug hover uses a fixed regular expression based heuristic to locate the hover input for expressions under the mouse.

A similar strategy is used in two other places:

• for the "Create Watch expression" action which creates a new watch expression based on the mouse location,
• for showing variable values inline in the editor, the variables are found by a fixed regular expression.

The consequence of the fixed heuristics is that not all language idiosyncrasies are covered (e.g. see #84044 (comment), #89603, or #89332).

This API request should address these problems.

[weinand]

This proposal adds a new API that provides an expression for a given location in a document:

/**
* An EvaluatableExpression represents an expression in a document that can be evaluated by an active runtime or debugger.
* The result of this evaluation is shown in a tooltip-like widget.
* If only a range is specified, the expression will be extracted from the underlying document.
* An optional expression can be used to override the extracted expression.
* In this case the range is still used to highlight the range in the document.
*/
export interface EvaluatableExpression {
	/*
	 * The range is used to extract the evaluatable expression from the underlying document and to highlight it.
	 */
	range: Range;
	/*
	 * If specified the expression overrides the extracted expression.
	 */
	expression?: string;
}

/**
 */
export interface DebugEvaluatableExpressionProvider {

	/**
	 * Provide an evaluatable expression for the given position and document.
	 * The expression can be implicitly specified by the range in the underlying document or by explicitly returning an expression.
	 *
	 * @param document The document in which the command was invoked.
	 * @param position The position at which the command was invoked.
	 * @param token A cancellation token.
	 * @return A EvaluatableExpression or a thenable that resolves to such. The lack of a result can be
	 * signaled by returning `undefined` or `null`.
	 */
	provideEvaluatableExpression(document: TextDocument, position: Position, token: CancellationToken): ProviderResult<EvaluatableExpression>;
}

export namespace languages {
	/**
	 * Register a provider that locates evaluatable expressions in source.
	 *
	 * If multiple providers are registered for a language an arbitrary provider is used.
	 *
	 * @param selector A selector that defines the documents this provider is applicable to.
	 * @param provider An evaluatable expression provider.
	 * @return A [disposable](#Disposable) that unregisters this provider when being disposed.
	 */
	export function registerEvaluatableExpressionProvider(selector: DocumentSelector, provider: DebugEvaluatableExpressionProvider): Disposable;
}

[weinand]

One open issue is:
In what namespace does the registerEvaluatableExpressionProvider live?
Since it is debugger related, an obvious choice would be debug.
I picked languages because the implementation of a DebugEvaluatableExpressionProvider depends on the language only and is really independent from the debugger. So a language extension author would be expected to implement and register it, not a debugger extension author.

[weinand]

@DanTup @DonJayamanne @MSLaguana @WebFreak001 @andysterland @daviwil @delmyers @devoncarew @felixfbecker @gregg-miskelly @hbenl @lukaszunity @ramya-rao-a @rebornix @rkeithhill @roblourens @svaarala @vadimcn
FYI, any feedback would be greatly appreciated.

[gjsjohnmurray]

I picked languages because the implementation of a DebugEvaluatableExpressionProvider depends on the language only and is really independent from the debugger. So a language extension author would be expected to implement and register it, not a debugger extension author.

@weinand as an author of a language- and debug-extension I agree with your choice of the languages namespace for this.

[lukaszunity]

Ping @miniwolf who is the current owner of the Unity debugger extension for VS Code. Is there a page where the owners are listed, so it can be updated?

[weinand]

After a first round of feedback received, the new proposal is this:

/**
* An EvaluatableExpression represents an expression in a document that can be evaluated by an active runtime or debugger.
* The result of this evaluation is shown in a tooltip-like widget.
* If only a range is specified, the expression will be extracted from the underlying document.
* An optional expression can be used to override the extracted expression.
* In this case the range is still used to highlight the range in the document.
*/
export class EvaluatableExpression {
  /*
  * The range is used to extract the evaluatable expression from the underlying document and to highlight it.
  */
  readonly range: Range;
  /*
  * If specified the expression overrides the extracted expression.
  */
  readonly expression?: string;

  /**
   * Creates a new evaluatable expression object.
   *
   * @param range The range in the underlying document from which the evaluatable expression is extracted.
   * @param expression If specified overrides the extracted expression.
   */
  constructor(range: Range, expression?: string);
}

/**
 * The evaluatable expression provider interface defines the contract between extensions and
 * the debug hover.
 */
export interface EvaluatableExpressionProvider {

  /**
   * Provide an evaluatable expression for the given position and document.
   * The expression can be implicitly specified by the range in the underlying document or by explicitly returning an expression.
   *
   * @param document The document in which the debug hover is opened.
   * @param position The position in the document where the debug hover is opened.
   * @param token A cancellation token.
   * @return A EvaluatableExpression or a thenable that resolves to such. The lack of a result can be
   * signaled by returning `undefined` or `null`.
   */
  provideEvaluatableExpression(document: TextDocument, position: Position, token: CancellationToken): ProviderResult<EvaluatableExpression>;
}

export namespace languages {
  /**
   * Register a provider that locates evaluatable expressions in text documents.
   *
   * If multiple providers are registered for a language an arbitrary provider will be used.
   *
   * @param selector A selector that defines the documents this provider is applicable to.
   * @param provider An evaluatable expression provider.
   * @return A [disposable](#Disposable) that unregisters this provider when being disposed.
   */
  export function registerEvaluatableExpressionProvider(selector: DocumentSelector, provider: EvaluatableExpressionProvider): Disposable;
}

[hbenl]

I also agree that this should be in the languages namespace. Debug adapters usually don't need to "understand" the source code, whereas language providers always do.
The API looks exactly like what I would expect for this functionality.

[weinand]

I've released a first cut of the proposed API and its implementation. It will be in the next Insider (Monday 2/17).

Sample code for how to use the API can be found in Mock Debug.

Be aware that you cannot publish extensions to the marketplace that use proposed API.

[kalberes]

Works perfectly for the AL debug adapter. We finally have means to make evaluation possible also on hovering on various expressions (example: rhs on member access), quoted identifiers, etc.

[weinand]

Reopened for final discussion in API call.

[weinand]

The new evaluatable expression API is now official (no longer proposed).
It is available in Insiders and will appear in VS Code stable at the end of the month.

[paulo-fernando-silva]

@weinand , since the grammars used in syntax highlighting already provide information about what each block of text represents, it would be nice if we could use that information to also decide what can be evaluated.

For example, if the grammar states that this block of text is a comment, I want to cancel evaluation. Can I do this? How would one go about accessing that grammar information from the registerEvaluatableExpressionProvider?

Thanks

[weinand]

Typically an EvaluatableExpressionProvider is implemented in a language extension which should have access to language information, like comments.

A generic way for finding out whether a source range is a comment would be via DocumentSymbol.