Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Clarify <old_text> requirement and difference between create/update in suggest_edits prompt. #22340

Closed
1 task done
mattprodani opened this issue Dec 21, 2024 · 0 comments · Fixed by #22341
Closed
1 task done

Clarify <old_text> requirement and difference between create/update in suggest_edits prompt. #22340

mattprodani opened this issue Dec 21, 2024 · 0 comments · Fixed by #22341
Labels
ai Improvement related to Assistant, Copilot, or other AI features assistant AI feedback for Assistant (inline or panel) enhancement [core label]

Comments

@mattprodani
Copy link
Contributor

mattprodani commented Dec 21, 2024 •
edited
Loading

Check for existing issues

  • Completed

Describe the feature

<old_text> is marked optional in suggest edits prompt:

zed/assets/prompts/suggest_edits.hbs

Line 16 in 6b92e0b

identifies a range within the file where the edit should occur. If this tag is not
.
This leads to the LLM occasionally attempting to suggest edits using the update operation, without providing <old_text>, when attempting to override the entire file. (From my experience with Claude 3, this happens very often). Opening this as a request to clarify that create can be used when replacing an entire file. Models often prefer to fully rewrite a file, rather than patch, possibly even more so with rewrite models in the future.

I feel this would be nice, as it is confusing for users new to zed to realize why their LLM edit suggestions are failing to process. There's no indication of the prompt used, syntax expected, or reasoning for the suggestion not appearing correctly to the user.

note: I realized after looking at the code, this isn't used for delete either, which I assumed was true due to If this tag is not specified, then the entire file will be used as the range. I guess that's likely the culprit

Zed Version and System Specs

mac / unrelated

If applicable, add mockups / screenshots to help present your vision of the feature

No response
@mattprodani mattprodani added admin read Pending admin review enhancement [core label] triage Maintainer needs to classify the issue labels Dec 21, 2024
@mattprodani mattprodani mentioned this issue Dec 21, 2024
Merged
@github-actions github-actions bot mentioned this issue Dec 23, 2024
Open
github-merge-queue bot pushed a commit that referenced this issue Jan 9, 2025
@mattprodani
Update suggest_edits prompt to clarify usage of <old_text> when u…
a9d2628 
…sing update/create operations (#22341)

Update `suggest_edits` prompt to clarify usage of `<old_text>` when
using update/create operations using update/create operations.

- Add a mention that `old_text` is required for all but create.
- Change definition of `create` operation to also mean overwrite, as
some models heavily prefer rewrites.
- Remove mention of `If this tag is not specified, then the entire file
will be used as the range.` which is not current behavior.


Closes #22340

Release Notes:

- N/A (not sure if this requires a release note)
@JosephTLyons JosephTLyons added ai Improvement related to Assistant, Copilot, or other AI features assistant AI feedback for Assistant (inline or panel) and removed triage Maintainer needs to classify the issue admin read Pending admin review labels Jan 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ai Improvement related to Assistant, Copilot, or other AI features assistant AI feedback for Assistant (inline or panel) enhancement [core label]
Projects
None yet
Milestone
No milestone
2 participants
@JosephTLyons @mattprodani