Better feedback when gh pr create --reviewer fails due to fine-grained access token not having "Organization: Member" permissions

[Thunderforge]

CLI Feedback

Consider the following:

1. A fine-grained access token exists for a particular repo
2. That token has the "Repository: Pull requests" permission and nothing else
3. An end-user runs gh pr create […] --reviewer FooTeam to create a new PR with the FooTeam Team as a reviewer

In this case, the request fails with this message:

GraphQL: Resource not accessible by personal access token (organization.t000)

The problem

The error does not clearly indicate what the problem is and is confusing to debug.

The reason the request was rejected is that the fine-grained access token does not have the "Organization: Member" permission and thus cannot validate that the Team in question exists.

As-is, this is not communicated well. It is not immediately apparent that the request failed due to the --reviewer flag and that organization.t000 corresponds with FooTeam. No information is given about how to resolve the issue.

Possible solutions

Some possible alternative error messages that would clarify the situation:

Cannot set reviewers because personal access token does not have "Organization: Member" permissions

Personal access token is unable to access reviewer "FooTeam"

Reviewer not accessible by personal access token (FooTeam)

Personal access token cannot set reviewers due to lacking "Organization: Member" permissions

[williammartin]

Hi @Thunderforge, thanks for the feedback and sorry you ran into some confusing behaviour. I've collected the output with GH_DEBUG=api here for posterity:

➜  GH_DEBUG=api GH_TOKEN=<PAT_WITH_JUST_PR_PERMS> pr create --title wm/7978-triage --body test --reviewer "cli/codespaces"
[git remote -v]
[git config --get-regexp ^remote\..*\.gh-resolved$]
* Request at 2023-09-18 12:30:22.866187 +0200 CEST m=+0.084446543
* Request to https://api.github.com/graphql
> POST /graphql HTTP/1.1
> Host: api.github.com
> Accept: application/vnd.github.merge-info-preview+json, application/vnd.github.nebula-preview
> Authorization: token ████████████████████
> Content-Length: 508
> Content-Type: application/json; charset=utf-8
> Graphql-Features: merge_queue
> Time-Zone: Europe/Amsterdam
> User-Agent: GitHub CLI 2.34.0

GraphQL query:
fragment repo on Repository {
    id
    name
    owner { login }
    hasIssuesEnabled
    description
    hasWikiEnabled
    viewerPermission
    defaultBranchRef {
      name
    }
  }

  query RepositoryInfo($owner: String!, $name: String!) {
    repository(owner: $owner, name: $name) {
      ...repo
      parent {
        ...repo
      }
      mergeCommitAllowed
      rebaseMergeAllowed
      squashMergeAllowed
    }
  }
GraphQL variables: {"name":"cli","owner":"cli"}

< HTTP/2.0 200 OK
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
< Content-Security-Policy: default-src 'none'
< Content-Type: application/json; charset=utf-8
< Date: Mon, 18 Sep 2023 10:30:23 GMT
< Github-Authentication-Token-Expiration: 2023-09-25 12:21:26 +0200
< Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
< Server: GitHub.com
< Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
< Vary: Accept-Encoding, Accept, X-Requested-With
< X-Content-Type-Options: nosniff
< X-Frame-Options: deny
< X-Github-Media-Type: github.merge-info-preview; param=nebula-preview; format=json
< X-Github-Request-Id: C790:4EFE:16881252:16C13E4A:650826BE
< X-Ratelimit-Limit: 5000
< X-Ratelimit-Remaining: 4941
< X-Ratelimit-Reset: 1695035058
< X-Ratelimit-Resource: graphql
< X-Ratelimit-Used: 59
< X-Xss-Protection: 0

{
  "data": {
    "repository": {
      "id": "MDEwOlJlcG9zaXRvcnkyMTI2MTMwNDk=",
      "name": "cli",
      "owner": {
        "login": "cli"
      },
      "hasIssuesEnabled": true,
      "description": "GitHub’s official command line tool",
      "hasWikiEnabled": false,
      "viewerPermission": "ADMIN",
      "defaultBranchRef": {
        "name": "trunk"
      },
      "parent": null,
      "mergeCommitAllowed": true,
      "rebaseMergeAllowed": true,
      "squashMergeAllowed": true
    }
  }
}

* Request took 574.875333ms
[git symbolic-ref --quiet HEAD]
[git status --porcelain]
[git config --get-regexp ^branch\.wm/7978-triage\.(remote|merge)$]
[git show-ref --verify -- HEAD refs/remotes/origin/wm/7978-triage refs/remotes/origin/wm/7978-triage]
[git remote -v]
[git config --get-regexp ^remote\..*\.gh-resolved$]
* Request at 2023-09-18 12:30:23.569517 +0200 CEST m=+0.787782710
* Request to https://api.github.com/graphql
> POST /graphql HTTP/1.1
> Host: api.github.com
⣾> Accept: application/vnd.github.merge-info-preview+json, application/vnd.github.nebula-preview
> Authorization: token ████████████████████
> Content-Length: 572
> Content-Type: application/json; charset=utf-8
> Graphql-Features: merge_queue
> Time-Zone: Europe/Amsterdam
> User-Agent: GitHub CLI 2.34.0

GraphQL query:
query PullRequestForBranch($owner: String!, $repo: String!, $headRefName: String!, $states: [PullRequestState!]) {
    repository(owner: $owner, name: $repo) {
      pullRequests(headRefName: $headRefName, states: $states, first: 30, orderBy: { field: CREATED_AT, direction: DESC }) {
        nodes {url,id,number,state,baseRefName,headRefName,isCrossRepository,headRepositoryOwner{id,login,...on User{name}}}
      }
      defaultBranchRef { name }
    }
  }
GraphQL variables: {"headRefName":"wm/7978-triage","owner":"cli","repo":"cli","states":["OPEN"]}

⢿< HTTP/2.0 200 OK
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
< Content-Security-Policy: default-src 'none'
< Content-Type: application/json; charset=utf-8
< Date: Mon, 18 Sep 2023 10:30:23 GMT
< Github-Authentication-Token-Expiration: 2023-09-25 12:21:26 +0200
< Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
< Server: GitHub.com
< Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
< Vary: Accept-Encoding, Accept, X-Requested-With
< X-Content-Type-Options: nosniff
< X-Frame-Options: deny
< X-Github-Media-Type: github.merge-info-preview; param=nebula-preview; format=json
< X-Github-Request-Id: C790:4EFE:1688152E:16C1413A:650826BF
< X-Ratelimit-Limit: 5000
< X-Ratelimit-Remaining: 4940
< X-Ratelimit-Reset: 1695035058
< X-Ratelimit-Resource: graphql
< X-Ratelimit-Used: 60
< X-Xss-Protection: 0

{
  "data": {
    "repository": {
      "pullRequests": {
        "nodes": []
      },
      "defaultBranchRef": {
        "name": "trunk"
      }
    }
  }
}

* Request took 379.274958ms

Creating pull request for wm/7978-triage into trunk in cli/cli

* Request at 2023-09-18 12:30:23.951204 +0200 CEST m=+1.169472460
* Request to https://api.github.com/graphql
> POST /graphql HTTP/1.1
> Host: api.github.com
> Accept: application/vnd.github.merge-info-preview+json, application/vnd.github.nebula-preview
> Authorization: token ████████████████████
> Content-Length: 145
> Content-Type: application/json; charset=utf-8
> Graphql-Features: merge_queue
> Time-Zone: Europe/Amsterdam
> User-Agent: GitHub CLI 2.34.0

GraphQL query:
query RepositoryResolveMetadataIDs {
organization(login:"cli"){
t000: team(slug:"codespaces"){id,slug}
}
}
GraphQL variables: null

< HTTP/2.0 200 OK
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
< Content-Security-Policy: default-src 'none'
< Content-Type: application/json; charset=utf-8
< Date: Mon, 18 Sep 2023 10:30:24 GMT
< Github-Authentication-Token-Expiration: 2023-09-25 12:21:26 +0200
< Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
< Server: GitHub.com
< Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
< Vary: Accept-Encoding, Accept, X-Requested-With
< X-Content-Type-Options: nosniff
< X-Frame-Options: deny
< X-Github-Media-Type: github.merge-info-preview; param=nebula-preview; format=json
< X-Github-Request-Id: C790:4EFE:1688171B:16C1432F:650826BF
< X-Ratelimit-Limit: 5000
< X-Ratelimit-Remaining: 4939
< X-Ratelimit-Reset: 1695035058
< X-Ratelimit-Resource: graphql
< X-Ratelimit-Used: 61
< X-Xss-Protection: 0

{
  "data": {
    "organization": {
      "t000": null
    }
  },
  "errors": [
    {
      "type": "FORBIDDEN",
      "path": [
        "organization",
        "t000"
      ],
      "extensions": {
        "saml_failure": false
      },
      "locations": [
        {
          "line": 3,
          "column": 1
        }
      ],
      "message": "Resource not accessible by personal access token"
    }
  ]
}

* Request took 411.879958ms
GraphQL: Resource not accessible by personal access token (organization.t000)

And indeed, after providing read-only access to Organization: members this was successful.

What can we do?

As we can see from the GQL response, we don't get much back from the server in terms of missing permission information, and I would hesitate to try and recreate any of the resource checks within the CLI. That said, if we look at the error message:

GraphQL: Resource not accessible by personal access token (organization.t000)

And compare it to our query:

query RepositoryResolveMetadataIDs {
organization(login:"cli"){
t000: team(slug:"codespaces"){id,slug}
}
}

Perhaps we could provide a more obvious mapping between t000 and the codespaces slug.

I'm not aware of the historical context behind these t000 shaped names that were introduced in #908, so I'll need to do more digging.

[Thunderforge]

I think that a more obvious mapping between t000 and the codespaces slug would indeed help.

I've had some time to think about why I was confused about the error message.

GraphQL: Resource not accessible by personal access token (organization.t000)

The problems are:

1. It was not immediately clear which "resource" is being discussed
2. "Organization" did not bring to mind Teams
3. I didn't know that the "Organization: Member" permission even existed or was tied to PR reviewers

1. It was not immediately clear which "resource" is being discussed

Because I was trying to create a pull request, my assumption was that the pull request itself (or alternatively the branch or repo) was the resource it said not accessible. Therefore I spent lots of time double-checking that my token had the "Pull Requests" and "Contents" permissions.

This was probably a tunnel vision problem. I had a very specific goal and reviewers being at fault just wasn't on my radar. It's operator error, but if I wasn't making the mental link, others are likely to make the same.

(For what it's worth though: I did assume that "Pull Requests" permission would be sufficient for all flags to work as intended. I'm guessing it's a long shot, but if that could happen, that would be super helpful).

2. "Organization" did not bring to mind Teams

For instance, we have our private repo at FooCompany/BarRepo, and it's associated with "FooTeam" and "BazTeam".

I think I've seen "Organization" used to refer to what "FooCompany" is, but I don't think I've ever seen it applied to what "FooTeam" is. Therefore it seemed like organization.t000 was referring to the entire namespace of FooCompany, which probably meant some sort of repo access issue.

If instead it said team.t000, that might have brought to mind it faster. Of course, the exact name of the team would have been even better.

Edit: After I had written up this post, it finally hit me that the t in organization.t000 might stand for "Team". That certainly wasn't obvious to me at the time; it took me a full week after making my original post to make that connection! Had it said organization.team000, I still would have been confused about why the PR wasn't getting created, but I would have maybe figured it out faster.

3. I didn't know that the "Organization: Member" permission even existed or was tied to PR reviewers

So far, all of my investigation of fine-grained token permissions had been focused on the repository groups. I took one look at Organization, saw that it was for a bunch of stuff that didn't seem to apply to pull requests, and I ignored it.

Honestly, the way I figured out the need for that permission was:

1. Carefully check if I had repository permissions I thought I needed (e.g. "Repository: Pull Request"), only to see it fail
2. In desperation, enable all Repository permissions, only to see it fail
3. In desperation, enable all Organization permissions, which succeeded
4. Systematically turn them off until I figured out that I needed "Organization: Members"

Therefore, my ideal solution would be an error message that specifically tells you that the fine-grained token lacks the particular permission, referred to by name. The actionable item is clear: if you want to use the token for this operation, you need to give it that permission.

---

Anyway, I hope this feedback is helpful. Thank you for the detailed investigation you have done so far. Let me know if I can help in any other way.

[williammartin]

@Thunderforge this is an excellent write up and I really appreciate you taking the time. Indeed, the t in t000 does stand for team: https://github.com/cli/cli/blob/trunk/api/queries_repo.go#L979.

Personally, I'd like to understand what the consequence of changing this to be the actual team name would be in terms of our query. In this case, the error would read:

GraphQL: Resource not accessible by personal access token (organization.FooTeam)

Which would hopefully far more quickly address the very reasonable mental connections you made in points 1 and 2.

In terms of point 3, we are limited by the responses of the API. In good news, they do have a plan to provide more granular errors but there's no indication of when that work will actually occur. I have included this issue as motivation (so thanks for the detailed write up again).

[Thunderforge]

@williammartin Thanks for doing this, and for looking at if it's possible to make the change you want. If that can be made, I think that would satisfy this feature request. Had the error said

GraphQL: Resource not accessible by personal access token (organization.FooTeam)

I think I would have definitely figured out that the issue was that it couldn't access the team, rather than some access issue with the pull request as I had assumed (the "organization" would have still confused me, but it probably wouldn't have stopped me from figuring it out eventually).

I'm happy to hear that there are more granular errors planned for the future, and that my issue is helping to spur them on.

[williammartin]

I'm labelling this core as I think there is value in modifying the t000 schema to something that provides a more useful error message but I don't feel confident enough that there are no side effects of that to ask for help from the community without doing some more investigation (unless someone wants to spend the time looking into that separately, which I would welcome).

Thanks again for all the feedback @Thunderforge