Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

users prune - Paginate graphql site users - fix bug for soft deleted users #909

Merged
merged 10 commits into from
Jan 16, 2023

Conversation

DaedalusG
Copy link
Contributor

@DaedalusG DaedalusG commented Jan 4, 2023

This PR fixes an issue where user removal graphQL queries were issued to delete users who had already been soft deleted (these users are returned by the site.users graphql api)

It also adds pagination to pull 100 users per graphql request until all users in site.users have been checked against removal conditions

For a better UX on large instances printing a table of users to be deleted during the confirmation check to delete users has been placed behind a flag

This PR is primarily a bug fix since currently the command is broken -- further documentation of the command will be added when its more robust (for instance allowing input of a json user exclusion list, and the addition of unit tests)

λ ~/src-cli/ paginate-users-prune* src users prune -h
Usage of 'src users prune':
  -days int
    	Days threshold on which to remove users, must be 60 days or greater and defaults to this value  (default 60)
  -display-users
    	display table of users to be deleted by prune
  -dump-requests
    	Log GraphQL requests and responses to stdout
  -force
    	skips user confirmation step allowing programmatic use
  -get-curl
    	Print the curl command for executing this query and exit (WARNING: includes printing your access token!)
  -insecure-skip-verify
    	Skip validation of TLS certificates against trusted chains
  -remove-admin
    	prune admin accounts
  -remove-null-users
    	removes users with no last active value
  -trace
    	Log the trace ID for requests. See https://docs.sourcegraph.com/admin/observability/tracing
  -user-agent-telemetry
    	Include the operating system and architecture in the User-Agent sent with requests to Sourcegraph (default true)

This command removes users from a Sourcegraph instance who have been inactive for 60 or more days. Admin accounts are omitted by default.

Examples:

	$ src users prune -days 182

	$ src users prune -remove-admin -remove-null-users

λ ~/src-cli/ paginate-users-prune* src users prune
49 of 209 users were inactive for more than 60 days on https://k8s.sgdev.org.
Do you  wish to proceed with user removal [y/N]: n
Aborting removal
λ ~/src-cli/ paginate-users-prune* src users prune -display-users
Users to remove from https://k8s.sgdev.org
╭─────────────────────────┬─────────────────────────────────────────┬────────────────────────╮
│ USERNAME                │ EMAIL                                   │ DAYS SINCE LAST ACTIVE │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ rijnard                 │ rijnard@sourcegraph.com                 │                    202 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ christine               │ christine@sourcegraph.com               │                    211 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ christina               │ christina@sourcegraph.com               │                    124 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ mike                    │ mike@sourcegraph.com                    │                     91 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ jean                    │ jean@sourcegraph.com                    │                    206 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ loic                    │ loic@sourcegraph.com                    │                    117 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ jonah                   │ jonah@sourcegraph.com                   │                    154 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ mariam                  │ mariam@sourcegraph.com                  │                    120 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ emily                   │ emily@sourcegraph.com                   │                    198 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ maria                   │ maria@sourcegraph.com                   │                    232 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ afogg                   │ afogg@sourcegraph.com                   │                    211 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ ben.venker              │ ben.venker@sourcegraph.com              │                    145 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ felix.kling             │ felix.kling@sourcegraph.com             │                     89 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ beatrix-test            │ buildkite@perforce.sgdev.org            │                     82 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ mohammad.alam           │ mohammad.alam@sourcegraph.com           │                    173 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ elzanne.wentzel         │ elzanne.wentzel@sourcegraph.com         │                    170 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ chrispine               │ chrispine@sourcegraph.com               │                    196 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ nick                    │ nick@sourcegraph.com                    │                    148 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ filip.haftek            │ filip.haftek@sourcegraph.com            │                    146 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ alex.jean-baptiste      │ alex.jean-baptiste@sourcegraph.com      │                     84 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ michal.vrtiak           │ michal.vrtiak@sourcegraph.com           │                     91 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ taylor.sperry           │ taylor.sperry@sourcegraph.com           │                     75 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ justin.dorfman          │ justin.dorfman@sourcegraph.com          │                    100 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ ajay.sridhar            │ ajay.sridhar@sourcegraph.com            │                    238 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ nate.freng              │ nate.freng@sourcegraph.com              │                    230 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ tech-ops-admin          │ tech-ops-admin@sourcegraph.com          │                     79 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ michelle.veronese       │ michelle.veronese@sourcegraph.com       │                     68 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ jeff.wayman             │ jeff.wayman@sourcegraph.com             │                    218 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ marisa.kanemoto         │ marisa.kanemoto@sourcegraph.com         │                    208 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ rakesh.joshi            │ rakesh.joshi@sourcegraph.com            │                    218 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ kristen.stretch         │ kristen.stretch@sourcegraph.com         │                    239 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ qkeast                  │ qkeast@sourcegraph.com                  │                    105 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ joe.johnson             │ joe.johnson@sourcegraph.com             │                    236 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ james.cotter            │ james.cotter@sourcegraph.com            │                    189 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ isuru.fonseka           │ isuru.fonseka@sourcegraph.com           │                    231 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ aditya.kalia            │ aditya.kalia@sourcegraph.com            │                    198 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ dulani.wallace          │ dulani.wallace@sourcegraph.com          │                    230 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ nancy.shah              │ nancy.shah@sourcegraph.com              │                    225 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ megan.standrew          │ megan.standrew@sourcegraph.com          │                    203 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ quinn.hare              │ quinn.hare@sourcegraph.com              │                    188 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ tbliu98                 │ thomas.liu@sourcegraph.com              │                     69 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ vincent.ruijter.appsec1 │ vincent.ruijter+appsec1@sourcegraph.com │                    182 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ sanjeev.sekar           │ sanjeev.sekar@sourcegraph.com           │                    180 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ nathan.downs            │ nathan.downs@sourcegraph.com            │                    166 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ alex.isken              │ alex.isken@sourcegraph.com              │                    112 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ nate.maynard            │ nate.maynard@sourcegraph.com            │                     98 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ jordan.plahn            │ jordan.plahn@sourcegraph.com            │                     91 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ vdavid                  │ david.veszelovszki@sourcegraph.com      │                     71 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ jon.galindo             │ jon.galindo@sourcegraph.com             │                     70 │
╰─────────────────────────┴─────────────────────────────────────────┴────────────────────────╯
49 of 209 users were inactive for more than 60 days on https://k8s.sgdev.org.
Do you  wish to proceed with user removal [y/N]: n
Aborting removal

Test plan

This command was tested against our dogfood kubernetes instance and cse-aws docker compose instance.

…been deleted but exist in the aggregated_user_stats table
@DaedalusG DaedalusG changed the title Paginate for through users in users prune - fix bug for soft deleted users users prune - Paginate graphql site users - fix bug for soft deleted users Jan 4, 2023
@DaedalusG DaedalusG marked this pull request as ready for review January 11, 2023 23:03
cmd/src/users.go Show resolved Hide resolved
cmd/src/users_prune.go Show resolved Hide resolved
cmd/src/users_prune.go Show resolved Hide resolved
cmd/src/users_prune.go Show resolved Hide resolved
cmd/src/users_prune.go Outdated Show resolved Hide resolved
cmd/src/users_prune.go Outdated Show resolved Hide resolved
cmd/src/users_prune.go Outdated Show resolved Hide resolved
@DaedalusG DaedalusG enabled auto-merge (squash) January 14, 2023 07:52
@DaedalusG DaedalusG merged commit 2a63033 into main Jan 16, 2023
@DaedalusG DaedalusG deleted the paginate-users-prune branch January 16, 2023 02:25
jdpleiness pushed a commit that referenced this pull request Jan 19, 2023
…d users (#909)

* fixes bug in which request is made to delete users that have already been deleted but exist in the aggregated_user_stats table

* implemented some pagination -- totalCount of nodes is always 0 -- needing to fix that for break condition

* fixed totalCount issue -- switched DoRaw methods to Do

* remove development print statements; hide users table behind flag; improve comments

* gofmt

* added a few more comments

* style improvements

* better graphQL querry formatting for single use queries
scjohns pushed a commit that referenced this pull request Apr 24, 2023
…d users (#909)

* fixes bug in which request is made to delete users that have already been deleted but exist in the aggregated_user_stats table

* implemented some pagination -- totalCount of nodes is always 0 -- needing to fix that for break condition

* fixed totalCount issue -- switched DoRaw methods to Do

* remove development print statements; hide users table behind flag; improve comments

* gofmt

* added a few more comments

* style improvements

* better graphQL querry formatting for single use queries
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants