Galaxy server clean-up script along with some changes to galaxykit to support it #95

ironfroggy · 2023-06-27T16:07:20Z

Changes to galaxykit include:

The collection list command gains repository, limit, offset, keywords parameters
- repository: Can specify multiple repositories to list from. Default: "published"
- `limit': How many collections to return at maximum.
- offset: How many collections to skip, to allow paging.
- keywords: Filter results with a keyword search.
Added a new flag to all http methods: include_response. Causes to return both the parsed JSON and the original HttpResponse object.
get() can take a keyword argument "params" to pass querystring parameters easily.
delete_collection() has many changes to support the cleanup script:
- Takes a repositories parameter instead of a repository parameter.
- Takes a boolean dependencies parameter to recursively delete dependencies of the collection before deleting the collection itself.

Finally, of course, added the actual clean up script in scripts/galaxy-cleaner.sh.

galaxy-cleaner.sh
script usage: ./cleanup.sh [-l LIMIT] [-b BATCH_SIZE] [-d]

    -l          Limit. Number of collections to remove.
    -b          Batch size. Number of collections to fetch in each batch
                to then delete. Adjust this for performance and/or rate limit
                problems.
    -d          Debug flag. Enables bash -xv flags and displays extra debug
                information while running.
    -t          Token file. Path to a file to read the access token from.
    -T          Token. Directly pass the access token as a parameter.
    -p          Proxy. Set the HTTP_PROXY value to access the Galaxy API via.
    -w          Wait. Pause between steps to reduce server impact or avoid rate
                limiting.galaxy-cleaner.sh

…the cleanup script.

github-actions · 2023-06-27T16:07:54Z

galaxykit/client.py

+        include_response = kwargs.pop("include_response", False)
+
+        if include_response and not parse_json:
+            raise ValueError("GalaxyClient._http() called with include_response=True only valid when parse_json=True!")


[black] _{reported by reviewdog 🐶}

Suggested change

raise ValueError("GalaxyClient._http() called with include_response=True only valid when parse_json=True!")

raise ValueError(

"GalaxyClient._http() called with include_response=True only valid when parse_json=True!"

)

github-actions · 2023-06-27T16:07:54Z

galaxykit/collections.py

-def get_collection_list(client):
-    url = "_ui/v1/collection-versions/?limit=999999"
-    return client.get(url)
+def get_collection_list(client, repo="published", limit=None, offset=None, keywords=None):


[black] _{reported by reviewdog 🐶}

Suggested change

def get_collection_list(client, repo="published", limit=None, offset=None, keywords=None):

def get_collection_list(

client, repo="published", limit=None, offset=None, keywords=None

):

github-actions · 2023-06-27T16:07:54Z

galaxykit/collections.py

@@ -201,19 +216,75 @@ def move_or_copy_collection(


 def delete_collection(
-    client, namespace, collection, version=None, repository="published"
+    client, namespace, collection, version=None, repositories=("published",), dependents=False


[black] _{reported by reviewdog 🐶}

Suggested change

client, namespace, collection, version=None, repositories=("published",), dependents=False

client,

namespace,

collection,

version=None,

repositories=("published",),

dependents=False,

github-actions · 2023-06-27T16:07:54Z

galaxykit/collections.py

+            resp = client.delete(delete_url)
+        except GalaxyClientError as e:
+            if e.response.status_code == 404:
+                logger.debug(f"Ignoring (maybe) already deleted {coll_name}", file=sys.stderr)


[black] _{reported by reviewdog 🐶}

Suggested change

logger.debug(f"Ignoring (maybe) already deleted {coll_name}", file=sys.stderr)

logger.debug(

f"Ignoring (maybe) already deleted {coll_name}", file=sys.stderr

)

github-actions · 2023-06-27T16:07:55Z

galaxykit/collections.py

+                    combined = {
+                        "original": None,
+                        "dependents": []
+                    }


[black] _{reported by reviewdog 🐶}

Suggested change

combined = {

"original": None,

"dependents": []

}

combined = {"original": None, "dependents": []}

github-actions · 2023-06-27T16:07:55Z

galaxykit/collections.py

+                    for dep in resp["dependent_collection_versions"]:
+                        dep_coll, dep_ver = dep.split(" ")
+                        dep_ns, dep_name = dep_coll.split(".")
+                        dep_resp = delete_collection(client, dep_ns, dep_name, dep_ver, repositories, dependents=True)


[black] _{reported by reviewdog 🐶}

Suggested change

dep_resp = delete_collection(client, dep_ns, dep_name, dep_ver, repositories, dependents=True)

dep_resp = delete_collection(

client,

dep_ns,

dep_name,

dep_ver,

repositories,

dependents=True,

)

github-actions · 2023-06-27T16:07:55Z

galaxykit/collections.py

+        final_resp["responses"].append({
+            "collection": coll_name,
+            "response": resp,
+        })


[black] _{reported by reviewdog 🐶}

Suggested change

final_resp["responses"].append({

"collection": coll_name,

"response": resp,

})

final_resp["responses"].append(

{

"collection": coll_name,

"response": resp,

}

)

github-actions · 2023-06-27T16:07:55Z

galaxykit/command.py

@@ -489,7 +526,15 @@ def report_error(resp):

 def parse_args(parser, args):
    for arg in args:
-        parser.add_argument(arg, **(args[arg]))
+        flag_args=[]


[black] _{reported by reviewdog 🐶}

Suggested change

flag_args=[]

flag_args = []

github-actions · 2023-06-27T16:07:55Z

galaxykit/command.py

+            flag_args.append("--" + args[arg].pop("long").strip("-"))
+        if not flag_args:
+            flag_args.append(arg)
+


[black] _{reported by reviewdog 🐶}

Suggested change

github-actions · 2023-06-27T16:07:56Z

galaxykit/command.py

                    args.namespace,
                    args.collection,
                    args.version,
-                    args.repository or "published",
+                    args.repository.split(","),
+                    args.dependents


[black] _{reported by reviewdog 🐶}

Suggested change

args.dependents

args.dependents,

himdel · 2023-07-21T02:17:14Z

This is causing a lot of failures in UI tests (https://github.com/ansible/ansible-hub-ui/actions/runs/5617442384/job/15221548810?pr=4013#step:30:464)

but looks like the primary cause is the change to galaxykit collection list:

changes data from list of latest collection versions to a list of collections with a latest_version
no longer includes repository_list
no longer includes collections from other repositories (such as rejected)
default page_size went from 100 to 10

Which makes collection list less straightforward to use for finding and cleaning up all collections.

Is that an intentional change?
(If so, updating tests in ansible/ansible-hub-ui#4013)

(Also, if the goal is to update the API endpoint, shouldn't this be using v3/plugin/ansible/search/collection-versions/?)

Expanding the capability of the collection delete command for use by …

56d8d73

…the cleanup script.

github-actions bot reviewed Jun 27, 2023

View reviewed changes

ironfroggy force-pushed the collection-delete branch from 9a13c57 to 56d8d73 Compare June 27, 2023 16:08

Add help text to cleanup script.

396ebb0

chr-stian requested review from himdel, rochacbruno, jctanner, appuk, MilanPospisil, drodowic and hendersonreed July 20, 2023 08:48

himdel mentioned this pull request Jul 23, 2023

test galaxykit#95 ansible/ansible-hub-ui#4013

Closed

himdel removed request for himdel and hendersonreed January 26, 2024 21:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Galaxy server clean-up script along with some changes to galaxykit to support it #95

Galaxy server clean-up script along with some changes to galaxykit to support it #95

ironfroggy commented Jun 27, 2023 •

edited

Loading

github-actions bot Jun 27, 2023

github-actions bot Jun 27, 2023

github-actions bot Jun 27, 2023

github-actions bot Jun 27, 2023

github-actions bot Jun 27, 2023

github-actions bot Jun 27, 2023

github-actions bot Jun 27, 2023

github-actions bot Jun 27, 2023

github-actions bot Jun 27, 2023

github-actions bot Jun 27, 2023

himdel commented Jul 21, 2023 •

edited

Loading

Galaxy server clean-up script along with some changes to galaxykit to support it #95

Are you sure you want to change the base?

Galaxy server clean-up script along with some changes to galaxykit to support it #95

Conversation

ironfroggy commented Jun 27, 2023 • edited Loading

github-actions bot Jun 27, 2023

Choose a reason for hiding this comment

github-actions bot Jun 27, 2023

Choose a reason for hiding this comment

github-actions bot Jun 27, 2023

Choose a reason for hiding this comment

github-actions bot Jun 27, 2023

Choose a reason for hiding this comment

github-actions bot Jun 27, 2023

Choose a reason for hiding this comment

github-actions bot Jun 27, 2023

Choose a reason for hiding this comment

github-actions bot Jun 27, 2023

Choose a reason for hiding this comment

github-actions bot Jun 27, 2023

Choose a reason for hiding this comment

github-actions bot Jun 27, 2023

Choose a reason for hiding this comment

github-actions bot Jun 27, 2023

Choose a reason for hiding this comment

himdel commented Jul 21, 2023 • edited Loading

ironfroggy commented Jun 27, 2023 •

edited

Loading

himdel commented Jul 21, 2023 •

edited

Loading