Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Travis CI Configuration #94

Merged
merged 12 commits into from
Jan 23, 2018
8 changes: 8 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
language: python
python:
- "3.6"
install:
- pip install -r requirements.txt
script:
- pyflakes assigner
- pylint assigner
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's leave pylint commented out until I fix all the remaining warnings! I'll start working on that but it'll be a bit (got some 5201 stuff to do first).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already did it, sorry. It's soothing as hell.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're a madman!

2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
## Devel

- Removed lint, removed old baserepo standalone code, and added travis config.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Want to split this into two things? (lint/old code & travis)

I'm no stickler for "one change per PR"


## 1.0.0

- Rename `token` to `gitlab-token` in the configuration file
Expand Down
8 changes: 4 additions & 4 deletions assigner/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,11 +56,11 @@ def manage_repos(conf, args, action):
student_section = student["section"]
if "id" not in student:
logging.warning(
"Student {} does not have a gitlab account.".format(username)
"Student %s does not have a gitlab account.", username
)
continue
full_name = StudentRepo.name(semester, student_section,
hw_name, username)
full_name = StudentRepo.build_name(semester, student_section,
hw_name, username)

try:
repo = StudentRepo(host, namespace, full_name, token)
Expand Down Expand Up @@ -164,7 +164,7 @@ def main():
if args.tracebacks:
raise e
if isinstance(e, KeyError):
logger.error(str(e) + " is missing")
logger.error("%s is missing", e)
else:
logger.error(str(e))
raise SystemExit(1) from e
Expand Down
119 changes: 69 additions & 50 deletions assigner/baserepo.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@
import os
import re
import requests
import tempfile

from enum import Enum
from requests.exceptions import HTTPError
Expand Down Expand Up @@ -67,40 +66,56 @@ def from_url(cls, url, token):
namespace, name, token, url)

logging.debug(json.dumps(self.info))
logging.debug("{} is valid.".format(self.name_with_namespace))
logging.debug("%s is valid.", self.name_with_namespace)

return self

@classmethod
def _cls_gl_get(cls, url_base, path, token, params={}):
def _cls_gl_get(cls, url_base, path, token, params=None):
"""Make a Gitlab GET request"""
if not params:
params = {}

params.update({"private_token": token})
url = urljoin(url_base, "/api/v4" + path)
r = requests.get(url, params=params)
r.raise_for_status()
return r.json()

@classmethod
def _cls_gl_post(cls, url_base, path, token, payload={}, params={}):
def _cls_gl_post(cls, url_base, path, token, payload=None, params=None):
"""Make a Gitlab POST request"""
if not params:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd love a better way to do this. Per pylint, default parameters that are objects are shared across all invocations.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is true, although in this case it's not a problem and perhaps even cuts down on the amount of garbage allocated (although frankly I'm really not worried about that for Assigner 😄).

The reason why it's not a problem is that this function doesn't carry state in the default params or payload objects -- the default payload object just stays empty, and the default params object just has the private_token key which is updated every time around. I suppose we could worry about requests.post modifying them but it doesn't seem to right now.

In short: I'm ok with telling pylint to ignore that error in these cases.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, I will go back and update this.

params = {}
if not payload:
payload = {}

params.update({"private_token": token})
url = urljoin(url_base, "/api/v4" + path)
r = requests.post(url, params=params, data=payload)
r.raise_for_status()
return r.json()

@classmethod
def _cls_gl_put(cls, url_base, path, token, payload={}, params={}):
def _cls_gl_put(cls, url_base, path, token, payload=None, params=None):
"""Make a Gitlab PUT request"""
if not params:
params = {}
if not payload:
payload = {}

params.update({"private_token": token})
url = urljoin(url_base, "/api/v4" + path)
r = requests.put(url, params=params, data=payload)
r.raise_for_status()
return r.json()

@classmethod
def _cls_gl_delete(cls, url_base, path, token, params={}):
def _cls_gl_delete(cls, url_base, path, token, params=None):
"""Make a Gitlab DELETE request"""
if not params:
params = {}

params.update({"private_token": token})
url = urljoin(url_base, "/api/v4" + path)
r = requests.delete(url, params=params)
Expand Down Expand Up @@ -145,7 +160,8 @@ def info(self):
self._info = self._gl_get(url)
except HTTPError as e:
if e.response.status_code == 404:
logging.debug("Could not find repo with url {}/api/v4{}.".format(self.url_base,url))
logging.debug("Could not find repo with url %s/api/v4%s.",
self.url_base, url)
self._info = None
else:
raise
Expand Down Expand Up @@ -178,7 +194,7 @@ def get_head(self, branch):
if head.name == branch:
return head

return self.repo.create_head(b, "origin/{}".format(b))
return self.repo.create_head(branch, "origin/{}".format(branch))

def checkout(self, branch):
return self.get_head(branch).checkout()
Expand All @@ -190,18 +206,18 @@ def pull(self, branch):
self.repo.remote().pull(branch)

def clone_to(self, dir_name, branch=None):
logging.debug("Cloning {}...".format(self.ssh_url))
logging.debug("Cloning %s...", self.ssh_url)
try:
if branch:
self._repo = git.Repo.clone_from(self.ssh_url, dir_name,
branch=branch)
branch=branch)
for b in branch:
self._repo.create_head(b, "origin/{}".format(b))

logging.debug(self._repo.heads)
else:
self._repo = git.Repo.clone_from(self.ssh_url, dir_name)
logging.debug("Cloned {}.".format(self.name))
logging.debug("Cloned %s.", self.name)
except git.exc.GitCommandError as e:
# GitPython may delete this directory
# and the caller may have opinions about that,
Expand All @@ -213,40 +229,39 @@ def clone_to(self, dir_name, branch=None):

def add_local_copy(self, dir_name):
if self.repo is not None:
logging.warn("You already have a local copy associated with this repo")
logging.warning("You already have a local copy associated with this repo")
return

logging.debug("Using {} for the local repo...".format(dir_name))
logging.debug("Using %s for the local repo...", dir_name)
self._repo = git.Repo(dir_name)

def delete(self):
self._gl_delete("/projects/{}".format(self.id))
logging.debug("Deleted {}.".format(self.name))
logging.debug("Deleted %s.", self.name)

# TODO: this should really go elsewhere
@classmethod
def get_user_id(cls, username, url_base, token):
data = cls._cls_gl_get(url_base, "/users", token,
params={"search": username})

if len(data) == 0:
logging.warn(
"Did not find any users matching {}.".format(username)
if not data:
logging.warning(
"Did not find any users matching %s.", username
)
raise RepoError("No user {}.".format(username))

for result in data:
if result["username"] == username:
logging.info(
"Got id {} for user {}.".format(data[0]["id"], username)
"Got id %s for user %s.", data[0]["id"], username
)
return result["id"]

# Fall back to first result if all else fails
logging.warn("Got {} users for {}.".format(len(data), username))
logging.warn("Failed to find an exact match for {}.".format(username))
logging.warning("Got %s users for %s.", len(data), username)
logging.warning("Failed to find an exact match for %s.", username)
logging.info(
"Got id {} for user {}.".format(data[0]["id"], data[0]["username"])
"Got id %s for user %s.", data[0]["id"], data[0]["username"]
)
return data[0]["id"]

Expand Down Expand Up @@ -332,32 +347,51 @@ def archive(self):
def unarchive(self):
return self._gl_post("/projects/{}/unarchive".format(self.id))

def protect(self, branch="master", developer_push=True, developer_merge=True): # NOTE: these are not the same defaults that Gitlab uses
# NOTE: these are not the same defaults that Gitlab uses
def protect(self, branch="master", developer_push=True, developer_merge=True):
params = {
"developers_can_push": developer_push,
"developers_can_merge": developer_merge,
}
return self._gl_put("/projects/{}/repository/branches/{}/protect".format(self.id, branch), params)
}
return self._gl_put("/projects/{}/repository/branches/{}/protect"
.format(self.id, branch), params)

def unprotect(self, branch="master"):
return self._gl_put("/projects/{}/repository/branches/{}/unprotect".format(self.id, branch))
return self._gl_put("/projects/{}/repository/branches/{}/unprotect"
.format(self.id, branch))

def _gl_get(self, path, params=None):
if not params:
params = {}

def _gl_get(self, path, params={}):
return self.__class__._cls_gl_get(
self.url_base, path, self.token, params
)

def _gl_post(self, path, payload={}, params={}):
def _gl_post(self, path, payload=None, params=None):
if not payload:
payload = {}
if not params:
params = {}

return self.__class__._cls_gl_post(
self.url_base, path, self.token, payload
)

def _gl_put(self, path, payload={}, params={}):
def _gl_put(self, path, payload=None, params=None):
if not payload:
payload = {}
if not params:
params = {}

return self.__class__._cls_gl_put(
self.url_base, path, self.token, payload
)

def _gl_delete(self, path, params={}):
def _gl_delete(self, path, params=None):
if not params:
params = {}

return self.__class__._cls_gl_delete(
self.url_base, path, self.token, params
)
Expand All @@ -370,11 +404,11 @@ def new(cls, name, namespace, url_base, token):
namespaces = cls._cls_gl_get(url_base, "/namespaces",
token, {"search": namespace})
logging.debug(
"Got {} namespaces matching {}.".format(len(namespaces), namespace)
"Got %s namespaces matching %s.", len(namespaces), namespace
)
logging.debug(
"Using namespace " +
"{} with ID {}.".format(namespaces[0]["path"], namespaces[0]["id"])
"%s with ID %s.", namespaces[0]["path"], namespaces[0]["id"]
)

payload = {
Expand All @@ -396,7 +430,7 @@ def push_to(self, student_repo, branch="master"):
r = git.Remote.add(self.repo, student_repo.name,
student_repo.ssh_url)
r.push(branch)
logging.debug("Pushed {} to {}.".format(self.name, student_repo.name))
logging.debug("Pushed %s to %s.", self.name, student_repo.name)


class StudentRepo(Repo):
Expand All @@ -406,7 +440,7 @@ class StudentRepo(Repo):
def new(cls, base_repo, semester, section, username, token):
"""Create a new repository on GitLab"""
payload = {
"name": cls.name(semester, section, base_repo.name, username),
"name": cls.build_name(semester, section, base_repo.name, username),
"namespace_id": base_repo.namespace_id,
"issues_enabled": False,
"merge_requests_enabled": False,
Expand All @@ -422,7 +456,7 @@ def new(cls, base_repo, semester, section, username, token):
return cls.from_url(result["http_url_to_repo"], token)

@classmethod
def name(cls, semester, section, assignment, user):
def build_name(cls, semester, section, assignment, user):
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method had some weird conflict with Repo.name set in init. I highly doubt this ever caused a bug.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wouldn't work anyway... functions were missing args.

fmt = {
"semester": semester,
"section": section,
Expand All @@ -436,18 +470,3 @@ def name(cls, semester, section, assignment, user):
def push(self, base_repo, branch="master"):
"""Push base_repo code to this repo"""
base_repo.push_to(self, branch)


if __name__ == "__main__":
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assumed this wasn't being used and was for testing / from the old one file assigner.
@LinuxMercedes @michaelwisely confirm?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, that all can go!

# Need to make this module if'n you're going to run this
from secret import token

logging.basicConfig(level=logging.INFO)

b = BaseRepo("https://git.mst.edu/2016-Spring-CS-2001/hw01.git", token)
print(json.dumps(b.info, indent=8))
with tempfile.TemporaryDirectory() as tmpdirname:
b.clone_to(tmpdirname)
print(os.listdir(tmpdirname))

StudentRepo.new(b, "2016SP", "A", "mwwcp2", token)
8 changes: 4 additions & 4 deletions assigner/canvas.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,10 @@ def _request(self, method: str, url: str, params: dict = None, retries: int = 3)
header = self.REQUEST_HEADER
connection.request(method, url, (json.dumps(params) if params is not None else None), header)
return connection.getresponse()
except Exception as ex:
except http.client.HTTPException:
tries += 1
if tries > retries:
raise ex
raise
logging.warning("Caught exception in request after %d tries. Will retry %d more times.",
tries, retries - tries, exc_info=True)
time.sleep(1)
Expand Down Expand Up @@ -59,7 +59,7 @@ def _get_all_pages(self, url: str, params: dict = None) -> list:
count = len(result)
page = 1
while 'next' in links:
logging.debug("Got links:\n" + json.dumps(links, sort_keys=True, indent=2))
logging.debug("Got links:\n%s", json.dumps(links, sort_keys=True, indent=2))
page += 1
next_url = url.split('?')[0] + '?page=%s&per_page=%s' % (page, count)
logging.info("Getting next page for " + url + " via " + next_url)
Expand All @@ -71,7 +71,7 @@ def _get_all_pages(self, url: str, params: dict = None) -> list:

def get_instructor_courses(self):
get = lambda x: self._get_all_pages('/api/v1/courses',
{'enrollment_type': x, 'state': ['available']})
{'enrollment_type': x, 'state': ['available']})
result = get('teacher')
result.extend(get('ta'))
result.extend(get('grader'))
Expand Down
Loading