Skip to content

Commit

Permalink
Added abstract backend base with Gitlab implementation (#102)
Browse files Browse the repository at this point in the history
* Fixed pyflakes errors, upgraded to 1.6.0.

* Added *Base backend, separated Gitlab backends. Added decorator to get backend.

* Changelog update.

Fixed pyflakes error about imports in __init__.py

Converted to @require_backend calling @config_context. Added basic mock backend.

Updated double decorator calls.

Removed changelog message about backend.

* Added MockBackend with config hooks.

Added hook for MockBackend in config.

Mock backend now works but doesn't do much.

Updated doc / references to base repo to now reference template repo.

Added linting error exception.

* Moved Access into backends, renamed config/backend decorators.

Moved Access into backend, fixed unit tests and remaining TODOs.

Renamed requires_backend to requires_backend_and_config

renamed config_context to requires_config.
Added changelog messages.

Added note about base repo -> template repo.

Send integer Access level rather than enum name

* Added catch to print warnings for backend missing in config.

Updated tests.

Rembed unneeded pyflakes override.

Ran sed on files to appease nitpicks.

Moved changelog to the correct spot.

Fixed linting error with asigner_test test case.

* Fixed merge conflicts, updated tests.

Removed artifact config.py

Fixed get integration test errors.

* Select backend based on backend name

* Split name-to-backend conversion into its own function and associated exception

* py3.4/3.5 compatible type annotations

* Add classmethod return type signatures

Also add a couple git ones that got missed.
  • Loading branch information
brhoades authored Jan 9, 2019
1 parent 28e0933 commit ace2561
Show file tree
Hide file tree
Showing 27 changed files with 643 additions and 123 deletions.
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@
- Make GitLab backend configuration generic
- Tidy up progress bars with logging output
- Test all supported python versions (3.4-3.7) automatically
- Added `BackendBase`, `RepoBase`, `StudentRepoBase`, and `TemplateRepoBase`.
- Base Repos are now called Template Repos.
- Separated Gitlab backend from most of the code. Added Base implementations for Gitlab.
- Added `requires_backend_and_config` decorator with backend config option to load the desired backend.
- Renamed `config_context` decorator to `requires_config`.
- pyflakes 1.6.0 upgrade for type annotations fixes.

## 1.2.0

Expand Down
14 changes: 7 additions & 7 deletions TUTORIAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,14 +101,14 @@ Lastly, you can remove students by running `assigner roster remove <student GitL

Once you have set Assigner up for your class, you can use it to make and assign homeworks and to fetch submissions from your students.

Each assignment has a **base repository** that you add assignment materials to.
The base repository is then copied to **student repositories**, one per student.
Each assignment has a **template repository** that you add assignment materials to.
The template repository is then copied to **student repositories**, one per student.

This section will walk you through making a new assignment named 'hw1' and performing various actions on it.

### Creating a new base repository
### Creating a new template repository

To create the base repository for the new assignment, run `assigner new hw1`.
To create the template repository for the new assignment, run `assigner new hw1`.
Assigner will create a new repository named 'hw1' in the GitLab group specified in `_config.yml`.
It then will print out both an HTTPS and an SSH URL that you can use to view this repo in the GitLab UI or to clone a local copy.

Expand All @@ -118,15 +118,15 @@ Commit your changes and push them to GitLab.

### Creating student repositories

Once you have created a base repo, you can make student repos from it by running `assigner assign hw1`.
Once you have created a template repo, you can make student repos from it by running `assigner assign hw1`.
This step *only* creates the repositories; it does not add the students to them.
You should open GitLab and verify that the students' repo contents look correct.
Each repository should be named something along the lines of `2017-FS-A-hw1-bob123`.

By default, `assigner assign` copies only the `master` branch from the base repo.
By default, `assigner assign` copies only the `master` branch from the template repo.
Typically, this is what you want.
If you wish to upload different branches, you can pass the `--branch` flag and list the branches you want to push.
For example, let's say that in addition to the `master` branch, you want to provide your students with a copy of the `devel` branch from the base repo.
For example, let's say that in addition to the `master` branch, you want to provide your students with a copy of the `devel` branch from the template repo.
To do so, you'd run `assigner assign hw1 --branch master devel`.

### Opening the assignment to students
Expand Down
13 changes: 6 additions & 7 deletions assigner/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,7 @@
from colorlog import ColoredFormatter
from git.cmd import GitCommandNotFound

from assigner.baserepo import StudentRepo
from assigner.config import config_context
from assigner.backends.decorators import requires_config_and_backend
from assigner.roster_util import get_filtered_roster
from assigner import progress

Expand Down Expand Up @@ -56,8 +55,8 @@
]


@config_context
def manage_repos(conf, args, action):
@requires_config_and_backend
def manage_repos(conf, backend, args, action):
"""Performs an action (lambda) on all student repos
"""
hw_name = args.name
Expand All @@ -78,10 +77,10 @@ def manage_repos(conf, args, action):
"Student %s does not have a gitlab account.", username
)
continue
full_name = StudentRepo.build_name(semester, student_section,
hw_name, username)
full_name = backend.student_repo.build_name(semester, student_section,
hw_name, username)

repo = StudentRepo(backend_conf, namespace, full_name)
repo = backend.student_repo(backend_conf, namespace, full_name)
if not dry_run:
action(repo, student)
count += 1
Expand Down
19 changes: 19 additions & 0 deletions assigner/backends/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
from assigner.backends.base import BackendBase, RepoError
from assigner.backends.gitlab import GitlabBackend
from assigner.backends.mock import MockBackend

pyflakes = [BackendBase, RepoError, GitlabBackend, MockBackend]

backend_names = {
"gitlab": GitlabBackend,
"mock": MockBackend,
}

class NoSuchBackend(Exception):
pass

def from_name(name: str):
try:
return backend_names[name]
except KeyError:
raise NoSuchBackend("Cannot find backend with name {}".format(name))
163 changes: 163 additions & 0 deletions assigner/backends/base.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
import git
import re
from typing import Optional, Type, TypeVar


class RepoError(Exception):
pass


T = TypeVar('T', bound='RepoBase')
class RepoBase:
"""Generic Repo base"""

PATH_RE = re.compile(
r"^/(?P<namespace>[\w\-\.]+)/(?P<name>[\w\-\.]+)\.git$"
)

def __init__(self, url_base: str, namespace: str, name: str, token: str, url: Optional[str] = None):
raise NotImplementedError

@classmethod
def build_url(cls, config, namespace: str, name: str) -> str:
""" Builds a url for a repository """
raise NotImplementedError

@classmethod
def from_url(cls: Type[T], url: str, token: str) -> T:
raise NotImplementedError

@property
def name_with_namespace(self) -> str:
raise NotImplementedError

@property
def namespace_id(self) -> str:
raise NotImplementedError

@property
def id(self) -> str:
raise NotImplementedError

@property
def info(self) -> Optional[str]:
raise NotImplementedError

@property
def repo(self) -> Optional[git.Repo]:
raise NotImplementedError

@property
def ssh_url(self) -> str:
raise NotImplementedError

def already_exists(self) -> bool:
raise NotImplementedError

def get_head(self, branch: str) -> git.refs.head.Head:
raise NotImplementedError

def checkout(self, branch: str) -> git.refs.head.Head:
raise NotImplementedError

def pull(self, branch: str) -> None:
raise NotImplementedError

def clone_to(self, dir_name: str, branch: Optional[str]) -> None:
raise NotImplementedError

def add_local_copy(self, dir_name: str) -> None:
raise NotImplementedError

def delete(self) -> None:
raise NotImplementedError

@classmethod
def get_user_id(cls, username: str, config) -> str:
raise NotImplementedError

def list_members(self) -> str:
raise NotImplementedError

def get_member(self, user_id: str) -> str:
raise NotImplementedError

def add_member(self, user_id: str, level: str) -> str:
raise NotImplementedError

def edit_member(self, user_id: str, level: str) -> str:
raise NotImplementedError

def delete_member(self, user_id: str) -> str:
raise NotImplementedError

def list_commits(self, ref_name: str = "master") -> str:
raise NotImplementedError

def list_pushes(self) -> str:
raise NotImplementedError

def get_last_HEAD_commit(self, ref: str = "master") -> str:
raise NotImplementedError

def list_branches(self) -> str:
raise NotImplementedError

def get_branch(self, branch: str) -> str:
raise NotImplementedError

def archive(self) -> str:
raise NotImplementedError

def unarchive(self) -> str:
raise NotImplementedError

def protect(self, branch: str = "master", developer_push: bool = True, developer_merge: bool = True) -> str:
raise NotImplementedError

def unprotect(self, branch: str = "master") -> str:
raise NotImplementedError

def unlock(self, student_id: str) -> None:
raise NotImplementedError

def lock(self, student_id: str) -> None:
raise NotImplementedError


T = TypeVar('T', bound='StudentRepoBase')
class StudentRepoBase(RepoBase):
"""Repository for a student's solution to a homework assignment"""

@classmethod
def new(cls, base_repo: str, semester: str, section: str, username: str) -> T:
raise NotImplementedError

@classmethod
def build_name(cls, semester: str, section: str, assignment: str, user: str) -> str:
raise NotImplementedError

def push(self, base_repo, branch: str = "master") -> None:
"""Push base_repo code to this repo"""
raise NotImplementedError


T = TypeVar('T', bound='StudentRepoBase')
class TemplateRepoBase(RepoBase):
"""A repo from which StudentRepos are cloned from (Homework Repo)."""
@classmethod
def new(cls, name: str, namespace: str, config) -> T:
raise NotImplementedError

def push_to(self, student_repo: StudentRepoBase, branch: str = "master") -> None:
raise NotImplementedError


class BackendBase:
"""
Common abstract base backend for all assigner backends (gitlab or mock).
"""
repo = RepoBase # type: RepoBase
template_repo = TemplateRepoBase # type: TemplateRepoBase
student_repo = StudentRepoBase # type: StudentRepoBase
access = None
18 changes: 18 additions & 0 deletions assigner/backends/decorators.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
import logging

from assigner.backends import from_name
from assigner.config import requires_config


logger = logging.getLogger(__name__)


def requires_config_and_backend(func):
"""Provides a backend depending on configuration."""
@requires_config
def wrapper(config, cmdargs, *args, **kwargs):

backend = from_name(config.backend["name"])

return func(config, backend, cmdargs, *args, **kwargs)
return wrapper
40 changes: 31 additions & 9 deletions assigner/baserepo.py → assigner/backends/gitlab.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,19 @@
from requests.exceptions import HTTPError
from urllib.parse import urlsplit, urlunsplit, urljoin, quote

from assigner.backends.base import (
BackendBase,
RepoBase,
RepoError,
StudentRepoBase,
TemplateRepoBase
)


# Transparently use a common TLS session for each request
requests = requests.Session()


class RepoError(Exception):
pass


class Visibility(Enum):
"""Gitlab API values for repo visibility"""
private = 0
Expand All @@ -34,7 +38,7 @@ class Access(Enum):
owner = 50


class Repo:
class GitlabRepo(RepoBase):
"""Gitlab repo; manages API requests and various metadata"""

PATH_RE = re.compile(
Expand Down Expand Up @@ -116,13 +120,14 @@ def _cls_gl_delete(cls, config, path, params={}):
r.raise_for_status()
return r.json()

#pylint: disable=super-init-not-called
def __init__(self, config, namespace, name, url=None):
self.config = config
self.namespace = namespace
self.name = name

if url is None:
self.url = Repo.build_url(self.config, namespace, name)
self.url = GitlabRepo.build_url(self.config, namespace, name)
else:
self.url = url

Expand Down Expand Up @@ -210,7 +215,8 @@ def clone_to(self, dir_name, branch):
else:
self._repo = git.Repo.clone_from(self.ssh_url, dir_name)
logging.debug("Cloned %s.", self.name)
except git.GitCommandError as e:
#pylint: disable=no-member
except git.exc.GitCommandError as e:
# GitPython may delete this directory
# and the caller may have opinions about that,
# so go ahead and re-create it just to be safe.
Expand Down Expand Up @@ -351,6 +357,12 @@ def unprotect(self, branch="master"):
return self._gl_put("/projects/{}/repository/branches/{}/unprotect"
.format(self.id, branch))

def unlock(self, student_id: str) -> None:
self.edit_member(student_id, Access.developer)

def lock(self, student_id: str) -> None:
self.edit_member(student_id, Access.reporter)

def _gl_get(self, path, params={}):
return self.__class__._cls_gl_get(
self.config, path, params
Expand All @@ -372,7 +384,7 @@ def _gl_delete(self, path, params={}):
)


class BaseRepo(Repo):
class GitlabTemplateRepo(GitlabRepo, TemplateRepoBase):

@classmethod
def new(cls, name, namespace, config):
Expand Down Expand Up @@ -415,7 +427,7 @@ def push_to(self, student_repo, branch="master"):
logging.debug("Pushed %s to %s.", self.name, student_repo.name)


class StudentRepo(Repo):
class GitlabStudentRepo(GitlabRepo, StudentRepoBase):
"""Repository for a student's solution to a homework assignment"""

@classmethod
Expand Down Expand Up @@ -451,3 +463,13 @@ def build_name(cls, semester, section, assignment, user):
def push(self, base_repo, branch="master"):
"""Push base_repo code to this repo"""
base_repo.push_to(self, branch)


class GitlabBackend(BackendBase):
"""
Common abstract base backend for all assigner backends (gitlab or mock).
"""
repo = GitlabRepo # type: RepoBase
template_repo = GitlabTemplateRepo # type: TemplateRepoBase
student_repo = GitlabStudentRepo # type: StudentRepoBase
access = Access
Loading

0 comments on commit ace2561

Please sign in to comment.