Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge dev into main - v0.2.0 #1

Merged
merged 153 commits into from
Nov 18, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
153 commits
Select commit Hold shift + click to select a range
2e8c803
Initial Commit
MLBZ521 Aug 10, 2022
b68ad35
Additional cleanup and standardization
MLBZ521 Aug 11, 2022
c712c95
Adding Tasks logic to endpoints and more
MLBZ521 Aug 12, 2022
4ed72ea
Starting to update README
MLBZ521 Aug 12, 2022
c16388f
Added scheduling functionality
MLBZ521 Aug 12, 2022
410426c
New/Updated LaunchDaemons
MLBZ521 Aug 13, 2022
5939d8a
Removing load of cli args from pkgbot.py
MLBZ521 Aug 13, 2022
4bc217a
Standardizing settings loading
MLBZ521 Aug 13, 2022
0edca72
Additional cleanup and standardization
MLBZ521 Aug 13, 2022
be6a6ef
Adding GitPython to manage cli git operations
MLBZ521 Aug 13, 2022
63ee2e2
Cleaned up new models
MLBZ521 Aug 13, 2022
71f3d30
Adding route to upload icon
MLBZ521 Aug 13, 2022
6168651
Updating to new function calls
MLBZ521 Aug 15, 2022
c8966b7
Adding updated Production Recipe Template
MLBZ521 Aug 16, 2022
0d997a6
Replaced the Recipe Checker/Promotion Processor
MLBZ521 Aug 16, 2022
435f23d
Updated README
MLBZ521 Aug 16, 2022
e7d3390
Add support for query parameters for recipes
MLBZ521 Aug 18, 2022
c32b048
Adding feature to track recurring recipe failures
MLBZ521 Aug 18, 2022
dd7a338
Updated README
MLBZ521 Aug 18, 2022
e77cc31
Fixed reference
MLBZ521 Aug 18, 2022
3cf41f7
Get absolute path to config file
MLBZ521 Aug 18, 2022
393b90c
Updated README
MLBZ521 Aug 18, 2022
f0eb1f7
Get absolute path to config file
MLBZ521 Aug 18, 2022
d307161
Corrected model
MLBZ521 Aug 18, 2022
2ed1766
Re-enable the ssl args
MLBZ521 Aug 19, 2022
be61b7f
Additional changes for static files/icons
MLBZ521 Aug 19, 2022
21e898c
Fixed logic issue
MLBZ521 Aug 19, 2022
c58b13a
_Actual_ support for recipes query parameters
MLBZ521 Aug 19, 2022
5152dd0
Correctly setting up StaticFiles usage
MLBZ521 Aug 19, 2022
eb6e3b0
Fixing jinja/html for static files
MLBZ521 Aug 19, 2022
2beb3bf
Improved the query parameter logic
MLBZ521 Aug 19, 2022
bf28482
Updating icon support for both legacy and future method
MLBZ521 Aug 19, 2022
4d6c9de
Additional improvements to query parameter logic
MLBZ521 Aug 19, 2022
5163b15
Updating icon support for both legacy and future method
MLBZ521 Aug 19, 2022
f259385
Disabling the recurring job for now
MLBZ521 Aug 20, 2022
1a004e9
Commenting
MLBZ521 Aug 20, 2022
a9bdbf3
Fixing issues with verify_trust route
MLBZ521 Aug 20, 2022
7109975
Setup parsing of autopkg arguments
MLBZ521 Aug 20, 2022
561b809
Clean Up
MLBZ521 Aug 20, 2022
8f90c63
Updated README
MLBZ521 Aug 20, 2022
d3a6238
Resolved issues in the webhook logic
MLBZ521 Aug 21, 2022
9d3f25d
Massively updated config loading logic
MLBZ521 Aug 21, 2022
a64ea63
Restructured package & updated config loading
MLBZ521 Aug 23, 2022
ebc255f
Restructured package & updated config loading
MLBZ521 Aug 23, 2022
991a469
Updating suggested directories
MLBZ521 Aug 23, 2022
4b8b198
Updating html views for new attribute
MLBZ521 Aug 23, 2022
f160854
Missed a config update
MLBZ521 Aug 23, 2022
57688f9
Adding defaults
MLBZ521 Aug 23, 2022
457c40c
Updates to the SQLAlchemy usage
MLBZ521 Aug 23, 2022
028107f
Updates to the SQLAlchemy usage
MLBZ521 Aug 23, 2022
570d6d9
Resolved issue with task result lookups
MLBZ521 Aug 23, 2022
61bd945
Fixing issues in task logic
MLBZ521 Aug 25, 2022
f8757a7
Bug fix
MLBZ521 Aug 25, 2022
21f312d
Logic Improvement
MLBZ521 Aug 25, 2022
1677cbe
Bug fix
MLBZ521 Aug 26, 2022
c6394ca
Finish Icon Upload Logic
MLBZ521 Aug 26, 2022
447a031
Confirmed proper logic
MLBZ521 Aug 26, 2022
4e2ea4c
Improved task logic
MLBZ521 Aug 26, 2022
3812927
Updating Blocks to use PkgBot hosted icons
MLBZ521 Aug 26, 2022
c3dab05
Fixed dev workflow logic
MLBZ521 Aug 26, 2022
0357d41
Changed conditional logic for dev workflow
MLBZ521 Aug 26, 2022
1983099
Added missing await
MLBZ521 Aug 26, 2022
23b548f
Slight adjustment to order of logic
MLBZ521 Aug 27, 2022
131d1dc
Fixing promotion logic
MLBZ521 Aug 27, 2022
d9045d2
Quoting options for safety
MLBZ521 Aug 27, 2022
2ff8f7c
Combining different options for various uses
MLBZ521 Aug 27, 2022
8d48d9b
Only disable recipe if it exists
MLBZ521 Aug 27, 2022
980f16b
Fixed file name and add proper override variables
MLBZ521 Aug 27, 2022
c7f0a17
Improved search logic
MLBZ521 Aug 27, 2022
b5ac1bf
Moved endpoint and improved
MLBZ521 Aug 27, 2022
6757910
Additional improvements and updates
MLBZ521 Aug 28, 2022
5268b8e
Updates and fixes to pkg promotion logic
MLBZ521 Aug 28, 2022
2b6f5fa
Fixed icon not showing up in Slack msg
MLBZ521 Aug 28, 2022
4358828
Added ability
MLBZ521 Aug 28, 2022
89479fc
Improved pkg promotion error messages in Slack
MLBZ521 Aug 28, 2022
4f151fc
Minor adjustment to the `recipe_trust_verify_failed` endpoint
MLBZ521 Aug 28, 2022
8ba48b0
Fixed missing called_by parameter
MLBZ521 Sep 2, 2022
bfd3399
Fixed issues with the recipe filter
MLBZ521 Sep 3, 2022
0dbdac8
Directory staging
MLBZ521 Sep 3, 2022
a80d30f
Removing files
MLBZ521 Sep 3, 2022
41ea813
Using full path to files
MLBZ521 Sep 13, 2022
6d76967
Updating verbiage
MLBZ521 Sep 13, 2022
d9b70e4
Updating README
MLBZ521 Sep 13, 2022
07f9875
Using nginx
MLBZ521 Sep 13, 2022
3508ce1
Misc cleanup and standarization
MLBZ521 Sep 13, 2022
70fe675
Using full path to files
MLBZ521 Sep 14, 2022
f33a0ad
Adding link
MLBZ521 Sep 14, 2022
d33dc8f
Bug fix
MLBZ521 Sep 14, 2022
c5bb041
Moved function to execute in proper timeline
MLBZ521 Sep 16, 2022
971560a
Fixed issue with Deny Package route
MLBZ521 Sep 16, 2022
760f6fd
Renamed directory
MLBZ521 Sep 16, 2022
4956ab4
Switching to Celery's Result.AsyncResult
MLBZ521 Sep 16, 2022
6c19de7
Fixed issue with Trust Info Failures not posting to Slack
MLBZ521 Sep 16, 2022
433ccbb
Getting SlackBot name from Config
MLBZ521 Sep 16, 2022
93c45ce
Tweaking Celery queues to account for issues with prioritization
MLBZ521 Sep 16, 2022
0d4b85f
Switching to Celery's Result.AsyncResult
MLBZ521 Sep 16, 2022
afe549a
Switching to Celery's Result.AsyncResult
MLBZ521 Sep 16, 2022
0878cf2
Switching to Celery's Result.AsyncResult
MLBZ521 Sep 16, 2022
4c508f9
Fixing Task Prioritization
MLBZ521 Sep 16, 2022
4326671
Missing awaits
MLBZ521 Sep 16, 2022
42c402c
Bug fix after previous change
MLBZ521 Sep 16, 2022
1b7f21c
Fixing Task Prioritization
MLBZ521 Sep 17, 2022
5e33d27
Fixing bug with promotion via the `autopkg/run/recipe/{recipe_id}` route
MLBZ521 Sep 17, 2022
98381ba
Adding catch just in case
MLBZ521 Sep 17, 2022
51164de
Potential Bug Fix
MLBZ521 Sep 17, 2022
5dbe4f9
Bug Fix
MLBZ521 Sep 17, 2022
a81018b
Updated receive > "verify_trust_info" event to support called_by methods
MLBZ521 Sep 17, 2022
5c83106
Added missing called_by references
MLBZ521 Sep 17, 2022
d0e5673
Added update-trust-info logic
MLBZ521 Sep 17, 2022
fe20eeb
Bug Fix
MLBZ521 Sep 17, 2022
47ee773
Bug Fix
MLBZ521 Sep 17, 2022
615e1c6
Fixed issue where recipe names are parsed as links
MLBZ521 Sep 17, 2022
af722c7
Updated & enabled the `git_pull_private_repo` Task
MLBZ521 Sep 19, 2022
6972548
`git` logic improvements
MLBZ521 Sep 19, 2022
41f0703
Further `git` logic improvements
MLBZ521 Sep 20, 2022
f0bb06f
Enabling scheduled runs
MLBZ521 Sep 21, 2022
e0803f6
Fixing Model Name
MLBZ521 Sep 21, 2022
f2dd6af
Bug Fix
MLBZ521 Sep 21, 2022
a13b71b
Removing misplaced `@` character
MLBZ521 Sep 22, 2022
0b101cf
Fixed issue where recipe names are parsed as links
MLBZ521 Sep 22, 2022
1281a6f
Bug Fix: Reference proper Model
MLBZ521 Sep 22, 2022
0dde132
Message Improvements
MLBZ521 Oct 2, 2022
3f297c5
Adding default icons
MLBZ521 Oct 2, 2022
751fc04
Add route to get Celery Task Results
MLBZ521 Oct 13, 2022
cc8a649
Added new default images with different message types
MLBZ521 Oct 13, 2022
e0d26c9
Add an "Acknowledge" Button to Error Messages
MLBZ521 Oct 21, 2022
67bf718
Add authentication to new endpoints
MLBZ521 Oct 21, 2022
ecae3af
Fixed Deny Trust Button Bug
MLBZ521 Oct 21, 2022
2d52f20
Set default value for recurring_fail_count
MLBZ521 Oct 23, 2022
bcd09b2
Updates to `trust`-related functions
MLBZ521 Nov 3, 2022
e9b6ef6
Fixed issue when overriding autopkg options
MLBZ521 Nov 13, 2022
b3595af
Improved Parent/Child Task tracking and reporting
MLBZ521 Nov 13, 2022
d042c7d
Forgot to commit this import
MLBZ521 Nov 14, 2022
53c452d
Standardizing variable reference
MLBZ521 Nov 14, 2022
ac99866
Updating suggested directories
MLBZ521 Nov 14, 2022
2caffa5
Updating README
MLBZ521 Nov 14, 2022
ee3caf6
Improved Parent/Child Task tracking and reporting
MLBZ521 Nov 14, 2022
44a789a
Improved redaction of sensitive strings logic
MLBZ521 Nov 16, 2022
af6ef26
Redact sensitive strings before returning Task results
MLBZ521 Nov 16, 2022
7d0d6d7
Move "locate icon" logic to dev runs only
MLBZ521 Nov 16, 2022
d7020c1
Improved redaction of sensitive strings logic
MLBZ521 Nov 16, 2022
9502cef
Bug Fix
MLBZ521 Nov 18, 2022
0698281
Removed use of `replace_sensitive_strings` in Slack API commands
MLBZ521 Nov 18, 2022
d10fca4
Ignore Additional Files
MLBZ521 Nov 18, 2022
f0dc3e1
Setting more appropriate variable name
MLBZ521 Nov 18, 2022
0123f15
Updating default verbose level
MLBZ521 Nov 18, 2022
c6d79f2
Removing previous revision code
MLBZ521 Nov 18, 2022
18e87a0
Switch from sync `requests` to async `httpx`
MLBZ521 Nov 18, 2022
e009b7e
Changing authentication level dependency for `autopkg/results/task_id`
MLBZ521 Nov 18, 2022
7d92506
Cleanup and standardization
MLBZ521 Nov 18, 2022
112eb20
Updating description on the homepage
MLBZ521 Nov 18, 2022
4cb043a
Updated Error Message Format
MLBZ521 Nov 18, 2022
cd8a337
Updated README
MLBZ521 Nov 18, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 137 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
# .gitignore itself
# .gitignore

# Visual Studio Code
.vscode

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-shm
db.sqlite3-wal
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
.python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# PEP 582; used by e.g. github.com/David-OConnor/pyflow
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/
File renamed without changes.
115 changes: 115 additions & 0 deletions PkgBot.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
#!/usr/local/autopkg/python

import multiprocessing
import sys

# import asyncio
import secure
import uvicorn

from fastapi import FastAPI
from fastapi.responses import RedirectResponse
from fastapi.staticfiles import StaticFiles
from fastapi.middleware.httpsredirect import HTTPSRedirectMiddleware

from tortoise.contrib.fastapi import register_tortoise

from pkgbot import config, settings

config = config.load_config(cli_args=tuple(sys.argv[1:]))

from pkgbot.utilities import common as utility
from pkgbot.db import models
from pkgbot import api


log = utility.log

app = FastAPI(
title="PkgBot API",
description="A framework to manage software packaging, testing, and promoting from a "
"development to production environment.",
version="0.2.0",
openapi_tags=settings.api.tags_metadata,
docs_url="/api"
)

app.mount("/static", StaticFiles(directory="/Library/AutoPkg/PkgBot/pkgbot/static"), name="static")
app.include_router(api.views.router)
app.include_router(api.auth.router)
app.include_router(api.autopkg.router)
app.include_router(api.package.router)
app.include_router(api.recipe.router)
app.include_router(api.bot.router)
app.include_router(api.build_msg.router)
app.include_router(api.send_msg.router)
app.include_router(api.user.router)

register_tortoise(
app,
config = settings.db.TORTOISE_CONFIG,
generate_schemas = True,
add_exception_handlers = True
)

# Add an exception handler to the app instance
# Used for the login/auth logic for the HTTP views
app.add_exception_handler(api.auth.NotAuthenticatedException, api.auth.exc_handler)
api.auth.login_manager.useRequest(app)

if config.PkgBot.get("enable_ssl"):

# Enforces that all incoming requests must be https.
app.add_middleware(HTTPSRedirectMiddleware)
server = secure.Server().set("Secure")
hsts = secure.StrictTransportSecurity().include_subdomains().preload().max_age(2592000)
cache_value = secure.CacheControl().must_revalidate()
secure_headers = secure.Secure(
server=server,
# csp=csp,
hsts=hsts,
# referrer=referrer,
# permissions=permissions_value,
cache=cache_value,
)

@app.middleware("http")
async def set_secure_headers(request, call_next):
response = await call_next(request)
secure_headers.framework.fastapi(response)
return response


async def number_of_workers():
number_of_threads = (multiprocessing.cpu_count() * 2) - 1
log.debug(f"Number of workers: {number_of_threads}")
return number_of_threads


@app.on_event("startup")
async def startup_event():

pkgbot_admins = config.PkgBot.get("Admins")

for admin in pkgbot_admins:
user_object = models.PkgBotAdmin_In(
username = admin,
slack_id = pkgbot_admins.get(admin),
full_admin = True
)
await api.user.create_or_update_user(user_object)


if __name__ == "__main__":

uvicorn.run(
"PkgBot:app",
reload = config.PkgBot.get("keep_alive"),
host = config.PkgBot.get("host"),
port = config.PkgBot.get("port"),
log_config = config.PkgBot.get("log_config"),
log_level = config.PkgBot.get("uvicorn_log_level"),
# workers = asyncio.run(number_of_workers()),
ssl_keyfile = config.PkgBot.get("ssl_keyfile"),
ssl_certfile = config.PkgBot.get("ssl_certfile")
)
Loading