Merge pull request #1169 from Metro-Records/feat/quarto-docs

Migrate docs from wiki to quarto

.github/workflows/publish_docs.yml

@@ -0,0 +1,35 @@
+name: Publish documentation to GitHub Pages
+
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - main
+
+jobs:
+  build-and-deploy:
+    name: Build and publish documentation
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+
+      - uses: tj-actions/changed-files@v45
+        id: docs-changed
+        with:
+          files: |
+            docs/*
+      - name: Set up Quarto
+        if: steps.docs-changed.outputs.any_changed == 'true'
+        uses: quarto-dev/quarto-actions/setup@v2
+
+      - name: Render and Publish
+        if: steps.docs-changed.outputs.any_changed == 'true'
+        uses: quarto-dev/quarto-actions/publish@v2
+        with:
+          path: docs
+          target: gh-pages
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.gitignore

@@ -25,5 +25,6 @@ debug.log
 lametro/secrets.py
 .env
 .env.local
+.env
 
 .venv

README.md

@@ -207,6 +207,24 @@ see the [Locust documentation](https://docs.locust.io/en/stable/).
 ## Review Apps
 This repo is set up to deploy review apps on Heroku, and those pull from the staging database to match the experience of deploying as closely as possible! However, note that in order to prevent unapproved model changes from effecting the staging database, migrations are prevented from running on review apps. So those will still have to be reviewed locally.
 
+## Updating the documentation
+
+To make changes to the documentation, [install Quarto](https://quarto.org/docs/get-started/).
+
+Then, run the following in your terminal:
+
+```bash
+quarto preview docs
+```
+
+Make your changes to the `.qmd` files in the `docs/` directory. They will be automatically
+reflected in your local version of the docs.
+
+For more on authoring docs with Quarto, see [their Getting Started guide](https://quarto.org/docs/get-started/authoring/text-editor.html) and [documentation](https://quarto.org/docs/guide/).
+
+The GitHub Pages site will rebuild automatically when your documentation changes are
+merged into `main`.
+
 ## Errors / Bugs
 
 If something is not behaving intuitively, it is a bug, and should be reported.

docs/.gitignore

@@ -0,0 +1,2 @@
+/.quarto/
+/_site/

docs/_quarto.yml

@@ -0,0 +1,22 @@
+project:
+  type: website
+
+website:
+  title: "LA Metro Councilmatic Documentation"
+  sidebar:
+    style: "docked"
+    search: true
+    contents: auto
+    tools:
+      - icon: github
+        menu:
+          - text: Source Code
+            href: https://github.com/Metro-Records/la-metro-councilmatic
+          - text: Issue Tracker
+            href: https://github.com/Metro-Records/la-metro-councilmatic/issues
+
+format:
+  html:
+    theme: litera
+    css: styles.css
+    toc: true

docs/commands.qmd

@@ -0,0 +1,58 @@
+---
+title: "Commands to Know"
+order: 2
+---
+
+The LA Metro galaxy comes with several CLI commands and their various options.
+This section identifies some of the most significant commands for councilmatic, how to use them and where to execute them.
+
+Metro Councilmatic runs additional processes on the data, after it gets imported to the database.
+If you do need to run a particular management command, read on for more information about the commands that comprise `hourly_processing` in the Metro dashboard.
+
+### Refresh the Property Image Cache
+Metro caches PDFs of board reports and event agendas. [This can raise issues.](https://github.com/datamade/la-metro-councilmatic/issues/347)
+The [`refresh_pic` management command](https://github.com/datamade/django-councilmatic/blob/master/councilmatic_core/management/commands/refresh_pic.py) refreshes the document cache ([an S3 bucket connected to Metro Councilmatic via `property-image-cache`](https://github.com/datamade/property-image-cache)) by deleting potentially out-of-date versions of board reports and agendas.
+
+```bash
+# run the command and log the results (if on the server)
+python manage.py refresh_pic >> /var/log/councilmatic/lametro-refreshpic.log 2>&1
+```
+
+### Create PDF packets
+Metro Councilmatic has composite versions of the Event agendas (the event and all related board reports) and board reports (the report and its attachments). [A separate app assists in creating these PDF packets](https://github.com/datamade/metro-pdf-merger), and the [`compile_pdfs` command](https://github.com/datamade/la-metro-councilmatic/blob/master/lametro/management/commands/compile_pdfs.py) communicates with this app by telling it which packets to create.
+
+```bash
+# run the command and log the results (if on the server)
+# documented in the `metro-pdf-merger` README: https://github.com/datamade/metro-pdf-merger#get-started
+python manage.py compile_pdfs >> /var/log/councilmatic/lametro-compilepdfs.log 2>&1
+
+python manage.py compile_pdfs --all_documents
+```
+
+### Convert report attachments into plain text
+Metro Councilmatic allows users to query board reports via attachment text. The attachments must appear as plain text in the database: [`convert_attachment_text`](https://github.com/datamade/django-councilmatic/blob/master/councilmatic_core/management/commands/convert_attachment_text.py) helps accomplish this.
+
+```bash
+# run the command and log the results (if on the server)
+python manage.py convert_attachment_text >> /var/log/councilmatic/lametro-convertattachments.log 2>&1
+
+# update all documents
+python manage.py convert_attachment_text --update_all
+```
+
+### Rebuild or update the search index
+Haystack comes with a utility command for rebuilding and updating the search index. [Learn more in the Haystack docs.](https://django-haystack.readthedocs.io/en/master/management_commands.html)
+
+```bash
+# ideally, rebuild should be run with a small batch-size to avoid memory consumption issues
+# https://github.com/datamade/devops/issues/42
+# run the command and log the results (if on the server)
+python manage.py rebuild_index --batch-size=200 >> /var/log/councilmatic/lametro-updateindex.log 2>&1
+
+# update can be run with an age argument, which instructs SmartLogic to consider bills updated so many hours ago
+python manage.py update_index --age=2
+
+# update should be run in non-interactive mode, when logging the results
+# `noinput` tells Haystack to skips the prompts
+python manage.py update_index --noinput
+```

docs/debugging.qmd

@@ -0,0 +1,62 @@
+---
+title: "Debugging"
+order: 1
+---
+
+## Don't Panic!
+
+Many issues can arise in the Metro galaxy, from the shallowest part of the frontend to the deepest depths of the backend.
+However, these issues are generally due to either a metadata error or a scraper error.
+
+This documentation will focus on metadata errors. If you suspect you're experiencing a scraper issue, please refer to the
+[debug documentation for the scrapers](https://metro-records.github.io/scrapers-lametro/debugging.html).
+
+### Metadata Error
+Metro performs a series of ETL tasks against its database. You can view the full pipeline
+[here](https://github.com/datamade/la-metro-dashboard/blob/main/dags/hourly_processing.py).
+
+Failures in the ETL pipeline might have a corresponding issue
+[in the `la-metro-councilmatic` Sentry project](https://sentry.io/organizations/datamade/issues/?project=2131912),
+however sometimes steps run without failing but don't generate the desired result.
+Read on for more on each step of the pipeline, plus past failures and their resolutions.
+
+#### `refresh_pic`
+
+**Where it lives:** [Django Councilmatic](https://github.com/datamade/django-councilmatic/blob/2.5/councilmatic_core/management/commands/refresh_pic.py)<br />
+**What is does:** Deletes [cached documents](https://github.com/datamade/property-image-cache) for recently updated bills and events
+
+**Past issues:**
+
+- [Cached event agenda was out of sync with Legistar](https://github.com/datamade/la-metro-councilmatic/issues/443).
+We have since updated the logic for which documents to remove from the cache, so this error should be resolved,
+but the linked issue contains instructions for resolving this error manually, in case we see a regression.
+
+#### `compile_pdfs`
+
+**Where it lives:** [LA Metro Councilmatic](https://github.com/datamade/la-metro-councilmatic/blob/main/lametro/management/commands/compile_pdfs.py)<br />
+**What it does:** Notifies the [`metro-pdf-merger`](https://github.com/datamade/metro-pdf-merger) of new documents that need to be merged into a bill or event packet
+
+**Past issues:**
+
+- [Sometimes the worker fails to merge documents, resulting in missing packets](https://github.com/datamade/la-metro-councilmatic/issues/476).
+There should be a corresponding error for this [in the `metro-pdf-merger` Sentry project](https://sentry.io/organizations/datamade/issues/?project=155211),
+however the project is pretty noisy, so you can shell into the server (`metro-pdf-merger.datamade.us`) and tail or grep the worker logs to double check.
+- [The worker PDF merger has died mysteriously](https://github.com/datamade/metro-pdf-merger/issues/19).
+
+#### `convert_attachment_text`
+
+**Where it lives:** [Django Councilmatic](https://github.com/datamade/django-councilmatic/blob/2.5/councilmatic_core/management/commands/convert_attachment_text.py)<br />
+**What it does:** Extracts text from bill attachments for indexing
+
+**Past issues:**
+
+- N/A
+
+#### `update_index`
+
+**Where it lives:** [Haystack](https://django-haystack.readthedocs.io/en/master/management_commands.html#update-index)<br />
+**What it does:** Updates the search index
+
+**Past issues:**
+
+- N/A

docs/deployment.qmd

@@ -0,0 +1,106 @@
+---
+title: "Deployment"
+order: 3
+---
+
+DataMade hosts a [staging version](https://la-metro-councilmatic-staging.herokuapp.com/) and [production version](https://boardagendas.metro.net/) of Metro Councilmatic.
+
+Both sites - as well as review apps for active PRs - can be found in the [corresponding Heroku pipeline](https://dashboard.heroku.com/pipelines/d77360c3-dcda-43d4-990e-b898b51ba37d/). Metro developers should be able to shell into these servers, (e.g., [for executing management commands](./commands.qmd)). If you cannot, then talk to your friendly DataMade devops team.
+
+### Patterns and practices
+
+Triggering a Metro deployment follows the typical DataMade pattern: merging or pushing to `main` deploys to the staging site, and pushing changes from `main` to `deploy` deploys to the production site.
+
+The Metro deployment script runs a few steps for the production and staging environments - but not the review apps.
+It will apply any new migrations, create any missing cache tables, load in shapes of districts, and clear the site-wide cache.
+
+These steps are not run for review apps because those use the staging environment's database. This is a deviation from most of DataMade's other Heroku pipelines.
+This gives us access to review apps without having to run lengthy scrapes each time we'd like to test a change of any size!
+However, this does also mean that any PRs that include a modification to the database will need to be tested locally instead of on those apps.
+
+### Working in Heroku
+
+At times, we'll need to shell directly into these deployed environments to investigate or manipulate the data within.
+Luckily, heroku provides us with a couple options for how to interact with them - the console on the site and the CLI.
+
+#### Using the Site Console
+To shell into the environment using the site console, click the Run Console link:
+![_The Run Console link as shown in Heroku_](images/heroku-console.png)
+
+And then enter your command as normal. In this case, we're running `python manage.py shell`:
+![_A python command in the Heroku console_](images/heroku-console-cmd.jpg)
+
+
+#### Using the CLI
+To shell in using your terminal, first make sure the [Heroku CLI is installed](https://devcenter.heroku.com/articles/heroku-cli#install-the-heroku-cli)
+then run the following commands, replacing `<ENVIRONMENT_NAME>` with your chosen environment
+(`la-metro-councilmatic-staging` for staging, and `la-metro-councilmatic-prod` for production).
+
+```bash
+>>> heroku login
+heroku: Press any key to open up the browser to login or q to exit:
+Opening browser to https://cli-auth.heroku.com/...
+Logging in... done
+Logged in as <your_email>
+
+>>> heroku run -a <ENVIRONMENT_NAME> python manage.py shell
+Running python manage.py shell on ⬢ <ENVIRONMENT_NAME>... up, run.####
+...
+Type "help", "copyright", "credits" or "license" for more information.
+(InteractiveConsole)
+>>> |
+```
+
+
+## Inspecting Data
+
+### View an entity in the Councilmatic database
+#### Use the slug of the problematic Bill, Event, Person, or Committee
+Every detail page is accessed in the browser using a slug. Let's walk through an example.
+Visit the page for the [October 2024 Executive Management Committee meeting](https://la-metro-councilmatic-staging.herokuapp.com/event/executive-management-committee-8a0e5d3ba5e2/),
+and check the address bar for the slug (the portion after the entity type, "event" in this case).
+We can see that the slug for this meeting is `executive-management-committee-8a0e5d3ba5e2`.
+
+#### Retrieve the entity in the Django shell
+Shell into a running instance of LA Metro Councilmatic using either the CLI or the web-based console.
+Then retrieve the problematic entity using its slug:
+
+```python
+# In the Django shell
+>>> from lametro.models import *
+>>> entity = LAMetroEvent.objects.get(slug='executive-management-committee-8a0e5d3ba5e2')
+```
+
+You can use the same ORM query to retrieve any entity. Simply swap out `LAMetroEvent` for the correct model and, of course, update the slug.
+
+| Entity | Model |
+| -- | -- |
+| Person | LAMetroPerson |
+| Committee | LAMetroOrganization |
+| Bill | LAMetroBill |
+| Event | LAMetroEvent |
+
+#### View useful information
+Assuming you have retrieved the entity as illustrated in the previous step, you can view its last updated date like this:
+
+```python
+>>> entity.updated_at
+datetime.datetime(2020, 3, 25, 0, 47, 3, 471572, tzinfo=<UTC>)
+```
+
+You can also view its sources like this:
+
+```python
+>>> import pprint
+>>> pprint.pprint([(source.note, source.url) for source in entity.sources.all()])
+[('api', 'http://webapi.legistar.com/v1/metro/events/1384'),
+ ('api (sap)', 'http://webapi.legistar.com/v1/metro/events/1493'),
+ ('web',
+  'https://metro.legistar.com/MeetingDetail.aspx?LEGID=1384&GID=557&G=A5FAA737-A54D-4A6C-B1E8-FF70F765FA94'),
+ ('web (sap)',
+  'https://metro.legistar.com/MeetingDetail.aspx?LEGID=1493&GID=557&G=A5FAA737-A54D-4A6C-B1E8-FF70F765FA94')]
+```
+
+Events have Spanish language sources (e.g., "api (sap)"), as well. In initial debugging, focus on the "api" and "web" sources – by visiting these links and checking that the data in Legistar appears as expected.
+
+**Pro-tip:** If you are using iTerm2 on a Mac, hold down the command key to convert URLs into hyperlinks. Saved you a copy/paste!

docs/glossary.qmd

@@ -0,0 +1,64 @@
+---
+title: "Glossary"
+order: 4
+---
+
+## Thesaurus
+Many of the entities Metro deals with can be called multiple things. Here is a cheatsheet to reference:
+
+### Board of Directors, Committee
+- Body (Legistar API)
+- Meeting body (Legistar UI)
+- Organization (Open Civic Data universe, Councilmatic models)
+- Committee (Legistar UI, Councilmatic UI)
+
+### Board member
+- Member (Legistar UI)
+- Person/s (Legistar API, Open Civic Data universe, Councilmatic models)
+- Board member (Councilmatic UI)
+
+### Membership
+- Office record (Legistar API)
+- Membership (Open Civic Data models, Councilmatic models)
+- Term (pupa)
+
+### Bill
+- Matter (Legistar API)
+- Bill (Open Civic Data universe)
+- Board report (Legistar UI, Councilmatic universe)
+
+### Event
+- Event (Legistar API, Open Civic Data universe, Councilmatic models)
+- Meeting (Legistar UI, Councilmatic UI)
+
+
+## Dictionary
+Here are some concepts in Metro that benefit from some more context:
+
+### Division
+- Political geography
+- "Jurisdictions exist within a division, while Posts can represent a division"
+- Divisions relevant to LA Metro:
+  - City of Los Angeles
+  - [Los Angeles County supervisorial districts](https://experience.arcgis.com/experience/159e4b53494e47fe82bacd8016065843)
+  - [Statutorially defined transit sectors](https://boardagendas.metro.net/static/pdf/CitySelectionCommitte-MTA-Rules-and-Regulations.pdf)
+  - [Caltrans (Calfornia Department of Transportation) District](https://dot.ca.gov/caltrans-near-me/district-7)
+- Further reading: [https://open-civic-data.readthedocs.io/en/latest/proposals/0002.html](https://open-civic-data.readthedocs.io/en/latest/proposals/0002.html)
+
+### Jurisdiction
+- Logical unit of governance
+- Example: LA Metro
+- Further reading: [https://open-civic-data.readthedocs.io/en/latest/proposals/0003.html](https://open-civic-data.readthedocs.io/en/latest/proposals/0003.html)
+
+### Post
+- Position in organization
+- Posts define the core positions within an organization, and can optionally be associated with a Division, i.e., the political geography they represent
+- Examples:
+  - Post associated with a division: Appointee of the Mayor of the City of Los Angeles on the Board of Directors representing the City of Los Angeles
+  - Post not associated with a division: Chair of the Board of Directors
+- Further reading: [https://open-civic-data.readthedocs.io/en/latest/proposals/0005.html](https://open-civic-data.readthedocs.io/en/latest/proposals/0005.html)
+
+### Membership
+- "A relationship between a Person and an Organization, possibly including a Post"
+  - Committee Memberships are not associated with a Post because committees do not have a defined membership structure
+- Further reading: [https://open-civic-data.readthedocs.io/en/latest/proposals/0005.html](https://open-civic-data.readthedocs.io/en/latest/proposals/0005.html)