-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Docs: add in the mailmerge scaffold & scripts used for MaDs & documen…
…t their usage (#19) * docs: add in the scripts used for MaDs & document their usage * docs: add line to README about mailmerge folder * docs: change template for mailmerge to use placeholder * chore: make gitignore for mailmerge more thorough * devops: docker ignore mailmerge * chore: change a var to a let Co-authored-by: Nishant Aanjaney Jalan <cybercoder.nishant@gmail.com> --------- Co-authored-by: Nishant Aanjaney Jalan <cybercoder.nishant@gmail.com>
- Loading branch information
1 parent
6570aea
commit 22c7b10
Showing
12 changed files
with
289 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
# Fill these in to send emails | ||
DOCSOC_SMTP_SERVER=smtp-mail.outlook.com | ||
DOCSOC_SMTP_PORT=587 | ||
DOCSOC_OUTLOOK_USERNAME=docsoc@ic.ac.uk | ||
# Password to docsoc email | ||
DOCSOC_OUTLOOK_PASSWORD= | ||
|
||
# Optional: Fill these in to upload drafts (email send itself is only done using the SMTP server above though, it does not need these) | ||
# (you don't need these for Mums & Dads, or to send emails at all) | ||
# You will need to create an app registration in Entra ID, restricted to the organisation, | ||
# And grant it the following permissions: | ||
# - Mail.ReadWrite | ||
# - User.Read | ||
MS_ENTRA_CLIENT_ID= | ||
MS_ENTRA_CLIENT_SECRET= | ||
MS_ENTRA_TENANT_ID= |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
# INCOMPLETE!!!!!! | ||
|
||
!.env.template | ||
.env | ||
|
||
# Outputs you should never push | ||
output/* | ||
!output/.gitempty | ||
|
||
data/* | ||
!data/.gitempty | ||
|
||
attachments/* | ||
!attachments/.gitempty |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,85 @@ | ||
# Sending family allocations | ||
|
||
> [!CAUTION] | ||
> Ensure you properly gitignore files in this directory to prevent commiting the personal information of the families to the repo, or the generated emails themselves. | ||
> | ||
> It is recommended to make a copy of this directory outside the repo and run the scripts from there. | ||
> | ||
> When I wrote this README it was 1am in the morning after MaDs 2024, so please forgive the fact I didn't complete a full gitignore for you. | ||
The code in this folder is used to send out family allocations for DoCSoc Mums & Dads. It uses the `docsoc-mailmerge` CLI tool (see [https://github.com/icdocsoc/docsoc-tools/tree/main/email/mailmerge-cli](https://github.com/icdocsoc/docsoc-tools/tree/main/email/mailmerge-cli)) to send out emails to all the parents and children with their family allocations. | ||
|
||
How it works: | ||
1. Generate a CSV with the family allocations using the `generateCsv.ts` script at the root of the repo | ||
2. Copy it into the `data` folder (e.g. with filename `families.csv`) | ||
3. Run `scripts/transform-data.py` on it to generate a new CSV, `data/families-emails.csv`, suitable for use by the mailmerge tool (this has to happen as the mailmerge tool expects a record per email, not per family) | ||
4. Install the `docsoc-mailmerge` CLI tool with `npm install -g @docsoc/mailmerge-cli` (or install it from the docsoc-tools repo directly via `npm link`) | ||
5. Make a copy of `.env.template` as `.env` and fill in the details (you just need to fill the SMTP server information in for MaDs) | ||
6. Adjust the email template in `templates/template.md.njk` as needed (e.g. change the from line, date, time, etc.) | ||
7. Generate the emails with `docsoc-mailmerge generate nunjucks ./data/family-emails.csv ./templates/template.md.njk -o ./output --htmlTemplate ./templates/wrapper.html.njk` | ||
8. Manually inspect a subset of the generated email HTML files in `./output/<runname>` in a browser - if there are issues, edit the template and regenerate the emails as in 7. | ||
1. Double check the JSON files in that directory have valid `email.to` and `email.subject` fields as well; if not ensure when asked during the generation to map the `to` field to something, you map it to the `to` field from the options list. | ||
9. Test send the emails with `docsoc-mailmerge send ./output/<runname> -t docsoc@ic.ac.uk -n 5` (this will send the first 5 emails to `docsoc@ic.ac.uk`) | ||
10. Once you are happy with everything, send the emails with `docsoc-mailmerge send ./output/<runname> -s 5` (this will send all emails at 5 seconds intervals to prevent hitting rate limits) | ||
|
||
## Original README generated by `docsoc-mailmerge` | ||
|
||
To get started: | ||
|
||
1. Put your own CSV in the `data` folder, with at the minimum `to` and `subject` columns. | ||
1. You can also add `cc` and `bcc` columns (to use them you will need to pass the correct CLI option though) | ||
2. `to`, `cc`, and `bcc` can be a space-separated list of emails. | ||
3. You can add any other columns you like, and they will be available in the template. | ||
4. For attachments, add a column with the name `attachment` with a singular path to the file to attach relative to th workspace root (e.g. `./attachments/image1.jpg`). | ||
1. Or, pass the same attachment to every email using the `-a` flag to `generate` | ||
5. For multiple attachments, have separate columns e.g. `attachment1`, `attachment2`, etc. | ||
6. See `data/example.csv` for an example. | ||
2. Put your own nunjucks markdown email template in the `templates` folder. | ||
1. You can also edit the default `wrapper.html.njk` file - this is what the markdown HTML will be wrapped in when sending it. It muat _always_ include a `{{ content }}` tag, which will be replaced with the markdown HTML. | ||
3. Fill in the `.env` file with your email credentials. | ||
|
||
Then run the following commands: | ||
|
||
```bash | ||
docsoc-mailmerge generate nunjucks ./data/my-data.csv ./templates/my-template.md.njk -o ./output --htmlTemplate ./templates/wrapper.html.njk | ||
# make some edits to the outputs and regenerate them: | ||
docsoc-mailmerge regenerate ./output/<runname> | ||
# review them, then send: | ||
docsoc-mailmerge send ./output/<runname> | ||
``` | ||
|
||
The CLI tool has many options - use `--help` to see them all: | ||
|
||
```bash | ||
docsoc-mailmerge generate nunjucks --help | ||
docsoc-mailmerge regenerate --help | ||
docsoc-mailmerge send --help | ||
``` | ||
|
||
## What happen when you generate | ||
|
||
1. Each record in the CSV will result in 3 files in `./output/<runname>`: an editable markdown file to allow you to modify the email, a HTML rendering of the markdown that you should not edit, and a `.json` metadata file | ||
2. The HTML files, which is what is actually sent, can be regenerated after edting the markdown files with `regenerate` command (see below) | ||
3. If you want to edit the to address or subject after this point you will need to edit the JSON files; csv edits are ignored. If you edit the CSV, delete all outputs and run generate again. | ||
|
||
## If the .env file is missing | ||
|
||
Use this template: | ||
|
||
```bash | ||
# Fill these in to send emails | ||
DOCSOC_SMTP_SERVER=smtp-mail.outlook.com | ||
DOCSOC_SMTP_PORT=587 | ||
DOCSOC_OUTLOOK_USERNAME= | ||
# Password to docsoc email | ||
DOCSOC_OUTLOOK_PASSWORD= | ||
|
||
# Optional: Fill these in to uplod drafts | ||
# You will need to create an app registration in Entra ID, restricted to the organisation, | ||
# And grant it the following permissions: | ||
# - Mail.ReadWrite | ||
# - User.Read | ||
MS_ENTRA_CLIENT_ID= | ||
MS_ENTRA_CLIENT_SECRET= | ||
MS_ENTRA_TENANT_ID= | ||
``` |
Empty file.
Empty file.
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,124 @@ | ||
# Define the constant for the CSV filename | ||
|
||
# RUN FROM THE `mailmerge` DIRECTORY, NOT THE `scripts` DIRECTORY | ||
|
||
# Change these as needed | ||
CSV_FILENAME = "data/families.csv" | ||
CSV_OUTPUT = "data/families-emails.csv" | ||
|
||
# Example record: | ||
# NOTE: For data protection purposes all names are fictional and all shortcodes replaced with the same value. | ||
# {'child1': 'Omar', | ||
# 'child1shortcode': 'kss22', | ||
# 'child2': 'Alex', | ||
# 'child2shortcode': 'kss22', | ||
# 'child3': 'Simran', | ||
# 'child3shortcode': 'kss22', | ||
# 'child4': 'Jack', | ||
# 'child4shortcode': 'kss22', | ||
# 'id': '16', | ||
# 'parent1': 'Amy', | ||
# 'parent1shortcode': 'kss22', | ||
# 'parent2': 'Aaron', | ||
# 'parent2shortcode': 'kss22'}, | ||
|
||
import csv | ||
from pprint import pprint | ||
import re | ||
|
||
|
||
def read_csv_to_dicts(filename): | ||
""" | ||
Reads a CSV file and returns an array of dictionaries. | ||
:param filename: The name of the CSV file to read. | ||
:return: A list of dictionaries representing the rows in the CSV file. | ||
""" | ||
data = [] | ||
with open(filename, mode="r", newline="", encoding="utf-8") as csvfile: | ||
reader = csv.DictReader(csvfile) | ||
for row in reader: | ||
data.append(row) | ||
return data | ||
|
||
|
||
def gen_new_csv_dict(data): | ||
""" | ||
Takes the MaDs CSV generated by the generateCsv.ts file in the root, | ||
and then for each family, generates a new record for each member of that family with the same data. | ||
This is because the mailmerge tool requires a record per recipient. | ||
:param data: The data to transform. | ||
:return: A new dictionary. | ||
""" | ||
new_data = [] | ||
for row in data: | ||
# List of pairs of (name, shortcode) | ||
recipients = [ | ||
(row["parent1"], row["parent1shortcode"]), | ||
(row["parent2"], row["parent2shortcode"]), | ||
(row["child1"], row["child1shortcode"]), | ||
(row["child2"], row["child2shortcode"]), | ||
(row["child3"], row["child3shortcode"]), | ||
(row["child4"], row["child4shortcode"]), | ||
] | ||
|
||
# Filter empty names/shortcode | ||
recipients = [ | ||
(name, shortcode) | ||
for name, shortcode in recipients | ||
if name is not None and name.rstrip() != "" | ||
] | ||
|
||
# Generate a record per recipient | ||
for rname, rshortcode in recipients: | ||
new_record = { | ||
"id": row["id"], | ||
"subject": "Your Family for DoCSoc Mums and Dads 2024", | ||
"to": rname, | ||
"email": f"{rshortcode}@ic.ac.uk", | ||
"parent1": row["parent1"], | ||
"parent1shortcode": row["parent1shortcode"], | ||
"parent2": row["parent2"], | ||
"parent2shortcode": row["parent2shortcode"], | ||
"child1": row["child1"], | ||
"child1shortcode": row["child1shortcode"], | ||
"child2": row["child2"], | ||
"child2shortcode": row["child2shortcode"], | ||
# Some families have only 2 children | ||
"child3": row["child3"] if "child3" in row else "N/A", | ||
"child3shortcode": ( | ||
row["child3shortcode"] if "child3shortcode" in row else "N/A" | ||
), | ||
"child4": row["child4"] if "child4" in row else "N/A", | ||
"child4shortcode": ( | ||
row["child4shortcode"] if "child4shortcode" in row else "N/A" | ||
), | ||
} | ||
new_data.append(new_record) | ||
|
||
return new_data | ||
|
||
|
||
def main(): | ||
print("Reading data from", CSV_FILENAME) | ||
data = read_csv_to_dicts(CSV_FILENAME) | ||
|
||
print("Transforming data...") | ||
new_data = gen_new_csv_dict(data) | ||
|
||
# Output to CSV | ||
print("Writing data to", CSV_OUTPUT) | ||
with open(CSV_OUTPUT, mode="w", newline="", encoding="utf-8") as csvfile: | ||
fieldnames = new_data[0].keys() | ||
writer = csv.DictWriter(csvfile, fieldnames=fieldnames) | ||
writer.writeheader() | ||
for row in new_data: | ||
writer.writerow(row) | ||
print("Done!") | ||
|
||
|
||
# Example usage | ||
if __name__ == "__main__": | ||
main() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
Hi {{ recipient }}, | ||
|
||
We hope you are looking forward to DoCSoc Mums & Dads which will be starting **today at 12:00 in HXLY 340/341/342**! | ||
|
||
We have an exciting afternoon planned for you - including a quiz and pizza! | ||
|
||
On arrival please go to the DoCSoc desk in the hallway and a committee member will direct you to your table, where you will meet your family. | ||
|
||
Your family ID (table number) is **{{ id }}**. | ||
|
||
Please find information about your family below: | ||
|
||
**Parents:** | ||
- {{ parent1 }} ({{ parent1shortcode }}) | ||
- {{ parent2 }} ({{ parent2shortcode }}) | ||
|
||
**Children:** | ||
- {{ child1 }} ({{ child1shortcode }}) | ||
- {{ child2 }} ({{ child2shortcode }}) | ||
- {{ child3 }} ({{ child3shortcode }}) | ||
- {{ child4 }} ({{ child4shortcode }}) | ||
|
||
You can also find this information on the Mums & Dads website: [https://family.docsoc.co.uk](https://family.docsoc.co.uk) | ||
|
||
We look forward to seeing you today - remember to also go to Sponsors Exhibition in the afternoon, and that Internships 101 will be happening tomorrow at 6pm! | ||
|
||
Kind regards, | ||
[Your name here] ([pronouns]) | ||
DoCSoc [role], on behalf of the DoCSoc Committee. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
<!-- DoCSoc Mail Merge Wrapper - used to wrap results rendered from Markdown --> | ||
<!-- You probably don't need to edit this file, but you can if you want to! --> | ||
<!DOCTYPE html> | ||
<html lang="en"> | ||
<head> | ||
<meta charset="UTF-8"> | ||
<meta name="viewport" content="width=device-width, initial-scale=1.0"> | ||
</head> | ||
<body> | ||
<!-- All HTML wrappers must provide a content block as below to render the content fron the markdown--> | ||
{{ content }} | ||
</body> | ||
</html> |