Docs: add in the mailmerge scaffold & scripts used for MaDs & documen…

…t their usage (#19)

* docs: add in the scripts used for MaDs & document their usage

* docs: add line to README about mailmerge folder

* docs: change template for mailmerge to use placeholder

* chore: make gitignore for mailmerge more thorough

* devops: docker ignore mailmerge

* chore: change a var to a let

Co-authored-by: Nishant Aanjaney Jalan <cybercoder.nishant@gmail.com>

---------

Co-authored-by: Nishant Aanjaney Jalan <cybercoder.nishant@gmail.com>

.dockerignore

@@ -27,4 +27,7 @@ logs
 # the database, lol
 db/
 
-README.md
+README.md
+
+# Please don't include mailmerge as the image doesn't need it
+mailmerge/

README.md

@@ -69,6 +69,8 @@ There also exists an admin page at `/admin` restricted to webmasters as per the
 
 Should you intend to use this code outside of an Imperial context, you will need to change the sign in code, callback code, and `isFresherOrParent` function. These are found in [`hono/auth/auth.ts`](/hono/auth/auth.ts) and [`hono/auth/jwt.ts`](/hono/auth/jwt.ts).
 
+Finally, if you wish to send emails to the families (with their allocations), you can use the `mailmerge` folder. Check the README there for more information.
+
 ## Contributors
 
 ### 2025 Webmasters

generateCsv.ts

@@ -1,6 +1,7 @@
+/** After running this script, see the `mailmerge` dir for how to send out emails with allocations. */
 import type { Student } from './hono/types';
 
-var csv =
+let csv =
   'id,parent1,parent1shortcode,parent2,parent2shortcode,child1,child1shortcode,child2,child2shortcode,child3,child3shortcode,child4,child4shortcode\n';
 
 const authCookie = prompt(

mailmerge/.env.template

@@ -0,0 +1,16 @@
+# Fill these in to send emails
+DOCSOC_SMTP_SERVER=smtp-mail.outlook.com
+DOCSOC_SMTP_PORT=587
+DOCSOC_OUTLOOK_USERNAME=docsoc@ic.ac.uk
+# Password to docsoc email
+DOCSOC_OUTLOOK_PASSWORD=
+
+# Optional: Fill these in to upload drafts (email send itself is only done using the SMTP server above though, it does not need these)
+# (you don't need these for Mums & Dads, or to send emails at all)
+# You will need to create an app registration in Entra ID, restricted to the organisation,
+# And grant it the following permissions:
+# - Mail.ReadWrite
+# - User.Read
+MS_ENTRA_CLIENT_ID=
+MS_ENTRA_CLIENT_SECRET=
+MS_ENTRA_TENANT_ID=

mailmerge/.gitignore

@@ -0,0 +1,14 @@
+# INCOMPLETE!!!!!!
+
+!.env.template
+.env
+
+# Outputs you should never push
+output/*
+!output/.gitempty
+
+data/*
+!data/.gitempty
+
+attachments/*
+!attachments/.gitempty

mailmerge/README.md

@@ -0,0 +1,85 @@
+# Sending family allocations
+
+> [!CAUTION]
+> Ensure you properly gitignore files in this directory to prevent commiting the personal information of the families to the repo, or the generated emails themselves.
+>
+> It is recommended to make a copy of this directory outside the repo and run the scripts from there.
+>
+> When I wrote this README it was 1am in the morning after MaDs 2024, so please forgive the fact I didn't complete a full gitignore for you.
+
+The code in this folder is used to send out family allocations for DoCSoc Mums & Dads. It uses the `docsoc-mailmerge` CLI tool (see [https://github.com/icdocsoc/docsoc-tools/tree/main/email/mailmerge-cli](https://github.com/icdocsoc/docsoc-tools/tree/main/email/mailmerge-cli)) to send out emails to all the parents and children with their family allocations.
+
+How it works:
+1. Generate a CSV with the family allocations using the `generateCsv.ts` script at the root of the repo
+2. Copy it into the `data` folder (e.g. with filename `families.csv`)
+3. Run `scripts/transform-data.py` on it to generate a new CSV, `data/families-emails.csv`, suitable for use by the mailmerge tool (this has to happen as the mailmerge tool expects a record per email, not per family)
+4. Install the `docsoc-mailmerge` CLI tool with `npm install -g @docsoc/mailmerge-cli` (or install it from the docsoc-tools repo directly via `npm link`)
+5. Make a copy of `.env.template` as `.env` and fill in the details (you just need to fill the SMTP server information in for MaDs)
+6. Adjust the email template in `templates/template.md.njk` as needed (e.g. change the from line, date, time, etc.)
+7. Generate the emails with `docsoc-mailmerge generate nunjucks ./data/family-emails.csv ./templates/template.md.njk -o ./output --htmlTemplate ./templates/wrapper.html.njk`
+8. Manually inspect a subset of the generated email HTML files in `./output/<runname>` in a browser - if there are issues, edit the template and regenerate the emails as in 7.
+    1. Double check the JSON files in that directory have valid `email.to` and `email.subject` fields as well; if not ensure when asked during the generation to map the `to` field to something, you map it to the `to` field from the options list.
+9. Test send the emails with `docsoc-mailmerge send ./output/<runname> -t docsoc@ic.ac.uk -n 5` (this will send the first 5 emails to `docsoc@ic.ac.uk`)
+10. Once you are happy with everything, send the emails with `docsoc-mailmerge send ./output/<runname> -s 5` (this will send all emails at 5 seconds intervals to prevent hitting rate limits)
+
+## Original README generated by `docsoc-mailmerge`
+
+To get started:
+
+1. Put your own CSV in the `data` folder, with at the minimum `to` and `subject` columns.
+    1. You can also add `cc` and `bcc` columns (to use them you will need to pass the correct CLI option though)
+    2. `to`, `cc`, and `bcc` can be a space-separated list of emails.
+    3. You can add any other columns you like, and they will be available in the template.
+    4. For attachments, add a column with the name `attachment` with a singular path to the file to attach relative to th workspace root (e.g. `./attachments/image1.jpg`).
+        1. Or, pass the same attachment to every email using the `-a` flag to `generate`
+    5. For multiple attachments, have separate columns e.g. `attachment1`, `attachment2`, etc.
+    6. See `data/example.csv` for an example.
+2. Put your own nunjucks markdown email template in the `templates` folder.
+    1. You can also edit the default `wrapper.html.njk` file - this is what the markdown HTML will be wrapped in when sending it. It muat _always_ include a `{{ content }}` tag, which will be replaced with the markdown HTML.
+3. Fill in the `.env` file with your email credentials.
+
+Then run the following commands:
+
+```bash
+docsoc-mailmerge generate nunjucks ./data/my-data.csv ./templates/my-template.md.njk -o ./output --htmlTemplate ./templates/wrapper.html.njk
+# make some edits to the outputs and regenerate them:
+docsoc-mailmerge regenerate ./output/<runname>
+# review them, then send:
+docsoc-mailmerge send ./output/<runname>
+```
+
+The CLI tool has many options - use `--help` to see them all:
+
+```bash
+docsoc-mailmerge generate nunjucks --help
+docsoc-mailmerge regenerate --help
+docsoc-mailmerge send --help
+```
+
+## What happen when you generate
+
+1. Each record in the CSV will result in 3 files in `./output/<runname>`: an editable markdown file to allow you to modify the email, a HTML rendering of the markdown that you should not edit, and a `.json` metadata file
+2. The HTML files, which is what is actually sent, can be regenerated after edting the markdown files with `regenerate` command (see below)
+3. If you want to edit the to address or subject after this point you will need to edit the JSON files; csv edits are ignored. If you edit the CSV, delete all outputs and run generate again.
+
+## If the .env file is missing
+
+Use this template:
+
+```bash
+# Fill these in to send emails
+DOCSOC_SMTP_SERVER=smtp-mail.outlook.com
+DOCSOC_SMTP_PORT=587
+DOCSOC_OUTLOOK_USERNAME=
+# Password to docsoc email
+DOCSOC_OUTLOOK_PASSWORD=
+
+# Optional: Fill these in to uplod drafts
+# You will need to create an app registration in Entra ID, restricted to the organisation,
+# And grant it the following permissions:
+# - Mail.ReadWrite
+# - User.Read
+MS_ENTRA_CLIENT_ID=
+MS_ENTRA_CLIENT_SECRET=
+MS_ENTRA_TENANT_ID=
+```

mailmerge/scripts/transform-data.py

@@ -0,0 +1,124 @@
+# Define the constant for the CSV filename
+
+# RUN FROM THE `mailmerge` DIRECTORY, NOT THE `scripts` DIRECTORY
+
+# Change these as needed
+CSV_FILENAME = "data/families.csv"
+CSV_OUTPUT = "data/families-emails.csv"
+
+# Example record:
+# NOTE: For data protection purposes all names are fictional and all shortcodes replaced with the same value.
+# {'child1': 'Omar',
+#   'child1shortcode': 'kss22',
+#   'child2': 'Alex',
+#   'child2shortcode': 'kss22',
+#   'child3': 'Simran',
+#   'child3shortcode': 'kss22',
+#   'child4': 'Jack',
+#   'child4shortcode': 'kss22',
+#   'id': '16',
+#   'parent1': 'Amy',
+#   'parent1shortcode': 'kss22',
+#   'parent2': 'Aaron',
+#   'parent2shortcode': 'kss22'},
+
+import csv
+from pprint import pprint
+import re
+
+
+def read_csv_to_dicts(filename):
+    """
+    Reads a CSV file and returns an array of dictionaries.
+
+    :param filename: The name of the CSV file to read.
+    :return: A list of dictionaries representing the rows in the CSV file.
+    """
+    data = []
+    with open(filename, mode="r", newline="", encoding="utf-8") as csvfile:
+        reader = csv.DictReader(csvfile)
+        for row in reader:
+            data.append(row)
+    return data
+
+
+def gen_new_csv_dict(data):
+    """
+    Takes the MaDs CSV generated by the generateCsv.ts file in the root,
+    and then for each family, generates a new record for each member of that family with the same data.
+
+    This is because the mailmerge tool requires a record per recipient.
+
+    :param data: The data to transform.
+    :return: A new dictionary.
+    """
+    new_data = []
+    for row in data:
+        # List of pairs of (name, shortcode)
+        recipients = [
+            (row["parent1"], row["parent1shortcode"]),
+            (row["parent2"], row["parent2shortcode"]),
+            (row["child1"], row["child1shortcode"]),
+            (row["child2"], row["child2shortcode"]),
+            (row["child3"], row["child3shortcode"]),
+            (row["child4"], row["child4shortcode"]),
+        ]
+
+        # Filter empty names/shortcode
+        recipients = [
+            (name, shortcode)
+            for name, shortcode in recipients
+            if name is not None and name.rstrip() != ""
+        ]
+
+        # Generate a record per recipient
+        for rname, rshortcode in recipients:
+            new_record = {
+                "id": row["id"],
+                "subject": "Your Family for DoCSoc Mums and Dads 2024",
+                "to": rname,
+                "email": f"{rshortcode}@ic.ac.uk",
+                "parent1": row["parent1"],
+                "parent1shortcode": row["parent1shortcode"],
+                "parent2": row["parent2"],
+                "parent2shortcode": row["parent2shortcode"],
+                "child1": row["child1"],
+                "child1shortcode": row["child1shortcode"],
+                "child2": row["child2"],
+                "child2shortcode": row["child2shortcode"],
+                # Some families have only 2 children
+                "child3": row["child3"] if "child3" in row else "N/A",
+                "child3shortcode": (
+                    row["child3shortcode"] if "child3shortcode" in row else "N/A"
+                ),
+                "child4": row["child4"] if "child4" in row else "N/A",
+                "child4shortcode": (
+                    row["child4shortcode"] if "child4shortcode" in row else "N/A"
+                ),
+            }
+            new_data.append(new_record)
+
+    return new_data
+
+
+def main():
+    print("Reading data from", CSV_FILENAME)
+    data = read_csv_to_dicts(CSV_FILENAME)
+
+    print("Transforming data...")
+    new_data = gen_new_csv_dict(data)
+
+    # Output to CSV
+    print("Writing data to", CSV_OUTPUT)
+    with open(CSV_OUTPUT, mode="w", newline="", encoding="utf-8") as csvfile:
+        fieldnames = new_data[0].keys()
+        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
+        writer.writeheader()
+        for row in new_data:
+            writer.writerow(row)
+    print("Done!")
+
+
+# Example usage
+if __name__ == "__main__":
+    main()

mailmerge/templates/template.md.njk

@@ -0,0 +1,29 @@
+Hi {{ recipient }},
+
+We hope you are looking forward to DoCSoc Mums & Dads which will be starting **today at 12:00 in HXLY 340/341/342**!
+
+We have an exciting afternoon planned for you - including a quiz and pizza!
+
+On arrival please go to the DoCSoc desk in the hallway and a committee member will direct you to your table, where you will meet your family.
+
+Your family ID (table number) is **{{ id }}**.
+
+Please find information about your family below:
+
+**Parents:**
+- {{ parent1 }} ({{ parent1shortcode }})
+- {{ parent2 }} ({{ parent2shortcode }})
+
+**Children:**
+- {{ child1 }} ({{ child1shortcode }})
+- {{ child2 }} ({{ child2shortcode }})
+- {{ child3 }} ({{ child3shortcode }})
+- {{ child4 }} ({{ child4shortcode }})
+
+You can also find this information on the Mums & Dads website: [https://family.docsoc.co.uk](https://family.docsoc.co.uk)
+
+We look forward to seeing you today - remember to also go to Sponsors Exhibition in the afternoon, and that Internships 101 will be happening tomorrow at 6pm!
+
+Kind regards,  
+[Your name here] ([pronouns])
+DoCSoc [role], on behalf of the DoCSoc Committee.

mailmerge/templates/wrapper.html.njk

@@ -0,0 +1,13 @@
+<!-- DoCSoc Mail Merge Wrapper - used to wrap results rendered from Markdown -->
+<!-- You probably don't need to edit this file, but you can if you want to! -->
+<!DOCTYPE html>
+<html lang="en">
+	<head>
+		<meta charset="UTF-8">
+		<meta name="viewport" content="width=device-width, initial-scale=1.0">
+	</head>
+	<body>
+        <!-- All HTML wrappers must provide a content block as below to render the content fron the markdown-->
+		{{ content }}
+	</body>
+</html>