Dynamically pull repo information for HPC lessons #25

ocaisa · 2021-04-20T14:05:59Z

Fixes #24

.github/workflows/build-and-deploy.yml

ocaisa · 2021-04-20T14:34:18Z

See https://github.com/hpc-carpentry/hpc-carpentry.github.io/runs/2391477702 for where this change is checked

ocaisa · 2021-04-20T14:35:15Z

.github/workflows/build-and-deploy.yml

-          curl --remote-name-all https://feeds.carpentries.org/all_instructors_by_airport.geojson &&\
-          cp all_instructors_by_airport.geojson files/.
+      - name: Setup EESSI to give us an R module
+        uses: eessi/github-action-eessi@v1


All ye who use HPC, take note of this awesome action!

This is giving us R with over 900 packages installed in just over 30 seconds!

Super cool!

R/help_wanted_issues.R

ocaisa · 2021-04-20T14:41:01Z

For this to work completely we will need to start using GitHub topics consistently on our repos. In particular things need to be at least tagged with lesson and hpc-carpentry before they will appear. The nice thing is that they will still show up even if they get migrated to the Carpentries incubator.

tkphd · 2021-04-20T14:44:57Z

If those "tags" are Labels, we can configure Issue Templates for each repo to apply them automagically going forward. Then we only have to manually intervene with... all of history.

ocaisa · 2021-04-20T14:51:20Z

If those "tags" are Labels, we can configure Issue Templates for each repo to apply them automagically going forward. Then we only have to manually intervene with... all of history.

No, not labels (that's what I thought too first). The labels are used for the help wanted data, but the topics are set globally for the repo in the "About" section of the repo homepage (see https://github.com/hpc-carpentry/hpc-shell where I set them for testing this)

tkphd · 2021-04-20T14:52:27Z

Oh, cool!! That's even easier!

ocaisa · 2021-04-20T14:54:58Z

At the moment we don't actually have a "Help Wanted" page (I think) but with this we can add one based on the Carpentries.

It also opens the door to being able to include lessons from other providers under separate organisations (and we can group them accordingly)

tkphd · 2021-04-20T14:56:25Z

Yes, this is fantastic. I introduced a "Help Wanted" page in #22, but hybridizing this feed with The Carpentries' code is a much better solution.

tkphd · 2021-04-20T14:56:50Z

Applied the lesson and hpc-carpentry tags to all our lesson repositories, FYI. Also english, alpha, and (for Chapel) pre-alpha.

pages/community-lessons.md

ocaisa · 2021-04-20T15:39:14Z

Applied the lesson and hpc-carpentry tags to all our lesson repositories, FYI. Also english, alpha, and (for Chapel) pre-alpha.

I tested this locally and they all appear fine

Remove use of json plugin (for now) Synch help-wanted page with upstream

ocaisa · 2021-04-21T09:10:37Z

Gemfile

@@ -4,5 +4,5 @@ gem 'faraday', '0.17.3'

 group :jekyll_plugins do
  gem 'github-pages'
-  gem 'jekyll-get-json', "~> 0.0.2"
+#  gem 'jekyll-get-json', "~> 0.0.2"


Since we are not pulling data from AMY this is not needed (yet)

ocaisa · 2021-04-21T09:14:05Z

pages/community-lessons.md

+
+<hr> 
+
+## HPC Carpentry Listings


If this starts getting populated, it should really be broken off into a separate page

I have a lesson I would like to include here

pages/teach.md

ocaisa · 2021-04-21T13:35:04Z

For the record, here are the changes required to the upstream code at https://github.com/carpentries/feeds.carpentries.org/tree/main/R

diff --git a/home/alanc/feeds.carpentries.org/R/community_lessons.R b/R/hpc-carpentry_lessons.R
index b0d33b8a..a8ead1c5 100644
--- a/home/alanc/feeds.carpentries.org/R/community_lessons.R
+++ b/feeds/hpc-carpentry_lessons.R
@@ -1,6 +1,7 @@
 source("R/utils.R")
 
 LIFE_CYCLE_TAGS <- c("pre-alpha", "alpha", "beta", "stable")
+# The tags below will be filtered out in the json
 COMMON_TAGS <- c(
   "carpentries",
   "carpentries-incubator",
@@ -43,10 +44,11 @@ check_repo_info <- function(.d, fields) {
 make_community_lessons_feed <- function(path, ...) {
 
   carp_inc <- get_org_topics("carpentries-incubator")
-  carp_lab <- get_org_topics("carpentrieslab")
+  hpc_carp <- get_org_topics("hpc-carpentry")
 
-  res <- dplyr::bind_rows(carp_inc, carp_lab) %>%
+  res <- dplyr::bind_rows(carp_inc, hpc_carp) %>%
     dplyr::select(-private) %>%
+    dplyr::filter(grepl("hpc-carpentry", github_topics)) %>%
     dplyr::filter(grepl("lesson", github_topics)) %>%
     extract_tag(
       life_cycle_tag,
@@ -71,4 +73,4 @@ make_community_lessons_feed <- function(path, ...) {
 
 }
 
-make_community_lessons_feed("_data/community_lessons.json")
+make_community_lessons_feed("_data/hpc_lessons.json")

diff --git a/home/alanc/feeds.carpentries.org/R/help_wanted_issues.R b/feeds/help_wanted_issues.R
index d0f79ddd..cb154f6e 100644
--- a/home/alanc/feeds.carpentries.org/R/help_wanted_issues.R
+++ b/feeds/help_wanted_issues.R
@@ -1,4 +1,4 @@
-source("R/utils.R")
+source("feeds/utils.R")
 
 new_tbl_github_issues <- function(url = character(0),
                                   title = character(0),
@@ -97,40 +97,17 @@ get_gh_issues <- function(owner, repo, labels) {
     )
 }
 
-keep_opted_in <- function(orgs) {
-
-  at_opted_in <- airtabler::airtable(
-    base = "appeZJGnGremE1MYm",
-    tables = "Repositories"
+keep_hpc_carpentry_repos <- function(orgs) {
+  dplyr::filter(
+    orgs, carpentries_org == "hpc-carpentry"
   )
-
-  opted_in <- at_opted_in$Repositories$select_all() %>%
-    dplyr::mutate(lesson_program = tolower(lesson_program))
-
-  dplyr::left_join(
-    orgs, opted_in,
-    by = c(
-      "carpentries_org" = "lesson_program",
-      "repo" = "repository"
-    )
-  ) %>%
-    dplyr::filter(display_help_wanted)
-
 }
 
 keep_other_repos <- function(orgs) {
+  # Repos that wish to opt in should be manually added here
   other_repos <- tibble::tribble(
     ~carpentries_org, ~repo,
-    "carpentries", "glosario",
-    "carpentries", "glosario-r",
-    "carpentries", "glosario-py",
-    ##    "carpentries", "glosario-js",
-    "carpentries", "carpentries.org",
-    "carpentries", "docs.carpentries.org",
-    "datacarpentry", "datacarpentry.github.io",
-    "librarycarpentry", "librarycarpentry.github.io",
-    "carpentryconnect", "carpentryconnect.org",
-    "swcarpentry", "website"
+    "carpentries-incubator", "hpc-intro",
   )
 
   dplyr::inner_join(
@@ -141,13 +118,8 @@ keep_other_repos <- function(orgs) {
 
 
 list_organizations <- c(
-  "Data Carpentry" = "datacarpentry",
-  "Software Carpentry" = "swcarpentry",
-  "Library Carpentry" = "librarycarpentry",
-  "CarpentriesLab" = "carpentrieslab",
-  "The Carpentries Incubator" = "carpentries-incubator",
-  "The Carpentries" = "carpentries",
-  "CarpentryConnect" = "carpentryconnect"
+  "HPC Carpentry" = "hpc-carpentry",
+  "The Carpentries Incubator" = "carpentries-incubator"
 )
 
 list_help_wanted <- purrr::imap_dfr(
@@ -158,14 +130,14 @@ list_help_wanted <- purrr::imap_dfr(
       ignore_pattern = "^\\d{4}-\\d{2}-\\d{2}"
     )
 
-    lessons <- orgs %>%
-      keep_opted_in()
+    hpc_carpentry_repos <- orgs %>%
+      keep_hpc_carpentry_repos()
 
     other_repos <- orgs %>%
       keep_other_repos()
 
     dplyr::bind_rows(
-      lessons,
+      hpc_carpentry_repos,
       other_repos
     )  %>%
       dplyr::distinct(carpentries_org, repo, .keep_all = TRUE) %>%

tkphd

This looks great!!! A few minor quibbles, but this is really good stuff.

_config.yml

pages/community-lessons.md

pages/teach.md

R/help_wanted_issues.R

_config.yml

tkphd · 2021-04-21T13:44:33Z

@ocaisa the EESSI trick is neat for getting an R environment. Is there a way to run it locally, so I can make serve and see the updated list of lessons? Can we document that step?

ocaisa · 2021-04-21T14:09:36Z

@ocaisa the EESSI trick is neat for getting an R environment. Is there a way to run it locally, so I can make serve and see the updated list of lessons? Can we document that step?

@tkphd There are a few example installation scripts at https://github.com/EESSI/eessi-demo/tree/main/scripts but they are slightly outdated (but should still work). You can now use https://github.com/EESSI/filesystem-layer/releases/download/latest/cvmfs-config-eessi_latest_all.deb as the configuration file.

After that you need to source /cvmfs/pilot.eessi-hpc.org/latest/init/bash and that will give you access to the stack for your CPU. You can then follow the same steps as the CI:

GITHUB_PAT=XXXXX make data
make serve

…arpentry.github.io into generate_lesson_json

…ntry.github.io into generate_lesson_json

tkphd

This looks great! Thanks @ocaisa!

Alan O'Cais added 2 commits April 20, 2021 13:30

Dynamically pull repo information for HPC lessons

20267a6

Update to use latest json data

526c20b

ocaisa commented Apr 20, 2021

View reviewed changes

.github/workflows/build-and-deploy.yml Outdated Show resolved Hide resolved

Fix action

c760145

ocaisa commented Apr 20, 2021

View reviewed changes

.github/workflows/build-and-deploy.yml Outdated Show resolved Hide resolved

ocaisa added 2 commits April 20, 2021 16:16

Fix CI

fef50ed

Update build-and-deploy.yml

45747e2

tkphd mentioned this pull request Apr 20, 2021

restore HPC content, remove Carpentries content #22

Merged

ocaisa commented Apr 20, 2021

View reviewed changes

R/help_wanted_issues.R Outdated Show resolved Hide resolved

ocaisa commented Apr 20, 2021

View reviewed changes

pages/community-lessons.md Outdated Show resolved Hide resolved

Alan O'Cais added 3 commits April 21, 2021 09:19

Add hpc-carpentry as tag to filter

1216dd3

Merge branch 'main' into generate_lesson_json and update lesson page

15ddec2

Fix broken link

a70358a

Remove use of json plugin (for now) Synch help-wanted page with upstream

ocaisa commented Apr 21, 2021

View reviewed changes

pages/teach.md Show resolved Hide resolved

tkphd mentioned this pull request Apr 21, 2021

Teach is a 404 #28

Closed

Add a whitelist to Help Wanted

75f2b93

tkphd requested changes Apr 21, 2021

View reviewed changes

_config.yml Show resolved Hide resolved

pages/community-lessons.md Outdated Show resolved Hide resolved

pages/teach.md Show resolved Hide resolved

R/help_wanted_issues.R Outdated Show resolved Hide resolved

_config.yml Show resolved Hide resolved

Alan O'Cais added 2 commits April 21, 2021 13:45

Remove unnecessary duplication

7bb1e52

Address review comments

59c6fd7

Alan O'Cais and others added 7 commits April 21, 2021 14:38

Update README with new instructions

f0dc50d

Document feed script dependencies

f9dc447

Typo

344dc96

Merge branch 'generate_lesson_json' of github.com:hpc-carpentry/hpc-c…

6211c97

…arpentry.github.io into generate_lesson_json

Update README.md

d8b9972

Update R feed dependencies, install instructions

3069c2c

Merge branch 'generate_lesson_json' of github:hpc-carpentry/hpc-carpe…

377f9a1

…ntry.github.io into generate_lesson_json

tkphd approved these changes Apr 21, 2021

View reviewed changes

tkphd merged commit 0bf965d into main Apr 24, 2021

tkphd deleted the generate_lesson_json branch April 24, 2021 14:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamically pull repo information for HPC lessons #25

Dynamically pull repo information for HPC lessons #25

ocaisa commented Apr 20, 2021

ocaisa commented Apr 20, 2021

ocaisa Apr 20, 2021

ocaisa Apr 20, 2021

tkphd Apr 20, 2021

ocaisa commented Apr 20, 2021

tkphd commented Apr 20, 2021

ocaisa commented Apr 20, 2021

tkphd commented Apr 20, 2021

ocaisa commented Apr 20, 2021

tkphd commented Apr 20, 2021

tkphd commented Apr 20, 2021 •

edited

Loading

ocaisa commented Apr 20, 2021

ocaisa Apr 21, 2021

ocaisa Apr 21, 2021

ocaisa Apr 21, 2021

ocaisa commented Apr 21, 2021 •

edited

Loading

tkphd left a comment

tkphd commented Apr 21, 2021

ocaisa commented Apr 21, 2021

tkphd left a comment

Dynamically pull repo information for HPC lessons #25

Dynamically pull repo information for HPC lessons #25

Conversation

ocaisa commented Apr 20, 2021

ocaisa commented Apr 20, 2021

ocaisa Apr 20, 2021

Choose a reason for hiding this comment

ocaisa Apr 20, 2021

Choose a reason for hiding this comment

tkphd Apr 20, 2021

Choose a reason for hiding this comment

ocaisa commented Apr 20, 2021

tkphd commented Apr 20, 2021

ocaisa commented Apr 20, 2021

tkphd commented Apr 20, 2021

ocaisa commented Apr 20, 2021

tkphd commented Apr 20, 2021

tkphd commented Apr 20, 2021 • edited Loading

ocaisa commented Apr 20, 2021

ocaisa Apr 21, 2021

Choose a reason for hiding this comment

ocaisa Apr 21, 2021

Choose a reason for hiding this comment

ocaisa Apr 21, 2021

Choose a reason for hiding this comment

ocaisa commented Apr 21, 2021 • edited Loading

tkphd left a comment

Choose a reason for hiding this comment

tkphd commented Apr 21, 2021

ocaisa commented Apr 21, 2021

tkphd left a comment

Choose a reason for hiding this comment

tkphd commented Apr 20, 2021 •

edited

Loading

ocaisa commented Apr 21, 2021 •

edited

Loading