workflow #3 - adding content/PR to qubes-community

[ghost]

This issue describes how to submit content to qubes-community, from a regular user's standpoint, shortcomings, etc. The text of this post should be updated/edited to reflect changes arising from comments.

Edits:

• Mar 19: subfolders + naming suggestions.
• Mar 17: instructions are in the landing page + reflect awokd's comments

---

Workflow:

• content is submitted with PRs (either directly or with issue->PR)
• further discussion - if any - happens in the PR.
• optionally the PR may be "redirected" to the qubes-doc fork (or even directly to the official repo).
• if the issue is not deemed fit for inclusion the issue is closed indicating the reason of the content/suggestion/... rejection

Instructions: https://github.com/Qubes-Community/Qubes-Community.github.io/blob/master/README.md

Organization of the repo:

• agreed: no subfolder in the docs/ folder
• agreed: subfolders, with basic index/list in readme

TBD- Naming suggestions for subfolders:

multimedia/
productivity/
dom0/
backup/
vm-creation/ 
power-mgnt/
server/            (running servers in Qubes)
staging/          (temp)

the staging folder is a catch-all location for uploaded stuff that isn't complete (eg. wire-chat-server); we'll have to decide what to do with the left bits before the project is advertised

[awokd]

I was thinking subfolders in docs/ might help reduce clutter but it's probably over-complicated at this point. Let's remove them for now and add some later if needed? Will be easier to come up with categories then too. I guess content would be published here the same way as any other repo, but I'm not sure if that's what you are asking.

We could take the same approach to the scripts folder.

Drawback is if people link to them, links will change if they get moved to subfolders afterwards.

[ghost]

Documentation: OK without subfolders. The doc/README.md can make it clear that work-in-progress contributions are in issues or PRs and explain how to search them. Also, nothing prevents from temporarily linking work-in-progress docs in the readme if needed.

I don't know what to think about the scripts/ folder though:

1. without subfolders: it will quickly become a mess so someone will have to organize them in different lists in the readme; that's a bit error prone and might be quite some work
2. with subfolders (eg. 'productivity', 'power-management', ...): difficult to move the script later and require users to click into each subfolder to see what's in them.

Maybe 1- is the way to go. While we're at it, 'scripts' is maybe not the right name - there can be other stuff than scripts (eg. programs, data, ...)

[awokd]

Yes, maybe you're right and we need to figure out script (how about "code"?) categories first. Can give an overview of each folder in the readme so people don't necessarily have to click through them.

[Aekez]

Good point about the name, indeed perhaps instead call it code, or which word that may be the most suitable.

What are the thoughts about this page, https://github.com/Qubes-Community/Qubes-Community/blob/master/README.md this is sort of how I envisioned it, but we need to discuss if that's how we do it or not though. I.e. will the status icons serve as official statuses, etc. I think the status icons makes it very easy to keep it updated, clean and clear.

Done this way, as in the link above, I don't think folders become too big of an issue, if people only look at the first overview (and don't jump directly into a sub-folder without coming from the overview first), since it can link directly to both scripts and folders with multiple of scripts. So only keeping smaller README.md in sub-folders if there are multiple of scripts to pick between, otherwise keeping no extra overviews beyond the front one when linking to singular scripts. How do you guys feel about it this way? prefer it different? if so please share, I'm definitely open to discuss alternative folder systems.

The dilemma of the permission system
It should be possible to have a protected branch on the scripts in addition to the issue tracker, and then allow people to upload or submit changes to existing scripts in other branches, and then have moderators accept changes to scripts or new scripts? The moderator then only keeps track of changes, and isn't part of any review system. For example, the moderator can change an old stable status icon to new, requiring review status icon, if a script is updated, and now requires a new review. That both warns users, and also invites in reviewers from the community to help take a look at it.

We can then also have scripts from the issue system too, if people are not familiar with git/terminal, or don't know what to do. So we have two ways to work with scripts, either directly from the pull requests system, or from the issue tracker system.

[Aekez]

Perhaps the solution might be to remove the focus from github and move it to the webpages? I.e. people who can navigate the folders will do fine, but people who can't, can just use the website instead?

[awokd]

Assuming:

A) Everything's in the Contents repo except documents intended for official qubes-doc
B) We split scripts into subfolders

Then, https://github.com/Qubes-Community/Qubes-Community/blob/master/README.md should just have an overview of the general script subfolder contents. Each subfolder could have its own README/web page with more detailed descriptions/icons. Adding icons to newly submitted scripts doesn't seem too bad, but will there be ongoing maintenance required of them and who would handle? And yes, since this is a repo like any other, we could process submissions to it with PRs.

[awokd]

Misclick

[Aekez]

np's, that close button is daringly close to that comment button.. seems a bit too easy to miss-click, a bad button placement.

Putting scripts in a sub-folder for themselves as you're suggesting seems to make good sense. I believe below bullet-points are the thoughts you have in mind with it, in which case I also agree with;

• It makes it more easy for people to find specific scripts for smaller simpler purposes, rather than full codes or projects.
• It also removes the clutter from the front-content-page, making it easier to navigate content without getting buried in a long page to scroll down.

Perhaps we can modify the icons to require less work now, but in the future, it might be ideal to have a more proper review system in place, if we can get it working through volunteer effort. Maybe other ideas can come forward to replace status icons by then too. But generally, I think it's a good idea if we can make people aware of what we do know about the status of a script or project work. A picture is said to be more than a thousand words, though probably not a thousand words for a status icon, but it does make it more easy to express the status, rather than writing it down, which both takes longer to write, but also longer to read for the end-user.

So the worry about status icons, is whether it'll introduce too much work to keep it updated? we need a balance between sustainability and available user-information? So your thoughts here is to use status icons that doesn't require too frequent changes?

[Aekez]

In general it would be good to have an easy way to tell people if something lacks reviews, or if it has been changed since last review. Other information statuses are perhaps not needed as much.

Maybe this can be done in a more efficient manner?

[awokd]

I follow what you mean now, thanks! Since this is a repo, the only way content would be supplied to it is through a PR at some point, which means there would be some level of review on it. Were you picturing it being wide open to edits from anybody?

[Aekez]

np's! I also have some trouble keeping track, we're having multiple parallel discussions, and it gets tough keeping the dynamics in each one together in a single overall discussion :)

In general more open to edits yes, but I'm hoping that it can be made possible in a different way so it doesn't become too open and risks adding reliability and trust issues to the content. I hope the below work-flow system can solve this issue partially, although it can probably never be fully solved.

In the layout I'm thinking about I'm trying to separate moderators work-flow with the reviewers work-flow, but I'm still pondering about if that is a good idea to suggest or not. In general, a moderator can ensure that everything goes to the right places, keeping the structure, status icons, etc. in working order. While the reviewer instead focuses entirely on the content itself, and not where it's put etc. This way a moderator can help organize code/scripts which they cannot review, and forward it for reviewers to later get into the details.

Moderator - first work-flow: Accepting scripts/code/organizing-structure/keep-links-in-working-order.
Reviewers - second work-flow: Quality-checks the content of script/code/text/projects.

The thought here is also that moderators can do a bit of optional reviewing, while reviewers can do a bit of optional moderating, so that they're not locked into their roles, but can also do a more holistic work, however it also gives them freedom to avoid "too much work", which overall, might allow more people to help the community going, overall adding more work done as a whole. In general, keeping them separated into two volunteer tasks, I believe it can make it more easy and increase the overall content/quality.

So in order to get anything uploaded on the community site, it only needs to pass the moderator. But in order to stay on the site, it'll need to pass the reviewers judgment.

Strengths and weaknesses of a reviewer/moderator system

• Reviewers might be highly skilled people, but they are not interested in all the management work. By freeing them from all the management, it might be easier to attract highly skilled people who can quality-check code/scripts/docs/guides. They might be busy with something else, and can only stop by once in a while for a short moment, and may drop a few comments and a review, etc. before they move on again.
• By allowing working but unfinished content which hasn't been reviewed yet, it also opens up openness and transparency, and it allows greater cooperation, teamwork, discussions, etc. on how to finish up something half-done. Reviewers here can still step in if they feel like it, but the general idea is to allow un-finished work to be posted for others to view.
• Since moderators are not automatically burdened by having to quality review, they can more easily help users keeping everything updated and in right working order. For example, they can more easily help updates on accepting changes on work-in-progress updates, flag when something is ready for reviews, put files in the right folders, ensuring links work, updating the content pages, etc.
• other strengths/weaknesses, there is more to reflect on, but this post is already getting rather big, and this initial post should be enough to get the basic idea out.

A perspective
Purely speculative, say we got for example 10 moderators, and say, maybe 30 reviewers, some who are much more active, and others who are much less active. Then the work burden might not be so bad for everyone, and we might be able to have a system that avoids bottlenecks. If we merge the reviewer and the moderator into a single roles, it might become too heavy a full-time-job rather than being just a volunteer, and it might also slow down everything, having to wait for proper reviews before anything can be posted. It might also scare away a lot of people, especially the skilled programmers who are probably busy with many other things as well.

This is just some loose thoughts, I haven't thought it all through yet either, but in general I feel it might add good value if we keep reviewers and moderators separated. It also makes it easier to attract skilled people, as well as people with much less programming skills but can do just fine doing management/moderator work (like me for example, as I don't have deep programming skills, I certainly can't do proper reviews).

And if the skilled programmers are not bound to every-day tasks, and can just drop by and do some reviews here and there, as they feel like it, then it might give a boom to the number and amount of reviews.

[Aekez]

This is how I imagine it:
Owners - works to ensure the long-term survival; and supports the well-being of the community.
Reviewers - skilled programmers who is not responsibility bound, but can drop in any-time.
Moderators - Ensures the day-to-day details work, i.e. links, folders, titles, flags, copy/paste'ing, assignments, keeping used language civil, etc.
Producers - Actively or semi-actively building community content for consumption.
Members - Consumption. Engages in the community in some way or another not in the above categories.
Users - Consumption, but not engaging in the community in any way (invisible members).

[ghost]

The organization you propose is nice but short to mid term it's totally overkill for this project
Given that we'll only get a small subset of Qubes OS' user base, it doesn't really make sense to have a lot more levels than the Qubes OS organization.
I'd settle on 3 levels - owner, reviewer and user. Reviewers have write access to the repos, the "owner" level is only technical, it's there not prevent reviewers from deleting repositories or the whole organization. So basically the owners are reviewers and we come down to two categories: contributors and reviewers. Nothing prevents from adding categories when needed.

[Aekez]

Sounds good to me, and an excellent point too, I was thinking more long-term which isn't suitable for us right now. As you're suggesting we can just keep it as an example-template for future discussions laying around when whenever such a time may come.

I'm not quite comfortable with the permission write-access system yet, but I suppose I will need to get used to thinking differently about it. So to keep file-versions and backups, and be less worried about write-access disasters. When you're putting it that way it seems interesting, we could indeed remove a full top-level like that.

[ghost]

Re- @awokd's comment above: Looks like we're OK to split scripts (code/) into subfolders.

Moving to subfolders + updating the readme(s):

• subfolders names/categories ?
• changes: "direct" commits or PRs ?
• who'll tackle that ? [edit: I'm OK to give it a try if nobody's interested]

[awokd]

Not sure whose changes you're referring to. If it's our own, I say direct commits for now while we're getting set up, but at some point we should switch to PRs too. If you mean for general use, I think we want to require PRs so there will be at least some level of review on scripts in particular.

[ghost]

I was referring to our own changes. OK for direct commits.

[ghost]

fyi I've updated the OP with folder naming suggestions. Let's agree on those first so that we don't end up redoing the main readme over and over when the subfolder names change.

[awokd]

From what I remember of the scripts I've seen in the mailing list etc., those seem like good categories.

[ghost]

OK ; I'll go on with those names then, we can always rename the folders later if needed.

docs/ folder: we agreed not to have 'staging' subfolders like '1)-work-in-progress' and '3)-preparing-qubes-docs' because that's what PRs and issues are for. But what about subfolders for organizing the documentation ?

Choices:

1. no subfolders at all (= a well maintained index in readme.md)
2. mirror qubes-doc subfolders (maybe doesn't apply to the doc submissions we'll receive ?)
3. create our "own" categories/subfolders when needed ?

Thoughts ?

[awokd]

I guess the second option is best without knowing what will go in there yet.

[ghost]

fyi I reorganized the folders and readmes and I've updated the main page (landing page) with a 'content' section.

I've simplified things quite a bit

• removed icons ; reason: there were too many (couldn't figure what meant what). I like the idea of icons though, we should revisit that in the related discussion.
• removed disclaimers in readmes here and there; reason: listing it once in the main page is enough.

I've left some stuff in code/staging that I think we should remove or transform as PR.

I didn't create empty folders in docs/ and code/ ; we'll create them when needed.

just to be clear - the structure isn't meant to be set in stone so we can re-add content that I've removed where appropriate.

[Aekez]

Looks good to me. I also like the idea of being dynamic and flexible.

I won't be very active this weekend btw, though I might be able to post here and there, but can't do any work for a couple of days.

[ghost]

Looks good to me

thanks :)

before closing the issue:

• OK to delete icons ? (we can re-introduce it later if we decide to use icons)
• I'd like to delete code/staging: what to do with its content ?
  • The windows doc is similar to the official PR, but I see there's additional commands and explanations ; comment/modify the official PR ? create a doc (or PR?) here with only the additional stuff ?
  • create minimal vm: link to the related official doc, and list the commands as if they were run in the VM rather than using qvm-run in dom0 ?
  • qvm-backup-to-appvm: : isn't backup related ; +, why not use qvm-run vmname 'mv ...' instead of helper script. ?

[awokd]

How about renaming Contents/Icons to Contents/Resources? Then we can keep those icons around for later use.

I tried to check the history on those staging files but only see when they got moved to the subfolder; nothing earlier. Where did they come from? Could maybe move them to individual issues (I can do this if you want) and follow the Issue -> PR workflow on them.

[ghost]

Contents/icons : bikeshedding - what about Contents/_res instead of Contents/Resources ? I'd prefer to avoid 'resources' since that's a term used throughout the documentation and people will likely click on that folder thinking there's some community doc/code.

staging: the files are from @Aekez ; but yesterday I did a git push before git pull, it asked me to commit something, I cancelled, ran git pull and then changed some stuff and did git commit / git push ; Github then showd that I merged a branch - which I did not (still didn't have the time to learn git properly); I don't know what happened; maybe the history was flushed (but then I don't understand why - the commands I've run are pretty basic).

[ghost]

Could maybe move them to individual issues (I can do this if you want) and follow the Issue -> PR workflow on them

(forgot to reply): OK, you can go ahead... (If you don't have time I can do that tomorrow)

[awokd]

_res is good since they're sort of internal resources.

I was thinking tomorrow too, to give @Aekez a chance to respond. :)

[ghost]

fyi I've renamed icons/ to _res/ and I've removed the staging/ folder.

@one7two99 : I see that the howto-build-win7-appvm doc was yours ; I was thinking about creating a PR here but almost all the instructions are in this official qubes-doc PR. I spoted a few commands that aren't in the PR (eg. disable hibernation, ...), feel free to add comments to the official PR and I'll update the doc there.

@Aekez : create-minimal-sys-vms: I'm not sure about that one, anyone creating a minimal template will want to tune it to his/her needs (for instance I don't use gnome-terminal and I don't wan't sound in my minimal templates) so the unattended script will have to be adapted, which kind of defeats its "unattended" purpose. What about having a 'recommended packages' doc page instead, with copy/paste instructions ?

[ghost]

a few questions:

• I've added a link to a cheat sheet in the docs/ and was wondering what the policy on adding links to third party code/doc should be:
  1. need to ask permission before linking
  2. link and send a courtesy PM/mail to owner when possible (IMO the right way)
  3. simply link
• where to add links: in code/ or docs/ ?
  • take for instance the 'qubes-server' by Rudd-O: it can be linked in code/server/readme.d or in docs/readme.md (I've left the link in both place for now).
  • I wanted to link to a 'productivity' script (popup menu) but was wondering where to put it: in code/productivity ? or in docs/productivity ? More generally, do we assume that people look for content in both docs/ and code/? If not, maybe we can do away with the main code/readme.md and have the main content in docs/readme.md ; thoughts ?
• I've updated the docs/readme to stick (where possible) to the official qubes-doc folder structure ; any remark ?

[awokd]

link and send a courtesy PM/mail to owner when possible

is also good because it can let them know about this effort.

On the links, how about inside a 3rdparty document in root (i.e. not code or docs)? The readme seems fine to me.

[ghost]

how about inside a 3rdparty document in root

I don't know. It'll be yet another place where one has to look to find content.

On a related note, the more I think about it the less I see the purpose of the code/ folder:

• bitrot: without write privileges, how many contributors would bother to go through PRs to update their code (even for something as trivial as correcting a typo) ? And even if they do, changes have to be acked by a member with write privileges on the repo, which will lead to delays and increased work for those members. There's the same problem with hosted repos - but it's worse here because we can't give granular write privileges on single files or folders.
• despite the disclaimer there's a false sense of security if the code is hosted here (like if it was reviewed/QA's).

A solution is to only link to contributors' resources (on github, gitlab, ...). It could be an issue when a contributor doesn't have a public web page, but how often does that happen ?

(that's a bit of a radical change for the content here - not pushing to go this way but will be interesting to read what others think).

[awokd]

I think the code/ folder is still useful to keep around, even with the caveats you mentioned. We can link to contributors comfortable with Github already, but there are often items in the mailing list that would be good to capture (e.g. https://www.mail-archive.com/qubes-users@googlegroups.com/msg20088.html). One of our goals is to make is easy for people to contribute, so those who don't want to hassle with Github etc. can submit small scripts in Issues that get hosted here. That's my thought, at least!

[ghost]

OK. I also thought about copy/pasting content from ML posts but thought it could go in docs/ ; interestingly that bash autocomplete stuff is also in my list of doc suggestions.

I'll add a note in code/readme about linking to third party resources/pages when possible over hosting (for the reasons mentioned in my OP).

[awokd]

See also the discussion in QubesOS/qubes-doc#601.

[ghost]

The official link policy make sense for qubes-doc (to "keep qubes-doc a trustworthy place"), but does it for this project ? Either we link to some resources and take the risk that someone can update them with bad code/instructions, or we don't link to anything in which case it'll become a mess to keep the hosted content synchronized. What do you suggest ?

[edits]:

• it might not be a black/white choice - we could link to resources hosted by well-known qubes user and host stuff otherwise
• re-reading my post, it may sound a bit harsh - wasn't meant to. I'm simply wondering what is the proper policy

[awokd]

I wasn't very clear either! I meant PR601 as an example of content we could host here that has both code and a doc, not as a policy for us to follow. I think it's good for the official site to minimize use of 3rd party content, while ours is more caveat emptor. So we would link where it's most convenient, and host otherwise. Does that make sense?

[ghost]

yep - makes sense now :)

[ghost]

following up on deleting the dom0 folder (good idea): what to do with its current content ?
eg.:

• ls-qubes.sh -> monitoring/ folder ? The script is by @one7two99, it displays the nb. of qubes + mem used and its output can be displayed in a text applet in a panel - that's a good addition to qubes's applet.
• R4-universal-update-script.sh -> update-scripts/ folder ?
• qvm-copy-to-dom0 -> ?

[awokd]

Your suggestions sound fine to me. Maybe productivity for qvm-copy-to-dom0 (making this too easy to do makes me uncomfortable), but I can see that quickly becoming a dumping ground too. I guess let's just see what types of other scripts end up there and organize as needed with other folders?

[ghost]

OK
I also don't think it's a good idea to make it too easy to copy to dom0, so I'll delete the qvm-copy-to-dom0 script and I'll add the qvm-run --pass-io... instructions to a "how to copy files to dom0" documentation page (with the sparse stuff in #12).

[ edit: done ; new copy-to-dom0 doc is here ]

[ghost]

how about closing this issue ? we've tested the workflow a few times and we settled on the structure of the doc/ and code/ folders. Anything else to discuss ?

[edit: will close if no comment received within a few day... ]

[ghost]

asking here instead of opening another discussion: as explained here github doesn't show a file's history after it's been moved. We'll probably often move files, at least at the beginning until we find the right folder structure, so we'll loose credit/authorship (it's not 'lost' actually, the operations stays in git's log but they're not displayed in github).

Q:

1. do we suggest to add a contributors: x,y,z, ... line eg. at the bottom of docs ?
2. same as above, but mandatory ?
3. or, no credit at all ?

personally I don't care about authorship for my docs but I expect others to have a different opinion ; thoughts ?

[one7two99]

Hello, Ivan <notifications@github.com> schrieb am Fr., 6. Apr. 2018, 10:24:

asking here instead of opening another discussion: as explained here <#15 (comment)> github doesn't show a file's history after it's been moved. We'll probably often move files, at least at the beginning until we find the right folder structure, so we'll loose credit/authorship (it's not 'lost' actually, the operations stays in git's log but they're not displayed in github). Q: 1. do we suggest to add a contributors: x,y,z, ... line eg. at the bottom of docs ? 2. same as above, but mandatory ? 3. or, no credit at all ? personally I don't care about authorship for my docs but I expect others to have a different opinion ; thoughts ?

I don't need credit for "my" docs even more that it is likely that it will be improved by others (hopefully). One thing why I like the idea f contributors: This would allow others to contact them directly, something I like to offer and it is more likely to get a feedback like: "this was nice, everything worked out" or "help I am lost at step ...". Having an email address from the contributor (if they want to share it) would lower the barrier to ask questions as not everyone knowns how to use GitHub. [799]

…

[awokd]

email address from the contributor

I think original credit is important and can be covered by a line on the bottom but suggest their @ github ID instead so they don't get scraped by spambots.

[ghost]

thank you both for your comments.

a last question then: should we keep only a single contributor(s): ... line at the end of the pages, or should it be like a changelog, eg.

userX: initial version
userY: fix blahblah
userZ: add section xyz

(maybe with dates too ?)

[awokd]

My vote's for single contributor and to rely on Github history for the rest.

[awokd]

Nice summary of the contributor workflows on https://qubes-community.github.io/, @taradiddles .

[one7two99]

Thanks. Nice. I also need to learn more about GitHub. [799] awokd <notifications@github.com> schrieb am So., 8. Apr. 2018, 14:50:

…

Nice summary of the contributor workflows on https://qubes-community.github.io/, @taradiddles <https://github.com/taradiddles> . — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAyvllWNUMYOwCI6WbMYltGkbF0iO7W4ks5tmgehgaJpZM4SoQLD> .

[ghost]

looks like we've agreed on most topics here ; closing...