Some points:

• build directory included
• maybe leave the markdown and use a sphinx extension: https://www.sphinx-doc.org/en/master/usage/markdown.html - this will be a lot less churn
• consider deploying a specific website github pages-only repo then we can also add the historical versions of the manual in the same place (need to check what domains are used and whether we need to redirect old domains to sub-directories)

Some points:

* build directory included

* maybe leave the markdown and use a sphinx extension: https://www.sphinx-doc.org/en/master/usage/markdown.html - this will be a lot less churn

* consider deploying a specific website github pages-only repo then we can also add the historical versions of the manual in the same place (need to check what domains are used and whether we need to redirect old domains to sub-directories)

1. build dir 👍
2. I did that when I started working on this but the problem is that the the .md files essentially remain as is so the look is the old wiki not the goal of the PR which is to harmonize (unless I missed something), so running the .md through pandoc to convert them to .rst was the only way to maintain the look that matched the manual.
3. will do, I am just trying to get through the conversions and formatting first then will get your's and Pete's input as to where we want to put things.

When I get some time to finish off the formatting issues (thi sweek hopefully) I will convert this PR back from draft for you to have a look at.

I was thinking with my point 2 suggestion that using that sphinx extension allows you to leave the markdown content as-is (i.e. it does the conversion from md - as well as any rst) but render it in whatever sphinx theme we want.

ok I will check again because this was months ago that I had tried it maybe I missed something

Here's another repo that builds its docs from .rst, .md, and .ipynb files. It's conf.py file might be an example. Also visit the repo's pyproject.toml for the dependencies.

I must have missed something when I first looked at this its been so long I thought it was "the look" didn't change but the look is correct, regardless I am able to build the docs using myst_parser but there are numerous issues though, for instance the local file links do not render the href in such a way that the links work, instead it treats the local link as an anchor to the file it is referenced in. For example this is how myst renders the link specified in the .md:

* [Facilities](Facilities.html "wikilink") using NeXus

As:

<li><p><a class="reference internal" href="#Facilities.html"><span class="xref myst">Facilities</span></a> using NeXus</p></li>

I spent the better part of the morning trying to configure myst_parser to treat the links as local links but no joy, the only way i was able to solve it was to have a function connected to app.connect('build-finished'...) walk all the generated html files and remove the '#' character from the href=. This approach works and does produce html files with links that work but during the sphinx build there are a ton of WARNING: 'myst' cross-reference target not found: warnings (among others) that is just one example.

I have been approaching this from a build without warnings/errors but maybe I should ask do we care about warnings during the build? if not and we only care about functioning web pages then that would make this task much quicker, for me anyway.

With the way I had been doing it (converting all .md to .rst) I am nearly done making all the files look nice (straight pandoc conversion from .md to .rst is not very readable), so I would prefer to just finish that off as opposed to trying to retain the .md files and sort out all of the issues which I don't think is going to save me any time, plus I had organized the files into some semblance of order as opposed to everything in the content directory which I also prefer.

I freely admit that I'm no expert on this stuff so I don't have a problem if you guys want to take a crack at using the .md files I just think i am closer to being done the initial way and it more closely mirrors the manual documentation in terms of repo structure.

I had a quick look at that link:

[Facilities](Facilities.html "wikilink") using NeXus

And it looks like MyST is not recognising that the first word of the link is an external target. There's two ways that may fix this:

1. Modify the link so it's recognised as an external target
2. Use linkify

Hey @PeterC-DLS I had previously tried the following adjustments in conf.py without result:

html_file_suffix = ".html"
myst_enable_extensions = ["linkify"]
myst_linkify_fuzzy_links = False  # Turn off automatic anchor/bookmark treatment
myst_url_schemes = ("http", "https", "mailto", "file")  # Allow file paths

and for treating all links as external I was assured by chatGPT (#facepalm) that the way to achieve that was to specify links like this in the markdown:

[Facilities](Facilities.html)

which it already was and which didn't produce a functioning link, ..but actually looking at the documentation for myst (shame) I found myst_all_links_external and setting it to True ...fixed the href and the links work although there are still a great many warnings to sort out.

All that said I still prefer the converted files for the reasons I have already given, I am willing to do it either way but if the reason to keep everything in markdown was to save time on this task i would say that that ship has sailed :), and there would still be work anyway to get rid of all of the warnings and errors that are present during the build.

I've had a hack this afternoon and fixed a few issues:

1. clone this repo at my Fix reference that used fragments wrongly #15 branch,
2. remove from source all of these directories -active_projects, codecamps, content, extra_files, facilities, faq, img, niac, teleconferences from your branch
3. copy in the content from my branch
4. run two sed commands to correct some URLs:

# replace User URLs
sed -r -i~ -E -e '$!N;s/\[([^]]+)\]\(U.*html "wikilink"\)/\1 /;P;D' content/*.md
# replace .html of relative URLs
sed -r -i~ -E -e 's/\(([^/].+)\.html/\(\1.md/g' content/*.md
# repeat the last sed three more times

5. paste at the end of index.md the contents of the attached file toctree.txt
6. build the web pages

It gives reasonable output but needs re-arranging into the hierarchy that you had to get the H1 headers in place. As you note there's some plenty of cross-reference target not found messages in the sphinx output

Oops, I forgot that I modified conf.py:

@@ -43,6 +43,7 @@ extensions = [ 
  'sphinx.ext.todo',
  'sphinx_tabs.tabs',
  'contrib_ext',
+    'myst_parser',
 ]
 # Show `.. todo` directives in the output
@@ -61,6 +62,7 @@ templates_path = ['_templates']
 # This pattern also affects html_static_path and html_extra_path.                                                                                                
 exclude_patterns = []
+root_doc = 'content/index'                                                                                                                                      

to be clear I'm not saying that using the .md files and myst_parser wont work,
I'm saying that I prefer:

• a structure that resembles the manual repo
• editing rst files
• more organized file structure
  of what I have in this PR

As I have mentioned there are still rst files that need a more readable format but I would like you and Pete to have a look at what I have and see if yawl agree with the md to rst conversion approach or not :)

The one thing to note is that at the start of the build the logo files are retrieved from the nexusformat/NIAC repo so that they are always up to date with what the manual has.

Your new structure is great for organising things compared to the current single directory. However, there's one unstated use-case that I think most people add new items directly via GH's web interface and are used to writing MD when adding new files.

On a side note, it seems the conversion of the site from a wiki to Jekyll built static site was incomplete and lots of wiki links such as User:<blah>, Help:<blah>, Talk:<blah>, Media:<blah>. See the output of

$ grep -R -E --include=*.md '\([^:]+:[^/]{2,}.*\.html' content/

when run on master.