Lexicon 0.2 Roadmap.

[MichaelHatherly]

Planning issue for updating Lexicon.

• Switch from methods in Docile.Interface to Docile.Cache.
• More coherent and detailed REPL display for docs.
• Clean up display of method signatures.
• Improve consistency of query system.
  • more tests and documentation.
• Move to a separate query mode instead of replacing ?-mode. Just use the same approach as used in Lexicon currently. Once 0.4 is released perhaps add a separate mode.
• Compose.jl-like API for creating static documentation.
  • documentation.
• Modularise codebase, Query, REPL, Render, Doctest, etc.
• Improvements to doctest such as empty docstring checks, etc.
• Add additional metamacro definitions.

[peter1000]

hi,

I would be interested if you could share a bit what you have in mind for Compose.jl-like API for creating static documentation.

Anyway, I will busy for a couple of days otherwise - so have a nice weekend.

[MichaelHatherly]

Anyway, I will busy for a couple of days otherwise - so have a nice weekend.

Also totally snowed under right now, so I'll form my thoughts and get back to your question ASAP.

[MichaelHatherly]

I would be interested if you could share a bit what you have in mind for Compose.jl-like API for creating static documentation.

What I mean is something like, in pseudocode:

other_pages = [page(m, title  = t,
                       filter = obj -> isa(obj, Function)) for (m, t) in things]

out =
    document(
        config(...),
        section(
            # The manual section of the documentation.
            config(...),
            page(...),
            page(...),
            section(...))
        section(
            # The function reference section.
            page(...),
            other_pages...))

save(pdf("out.pdf", out))
save(markdown("out", out))
...

document creates a context in which we can set up a structure for the resulting documentation.

config would store things like title, filter, sorter, etc. for modifying whatever it's applied to.

section would be used to separate the documentation into different parts that could have different config settings. Can have sections within sections.

page would contain either the contents of an external file, such as manual.md or the docstrings from a module, or group of modules.

pdf would create an object suitable to save as a pdf. Similarly for markdown, but as a folder rather than single file.

(Total vapour-ware at the moment though, so don't take this as a completed idea yet.)

[peter1000]

I did a small graph to get a feeling how that would/could look like based on a joined Docile/Lexicon Documentation. Did not add all Docile modules - just to get a feeling.

It is basically based on the current Doc outlines.

Added possible (Section, Page, Header, SubHeader) which are probably needed especially for a single pdf.

Docile/Lexicon combined Documentation Project
├── Manual  (Section)
│   ├── Overview  (Page)
│   │   ├── Installation  (Header)
│   │   ├── Overview  (Header)
│   │   ├── Backwards compatibility with julia 0.4's @doc  (Header)
│   │   ├── Viewing Documentation  (Header)
│   │   ├── Generating Documentation  (Header)
│   │   └── Doctests  (Header)
│   └── Syntax  (Page)
│       ├── Docstring Syntax   (Header)
│       │   ├── Syntax Summary  (SubHeader)
│       │   ├── @doc  (SubHeader)
│       │   └── Plain strings  (SubHeader)
│       └── Metamacro Syntax  (Header)
│           ├── Metamacros  (SubHeader)
│           ├── Metamacro Syntax  (SubHeader)
│           └── Metamacro definitions  (SubHeader)
└── API  (Section)
    ├── Docile  (SubSection)
    │   ├── Docile  (Page)
    │   │   └── Modules  (Header)
    │   ├── Cache  (Page)
    │   │   ├── Exported  (Header)
    │   │   │   └── Methods  (SubHeader)
    │   │   └── Internal  (Header)
    │   │       ├── Methods  (SubHeader)
    │   │       ├── Macros  (SubHeader)
    │   │       └── Aside  (SubHeader)
    │   └── Collector  (Page)
    │       └── Internal  (Header)
    │           ├── Functions  (SubHeader)
    │           ├── Methods  (SubHeader)
    │           ├── Types  (SubHeader)
    │           └── Aside  (SubHeader)
    └── Lexicon  (SubSection)
        ├── Exported  (Page)
        └── Internal  (Page)

One can easily see that the minimum required structure would be (Document, Section, SubSection, Page, Header, SubHeader)

Some of the page, header could be replaced by more SubSubSections but I think for a consistence display it would be better to plan for

• Document
• Section
• Page
• Header
• SubHeader

structure - which is quite similar to what I had in mind her: Propose new section meta

[peter1000]

BTW: I send you some thought about aside-comments to gitter

[MichaelHatherly]

We don't need SubSection since you can just put a Section inside another Section. Anything internal to a Page is part of the page, like Header, and so don't need to be "composed".

[peter1000]

We don't need SubSection

Yes I know just wanted to put the structure out

Anything internal to a Page is part of the page, like Header, and so don't need to be "composed".

I don't understand exactly what you mean by: don't need to be "composed".

page would contain either the contents of an external file, such as manual.md or the docstrings from a module, or group of modules.

Seems to me that one has to somehow produce the right header with which ever group of objects one want below.

[MichaelHatherly]

I don't understand exactly what you mean by: don't need to be "composed".

So say you have a elem = page("manual.md"). That's a single block of content that can be placed into a document. Being able to manipulate elem is unnecessary since it's just an external file that you can edit manually. Having multiple routes to modifying a single thing is unnecessary.

Seems to me that one has to somehow produce the right header with which ever group of objects one want below.

That should be covered by filter, sorter and some additional options for specifying styling of entries.

[peter1000]

I plan to work next on this issues: which are all inter-related.

#118

[peter1000]

I was wondering about the docs node which accepts Union(AbstractString, Module))

If you look at my example

after appling the filter there where no objects for: External - Modules, Functions, Methods & Macros

1.
What do you think: should any AbstractString before a module only be printed if there are actual objects after filtering?

docs("### Modules, Functions, Methods & Macros", 
        Docile.Cache,
        filter = obj -> iscategory(Docile.Cache, obj, [:module, :function, :method, :macro])
       )

2
What should happen if there are multiple modules defined within one docs

docs("### Modules, Functions, Methods & Macros", 
        Docile.Cache,
        Docile.Collector,
        filter = obj -> iscategory(Docile.Cache, obj, [:module, :function, :method, :macro]) ||
                              iscategory(Docile.Collector, obj, [:module, :function, :method, :macro])
       )

Should all objects mixed together when sorting or each module by itself etc..

Or would it be better to allow only one module per docs - even make it mandatory?

[peter1000]

At the moment my own feeling for docs would be:

• allow only modules for docs - remove the AbstractString

• for an optional header text use a config:

  e.g. header = "### Modules, Functions, Methods & Macros"
  which is only written if ther are actual objects after filtering.

• all other text can be done in the page node.

• still allow multiple modules but their objects get completely mixed per filter and sorter

  • if that is not wanted just use 2 separate docs()

NOTE: for mixing modules into one it is also a bit annoying that the modules have to be passed on to any filter because we can not get all from the object themself.

Actual for small project mixing all objects in one docs might be not bad - probably better than having a single obj under each heading.

[MichaelHatherly]

What do you think: should any AbstractString before a module only be printed if there are actual objects after filtering?

I'd think it should stay. If you're building docs and you end up with a section with a header but no docs it's slightly more noticeable than a completely empty section.

At the moment my own feeling for docs would be:

Give it a try if you'd like. I'm not sure a see the issue with interspersing strings and modules in docs though, but I'm happy go that route if we don't loose any features.

multiple modules but their objects get completely mixed

Is mixing objects from different modules actually a good idea? If a project is so small then multiple modules probably isn't what should be used to start with.

[peter1000]

I'd think it should stay. If you're building docs and you end up with a section with a header but no docs it's slightly more noticeable than a completely empty section.

Also true

If we keep them the other things are not really relevant ...

thanks

[peter1000]

hi,

just catched up with your work on the julia github. I think I skip a couple of days before restarting on Lexicon.... who knows what changes the julia doctransition might bring for Docile/Lexicon.

[MichaelHatherly]

Yes, lets rather wait. There may be a few things worth taking from our work here once things have settled on master. There's momentum at the moment so it's not really worth splitting effort.

[hayd]

To get to #109 (comment) / #109 (comment)

As an alternative approach, if it exists read the ../mkdocs.yml file (and accept passing of a yml of a different name), and extract the pages section e.g.

pages:
- Introduction: index.md
- Manual:
  - Overview: manual.md
- API:
  - Lexicon: api/Lexicon.md
  - Index: api/index.md

Boom. You have the structure for a pdf.

I think this would be easier/clearer what's going:

1 you generate markdown files
2a. you statically host
2b. include parsed/latexed markdown, using the above titles etc. -> pdf

Edit: although perhaps we're better off just using pandoc to do the latter, since that handles html (Base.Markdown doesn't allow it)... in fact that's probably the way to go... (create a big markdown file and just throw it to pandoc to make the latex & pdf) ??

[MichaelHatherly]

As an alternative approach, if it exists read the ../mkdocs.yml file (and accept passing of a yml of a different name), and extract the pages section e.g.

That would work nicely I think. Would be nice to have it more general than relying on mkdocs for that though. Since if mkdocs goes and changes their format, which I think has happened already, then that would break Lexicon.

[MichaelHatherly]

create a big markdown file and just throw it to pandoc

Do you know what the licensing is like for pandoc? It's GLP I think, would that be a problem if bundled as a dep?

[hayd]

I wouldn't bother, you could just ask users install it, and shell out.... :/

all we'd need is a suitable --include-in-header or --template. The default isn't great IMO.

[hayd]

Maybe outputting to gitbook is the way forward (I still favor doing this via _generated), IMO their pdf rendering looks really nice. see https://www.gitbook.com/book/gitbookio/documentation/details (since you mentioned this earlier).

[MichaelHatherly]

Yeah, gitbook is quite nice from what I've tried so far.

So long as we just output plain old markdown with the docstrings spliced in it shouldn't make a difference what anyone uses after that stage. I'd rather not become totally tied to some external package if we can help it.

[hayd]

That's the good thing about most of these tools, it's usually some file e.g. SUMMARY.md or mkdocs.yml which glues the markdown files together. One annoying part (of gitbook) is the showing of wide code blocks cuts off (IIUC it uses phantomjs to render, which can't scroll).

[peter1000]

So long as we just output plain old markdown with the docstrings spliced in it shouldn't make a difference what anyone uses after that stage.

👍

hi Michael,

I saw you and one-more-minute merging quite a bit to julia master. Could you just shortly update me on your thoughts:

1. How does the new additions to julia effect Docile/Lexicon. Does not some of it overlap what Docile did? Do you plan to removed such from Docile/Lexicon source ...
2. Are you still planning to change to exicon-next ? I saw hayd has done some work / addition based on the 'old/current Lexicon'.

Cheers
P

[MichaelHatherly]

Does MichaelHatherly/Docile.jl#147 answer some of you're questions? There's probably lots of more specialised things that the base docsystem doesn't need to cover which can be left to Docile/Lexicon.

I saw hayd has done some work / addition based on the 'old/current Lexicon'.

Hadn't noticed the branch that was against. It's fine though if it turns out to be a minor addition then getting it into a tagged release would be easier than having it on lexicon-next.

[peter1000]

Does MichaelHatherly/Docile.jl#147 answer some of you're questions?

Yes. Thanks I had not seen that.

[hayd]

Also there's the rewrite branch. My PR was pretty independent. :s

Edit: well what I mean is... similar/the same code but in different files so will merge cleanly.

[MichaelHatherly]

Also there's the rewrite branch.

Not sure what I was thinking when creating yet another branch... ignore that one for the moment I think.

[hayd]

If we thought the generate_for api was a good one, I would suggest that the api files should generate something parsable (i.e. intermediary markdown which includes the {{}} syntax) that can be hit with @file. Thoughts?

I think this would mean

1. bringing in all features build doesn't have yet.
2. separate docstring rendering of methods that do not share docstrings. (Would use this always).
3. be able to do it based on types (e.g. you may be interested in a methods on TypeA in one page, and TypeB in another). IIUC This wouldn't be necessary if you used submodules. Not required at first.

I think the API, at it's simplest would remain:

generate_for(Lexicon, "index.md", ...)

perhaps with a kwarg to (not) build and @file the api markdown files for submodules(Lexicon), and a kwarg for no_submodules (IIRC you have a better name for that).

[MichaelHatherly]

I would suggest that the api files should generate something parsable

Could you provide an example of what you mean by this? Just markdown files containing {{}} that expand into the @file macro calls?

generate_for(Lexicon, "index.md", ...)

Naming-wise would build, save, or generate sound better instead?

[hayd]

e.g. generate api/Lexicon.md as:

# Lexicon

## Exported

{{
  doctest
  query
  save
  Config
  EachEntry
  @query
}}

## Internal

{{
  calculate_score
  ...
}}

This could be done temporarily i.e. generate _temp/api/Lexicon.md apply @file to this into _generated/api/Lexicon.md (maybe) remove the temp file.

Similarly for (API-)Index.md.

Make sense?

---

I like readability of generate_for, "generate for Lexicon the following files", I don't think brevity is important in naming here. :/

[peter1000]

I don't mind much about generate, build, save vs. generate_for but in my past experience MichaelHatherly doesn't like the underscore _for part.

Considering the present and the lexicon-next situation I think save would be the most appropriate one but generate is also good as a tempfile is suggested to be generated.

[hayd]

I guess I'd be okay with generate, I really dislike "save" for this (that kinda makes sense for creating a single page - from the generated markdowns, or for creating a mkdocs.yml*)

* would make more sense if generate accepted Pairs (of filename => title).