path and directiries for optlib and xcmd

[masatake]

@majutsushi, forgive me if I'm impolite.
Are you have interested in improving ctags?
We, fishman-ctags developers, are interested in your wiki page:

https://github.com/majutsushi/tagbar/wiki

fishman-ctags has ability to use the contents of .ctags and command line as a first class library with a mechanism called optlib. In my opinion it is better for users that your wiki contents are part of ctags and
made runnable.

Technically I, who designed and implmented optlib mechanisum, can import code from your wiki page.
However, it is ... not so ... good.

First of all I want feedback for what I implemented from other person than myself.
In addition I'm not sure the copyright and license term about the contents of wiki.

So I wonder whether you have interests writing an optlib file.
If your answer was yes, could you look at
https://github.com/fishman/ctags/blob/master/docs/f-news.rst
https://github.com/fishman/ctags/blob/master/docs/f-optlib.rst
and
https://github.com/fishman/ctags/blob/master/docs/f-units.rst
?

Questions are welcome.
Thank you very much for reading this.

[majutsushi]

Sorry for the late reply. Before responding to your main question I have a few comments about the optlib mechanism in general:

• The way additional definitions are currently being loaded with the optlib and preload directories strikes me as a bit overcomplicated. If some filetype plugin wants to add a new definition to fishman-ctags from its own directory, it has to add both the path to the data-path env var (or command-line argument) and add the name of language bundle to the --options argument. I would suggest merging the optlib and preload mechanisms into one with these properties:

  • New directories/files can be added as with the current optlib mechanism.
  • Directories/files in higher-precedence data directories take priority over ones in lower-precedence directories as with the current preload mechanism, and only the highest-priority one found gets loaded. It probably doesn't make much sense to mix language configurations from different sources.
  • All of the directories/files found get automatically loaded without having to use the --options option. If loading a specific bundle is not desired then it can be overridden with an empty file/directory as with the current preload mechanism. This would for example make it possible to just symlink a configuration directory into for example ~/.ctags.d/optlib and have it work automatically even when calling ctags directly from the command line.

  This would remove the need for the preload directories and make adding new languages more straightforward. Does this make sense or am I missing a use case?

• The /usr/libexec directory seems to be Red Hat-specific, for example neither Debian nor Arch use it since it is apparently not FHS-compliant: https://lists.archlinux.org/pipermail/arch-general/2010-May/013410.html. So it would probably make sense to change the default search path here, and possible the name of the "libexec" options as well.

• There seems to be a discrepancy with the naming of the libexec options: The environment variable is called $CTAGS_LIBEXEC_PATH, but the command line argument is --libexec-dir.

• For both command line path arguments it is not quite clear whether you can call it with more than one directory and whether you can give this argument more than once.

Okay, now for the actual optlib configurations. I am actually currently working on extracting the lower-level ctags parts from Tagbar into its own plugin in order to make it usable by any Vim plugin that wants to make use of ctags functionality, and part of that is that I want to convert the wiki data into configurations files directly in the plugin. Originally I thought that it might be a good idea to have a common repository that both fishman-ctags and my new plugin can use, but after thinking about it some more I don't think this is really a good idea: one, it would mean that people who want to contribute a new filetype have to add it to two repositories (the ctags configuration one and the Vim-plugin one for the Vim-specific configuration), and two, since I would also have to support the standard exuberant-ctags it would mean that the configuration wouldn't be able to make use of the new fishman-ctags features. So maybe for now it would be better to have the configurations separate and maybe sync them occasionally.

As for the license of the wiki contents, I don't think there should be any issue – it's just some simple configurations. I would certainly consider it public domain.

I'll hopefully have some time next to do check things out with fishman-ctags and do some test conversions.

[masatake]

Thank you very much for great comment.
I will read carefully and reflect your comment to the implementation.
I will work on this topic after solving mysterious issue #95.

[majutsushi]

Have you thought about my comments? I think it would be a good idea to reach a consensus here before the first release to avoid having to change things later.

[masatake]

I read again and again your comments.

• I need both optlib and preload directories store the different type of
  files However, I understand the complexity from your comment. I found
  where the complexity comes from.
• About libexec, I revise the directory structure and Makefile.

---

I think default.ctags causes the complexity. In current
implementation (in the git repository), optlib files(coffee.ctags,
ctags.ctags and mib.ctags) are installed to optlib directory. They are
referred from preload/default.ctags, which are generated by Makefile.

To avoid the meaningless complexity.
The bundled optlib files(coffee.ctags, ctags.ctags and mib.ctags)
should be directly installed to preload directory. optlib directory
must be empty now.

I found both two type of directories, optlib and preload.
However, I'm not sure the name is the best or not.

Background and requirement

I assumed there are two types of optlib file; one should be turned on
by default and the other should be turned off by default. I introduced
preload directory and optlib directory to deal the difference. The
bundled optlib files(coffee.ctags, ctags.ctags and mib.ctags) should
be turned on by default.

Informal specification

preload directory family is for storing optlib files which should be
turned on by default.

optlib directory family is for storing optlib files which should be
turned off by default.

As far as conflicting the name of files, files under preload directory
family are loaded automatically when starting ctags command.

Files under optlib directory family are not loaded automatically.
They are loaded only when a file under the directory is specified
in --options option.

--options option can be used in files under preload directory, this
technique is used in current implementation. However, this technique
should be used only if it is really needed. This technique just
introduces the complexity.

Following directories are involved.

A. /usr/lib/ctags/preload/.ctags (or /usr/local...)
B. /usr/lib/ctags/optlib/.ctags (or /usr/local...)
C. /etc/ctags.d/preload/.ctags
D. /etc/ctags.d/optlib/.ctags
E. ~/.ctags.d/preload/.ctags
F. ~/.ctags.d/optlib/.ctags

Ctags behavior
..................................
If file names are not conflicted, all
files under preload are loaded automatically:
E -> C -> A. Once a file is loaded, another file
having the same name is not loaded(shadowing).
--options doesn't forbidden for chain-loading files
under optlib directory but generally it is normal
technique.

Files under optlib can be used with specifying
--options. If a name, not a file name is passed
to --options as an option argument, ${name}.ctags
are searched in directories with following order
F -> D -> B.

Role of directories
..................................
You may want to maintain, redistribute and share
your optlib files.

For the personal purpose, use E or F.
If you want to it turn on by default, put
your file to E. If not, put it to F.
If you don't want to load an optlib file X under
C or A in starting
up, put an empty file with name X to E.

For the site-private purpose, use C or D.
If you want to it turn on by default, put
your file to C. If not, put it to D.
If you don't want to load an optlib file X under
A in starting up, put an empty file with name X
to C.

If you are a ctags developer, use A or B.
If you want to it turn on by default, put
your file to A. If not, put it to B.

[majutsushi]

Thanks for the explanation, I now understand your idea better. I think it would be good to clarify this main difference more in the documentation. I'm still not entirely sure though why you don't want to load all configurations by default -- what would be the downside to that? Performance? Potential incompatibilities? Considering that your mechanism allows for the disabling of configurations if they create problems I'm not sure the extra complexity of two mechanisms is worth it.

I have two other questions:

• In the optlib mechanism, will the loading also stop once it finds the first occurrence, like with the preload mechanism?
• Does preload support additional data paths just like optlib?

In case that the separation of these two mechanism will remain I would suggest renaming them to make their intent more clear, for example rename preload to lib and optlib to extra.

By the way, I noticed a small issue in the news file: In the optlib section you load *.conf before *.ctags, but in the "recursive" section it is the other way around.

[viccuad]

Here is my humble opinion; let me know what you think.

I have the same opinion as @majutsushi, I would prefer to simplify and merge both optlib and preload dirs into only one.

---

I would do a lib dir that would be in:

A. /etc/ctags.d/lib/*.ctags (or /usr/local/etc/ctags.d/.. if using make install)
B. ~/.ctags.d/lib/*.ctags

Users are not expected to touch A since it is owned by root.

If names are not in conflict, all files are loaded automatically:
B -> A. Once a file is loaded, another file having the same name is not loaded (shadowing). This will work too for /etc/ctags.d/ctags.conf and ~/ctags.d/ctags.conf if there is plan for that conf file.

If necessary, we could have a --options flag or similar (I would prefer a flag name that shows that we are loading something instead of A and B, eg: --instead-load).

This flag makes ctags not load at all A and B and only load whatever is passed with it. You can now select exactly what you want to load by passing it to the flag (you can pass A to the flag, A + something, B + something, only something, etc).

How do we enable and disable certain *.ctags so they don't get loaded?

(Which is the same question as: How we provide optlib functionality?)

By not loading files which suffix isn't one we recognize (eg: *.ctags). Eg: ~/.ctags.d/lib/mib.ctags gets loaded, but ~/.ctags.d/lib/mib.ctags.bkp does not.

This is consistent to what happens to other conf files in other projects (eg: .vimrc vs .vimrc.bkp or .vimrc_bkp), and it simplifies things.

Why there isn't a /usr/lib/ctags.d/?

X. /usr/lib/ctags.d/lib/*.ctags (or /usr/local/lib/ctags.d/..)
A. /etc/ctags.d/lib/*.ctags (or /usr/local/etc/ctags.d/..)
B. ~/.ctags.d/lib/*.ctags

The normal user is not expected to touch either A or X (since they are property of root). So if we add X, it should only aid the developers. An open question for which I don't have an answer is,

Do the developers need `/usr/lib/ctags.d/`?  (or `/usr/local/lib/ctags.d/..`)

I don't see the point, since everything that can be done in /usr/lib/ctags.d can be done in /etc/ctags.d/lib.

Anyway, universal-ctags project shouldn't worry about using /usr or /etc in my opinion, since there are variations on how it is used from distro to distro. If universal-ctags provides some flags in configure for taking care of the A option, the distro can put it for example in /usr/lib/ctags.d and put themselves a link from /etc/ctags.d to /usr/lib/ctags.d.

---

@masatake said:

I assumed there are two types of optlib file; one should be turned on
by default and the other should be turned off by default.

I may be missing use-cases, and that's why I simplified this dir scheme. If there are use-cases, please tell me.

[masatake]

@viccuad, thank you for comments. I have been thinking again and again this issue.
The current result of my thinking is that we should not put any file at /usr.
Instead all *.ctags file in universal ctags source tree should be linked to ctags executable at building-time.
The basic idea is shown in my comment #544.
These days @p-montanus have been worked hard in building script. so that I can integrate my optlib translator into Makefile.in easily with following small target:

.ctags.c:
    $(SILENT) dir="$$(dirname $@)"; mkdir -p "$$dir"
    $(V_CTAGST) $(CTAGST) --options=$< > $@

It is just an experimental code but when I successfully integrate the translator to the build script, we can make this issue much simpler; we can focus on /etc and home directory.

The critial issue is other members allow me to use bash in building process:-P

[cweagans]

Sorry for the noise. Just switching the "Release blocker" tag for the "Initial release" milestone so that everything is tracked in one place.

[masatake]

Mach simlified mechanisms are introduced in #1519.
Thank you for comments people joning the discussion.