Pythonic preprocessor

[szaghi]

Fortran has not its own standard preprocessor (yes fpp has been discussed during 2003/2008 standard definition, but as far as I know it is not a standard and it not widely used). Presently, cpp is the most used preprocessor. However, cpp has many limitations. It would be very useful to have a modern, features-rich, python preprocessor. Even more nice, this preprocessor should be integrated within FoBiS.py.

[Tobychev]

No! The preprocessor needs to be separate, but well integrated with FoBiS.py. Of the reasons I can think of to keep them separate, the most immediate is that I don't want to update my FoBiS.py file often: it is the start of the build system, and I would like it to stay very much the same.

I would very much like an actual pre-processor intended for fortran, but I'm not entirely sure what "intended for fortran" that really means. Further, what do you mean with a pre-processor that is pythonic, does that mean it is written in python?

These questions that indicate the need for some sort of specification of what the pre-processor should do, and I think that means it should be a separate project. FoBiS is for simple fortran-men, who need a simple and reliable too. The Glorious Fortran Pre-Processor is not going to be simple, so it should live alongside FoBiS, not within it.

[szaghi]

Dear Tomas,

you are great!
My bad English has made not clear my intentions.

• I absolutely agree that the pre-processor must be separate from FoBiS.py, but it should be very "integrated". In past years I have already written a python preprocessor for Fortran source aimed to create latex documentation, FortranDOC. I have already planned to re-use this old code for new goals...
• which goals? I ask for a preprocessor dedicated only to Fortran having some of the cpp feature, but that overcomes its limits. In particular one of the first feature I absolutely need is the "loop-preprocessing". I try to be more clear. Let us suppose we want to write a generic interface for three different procedures that differ only in the dummy argument kind-type-shape, but are identical for all the rest (and we do not want to use polymorphic class...); we write something like:

...
interface foo
  module procedure foo1,foo2,foo3
endinterface
contains
  function foo1(in) result(out)
  type(first), intent(IN):: in
  logical:: out
  out = in%logical_test()
  endfunction foo1

  function foo2(in) result(out)
  type(second), intent(IN):: in
  logical:: out
  out = in%logical_test()
  endfunction foo2

  function foo3(in) result(out)
  type(third), intent(IN):: in
  logical:: out
  out = in%logical_test()
  endfunction foo3
...

The identical result can be obtained using a preprocessor cleaver than cpp that is able to loop over some pseudo code. I am thinking to something similar to:

...
interface foo
  module procedure foo1,foo2,foo3
endinterface
contains
  !$PP for i in [1,2,3] and t in [first,second,third]:
  function foo$i(in) result(out)
  type($t), intent(IN):: in
  logical:: out
  out = in%logical_test()
  endfunction foo$i
  !$PP endfor

The pseudo-syntax of preprocessor is absolutely not definitive... just to make a more clear example.
This is just the first feature I am searching for, but more others are welcome.

• This preprocessor should be very simple, as FoBiS.py it should be designed for Fortran poor-man.
• Pythonic for me has twofold meaning: 1) it will be written in python and 2) it should have pythonic-syntax as much as possible.

The discussion is open. Your suggestions are always welcome!

[Tobychev]

Hmm, you know from that example I'm immediately reminded of template systems used in web applications. I guess they are in principle the same, being systems that takes highly structured text and dynamically generate an output that is then parsed by a further application. But maybe that is just me not having used pre-processors enough to know the real differences.

In any case, some things in no particular order:

• Here's a list of lots of python templating engines, if nothing else there is inspiration for syntax there.
• I notice your loop syntax is such that it is valid fortran code, but the function body itself will not pass because it contains invalid names. In this example it seems senseless that the unprocessed file would be valid fortran, as the final executable would likely not work as intended, so the syntax should probably not be valid fortran.
• In your example it does not seem like the pre-processor actually has to know anything at all about fortran, you could probably achieve the desired result trivially with almost any of the templating libraries in the list above. So while a good example what you want, it doesn't really show anything related to fortran.

I've not used pre-processing much actually, so if you could give more examples that we can talk about that is probably a good way to proceed.

[szaghi]

Yes, you are focused the problem, however:

• a pre-processor is similar to a template system (maybe historically come first pre-processor applications, I do not known), but in some circumstances a pre-processor can do more than a template (see the following);
• it is very common that a source file pulled with pre-processing directives has not a valid syntax until the file is pre-processed; in any case, it could be very dangerous to use a file with a pre-processing directives without pre-processes it even if the file has always a valid syntax; as an example let us consider one of the most common usage of the cpp the #ifdef directive; it is very common (I widely use it) to use such a directive to maximize the code efficiency by means of algorithm differentiation accordingly to the specific case without use complex if-then-else runtime check that may have a role on the performances degrade, i.e.

...
#idef VISCOUS
call viscous_solver
do i = 1, N
  ! perform viscous stuff
  ...
enddo
#else
  call eulerian_solver
do i = 1, N
  ! perform Eulerian stuff
  ...
enddo
#endif
...
! perform common tasks for both visous and Eulerian case

This is a very typical case (see for example this procedure at line 1389) where the syntax is valid even without pre-processing the file; however, if you compile it without pre-processing the results you obtain running your code will be algorithmically wrong or unpredictable. This is the reason why in general the source file that contains pre-processing directives are marked with an upper case extension (.F, .F90 etc...) in order to be automatically pre-processed from the compiler supporting cpp or fpp. Consequently, I am not afraid that syntax is invalid until the file is pre-processed, it is very common, e.g. I often use such approach, see this procedure at line 747: the name neq_scal_ is a cpp name and it is not valid without pre-processing the file.

• you are right, in the examples I have made there is no need of a Fortran-aware prep-processor, a template system should be enough; however in some cases it is not; recently I have found an interesting project on githu, Hybrid Fortran. It can be view as an extension of Fortran obtained by means of pre-processing directives that transform Fortran in CUDA kernels or OpenMP Fortran... I have not yet read the source but I suspect that the pre-processor of Hybrid Fortran is more than a simple template system.

Your python template list is very interesting I will read asap. For my workflow a pre-processor is very important; indeed I would like cpp if it would support loop-template, but it does not (and it has other limits). Consequently, I am seriously thinking to a fortran-poor-men pre-processor being probably a step more than a template system. The problem is to find the time to do it....

[Tobychev]

I think we both agree that compiling a file without pre-processing even though it has pre-processor instructions will have unpredictable results. So do you agree that it is desirable that the pre-processor syntax is not valid fortran statements to ensure people can't compile unprocessed files by mistake?

Moving on: templating systems for the web typically allow for the execution of arbitrary code during the rendering of the output file, so basically any task that require a comprehension of static surroundings in the input file can be done by the template (by the human properly designing the function to be executed ).

I'm quite sure that a (web) template system would be a good starting point at least, although the exact choice is probably critically important as it will dictate how easily it is modified.

Another principle question is how to proceed with the development of poor fortran-men pre-processor, and I would favour getting something simple working first, but constructed such that it is easily extended. This first simple bit should probably be totally fortran agnostic, as both the if-else and the loop construct example. From this it follows that the initial pre-processor works on a file-by-file basis, with no knowledge of the module dependency for example.

Since the module dependency is already discovered by FoBiS, maybe the pre-processor should get this information from FoBiS and that would be an example of the integration between the two? Of course, this might severely impact the design of FoBiS and end up coupling the two programs too tightly. I imagine more advanced functionality would could be done if you know the module graph, but I'm not sure what that functionality would be. In any case this should be left for later, but it serves as an example of things we should consider when we make the first, simple, version.

One more point is that it will probably be highly desirable to others if their cpp-preprocessor directives can be well mapped to the fortran preprocessor, since then an automatic conversion script between the two can be produced. This means that an early partial goal (if we wish to get users fast) could be to reach parity with cpp functionality and produce a script to make the conversion away from cpp possible.

Finally, some of the template engines have a syntax that I like for delimiting their commands (because I find it pretty and that's basically all that a template syntax is about): "{{ }}" . (Typically they also come with "{% %}" that represents a different mode of execution of the contained commands, but I don't find that as pretty)

[szaghi]

• Yes, I agree that it is desirable that the pre-processor syntax is not valid fortran statements; moreover, following one the suggestions you have just done, I think that it is desirable that each processor directive starts with # symbol as cpp directive;
• yes, I agree that a template system is a good starting point, but which is the best?
• yes, I agree that the first step is to a very small set of features, the preprocessor remaining fortran-agnostic;
• as the coupling of FoBiS.py and the future preprocessor is concerned, I am not sure I understand you; FoBiS.py already produces something similar to a graph of the dependency: a log of the dependency-hierarchy can be found into the building log (is activated) or into the makefile (a new feature just implemented: FoBiS.py build -m my-makefile generate a valid makefile...); consequently, I think that the modifications that must be done for generating a graph in FoBiS.py are small, but I do not understand the possible applications for the preprocessor;
• you are absolutely right: it is highly desirable to support cpp directives; our starting point should be this, as you suggest;
• mmm, indeed I prefer markdown syntax rather than markup; I think that surroundings make heavy the syntax... but I think that you can persuade me with more examples.

I suggest to make first a research on a possible template system to use as starting point.

[szaghi]

Dear Tomas,

after a very quick search, I decided to write a new preprocessor/template system from scratch. I have created a github repo:

PreForM

It is a very beta script with only a few lines of statement. It is totally Fortran-agnostic. Presently it supports only a very small set of cpp directive:

• #ifdef-#else-#endif;
• #ifndef-#else-#endif;
• #if-#elif-#else-#endif;
• #define;
• #undef;
• #include;

The present implementation is very dirty, it is just a prototype for demonstrating that our goals can be obtained with small efforts.

I would be very happy if you want to contribute actively to this project. If you want I can share with you the the ownership of the github repo thus you can pull your patches autonomously. In any case, I would very happy if you can help me to write the specifications of the code. How many cpp directive we should support? Which directives? What kind of extensions of cpp we should implement? I have my list of todos, but I would like to know your opinion.

See you soon.

[szaghi]

Hi Tomas,

I have gone over PreForM.py, see https://github.com/szaghi/PreForM. Many cpp directive is now working. In my list of priorities the next cpp directive to implement if the function-like macro. I am very doubtful if implementing or not the following cpp features:

• stringification;
• concatenation;
• variadic macros.

These are not straightforward to implement, and not so widely diffused in Fortran workflows.

The templating features are still missing, but they should simple to implement. As I wrote in README.md, I have chosen a prefix for the templating directive that invalidates the sources syntax : #PFM directive. Do you agree with this choice?

See you soon.

[Tobychev]

Hello Stefano,
I'm sorry I haven't replied earlier, I've been trying to finish some school work and it has been taking a lot of time. I would like to help out, but I can't promise you much time unfortunately.

As I've said before, I have not had much experience with preprocessors so I have very little understanding of the crazy use cases people dream up. For instance I don't know what stringification is, but if it is complex and little used I'm not sure it is relevant.

I think generally cpp is used to do some crazy things because much C code is supposed to be very portable over a wide array of architectures, like laptops, cars and microwave ovens. This is not really something demanded of fortran code, so maybe people don't put the preprocessor to such crazy use? I simply know too little to be of much use unless you explain more about what things do :(

I also mentioned which syntax I favour: {{ }}

But I won't protest the choice, there is not much argument in either direction apart from aesthetic ones as far as I can tell.

[szaghi]

Hi Tomas,

I am sorry to bore you. I just realize now that you are not "enthusiast" preprocessor user. Indeed, me too... but in some circumstances the preprocessor directives can "boost" your code enhancing performances. Anyhow, I am closing this issue. Anyone interested on the preprocessor is encouraged to post in the PreForM.py repo.

See you soon.