RFC: Simpelify packaging by splitting out collection of git version info

[ivarne]

This is intended to be part of the solution to #5068. I tried to look at the submodules also, but I was confused, and hope someone else will tackle that problem.

It would be nice to get some feedback on the changes. I have only tried it on Ubuntu, because I tried to dual boot my MacBook Pro, but now I can't boot into OSX.

• Renamed BUILD_INFO to GIT_VERSION_INFO (I'm not sure about this one, but it makes sense)
• Put the git parts of build_h.jl into version_git.jl
• Made sure that an existing version_git.jl file does not get overwritten if
  if no git information is availible.

[ivarne]

Hmm.. Adding another file that must not be committed has the unfortunate consequence that it will suddenly show up when you checkout another branch (or a older tagged commit). Now, if you do make clean before you change branch, you'll not get that orphaned file.

[staticfloat]

My shell (bash in OSX) needs a semicolon at the end of this command

[ivarne]

@staticfloat Thanks, I'll try to implement your suggestions

NO_GIT flag seems like a good suggestion. I'll also issue a warning if git status fails, to instruct users to pregenerate version_git.jl and set the flag.

[ivarne]

These variables JULIA_COMMIT and JULIA_VERSION are almost not used. See https://gist.github.com/ivarne/8130d4cac3aa3e7772ee

It should be the same info that is used in all processes involving version and commit

[staticfloat]

Since JULIA_COMMIT is only used for file naming, I think it'd be better to just set JULIA_COMMIT = JULIA_VERSION instead of "Not Available".

[ivarne]

That seems right

[ivarne]

Now we have to decide if we want to name-space the other build constants that I left in build_h.jl. Having them in a type or dictionary is kind of nicer than global constants.

[staticfloat]

Let's leave the rest of build_h.jl alone for now. This works well for me on OSX. (Tested both with git information, and with .git moved away). I'd say this is good to merge.

[ivarne]

It would be nice if it could be tested in the supported environmens on windows first. I will also add a notice in DISTRIBUTING.md, but that will have to wait until tomorrow.

[staticfloat]

@vtjnash @loladiro Could one of you two test on windows?

[ivarne]

@nalimilan Have you tried this patch? Does it solve your problems?

PS. Sorry that I did not recognize you in the other issue.

[nalimilan]

I've grabbed the tarball from this branch, disabled my no-git patch and built the RPM, and I get an error:

Warning: git information unavailable; versioning information limited
Warning: git information unavailable; versioning information limited
version_git.jl
error during bootstrap: LoadError(at "sysimg.jl" line 25: LoadError(at "version_git.jl" line 2: ErrorException("syntax: extra token "commit_short" after end of expression")))
make[1]: *** [/home/makerpm/rpmbuild/BUILD/julia-1bed8950421f195b7cd6c8305009f4636cf1aeab/usr/lib64/julia/sys0.ji] Error 1

The file itself contains this:

# This file was autogenerated in base/version_git.sh
immutable GitVersionInfo\n    commit::String\n    commit_short::String\n    branch::String\n    build_number::Int\n    date_string::String\n    tagged_commit::Bool\n    fork_master_distance::Int\n    fork_master_timestamp::Float64\nend\n
# Default output if git is not availible.
const GIT_VERSION_INFO = GitVersionInfo("" ,"" ,"" ,0 ,"" ,true ,0 ,0.)

[ivarne]

Thanks for trying this out. Do you really get the \n in the output?

If so, your system probably have a echo command that does not substitute \n with newlines. I will make a fix, where i have a new echo statement for every line instead of a multiline string with newlines inside.

[ivarne]

Please also read my modified DISTRIBUTING.md to find out how to pregenerate the version_git.jl file on a git enabled system before bundeling.

[nalimilan]

Cool, with the new commit it works.

Why don't you build version_git.jl automatically if .git does not exist? It's not very nice to require packagers (and users installing from the tarball) to find out that very specific trick. make && make install should be enough if we want Julia to be packaged everywhere. :-)

[ivarne]

The problem is that there is no way to put anything useful in the version_git.sh if there is no git repository. On my machine I can make without git info, but then GIT_VERSION_INFO will only be empty strings, and make will issue a warning. So it kind of works the way you describe. Does it not work for you?

[ivarne]

I also think I will try to convince the core maintainers to check in a version_git.jl before tagging releases, and delete it in the next commit. That will probably fix that issue, but there is still the submodules that need to be handled somehow, if we are going to support building from a tarball.

[StefanKarpinski]

We've already reached the point where tagging a release has enough steps that we should automate it. Creating version_git.jl, checking it in, tagging a release, and git rm:ing it can be part of that process. But we can only make that part of the release process if it becomes automated.

[nalimilan]

Agreed. You should also remove all .gitignore files as they make package checkers complain.

@ivarne When .git is missing, assume the VERSION file contains the correct version, and not git commit is needed. The way it currently works when you call make -C base version_git.jl.phony seems fine to me, it should just happen automatically.

FWIW, I've spotted a few typos in the code:

+  $(warn "Submodules could not be updated because git is unavailible")

+  @# This to avoid touching git_version.jl when it is not modified, so that the
+  @# so that the system image does not need to be rebuilt.

+  @if grep -q "Default output if git is not availible" version_git.jl; then \

+# Collect tempoarty variables

[ivarne]

If the package checker complains about .gitignore you should delete them. I think they should be part of the repository, and it will mess up the history if they get deleted every now and then. Maybe we should just add a make target for a git free tarball instead. The problem is that it needs to be maintained, but that is the case if it is done in packaging scripts also.

What do you mean automatically? If the code works the same for you as for me, I get two warnings if .git is missing. One because submodules could not be updated, and one that tells you that you use a old version_git.jl or that an empty (automatically generated?) version_git.jl is used. Is the warnings a problem?

I'll try to fix the spelling issues tomorrow, thanks for noticing.

[StefanKarpinski]

Having a target that creates a git-free tarball without .gitignore and with a generated version_git.jl file seems like the better option.

[nalimilan]

@ivarne Currently indeed I delete .gitignore files, but I was asked to ask by Fedora people to ask Julia maintainers to produce a proper tarball free of any revision control elements; this makes life simpler for everybody. A make target to build the tarball is the way to go.

By "automatically", I meant that people should not need to call make -C base version_git.jl.phony. But I just tried and actually it's not needed. I misunderstood what you wrote in DISTRIBUTING.md. Maybe you should make it clear that the command is only needed if you care about git version information. When using a release tarball, that's not useful.

[ivarne]

My guess was that "nobody reads documentation", especially if it is long and convoluted. Therefore I made it so that it should never fail, but do the best job possible, and issue a warning if I think I miss some information. In the documentation I tell people how to do it correctly, not how they can be lazy and get a less desirable result.

[nalimilan]

Sure. After #5181 is fixed, it would be cool to avoid printing the warning when building from a tarball, and to update DISTRIBUTING.md to explain that the git stuff is not useful in that case.

[ivarne]

@staticfloat While cleaning up this I found out that all the dependencies that require updated submodules also update submodules. Is there then any reason to update submodules in the main Makefile?

[vtjnash]

That was originally done because the submodules don't know they need to update unless that is handled by the main Makefile. Conversely, the rules can be run separately, independent of the main Makefile updating them.

[staticfloat]

@ivarne any chance this can get rebased/merged soon? I'd love to have this to streamline the ubuntu nightly process.

[ivarne]

Merged in #5528