Release v0.13.0

Release v0.13.0 is fresh from the oven.

This release ships with major improvements in the xrootd implementation and a few fixes in rootio.

rootio

• leveraging the work that happened in xrootd, rootio is now able to read files over [x]root:

import (
    "go-hep.org/x/hep/rootio"
)

func foo() {
    f, err := rootio.Open("root://ccxrootdgotest.in2p3.fr:9001/tmp/rootio/testdata/small-flat-tree.root")
    if err != nil { ... }
    defer f.Close()
}

• all the root-xyz commands can now also leverage xrootd:

$> root-ls -t root://ccxrootdgotest.in2p3.fr:9001/tmp/rootio/testdata/small-flat-tree.root
=== [root://ccxrootdgotest.in2p3.fr:9001/tmp/rootio/testdata/small-flat-tree.root] ===
version: 60806
TTree          tree                 my tree title (entries=100)
│ Int32        "Int32/I"            TBranch
│ Int64        "Int64/L"            TBranch
│ UInt32       "UInt32/i"           TBranch
│ UInt64       "UInt64/l"           TBranch
│ Float32      "Float32/F"          TBranch
│ Float64      "Float64/D"          TBranch
│ Str          "Str/C"              TBranch
│ ArrayInt32   "ArrayInt32[10]/I"   TBranch
│ ArrayInt64   "ArrayInt64[10]/L"   TBranch
│ ArrayUInt32  "ArrayInt32[10]/i"   TBranch
│ ArrayUInt64  "ArrayInt64[10]/l"   TBranch
│ ArrayFloat32 "ArrayFloat32[10]/F" TBranch
│ ArrayFloat64 "ArrayFloat64[10]/D" TBranch
│ N            "N/I"                TBranch
│ SliceInt32   "SliceInt32[N]/I"    TBranch
│ SliceInt64   "SliceInt64[N]/L"    TBranch
│ SliceUInt32  "SliceInt32[N]/i"    TBranch
│ SliceUInt64  "SliceInt64[N]/l"    TBranch
│ SliceFloat32 "SliceFloat32[N]/F"  TBranch
│ SliceFloat64 "SliceFloat64[N]/D"  TBranch

• support for seeking (i.e.: event random access) has been added to scanners connected to chains of rootio.Trees,

• rootio can now automatically generate streamers for std::vector<T> when a streamer for T exists,

• rootio has been updated to v2 of pierrec/lz4 library to decode LZ4 compressed ROOT files.

xrootd

• support for ping and protocol requests
• support for dirlist, open, close and sync requests
• support for read and write requests
• support for rm, rmdir and truncate requests
• support for stat, vstat, statx, mkdir, mv and chmod requests
• support for signing requests
• support for auth+unix request
• introduction of the xrd-cp command to copy files from a remote xrootd server:

$> go doc go-hep.org/x/hep/xrootd/cmd/xrd-cp
Command xrd-cp copies files and directories from a remote xrootd server to
local storage.

Usage:

    $> xrd-cp [OPTIONS] <src-1> [<src-2> [...]] <dst>

Example:

    $> xrd-cp root://server.example.com/some/file1.txt .
    $> xrd-cp root://gopher@server.example.com/some/file1.txt .
    $> xrd-cp root://server.example.com/some/file1.txt foo.txt
    $> xrd-cp root://server.example.com/some/file1.txt - > foo.txt
    $> xrd-cp -r root://server.example.com/some/dir .
    $> xrd-cp -r root://server.example.com/some/dir outdir

Options:

    -r	copy directories recursively
    -v	enable verbose mode

• introduction of the xrd-ls command to list the contents of directories on a remote xrootd server:

$> go doc go-hep.org/x/hep/xrootd/cmd/xrd-ls
Command xrd-ls lists directory contents on a remote xrootd server.

Usage:

    $> xrd-ls [OPTIONS] <dir-1> [<dir-2> [...]]

Example:

    $> xrd-ls root://server.example.com/some/dir
    $> xrd-ls -l root://server.example.com/some/dir
    $> xrd-ls -R root://server.example.com/some/dir
    $> xrd-ls -l -R root://server.example.com/some/dir

Options:

    -R	list subdirectories recursively
    -l	use a long listing format

• a convenience xrootd/xrdfs.FileSystem interface has been introduced to model interacting with the remote xrootd server's filesystem, following the os package API
• a convenience xrootd/xrdfs.File interface has been introduced to model interacting with the remote xrootd file, following the os.File API
• a convenience xrootd/xrdio.File type, implementing various io.Xyz interfaces has been introduced as well.

Release v0.12.0

Release v0.12.0 is fresh from the oven.

This release is first one to introduce preliminary support for vgo, the official Go way to handle versioning.
vgo is still in flux: the first Go version with experimental opt-in support should be Go 1.11 (to be released in August 2018.)
Obviously, on the Go-HEP side, adjustments will probably still be required as the user story solidifies and experience is accumulated.

Nonetheless, it is still an interesting new development!

geo/gdml

This release adds preliminary support for parsing Geometry Description Markup Language (GDML) files, a de facto standard for describing (detector) geometries.
The documentation for this new package is here: geo/gdml.

Help wanted and (gladly) accepted to get this package in a shape where it could be used for detailed detector studies!
This is tracked here:

• https://github.com/go-hep/hep/projects/4

hplot

hplot was slightly updated to cope with an interesting development percolating from upstream gonum/plot, namely: the migration to a new PDF backend that allows to embed fonts inside the output PDF file.
No more PDFs that display weirdly on foreign computer. Yay!

This obviously means the resulting PDF files may be quite larger than with previous versions.
(You can't have your cake and eat it.)
You can use vgpdf.Canvas.EmbedFonts to get the old behaviour.

rootio

This release adds preliminary support for chaining multiple rootio.Trees into a logical view: the famed rootio.Chain.

Mohamed Amine El Gnaoui (a.k.a @maloft), our new summer student @LPC-Clermont, provided the initial implementation: thanks!
More tests and benchmarks improvements yet to come :)

Another noteworthy change: rootio/cmd/root-srv dropped its dependency against github.com/satori/go.uuid in favor of github.com/pborman/uuid.
The latter exposes a more stable API.

xrootd

This release adds yet another new package: xrootd.
This package will provide (eventually) a pure-Go implementation of an XRootD client as well as a server.

Mikhail Ivchenko (a.k.a @EgorMatirov), our Google Summer of Code 2018 student has been already hard at work, providing support for:

• the initial xrootd client,
• the handshake with an XRootD-compliant server (C++ or otherwise), and
• the protocol and login requests/responses.

The dirlist request/response is already in the pipe.

Release v0.11

Release v0.11 is fresh from the oven.

This release drops official support for Go-1.7 and earlier, adds support for Go-1.10.

This allowed to migrate the fwk package away from golang.org/x/net/context.
fwk now directly uses the context.Context type from the standard library.

rootio

This release brings quite a few improvements in the rootio area:

• add support for empty ROOT files,
• add support for reading remote ROOT files over HTTP.
  This is implemented in a quite naive way, using net/http.Get to
  download the whole file under a temporary directory.
  See go-hep/hep#142 for ideas on
  how to improve this,
• add support for TH1-v6,
• add support for streamer-less TDirectoryFile,
• add support for TBranchElements in cmd/root-dump,
• add support for displaying TH1, TH2 and TGraph{,Error}s in cmd/root-dump,
• add support for branches and leaves with names only differing by case in
  cmd/root-gen-datareader.
  Now, the struct type being generated contains fields whose names start
  with ROOT_.
  The original branch or leaf name is then appended after ROOT_, unmodified.
• add support for TKeys with a large binary payload,
• add preliminary support for creating new Go types at runtime directly from
  a StreamerInfo value,
• add more documentation about ROOT binary file format (TFile, TKey and a bit about
  TStreamerInfo so far.)

hbook

• fixed a bug in the YODA ASCII file parsing code that would choke on
  analysis objects (histograms, scatters, ...) that contain whitespace.

AOB

LPC-Clermont has funded a 5 months internship
student, starting now.
Mohamed Amine El Gnaoui (@maloft) will:

• implement reading all ROOT files created with ROOT-6,
• implement writing ROOT files, in a ROOT-6 compatible way,
• improve the read/write performances of Go-HEP to be on par with that of ROOT/C++ (using the builtin performance tools of the Go toolchain: pprof, execution tracer, ... and/or linux perf)
• extend the test suite of Go-HEP for reading and writing ROOT files,
• add benchmark tests for reading and writing ROOT files,
• document the ROOT file format as understood by Go-HEP.

Welcome to Go-HEP, Amine!

Release v0.10

Release v0.10 is fresh from the oven.

This release brings quite a few improvements in the rootio area:

• support for ROOT files compressed with LZMA
• support for ROOT files compressed with LZ4
• support scanning ROOT trees with branches holding multiple leaves
• support for ROOT trees holding bool, [X]bool and std::vector<bool>
• support for ROOT files holding TBranch version 10
• support for ROOT files holding TBranchElement version 8
• support for ROOT files holding TH1 version 5
• support for ROOT trees and branches produced with ROOT-6.12

The rootio/cmd/root-srv command dropped support for Go-1.6 and earlier.
Please upgrade to a newer Go version.

In other news, the csvutil/csvdriver and hbook/ntup/ntcsv gained the ability to read remote CSV files (over HTTP.)

AOB

Better support for TStreamerElements and thus the ability to read nested std::vector<std::vector<T>> data or the ability to run root-dump and root-diff on all ROOT files, has started.

Google Summer of Code has started and Go-HEP is part of it, thanks to the CERN-HSF umbrella.
Check out the 2 proposals:

• Go-HEP/xrootd
• Go-HEP/FPGA

Stay tuned.

Release v0.9

Release v0.9 is fresh from the oven.

This is a quick and simple release fix with the following main features:

• fix a broken link to the PDF LaTeX example (back from the multi- to mono-repo migration)
• fix a compilation failure in rootio/cmd/root-srv originating from an API modification in a 3rd-party dependency (github.com/satori/go.uuid)

One interesting new feature is the introduction of a simple ROOT::TTree to CSV file conversion command: root2csv.

$> go get go-hep.org/x/hep/cmd/root2csv
$> root2csv --help
root2csv -h
Usage of root2csv:
  -f string
        path to input ROOT file name
  -o string
        path to output CSV file name (default "output.csv")
  -t string
        name of the tree to convert (default "tree")

$> go doc go-hep.org/x/hep/cmd/root2csv 
root2csv converts the content of a ROOT TTree to a CSV file.

    Usage of root2csv:
      -f string
                path to input ROOT file name
      -o string
                path to output CSV file name (default "output.csv")
      -t string
                name of the tree to convert (default "tree")

By default, root2csv will write out a CSV file with ';' as a column
delimiter. root2csv ignores the branches of the TTree that are not supported
by CSV:

    - slices/arrays
    - C++ objects

Example:

    $> root2csv -o out.csv -t tree -f testdata/small-flat-tree.root
    $> head out.csv
    ## Automatically generated from "testdata/small-flat-tree.root"
    Int32;Int64;UInt32;UInt64;Float32;Float64;Str;N
    0;0;0;0;0;0;evt-000;0
    1;1;1;1;1;1;evt-001;1
    2;2;2;2;2;2;evt-002;2
    3;3;3;3;3;3;evt-003;3
    4;4;4;4;4;4;evt-004;4
    5;5;5;5;5;5;evt-005;5
    6;6;6;6;6;6;evt-006;6
    7;7;7;7;7;7;evt-007;7

AOB

The next release cycle will see improvements in the rootio department (read and write.)

Stay tuned.

Release v0.8

Release v0.8 is fresh from the oven.

No API changes in go-hep itself, just a few modifications to cater with changes in gonum/v1/gonum and gonum/v1/plot:

• gonum/v1/gonum/stat/distuv distributions replaced the name of their rand.Rand source field from Source to Src (to align with the standard library)
• gonum/v1/gonum/stat/... packages now use golang.org/x/exp/rand.Rand instead of math/rand.Rand

The biggest news (and where the work mostly happened between v0.7 and v0.8) is that there's now a Jupyter instance with a Go kernel installed with Go-HEP and Gonum libraries pre-installed.
This has been packaged up at go-hep/binder.
You can try it there:

fmom

fmom gained a new top-level function fmom.InvMass(p1, p2 P4) float64 :

// InvMass computes the invariant mass of two incoming 4-vectors p1 and p2.
func InvMass(p1, p2 P4) float64 {
       p := Add(p1, p2)
       return p.M()
}

hplot

hplot gained a new top-level function Show:

// Show displays the plot according to format, returning the raw bytes and
// an error, if any.
//
// If format is the empty string, then "png" is selected.
// The list of accepted format strings is the same one than from
// the gonum.org/v1/plot/vg/draw.NewFormattedCanvas function.
func Show(p *Plot, w, h vg.Length, format string) ([]byte, error) { ... }

Show is especially useful for the new support for Go in Jupyter notebooks (via the Neugram interpreter)

See https://github.com/go-hep/binder for more details and examples on using Go and Go-HEP in Jupyter.

David Blyth provided a new hplot.HInfoStdDev HInfoStyle value and made them bitset-like.
Thanks David!

rootio

• cmd/root-dump can now recursively handle rootio.Directory and thus walk an entire ROOT file.
• cmd/root-ls got the same handling of rootio.Directory.

Release v0.7

Release v0.7 is fresh from the oven with a few breaking API changes.

cmd/yoda

The cmd/rio2yoda, cmd/root2yoda and cmd/yoda2rio commands now support gzipped YODA files.

The cmd/yoda2rio is also more lenient when dealing with YODA objects that are not yet supported on the hbook side (Counter, S1D, ...) (thanks Lukas Heinrich for the report.)

fastjet

Bastian added the ability to plot the Voronoi diagram from a set of 2D-points.

hbook

hbook/yodacnv was modified to support the more lenient approach with regard to unsupported (yet!) YODA objects.

hplot

hplot has seen the most user-facing work:

• a new default style that is more in-line with current aesthetic standards (ie: matplotlib-like)
• hplot.New uses this new default style and thus is now able to ensure it won't fail loading the fonts
• hplot.New thus only returns *hplot.Plot, without an error value
• hplot.NewH1D is also able to ensure no fonts-loading error will araise and thus only return a *hplot.H1D

So, where you were doing:

p, err := hplot.New()
if err != nil {
	log.Fatal(err)
}

h, err := hplot.NewH1D(h1d)
if err != nil {
	log.Fatal(err)
}

you can now just write:

p := hplot.New()
h := hplot.NewH1D(h1d)

pawgo

pawgo has been slightly updated to support plotting 2-dim histograms.
pawgo was also fixed to correctly handle YODA files converted to RIO (thanks Lukas Heinrich for the report.)

rootio

rootio gained 2 new commands:

• rootio/cmd/root-diff: a command to print the differences between 2 ROOT files, including the content of their TTrees,
• rootio/cmd/root-print: a command to print histograms contained in ROOT files into PDF, PNG, ... files.

rootio now also defines and exports the rootio.H1 and rootio.H2 interfaces that are implemented by the TH1x and TH2x (respectively) concrete types.

Release v0.6

Release v0.6 is a small iteration (code-wise) but contains:

• a software paper submitted to the Journal of OpenSource Software (JOSS):
• a new WIP command, rootio/cmd/root-dump, to dump the content of ROOT files, including the entries of TTrees,
• documentation fixes and updates in the rootio package,
• still more work in the Delaunay and Voronoi area (thanks Bastian!)

One API change that is worth noting, is the retrieval of rootio.Keys from a rootio.Directory.
The d4823f0 changes:

type Directory interface {
    Get(namecycle string) (Object, bool)
}

to:

type Directory interface {
    Get(namecycle string) (Object, error)
}

this allows to more quickly bubble-up errors while reducing boilerplate code at the Directory.Get(...) call site:

diff --git a/cmd/root2npy/main.go b/cmd/root2npy/main.go
index c81e81d..e3827fa 100644
--- a/cmd/root2npy/main.go
+++ b/cmd/root2npy/main.go
@@ -159,9 +159,9 @@ func main() {
        }
        defer f.Close()
 
-       obj, ok := f.Get(*tname)
-       if !ok {
-               log.Fatalf("no object named %q in file %q", *tname, *fname)
+       obj, err := f.Get(*tname)
+       if err != nil {
+               log.Fatal(err)
        }

Release v0.5

Release v0.5 has seen some interesting work.

General

• Go-HEP has migrated all its Gonum imports to use "gonum.org/v1/gonum/{mat,stat,...}", instead of the old "github.com/gonum/{mat64,stat,...}"
• Go-HEP has also migrated all its use of gonum/plot from "github.com/gonum/plot/..." to the new "gonum.org/v1/plot/..."
• Go-1.6's support has been dropped from the Travis-CI matrix
• Go-1.9's support has been added to the Travis-CI matrix

rootio

The rootio package has seen most of the work: performance improvements for reading ROOT files, a new root2npy command and the beginning of the work to provide write support.

The performances of rootio have been assessed for the ACAT conference.
The talk has all the details and can be read here.
In a nutshell, rootio is now "just" 2 times slower than C++ ROOT to read files, and uses less than 10 times the VMem of C++ ROOT.
More work to match the read speed of C++ ROOT is of course needed but we think we know how to recoup most of performance delta.

cmd/root2npy is a simple command to quickly convert a ROOT file containing a TTree into a NumPy array data file format.

cmd/root-srv has been updated to support https (thanks Michaël Ughetto).

cmd/root2yoda is a simple command to quickly convert a ROOT file containing histograms and profiles into a YODA file.

hbook & csvutil

csvutil/csvdriver has been improved to correctly handle CSV headers.
You need to tell the driver that the CSV file contains a header describing the columns of the CSV file, but once that's done, you can use the correct columns' names in your SQL queries.

hbook/ntup/ntcsv is a new package that leverages the new feature of csvdriver to easily create hbook/ntup.Ntuple values from a CSV file, with the correct columns' names.

fit

Initial support for automatically computing the Hessian matrix has been added.
If the user does not provide a Hess function, Gonum's diff/fd.Hessian will be used

fastjet

More of the GSoC work for the faster jet clustering has been integrated.
Thanks Bastian Wieck.

AOB

With release v0.5, Go-HEP gained its first external user: eicio, "an exploratory project testing ideas about forward-looking IO solutions to suit the needs of the Electron Ion Collider (EIC) community."
Welcome eicio!

Release v0.4

Release v0.4 is a small iteration (because of holidays.)

But still, it does bring interesting things to the table:

• root2npy, a new command to convert the content of simple TTrees into a NumPy data file,
• migration from the github.com/gonum/xyz packages to their new location -- gonum.org/v1/gonum/xyz,

fastjet

• preliminary work for implementing better/faster jet clustering, thanks Bastian Wieck

hplot

• make the hplot.H1D type implement gonum/plot.Thumbnailer (for better integration with legends and labels)

lcio

• fix a bug in the marshaling of lcio.TrackerRawData, thanks David Blyth