Skip to content

darothen/xbpch

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

90 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

xbpch: xarray interface for bpch files

PyPI version Documentation Status Zenodo DOI

xpbch is a simple utility for reading the proprietary binary punch format (bpch) outputs used in versions of GEOS-Chem earlier than v11-02. The utility allows a user to load this data into an xarray- and dask-powered workflow without necessarily pre-processing the data using GAMAP or IDL.

This package is maintained as part of a broader, community effort to tackle big data problems in geoscience.

What's the Deal?

The contemporary scientific Python software stack provides free, powerful tools for nearly all of your data processing, analysis, and visualization needs. These tools are well supported by a large community of heavily invested users and developers from academia, government, and industry. They are also developed (mostly) as part of community-based, open-source, and user-driven projects.

For nearly any application you might have in the geosciences, you can start using this powerful, free software stack today with minimal friction. However, one friction point that has tripped up adoption by GEOS-Chem users is that it is difficult to work with legacy bpch-format diagnostics files. xbpch solves this problem by providing a convenient and performant way to read these files into a modern Python-based analysis or workflow.

Furthermore, xbpch is 100% future-proof. In two years, when your GEOS-Chem simulations are writing NetCDF diagnostics, you won't need to change more than a single line of code in any of your scripts using xbpch. All you'll need to do is swap out xbpch's function for reading data and instead defer to it's parent package (xarray). It will literally take less than 10 keystrokes to make this change in your code. Plus - you'll be backwards compatible with any legacy output you need to analyze.

So give xbpch a try, and let me know what issues you run in to! If we solve them once today, they'll be solved in perpetuity, which means more time for you to do science and less time to worry about processing data.

Installation

Requirements

xbpch is only intended for use with Python 3, although with some modifications it would likely work with Python 2.7 (Pull Requests are welcome!). As the package description implies, it requires up-to-date copies of xarray (>= version 0.9) and dask (>= version 0.14). The best way to install these packages is by using the conda package management system, or the Anaconda Python distribution.

To install xbpch and its dependencies using conda, execute from a terminal:

$ conda install -c conda-forge xbpch xarray dask

Alternatively, you can install xbpch from PyPI:

$ pip install xbpch

You can also install xbpch from its source. To do this, you can either clone the source directory and manually install:

$ git clone https://github.com/darothen/xbpch.git
$ cd xbpch
$ python setup.py install

or, you can install via pip directly from git:

$ pip install git+https://github.com/darothen/xbpch.git

Please note that if you locally clone the repository from GitHub but do not explicitly install the package using setup.py, the file xbpch/version.py will not get written properly and you will not be able to use the package. We strongly recommend you install the package using traditional techniques to ensure that all dependencies are properly added to your environment.

Quick Start

If you're already familiar with loading and manipulating data with xarray, then it's easy to dive right into xbpch. Navigate to a directory on disk which contains your .bpch output, as well as tracerinfo.dat and diaginfo.dat, and execute from a Python interpeter:

from xbpch import open_bpchdataset
fn = "my_geos_chem_output.bpch"
ds = open_bpchdataset(fn)

After a few seconds (depending on your hard-drive speed) you should be able to interact with ds just as you would any xarray.Dataset object.

Caveats and Future Notes

xbpch should work for most simple workflows, especially if you need a quick-and-dirty way to ingest legacy GEOS-Chem output. It is not tested against the majority of output grids, including data for the Hg model or nested models. Grid information (at least for the vertical) is hard-coded and may not be accurate for the most recent versions of GEOS-Chem.

Most importantly, xbpch does not yet solve the problem of manually scanning bpch files before producing a dataset on disk. Because the bpch format does not encode metadata about what its contents actually are, we must manually process this from any output file we wish to load. For the time being, we do not short-circuit this process because we cannot necessarily predict file position offsets in the bpch files we read. In the future, I hope to come up with an elegant solution for solving this problem.

Acknowledgments

This utility packages together a few pre-existing toolkits which have been floating around the Python-GEOS-Chem community. In particular, I would like to acknowledge the following pieces of software which I have built this utility around:

Furthermore, the strategies used to load and process binary output on disk through xarray's DataStore API is heavily inspired by Ryan Abernathey's package xmitgcm.

License

Copyright (c) 2017 Daniel Rothenberg

This work is licensed under a permissive MIT License. I acknowledge important contributions from Benoît Bovy, Gerrit Kuhlmann, and Christoph Keller in the form of prior work which helped create the foundation for this package.

Contact

Daniel Rothenberg - darothen@mit.edu