Skip to content

Commit

Permalink
Update documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
@actions-user
actions-user committed Jul 1, 2024
1 parent 0679468 commit d9144e3
Show file tree
Hide file tree
Showing 3 changed files with 53 additions and 8 deletions.
45 changes: 41 additions & 4 deletions _modules/kerchunk/hdf.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -203,22 +203,39 @@ <h1>Source code for kerchunk.hdf</h1><div class="highlight"><pre>

<div class="viewcode-block" id="SingleHdf5ToZarr.translate">
<a class="viewcode-back" href="../../reference.html#kerchunk.hdf.SingleHdf5ToZarr.translate">[docs]</a>
<span class="k">def</span> <span class="nf">translate</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">translate</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">preserve_linked_dsets</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Translate content of one HDF5 file into Zarr storage format.</span>

<span class="sd"> This method is the main entry point to execute the workflow, and</span>
<span class="sd"> returns a &quot;reference&quot; structure to be used with zarr/kerchunk</span>

<span class="sd"> No data is copied out of the HDF5 file.</span>

<span class="sd"> Parameters</span>
<span class="sd"> ----------</span>
<span class="sd"> preserve_linked_dsets : bool (optional, default False)</span>
<span class="sd"> If True, translate HDF5 soft and hard links for each `h5py.Dataset`</span>
<span class="sd"> into the reference structure. Requires h5py version 3.11.0 or later.</span>
<span class="sd"> Will not translate external links or links to `h5py.Group` objects.</span>

<span class="sd"> Returns</span>
<span class="sd"> -------</span>
<span class="sd"> dict</span>
<span class="sd"> Dictionary containing reference structure.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">lggr</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">&quot;Translation begins&quot;</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_transfer_attrs</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_h5f</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_zroot</span><span class="p">)</span>

<span class="bp">self</span><span class="o">.</span><span class="n">_h5f</span><span class="o">.</span><span class="n">visititems</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_translator</span><span class="p">)</span>

<span class="k">if</span> <span class="n">preserve_linked_dsets</span><span class="p">:</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">has_visititems_links</span><span class="p">():</span>
<span class="k">raise</span> <span class="ne">RuntimeError</span><span class="p">(</span>
<span class="s2">&quot;&#39;preserve_linked_dsets&#39; kwarg requires h5py 3.11.0 or later &quot;</span>
<span class="sa">f</span><span class="s2">&quot;is installed, found </span><span class="si">{</span><span class="n">h5py</span><span class="o">.</span><span class="n">__version__</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_h5f</span><span class="o">.</span><span class="n">visititems_links</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_translator</span><span class="p">)</span>

<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">spec</span> <span class="o">&lt;</span> <span class="mi">1</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">store</span>
<span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">store</span><span class="p">,</span> <span class="n">LazyReferenceMapper</span><span class="p">):</span>
Expand Down Expand Up @@ -338,10 +355,26 @@ <h1>Source code for kerchunk.hdf</h1><div class="highlight"><pre>
<span class="p">)</span>
<span class="k">return</span> <span class="n">filters</span>

<span class="k">def</span> <span class="nf">_translator</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">h5obj</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="n">h5py</span><span class="o">.</span><span class="n">Dataset</span><span class="p">,</span> <span class="n">h5py</span><span class="o">.</span><span class="n">Group</span><span class="p">]):</span>
<span class="k">def</span> <span class="nf">_translator</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">name</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">h5obj</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span>
<span class="n">h5py</span><span class="o">.</span><span class="n">Dataset</span><span class="p">,</span> <span class="n">h5py</span><span class="o">.</span><span class="n">Group</span><span class="p">,</span> <span class="n">h5py</span><span class="o">.</span><span class="n">SoftLink</span><span class="p">,</span> <span class="n">h5py</span><span class="o">.</span><span class="n">HardLink</span><span class="p">,</span> <span class="n">h5py</span><span class="o">.</span><span class="n">ExternalLink</span>
<span class="p">],</span>
<span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Produce Zarr metadata for all groups and datasets in the HDF5 file.&quot;&quot;&quot;</span>
<span class="k">try</span><span class="p">:</span> <span class="c1"># method must not raise exception</span>
<span class="n">kwargs</span> <span class="o">=</span> <span class="p">{}</span>

<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">h5obj</span><span class="p">,</span> <span class="p">(</span><span class="n">h5py</span><span class="o">.</span><span class="n">SoftLink</span><span class="p">,</span> <span class="n">h5py</span><span class="o">.</span><span class="n">HardLink</span><span class="p">)):</span>
<span class="n">h5obj</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_h5f</span><span class="p">[</span><span class="n">name</span><span class="p">]</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">h5obj</span><span class="p">,</span> <span class="n">h5py</span><span class="o">.</span><span class="n">Group</span><span class="p">):</span>
<span class="c1"># continues iteration of visititems_links</span>
<span class="n">lggr</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;Skipping translation of HDF5 linked group: &#39;</span><span class="si">{</span><span class="n">h5obj</span><span class="o">.</span><span class="n">name</span><span class="si">}</span><span class="s2">&#39;&quot;</span>
<span class="p">)</span>
<span class="k">return</span> <span class="kc">None</span>

<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">h5obj</span><span class="p">,</span> <span class="n">h5py</span><span class="o">.</span><span class="n">Dataset</span><span class="p">):</span>
<span class="n">lggr</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;HDF5 dataset: </span><span class="si">{</span><span class="n">h5obj</span><span class="o">.</span><span class="n">name</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">lggr</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;HDF5 compression: </span><span class="si">{</span><span class="n">h5obj</span><span class="o">.</span><span class="n">compression</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
Expand Down Expand Up @@ -523,7 +556,7 @@ <h1>Source code for kerchunk.hdf</h1><div class="highlight"><pre>
<span class="p">)</span>

<span class="c1"># Create a Zarr array equivalent to this HDF5 dataset...</span>
<span class="n">za</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_zroot</span><span class="o">.</span><span class="n">create_dataset</span><span class="p">(</span>
<span class="n">za</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_zroot</span><span class="o">.</span><span class="n">require_dataset</span><span class="p">(</span>
<span class="n">h5obj</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
<span class="n">shape</span><span class="o">=</span><span class="n">h5obj</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span>
<span class="n">dtype</span><span class="o">=</span><span class="n">dt</span> <span class="ow">or</span> <span class="n">h5obj</span><span class="o">.</span><span class="n">dtype</span><span class="p">,</span>
Expand Down Expand Up @@ -571,7 +604,7 @@ <h1>Source code for kerchunk.hdf</h1><div class="highlight"><pre>

<span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">h5obj</span><span class="p">,</span> <span class="n">h5py</span><span class="o">.</span><span class="n">Group</span><span class="p">):</span>
<span class="n">lggr</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;HDF5 group: </span><span class="si">{</span><span class="n">h5obj</span><span class="o">.</span><span class="n">name</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">zgrp</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_zroot</span><span class="o">.</span><span class="n">create_group</span><span class="p">(</span><span class="n">h5obj</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
<span class="n">zgrp</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_zroot</span><span class="o">.</span><span class="n">require_group</span><span class="p">(</span><span class="n">h5obj</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_transfer_attrs</span><span class="p">(</span><span class="n">h5obj</span><span class="p">,</span> <span class="n">zgrp</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="kn">import</span> <span class="nn">traceback</span>
Expand Down Expand Up @@ -731,6 +764,10 @@ <h1>Source code for kerchunk.hdf</h1><div class="highlight"><pre>

<span class="k">def</span> <span class="nf">_is_netcdf_variable</span><span class="p">(</span><span class="n">dataset</span><span class="p">:</span> <span class="n">h5py</span><span class="o">.</span><span class="n">Dataset</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">any</span><span class="p">(</span><span class="s2">&quot;_Netcdf4&quot;</span> <span class="ow">in</span> <span class="n">_</span> <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="n">dataset</span><span class="o">.</span><span class="n">attrs</span><span class="p">)</span>


<span class="k">def</span> <span class="nf">has_visititems_links</span><span class="p">():</span>
<span class="k">return</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">h5py</span><span class="o">.</span><span class="n">Group</span><span class="p">,</span> <span class="s2">&quot;visititems_links&quot;</span><span class="p">)</span>
</pre></div>

</div>
Expand Down
14 changes: 11 additions & 3 deletions reference.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -216,21 +216,29 @@ <h2>File format backends<a class="headerlink" href="#file-format-backends" title
<p class="rubric">Methods</p>
<table class="autosummary longtable docutils align-default">
<tbody>
<tr class="row-odd"><td><p><a class="reference internal" href="#kerchunk.hdf.SingleHdf5ToZarr.translate" title="kerchunk.hdf.SingleHdf5ToZarr.translate"><code class="xref py py-obj docutils literal notranslate"><span class="pre">translate</span></code></a>()</p></td>
<tr class="row-odd"><td><p><a class="reference internal" href="#kerchunk.hdf.SingleHdf5ToZarr.translate" title="kerchunk.hdf.SingleHdf5ToZarr.translate"><code class="xref py py-obj docutils literal notranslate"><span class="pre">translate</span></code></a>([preserve_linked_dsets])</p></td>
<td><p>Translate content of one HDF5 file into Zarr storage format.</p></td>
</tr>
</tbody>
</table>
<dl class="py method">
<dt class="sig sig-object py" id="kerchunk.hdf.SingleHdf5ToZarr.translate">
<span class="sig-name descname"><span class="pre">translate</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/kerchunk/hdf.html#SingleHdf5ToZarr.translate"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#kerchunk.hdf.SingleHdf5ToZarr.translate" title="Link to this definition"></a></dt>
<span class="sig-name descname"><span class="pre">translate</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">preserve_linked_dsets</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/kerchunk/hdf.html#SingleHdf5ToZarr.translate"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#kerchunk.hdf.SingleHdf5ToZarr.translate" title="Link to this definition"></a></dt>
<dd><p>Translate content of one HDF5 file into Zarr storage format.</p>
<p>This method is the main entry point to execute the workflow, and
returns a “reference” structure to be used with zarr/kerchunk</p>
<p>No data is copied out of the HDF5 file.</p>
<dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><dl class="simple">
<dt><strong>preserve_linked_dsets</strong><span class="classifier">bool (optional, default False)</span></dt><dd><p>If True, translate HDF5 soft and hard links for each <cite>h5py.Dataset</cite>
into the reference structure. Requires h5py version 3.11.0 or later.
Will not translate external links or links to <cite>h5py.Group</cite> objects.</p>
</dd>
</dl>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><dl class="simple">
<dt>dict</dt><dd><p>Dictionary containing reference structure.</p>
</dd>
</dl>
Expand Down
Loading

0 comments on commit d9144e3

Please sign in to comment.