Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry pick docs 6.2 #177

Merged
merged 14 commits into from
Jul 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# Source code
* @kiritigowda @rrawther
# Documentation files
docs/* @saadrahim @LisaDelaney @kiritigowda @rrawther
*.md @saadrahim @LisaDelaney @kiritigowda @rrawther
*.rst @saadrahim @LisaDelaney
docs/ @ROCm/rocm-documentation @kiritigowda @rrawther
*.md @ROCm/rocm-documentation @kiritigowda @rrawther
*.rst @ROCm/rocm-documentation @kiritigowda @rrawther
.readthedocs.yaml @ROCm/rocm-documentation
# Header directory
library/include/* @saadrahim @LisaDelaney @kiritigowda @rrawther
# Source code
@kiritigowda @rrawther
library/include/ @ROCm/rocm-documentation @kiritigowda @rrawther
2 changes: 1 addition & 1 deletion .github/workflows/linting.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ on:
jobs:
call-workflow-passing-data:
name: Documentation
uses: RadeonOpenCompute/rocm-docs-core/.github/workflows/linting.yml@develop
uses: ROCm/rocm-docs-core/.github/workflows/linting.yml@develop
4 changes: 2 additions & 2 deletions .readthedocs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ python:
- requirements: docs/sphinx/requirements.txt

build:
os: ubuntu-20.04
os: ubuntu-22.04
tools:
python: "3.8"
python: "3.10"
apt_packages:
- "doxygen"
29 changes: 16 additions & 13 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
<p align="center"><img width="70%" src="data/rocAL_logo.png" /></p>

# rocAL User Guide

Today’s deep learning applications require loading and pre-processing data efficiently to achieve high processing throughput. This requires creating efficient processing pipelines fully utilizing the underlying hardware capabilities. Some examples are load and decode data, do a variety of augmentations, color-format conversions, etc.
Expand All @@ -9,12 +7,12 @@ AMD ROCm Augmentation Library (rocAL) is designed to efficiently do such process
These pipelines are programmable by the user using both C++ and Python APIs.

## User Guide Chapters
- [Chapter 1 - Overview](user_guide/ch1.md)
- [Chapter 2 - Architecture Components](user_guide/ch2.md)
- [Chapter 3 - Installation](user_guide/ch3.md)
- [Chapter 4 - Using with Python API](user_guide/ch4.md)
- [Chapter 5 - Framework Integration](user_guide/ch5.md)
- [Chapter 6 - Using with C++ API](user_guide/ch6.md)
* [Chapter 1 - Overview](user_guide/ch1.md)
* [Chapter 2 - Architecture Components](user_guide/ch2.md)
* [Chapter 3 - Installation](user_guide/ch3.md)
* [Chapter 4 - Using with Python API](user_guide/ch4.md)
* [Chapter 5 - Framework Integration](user_guide/ch5.md)
* [Chapter 6 - Using with C++ API](user_guide/ch6.md)

## Key Components of rocAL

Expand Down Expand Up @@ -88,7 +86,7 @@ amd.rocal.types are enums exported from C++ API to python. Some examples include

* Build and install RPP
* Build and install MIVisionX which installs rocAL c++ lib
* Go to [rocal_pybind](https://github.com/ROCm/rocAL/tree/master/rocAL_pybind) folder
* Go to the [rocal_pybind](https://github.com/ROCm/rocAL/tree/develop/rocAL_pybind) folder
* sudo ./run.sh

### Steps to run MLPerf Resnet50 classification training with rocAL on a system with MI50+ and ROCm
Expand All @@ -97,12 +95,15 @@ amd.rocal.types are enums exported from C++ API to python. Some examples include
* Step 2: Build [MIVisionX Pytorch docker](https://github.com/ROCm/rocAL/blob/master/docker/README.md)
* Step 3: Install rocAL python_pybind plugin as described above
* Step 4: Clone [MLPerf](https://github.com/rrawther/MLPerf-mGPU) branch and checkout mlperf-v1.1-rocal branch
```

``` bash
git clone -b mlperf-v1.1-rocal https://github.com/rrawther/MLPerf-mGPU
```

* Step 5: Modify RN50_AMP_LARS_8GPUS_NCHW.sh or RN50_AMP_LARS_8GPUS_NHWC.sh to reflect correct path for imagenet directory
* Step 8: Run RN50_AMP_LARS_8GPUS_NCHC.sh or RN50_AMP_LARS_8GPUS_NHWC.sh
```

``` bash
./RN50_AMP_LARS_8GPUS_NCHW.sh
(or)
./RN50_AMP_LARS_8GPUS_NHWC.sh
Expand All @@ -112,9 +113,11 @@ git clone -b mlperf-v1.1-rocal https://github.com/rrawther/MLPerf-mGPU

* Refer to the [docker](https://github.com/ROCm/MIVisionX#docker) page for prerequisites and information on building the docker
* Step 1: Run the docker image*
````

``` bash
sudo docker run -it -v <Path-To-Data-HostSystem>:/data -v /<Path-to-GitRepo>:/dockerx -w /dockerx --privileged --device=/dev/kfd --device=/dev/dri --group-add video --shm-size=4g --ipc="host" --network=host <docker-name>
````
```

* Optional: Map localhost directory on the docker image
* option to map the localhost directory with imagenet dataset folder to be accessed on the docker image.
* usage: -v {LOCAL_HOST_DIRECTORY_PATH}:{DOCKER_DIRECTORY_PATH}
30 changes: 14 additions & 16 deletions docs/how-to/framework.rst
Original file line number Diff line number Diff line change
Expand Up @@ -75,22 +75,22 @@ Follow these steps:
.. code-block:: python
:caption: Import libraries for PyTorch

import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim


4. Call the training pipeline with rocAL classification data `loader <https://github.com/ROCm/rocAL/blob/master/docs/examples/pytorch/test_training.py#L78>`_.

.. code-block:: python
:caption: Call the training pipeline

Def get_pytorch_train_loader(self):
print(“in get_pytorch_train_loader function”)
pipe_train = trainPipeline(self.data_path, self.batch_size, self.num_classes, self.one_hot, self.local_rank,
self.world_size, self.num_thread, self.crop, self.rocal_cpu, self.fp16)
pipe_train.build()
train_loader = ROCALClassificationIterator(pipe_train, device=”cpu” if self.rocal_cpu else “cuda”, device_id = self.local_rank)
Def get_pytorch_train_loader(self):
print(“in get_pytorch_train_loader function”)
pipe_train = trainPipeline(self.data_path, self.batch_size, self.num_classes, self.one_hot, self.local_rank,
self.world_size, self.num_thread, self.crop, self.rocal_cpu, self.fp16)
pipe_train.build()
train_loader = ROCALClassificationIterator(pipe_train, device=”cpu” if self.rocal_cpu else “cuda”, device_id = self.local_rank)


5. Run the `training script <https://github.com/ROCm/rocAL/blob/master/docs/examples/pytorch/test_training.py#L179>`_.
Expand Down Expand Up @@ -130,7 +130,7 @@ Create Data-loading Pipeline

Follow these steps:

1. Import libraries for `rocAL_pybind <https://github.com/ROCm/rocAL/blob/master/rocAL_pybind/examples/tf_petsTrainingExample/train_withROCAL_withTFRecordReader.py#L22>`_.
1. Import libraries for `rocAL_pybind <https://github.com/ROCm/rocAL/blob/master/docs/examples/tf/pets_training/train.py#L22>`_.

.. code-block:: python
:caption: Import libraries
Expand All @@ -141,7 +141,7 @@ Follow these steps:
import amd.rocal.types as types


2. See a rocAL pipeline for TensorFlow below. It reads data from the TFRecords using TFRecord Reader and uses ``fn.decoders.image`` to decode the raw `images <https://github.com/ROCm/rocAL/blob/master/rocAL_pybind/examples/tf_petsTrainingExample/train_withROCAL_withTFRecordReader.py#L128>`_.
2. See a rocAL pipeline for TensorFlow below. It reads data from the TFRecords using TFRecord Reader and uses ``fn.decoders.image`` to decode the raw `images <https://github.com/ROCm/rocAL/blob/master/docs/examples/tf/pets_training/train.py#L128>`_.

.. code-block:: python
:caption: Pipeline for TensorFlow
Expand Down Expand Up @@ -170,7 +170,7 @@ Follow these steps:
trainPipe.build()


3. Import libraries for `TensorFlow <https://github.com/ROCm/rocAL/blob/master/rocAL_pybind/examples/tf_petsTrainingExample/train_withROCAL_withTFRecordReader.py#L174>`_.
3. Import libraries for `TensorFlow <https://github.com/ROCm/rocAL/blob/master/docs/examples/tf/pets_training/train.py#L174>`_.

.. code-block:: python
:caption: Import libraries for TensorFlow
Expand All @@ -193,8 +193,7 @@ Follow these steps:

4. To see and run a sample training script, refer to `rocAL TensorFlow example <https://github.com/ROCm/MIVisionX/tree/master/rocAL/rocAL_pybind/examples/tf_petsTrainingExample>`_.


.. _ml-perf:
.. __resnet50:

Run MLPerf Resnet50 classification training with rocAL
=======================================================
Expand All @@ -216,8 +215,7 @@ Run MLPerf Resnet50 classification training with rocAL
* Option to map the localhost directory with imagenet dataset folder to be accessed on the docker image.
* Usage: ``-v {LOCAL_HOST_DIRECTORY_PATH}:{DOCKER_DIRECTORY_PATH}``

#. Install rocAL ``python_pybind`` plugin as described above
#. Clone `MLPerf <https://github.com/rrawther/MLPerf-mGPU>`_ repo and checkout ``mlperf-v1.1-rocal`` branch
#. To see and run a sample training script, refer to `rocAL Imagenet example <https://github.com/ROCm/rocAL/tree/master/docs/examples/pytorch/imagenet_training>`_.

.. code-block:: shell

Expand Down
12 changes: 5 additions & 7 deletions docs/how-to/using-with-python.rst
Original file line number Diff line number Diff line change
Expand Up @@ -91,11 +91,10 @@ Given below is an example of a file reader, which takes a folder of images as in
images = fn.decoders.image(jpegs, file_root=data_path, device=decoder_device, output_type=types.RGB, shard_id=0, num_shards=1, random_shuffle=True)
images = fn.resize(images, device=rocal_device, resize_x=300, resize_y=300)


Defining the Pipeline
------------------------

To define a pipeline, see `https://github.com/ROCm/rocAL/blob/master/rocAL_pybind/amd/rocal/pipeline.py#L29`.
To define a pipeline, see `<https://github.com/ROCm/rocAL/blob/master/rocAL_pybind/amd/rocal/pipeline.py#L29>`__.

.. code-block:: shell
:caption: Pipeline Class
Expand Down Expand Up @@ -192,7 +191,7 @@ Running the Pipeline

To run/use the pipeline, simply create a data loader using the pipeline and iterate through it to get the next batch of images with labels.

To run the pipeline, see `https://github.com/ROCm/rocAL/blob/master/rocAL_pybind/examples/rocAL_api_python_unittest.py#L168`
To run the pipeline, see `<https://github.com/ROCm/rocAL/blob/master/tests/python_api/unit_test.py#L168>`__.

.. code-block:: python
:caption: Run the Pipeline
Expand Down Expand Up @@ -220,7 +219,7 @@ Performing Augmentations

rocAL not only reads images from the disk and batches them into tensors, it can also perform various augmentations on those images.

To read images, decode them, and rotate them in the pipeline, see `https://github.com/ROCm/rocAL/blob/master/rocAL_pybind/examples/rocAL_api_python_unittest.py#L77`
To read images, decode them, and rotate them in the pipeline, see `<https://github.com/ROCm/rocAL/blob/master/tests/python_api/unit_test.py#L77>`__

.. code-block:: python
:caption: Perform Augmentations
Expand All @@ -246,11 +245,10 @@ To run the pipeline, see:
images, labels = pipe_out
show_images(images)


rocAL Data Types
=========================

All the rocAL data types are defined under `amd.rocal.types <https://github.com/ROCm/MIVisionX/blob/master/rocAL/rocAL_pybind/amd/rocal/types.py>`_. Import this library in the application to access the various data types such as rocAL status, processing mode, tensor output type, image size evaluation policy, image color, tensor layout, decode device, resize scaling mode, and resize interpolation type.
All the rocAL data types are defined under `amd.rocal.types <https://github.com/ROCm/rocAL/blob/master/rocAL_pybind/amd/rocal/types.py>`_. Import this library in the application to access the various data types such as rocAL status, processing mode, tensor output type, image size evaluation policy, image color, tensor layout, decode device, resize scaling mode, and resize interpolation type.

Here are some of the commonly used rocAL data types:

Expand All @@ -269,7 +267,7 @@ Here are some of the commonly used rocAL data types:
* tensor_dtype = types.FLOAT
* tensor_dtype = types.FLOAT16

To see the usage of the above-mentioned data types, see `https://github.com/ROCm/rocAL/blob/master/rocAL_pybind/amd/rocal/pipeline.py#L97`
To see the usage of the above-mentioned data types, see `<https://github.com/ROCm/rocAL/blob/master/rocAL_pybind/amd/rocal/pipeline.py#L97>`__.

.. code-block:: python

Expand Down
4 changes: 2 additions & 2 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ The rocAL documentation is structured as follows:

* :ref:`install`

.. grid-item-card:: How-to
.. grid-item-card:: How to

* :ref:`overview`
* :ref:`architecture`
Expand All @@ -54,7 +54,7 @@ The rocAL documentation is structured as follows:

* :ref:`examples`

To contribute to the documentation refer to `Contributing to ROCm <https://rocm.docs.amd.com/en/latest/contribute/index.html>`_.
To contribute to the documentation refer to `Contributing to ROCm Docs <https://rocm.docs.amd.com/en/latest/contribute/contributing.html>`_.

You can find licensing information on the `Licensing <https://rocm.docs.amd.com/en/latest/about/license.html>`_ page.

61 changes: 41 additions & 20 deletions docs/install/install.rst
Original file line number Diff line number Diff line change
Expand Up @@ -18,29 +18,52 @@ Prerequisites
- Ubuntu 20.04 or 22.04
- CentOS 7
- RedHat 8 or 9
- SLES 15-SP4

* `ROCm supported hardware <https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html>`_
* Install ROCm with `amdgpu-install <https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/amdgpu-install.html>`_ with ``--usecase=graphics,rocm --no-32``
* `RPP <https://github.com/ROCm/rpp>`_
* `AMD OpenVX™ <https://github.com/ROCm/MIVisionX/tree/master/amd_openvx>`_ and AMD OpenVX™ Extensions: ``VX_RPP`` and ``AMD Media`` - MIVisionX Components
* `Turbo JPEG <https://libjpeg-turbo.org/>`_ - Version 2.0.6.2 from ``https://github.com/rrawther/libjpeg-turbo.git``
* `Half-precision floating-point <https://half.sourceforge.net>`_ library - Version 1.12.0 or higher
* `Google Protobuf <https://developers.google.com/protocol-buffers>`_ - Version 3.12.4 or higher
* `LMBD Library <http://www.lmdb.tech/doc/>`_
* `RapidJSON <https://github.com/Tencent/rapidjson>`_
* `PyBind11 <https://github.com/pybind/pybind11>`_
* `HIP <https://github.com/ROCm/HIP>`_
* OpenMP
* C++17
- SLES 15-SP5

* ROCm-supported hardware

* Install ROCm `6.1.0` or later with amdgpu-install: Required usecase - rocm

* HIP

* RPP

* MIVisionX

* rocDecode

* Half-precision floating-point library - Version `1.12.0` or higher

* Google Protobuf

* LMBD Library

* Python3 and Python3 PIP

* Python Wheel

* PyBind11

* Turbo JPEG

* RapidJSON

* **Optional**: FFMPEG

* **Optional**: OpenCV

IMPORTANT
* Compiler features required
* OpenMP
* C++17

Installation instructions
================================

The installation process uses the following steps:

* `ROCm supported hardware install <https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html>`_
* Install ROCm with `amdgpu-install <https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/amdgpu-install.html>`_ with ``--usecase=graphics,rocm --no-32``
* Install ROCm with `amdgpu-install <https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/amdgpu-install.html>`_ with ``--usecase=rocm``
* Use either :ref:`package-install` or :ref:`source-install` as described below.

.. _package-install:
Expand Down Expand Up @@ -118,18 +141,17 @@ Prerequisites:
- Ubuntu 20.04 or 22.04
- CentOS 7
- RedHat 8 or 9
- SLES 15-SP4
- SLES 15-SP5

* `ROCm supported hardware <https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html>`_
* Install ROCm with `amdgpu-install <https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/amdgpu-install.html>`_ with ``--usecase=graphics,rocm --no-32``
* Install ROCm with `amdgpu-install <https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/amdgpu-install.html>`_ with ``--usecase=rocm``

Using ``rocAL-setup.py`` script:

.. code-block:: python

python rocAL-setup.py --directory [setup directory - optional (default:~/)]
--opencv [OpenCV Version - optional (default:4.6.0)]
--protobuf [ProtoBuf Version - optional (default:3.12.4)]
--pybind11 [PyBind11 Version - optional (default:v2.10.4)]
--reinstall [Remove previous setup and reinstall (default:OFF)[options:ON/OFF]]
--backend [rocAL Dependency Backend - optional (default:HIP) [options:OCL/HIP]]
Expand Down Expand Up @@ -205,4 +227,3 @@ Test package will install ctest module to test rocAL. Follow below steps to test
mkdir rocAL-test && cd rocAL-test
cmake /opt/rocm/share/rocal/test/
ctest -VV

2 changes: 1 addition & 1 deletion docs/sphinx/_toc.yml.in
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ subtrees:
- file: install/install.rst
title: Installation
- file: how-to/index.rst
title: How To
title: How to
subtrees:
- entries:
- file: how-to/overview.rst
Expand Down
2 changes: 1 addition & 1 deletion docs/sphinx/requirements.in
Original file line number Diff line number Diff line change
@@ -1 +1 @@
rocm-docs-core[api_reference]>=0.24.0
rocm-docs-core[api_reference]==1.4.1
Loading