Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Install the Catalyst CLI into user Python environment #1434

Open
wants to merge 4 commits into
base: v0.10.0-rc
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/check-catalyst.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -479,7 +479,7 @@ jobs:
echo "OQC_LIB_DIR=$(pwd)/oqc-build" >> $GITHUB_ENV
echo "OQD_LIB_DIR=$(pwd)/oqd-build" >> $GITHUB_ENV
echo "CATALYST_BIN_DIR=$(pwd)/quantum-build/bin" >> $GITHUB_ENV
chmod +x $(pwd)/quantum-build/bin/catalyst-cli # artifact upload does not preserve permissions
chmod +x $(pwd)/quantum-build/bin/catalyst # artifact upload does not preserve permissions
- name: Run Python Lit Tests
run: |
Expand Down Expand Up @@ -559,7 +559,7 @@ jobs:
echo "MLIR_LIB_DIR=$(pwd)/llvm-build/lib" >> $GITHUB_ENV
echo "OQD_LIB_DIR=$(pwd)/oqd-build" >> $GITHUB_ENV
echo "CATALYST_BIN_DIR=$(pwd)/quantum-build/bin" >> $GITHUB_ENV
chmod +x $(pwd)/quantum-build/bin/catalyst-cli # artifact upload does not preserve permissions
chmod +x $(pwd)/quantum-build/bin/catalyst # artifact upload does not preserve permissions
- name: Run Python Pytest Tests (backend=lightning.kokkos)
run: |
Expand Down Expand Up @@ -616,7 +616,7 @@ jobs:
echo "RUNTIME_LIB_DIR=$(pwd)/runtime-build/lib" >> $GITHUB_ENV
echo "MLIR_LIB_DIR=$(pwd)/llvm-build/lib" >> $GITHUB_ENV
echo "CATALYST_BIN_DIR=$(pwd)/quantum-build/bin" >> $GITHUB_ENV
chmod +x $(pwd)/quantum-build/bin/catalyst-cli # artifact upload does not preserve permissions
chmod +x $(pwd)/quantum-build/bin/catalyst # artifact upload does not preserve permissions
- name: Run Python Pytest Tests
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/check-pl-compat.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ jobs:
echo "RUNTIME_LIB_DIR=$(pwd)/runtime-build/lib" >> $GITHUB_ENV
echo "MLIR_LIB_DIR=$(pwd)/llvm-build/lib" >> $GITHUB_ENV
echo "CATALYST_BIN_DIR=$(pwd)/quantum-build/bin" >> $GITHUB_ENV
chmod +x $(pwd)/quantum-build/bin/catalyst-cli # artifact upload does not preserve permissions
chmod +x $(pwd)/quantum-build/bin/catalyst # artifact upload does not preserve permissions
- name: Run Frontend Tests
run: |
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,7 @@ wheel:
cp $(COPY_FLAGS) $(DIALECTS_BUILD_DIR)/python_packages/quantum/mlir_quantum/dialects/*$${file}* $(MK_DIR)/frontend/mlir_quantum/dialects ; \
done
mkdir -p $(MK_DIR)/frontend/catalyst/bin
cp $(COPY_FLAGS) $(DIALECTS_BUILD_DIR)/bin/catalyst-cli $(MK_DIR)/frontend/catalyst/bin
cp $(COPY_FLAGS) $(DIALECTS_BUILD_DIR)/bin/catalyst $(MK_DIR)/frontend/catalyst/bin
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be possible with CMake's installation support, rather than explicit copies?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably, happy to accept a PR on this :D

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll take a look once the release goes out --- should be a quick change.

find $(MK_DIR)/frontend -type d -name __pycache__ -exec rm -rf {} +

$(PYTHON) -m pip wheel --no-deps . -w dist
Expand Down
42 changes: 21 additions & 21 deletions doc/catalyst-cli/catalyst-cli.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Catalyst Command Line Interface
===============================

Catalyst includes a standalone command-line-interface compiler tool ``catalyst-cli`` that
Catalyst includes a standalone command-line tool, ``catalyst``, that
compiles quantum programs written in our MLIR dialects into an object file,
independent of the Catalyst Python frontend.

Expand All @@ -13,47 +13,47 @@ This compiler tool combines three stages of compilation:
`LLVM IR <https://llvm.org/docs/LangRef.html>`_.
#. ``llc``: Performs lower-level optimizations on the LLVM IR input and creates the object file.

``catalyst-cli`` runs all three stages under the hood by default, but it also has the ability to run
each stage individually. For example:
The ``catalyst`` tool will run all three stages under the hood by default, but it also has the
ability to run each stage individually. For example:

.. code-block:: console

# Creates both the optimized IR and an object file
catalyst-cli input.mlir -o output.o
catalyst input.mlir -o output.o

# Only performs MLIR optimizations and translates to LLVM dialect
catalyst-cli --tool=opt input.mlir -o llvm-dialect.mlir
catalyst --tool=opt input.mlir -o llvm-dialect.mlir

# Only lowers LLVM dialect input to LLVM IR
catalyst-cli --tool=translate llvm-dialect.mlir -o llvm-ir.ll
catalyst --tool=translate llvm-dialect.mlir -o llvm-ir.ll

# Only performs lower-level optimizations and creates object file (object.o)
catalyst-cli --tool=llc llvm-ir.ll -o output.ll --module-name object
catalyst --tool=llc llvm-ir.ll -o output.ll --module-name object

.. note::

If Catalyst is built from source, the ``catalyst-cli`` executable will be located in
If Catalyst is built from source, the ``catalyst`` executable will be located in
the ``mlir/build/bin/`` directory relative to the root of your Catalyst source directory.

If Catalyst is installed via pip or from wheels, the executable will be located
in the ``catalyst/bin/`` directory relative to the environments installation directory.
If Catalyst is installed via pip or from wheels, the executable will be located
in the ``catalyst/bin/`` directory relative to the environment's installation directory.

Usage
-----

.. code-block:: console

catalyst-cli [options] <input file>
catalyst [options] <input file>

Calling ``catalyst-cli`` without any options runs the three compilation stages (``quantum-opt``,
Calling ``catalyst`` without any options runs the three compilation stages (``quantum-opt``,
``mlir-translate`` and ``llc``) using all default configurations, and outputs by default an object
file named ``catalyst_module.o``. The name of the output file can be set by changing the output
file named ``catalyst_module.o``. The name of the output file can be set by changing the output
module name using the ``--module-name`` option (the default module name is ``catalyst_module``).

Command line options
^^^^^^^^^^^^^^^^^^^^

The complete list of options for the Catalyst CLI tool can be displayed by running ``catalyst-cli --help``.
The complete list of options for the Catalyst CLI tool can be displayed by running ``catalyst --help``.
As this list contains *all* available options, including those for configuring LLVM, the options
most relevant to the usage of the Catalyst CLI tool are covered in more detail below.

Expand Down Expand Up @@ -111,7 +111,7 @@ name. Currently, the following pipelines are available:
``hlo_lowering-pipeline``,
``quantum-compilation-pipeline``,
``bufferization-pipeline``,
``llvm-dialect-lowring-pipeline``, and finally
``llvm-dialect-lowering-pipeline``, and finally
``default-catalyst-pipeline`` which encompasses all the above as the default pipeline used by the
Catalyst CLI tool if no pass option is specified.

Expand All @@ -138,7 +138,7 @@ applies the pass ``inline-nested-module``, we would specify this pipeline config
The workspace directory where intermediate files are saved. The default is the current working
directory.

Note that the workspace directory must exist before running ``catalyst-cli`` with this option.
Note that the workspace directory must exist before running ``catalyst`` with this option.

``--module-name=<name>``
""""""""""""""""""""""""
Expand Down Expand Up @@ -210,7 +210,7 @@ pass that is applied, and the ``-o`` option to set the name of the output IR fil

.. code-block::

catalyst-cli my_circuit.mlir \
catalyst my_circuit.mlir \
--tool=opt \
--catalyst-pipeline="pipe(remove-chained-self-inverse;merge-rotations)" \
--mlir-print-ir-after-all \
Expand Down Expand Up @@ -259,23 +259,23 @@ optimized MLIR.

For a list of transformation passes currently available in Catalyst, see the
:ref:`catalyst-s-transformation-library` documentation. The available passes are also listed in the
``catalyst-cli --help`` message.
``catalyst --help`` message.

MLIR Plugins
------------

``mlir-opt``-like tools are able to take plugins as inputs.
These plugins are shared objects that include dialects and passes written by third parties.
This means that you can write dialects and passes that can be used with ``catalyst-cli`` and ``quantum-opt``.
This means that you can write dialects and passes that can be used with ``catalyst`` and ``quantum-opt``.

As an example, the `LLVM repository includes a very simple plugin <https://github.com/llvm/llvm-project/tree/main/mlir/examples/standalone/standalone-plugin>`_.
To build it, simply run ``make plugin`` and the standalone plugin
will be built in the root directory of the Catalyst project.

With this, you can now run your own passes by using the following flags:

``catalyst-cli --load-dialect-plugin=$YOUR_PLUGIN --load-pass-plugin=$YOUR_PLUGIN $YOUR_PASS_NAME file.mlir``
``catalyst --load-dialect-plugin=$YOUR_PLUGIN --load-pass-plugin=$YOUR_PLUGIN $YOUR_PASS_NAME file.mlir``

Concretely for the example plugin, you can use the following command:

``catalyst-cli --tool=opt --load-pass-plugin=standalone/build/lib/StandalonePlugin.so --load-dialect-plugin=standalone/build/lib/StandalonePlugin.so --pass-pipeline='builtin.module(standalone-switch-bar-foo)' a.mlir``
``catalyst --tool=opt --load-pass-plugin=standalone/build/lib/StandalonePlugin.so --load-dialect-plugin=standalone/build/lib/StandalonePlugin.so --pass-pipeline='builtin.module(standalone-switch-bar-foo)' a.mlir``
8 changes: 4 additions & 4 deletions doc/dev/plugins.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
MLIR Plugins
============

This page outlines documentation on how to start developping an MLIR plugin that can work with Catalyst.
This page outlines documentation on how to start developing an MLIR plugin that can work with Catalyst.
An MLIR plugin is a shared object that implements a compilation pass compatible with the MLIR framework.
Catalyst is built on top of MLIR, this means that MLIR plugins work with Catalyst.
This can enable anyone to build quantum compilation passes and new dialects as well.
Expand All @@ -20,7 +20,7 @@ It is intended to show how one would build an MLIR plugin, rather than showing a

You can use the ``StandalonePlugin.so`` plugin

* with either ``quantum-opt`` or ``catalyst-cli``,
* with either ``quantum-opt`` or ``catalyst``,
* load it from Python and transform a quantum program.

For example, if you are interested in using it from the command line interface, you can use the following flags to load the standalone plugin:
Expand Down Expand Up @@ -257,7 +257,7 @@ One also needs to link the MLIRQuantum library and change the plugin tool to cat
LINK_LIBS
MLIRStandalone
Please note that if you are using the Catalyst or Gradient dialects, you should also add MLIRCatalyst and MLIRGradient to the list of dependences and libraries to be linked.
Please note that if you are using the Catalyst or Gradient dialects, you should also add MLIRCatalyst and MLIRGradient to the list of dependencies and libraries to be linked.

4. Modify the standalone plugin to modify quantum operations.

Expand Down Expand Up @@ -325,7 +325,7 @@ The next step is changing the contents of the function itself:
And then we can run ``make all`` again.
The shared object of the standalone plugin should be available in ``mlir/build/lib/StandalonePlugin.so``.
This shared object can be used with ``catalyst-cli`` and ``quantum-opt``.
This shared object can be used with both the ``catalyst`` and ``quantum-opt`` tools.
From here, you can change the name of the pass, change the name of the shared object, and implement more complex transformations.


Expand Down
10 changes: 4 additions & 6 deletions frontend/catalyst/compiler.py
Original file line number Diff line number Diff line change
Expand Up @@ -267,17 +267,17 @@ def __init__(self, options: Optional[CompileOptions] = None):

@debug_logger
def get_cli_command(self, tmp_infile_name, output_ir_name, module_name, workspace):
"""Prepare the command for catalyst-cli to compile the file.
"""Prepare the command to run the Catalyst CLI to compile the file.
Args:
module_name (str): Module name to use for naming
workspace (Directory): directory that holds output files and/or debug dumps.
Returns:
cmd (str): The command to be executed.
"""
cli_build_path = get_bin_path("cli", "CATALYST_BIN_DIR") + "/catalyst-cli"
cli_build_path = get_bin_path("cli", "CATALYST_BIN_DIR") + "/catalyst"
if not path.isfile(cli_build_path):
raise FileNotFoundError("catalyst-cli executable was not found.") # pragma: nocover
raise FileNotFoundError("catalyst executable was not found.") # pragma: nocover
cmd = [cli_build_path]
cmd += [tmp_infile_name, "-o", output_ir_name]
cmd += ["--module-name", module_name, "--workspace", str(workspace)]
Expand Down Expand Up @@ -356,9 +356,7 @@ def run_from_ir(self, ir: str, module_name: str, workspace: Directory):
if result.stderr:
print(result.stderr.strip(), file=self.options.logfile)
except subprocess.CalledProcessError as e: # pragma: nocover
raise CompileError(
f"catalyst-cli failed with error code {e.returncode}: {e.stderr}"
) from e
raise CompileError(f"catalyst failed with error code {e.returncode}: {e.stderr}") from e

with open(output_ir_name, "r", encoding="utf-8") as f:
out_IR = f.read()
Expand Down
4 changes: 2 additions & 2 deletions frontend/catalyst/pipelines.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@
.. note::
for DEFAULT_PIPELINES pipeline and the pipelines in DEFAULT_ASYNC_PIPELINES,
For DEFAULT_PIPELINES pipeline and the pipelines in DEFAULT_ASYNC_PIPELINES,
any change should be reflected in the mlir/lib/Driver/Pipelines.cpp files as well.
This is to ensure that the command line tool catalyst-cli default pipelines are
This is to ensure that Catalyst's command line tool default pipelines are
in sync with the pipelines defined in the Python frontend.
"""
Expand Down
2 changes: 1 addition & 1 deletion frontend/test/lit/test_instrumentation_console.py
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ def circuit(weights):
# CHECK-NEXT: [DIAGNOSTICS] Running device_release
# CHECK-SAME: walltime: {{[0-9\.]+}} ms{{\s*}} cputime: {{[0-9\.]+}} ms
# CHECK: [DIAGNOSTICS] > Total run
# COM: As the output below is generated by catalyst-cli, checking the correct order may cause flaky results.
# COM: As the output below is generated by the Catalyst CLI, checking the correct order may cause flaky results.
# CHECK: [DIAGNOSTICS] Running parseMLIRSource
# CHECK-SAME: walltime: {{[0-9\.]+}} ms{{\s*}} cputime: {{[0-9\.]+}} ms{{\s*}} programsize: {{[0-9]+}} lines
# CHECK: [DIAGNOSTICS] Running {{[a-zA-Z]+}}Pass
Expand Down
7 changes: 3 additions & 4 deletions mlir/README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,8 @@ directory structure generally follows the recommended practices from the upstrea

- `tools <https://github.com/PennyLaneAI/catalyst/tree/main/mlir/tools>`_:
A collection of stand-alone executables built by the project. Currently includes the
``catalyst-cli`` compiler tool for orchestrating different stages of compilation,
the ``quantum-opt`` tool to run passes on ``.mlir`` files, and
``quantum-lsp-server``, a language
server for MLIR with support for the quantum and gradient dialect.
``catalyst`` CLI compiler tool for orchestrating different stages of compilation,
the ``quantum-opt`` tool to run passes on ``.mlir`` files, and ``quantum-lsp-server``,
a language server for MLIR with support for the quantum and gradient dialect.

.. mlir-end-inclusion-marker-do-not-remove
4 changes: 2 additions & 2 deletions mlir/lib/Driver/CompilerDriver.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -701,7 +701,7 @@ LogicalResult QuantumDriverMain(const CompilerOptions &options, CompilerOutput &
if (runOpt && (inType == InputType::MLIR)) {
TimingScope optTiming = timing.nest("Optimization");
// TODO: The enzymeRun flag will not travel correctly in the case where different
// stages of compilation are executed independently via the catalyst-cli executable.
// stages of compilation are executed independently via the Catalyst CLI.
// Ideally, It should be added to the IR via an attribute.
enzymeRun = containsGradients(*mlirModule);
if (failed(runLowering(options, &ctx, *mlirModule, output, optTiming))) {
Expand Down Expand Up @@ -865,7 +865,7 @@ int QuantumDriverMainFromCL(int argc, char **argv)
// ---------
// Any modifications made to the command-line interface should be documented in
// doc/catalyst-cli/catalyst-cli.rst
cl::OptionCategory CatalystCat("Catalyst-cli Options", "");
cl::OptionCategory CatalystCat("Catalyst CLI Options", "");
cl::opt<std::string> WorkspaceDir("workspace", cl::desc("Workspace directory"), cl::init("."),
cl::cat(CatalystCat));
cl::opt<std::string> ModuleName("module-name", cl::desc("Module name"),
Expand Down
2 changes: 1 addition & 1 deletion mlir/patches/test-plugin-with-catalyst.patch
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ diff -ruN standalone_unmodified/test/lit.cfg.py standalone/test/lit.cfg.py
"standalone-capi-test",
"standalone-opt",
"standalone-translate",
+ "catalyst-cli",
+ "catalyst",
]

llvm_config.add_tool_substitutions(tools, tool_dirs)
Expand Down
2 changes: 1 addition & 1 deletion mlir/test/cli/DumpAfterFailure.mlir
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.

// RUN: not catalyst-cli --tool=opt %s --catalyst-pipeline="pipeline(llvm-dialect-lowring-pipeline)" --mlir-print-ir-after-failure --verify-diagnostics 2>&1 | FileCheck %s
// RUN: not catalyst --tool=opt %s --catalyst-pipeline="pipeline(llvm-dialect-lowring-pipeline)" --mlir-print-ir-after-failure --verify-diagnostics 2>&1 | FileCheck %s

func.func @foo(%arg0: tensor<?xf64>) {
"catalyst.print"(%arg0) : (tensor<?xf64>) -> ()
Expand Down
10 changes: 5 additions & 5 deletions mlir/test/cli/DumpBeforeAfterPass.mlir
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,11 @@
// See the License for the specific language governing permissions and
// limitations under the License.

// RUN: catalyst-cli --tool=opt %s --catalyst-pipeline="pipe1(split-multiple-tapes;apply-transform-sequence),pipe2(inline-nested-module)" --mlir-print-ir-before-all --verify-diagnostics 2>&1 | FileCheck %s --check-prefix=CHECK-BEFORE
// RUN: catalyst-cli --tool=opt %s --catalyst-pipeline="pipe1(split-multiple-tapes;apply-transform-sequence),pipe2(inline-nested-module)" --mlir-print-ir-after-all --verify-diagnostics 2>&1 | FileCheck %s --check-prefix=CHECK-AFTER
// RUN: catalyst-cli --tool=opt %s --catalyst-pipeline="pipe1(split-multiple-tapes;apply-transform-sequence),pipe2(inline-nested-module)" --mlir-print-ir-before=inline-nested-module --verify-diagnostics 2>&1 | FileCheck %s --check-prefix=CHECK-BEFORE-ONE
// RUN: catalyst-cli --tool=opt %s --catalyst-pipeline="pipe1(split-multiple-tapes;apply-transform-sequence),pipe2(inline-nested-module)" --mlir-print-ir-after=inline-nested-module --verify-diagnostics 2>&1 | FileCheck %s --check-prefix=CHECK-AFTER-ONE
// RUN: catalyst-cli --tool=opt %s --catalyst-pipeline="pipe1(split-multiple-tapes;apply-transform-sequence),pipe2(inline-nested-module)" --mlir-print-op-generic --verify-diagnostics 2>&1 | FileCheck %s --check-prefix=CHECK-GENERIC
// RUN: catalyst --tool=opt %s --catalyst-pipeline="pipe1(split-multiple-tapes;apply-transform-sequence),pipe2(inline-nested-module)" --mlir-print-ir-before-all --verify-diagnostics 2>&1 | FileCheck %s --check-prefix=CHECK-BEFORE
// RUN: catalyst --tool=opt %s --catalyst-pipeline="pipe1(split-multiple-tapes;apply-transform-sequence),pipe2(inline-nested-module)" --mlir-print-ir-after-all --verify-diagnostics 2>&1 | FileCheck %s --check-prefix=CHECK-AFTER
// RUN: catalyst --tool=opt %s --catalyst-pipeline="pipe1(split-multiple-tapes;apply-transform-sequence),pipe2(inline-nested-module)" --mlir-print-ir-before=inline-nested-module --verify-diagnostics 2>&1 | FileCheck %s --check-prefix=CHECK-BEFORE-ONE
// RUN: catalyst --tool=opt %s --catalyst-pipeline="pipe1(split-multiple-tapes;apply-transform-sequence),pipe2(inline-nested-module)" --mlir-print-ir-after=inline-nested-module --verify-diagnostics 2>&1 | FileCheck %s --check-prefix=CHECK-AFTER-ONE
// RUN: catalyst --tool=opt %s --catalyst-pipeline="pipe1(split-multiple-tapes;apply-transform-sequence),pipe2(inline-nested-module)" --mlir-print-op-generic --verify-diagnostics 2>&1 | FileCheck %s --check-prefix=CHECK-GENERIC

func.func @foo() {
return
Expand Down
Loading
Loading