feat: add support for compiling programs to AVM bytecode, and referen…

…cing those programs within other contracts.

In particular:
* support getting compiled programs from contracts using `compile_contract` and address from logic signatures using `compile_logicsig`
* added `algopy.arc4.arc4_create`, and `algopy.arc4.arc4_update` for creating and updating ARC4 applications, automatically providing the required parameters
* new CLI options for controlling AVM bytecode output
  `--output-bytecode` enables outputting AVM bytecode to a `.bin` file
  `--match-algod-bytecode` outputs bytecode that will match algod output, if this is not specified bytecode output will be smaller where possible, but functionally equivalent
* new CLI options for providing template values when outputting to bytecode
  `--template-var` and `--template-vars-path` allows specifying template values to use during bytecode generation
  `--template-vars-prefix` allows overriding the default prefix (TMPL_) used for template variable expansion

.coveragerc

@@ -9,3 +9,4 @@ exclude_also =
     if .*TYPE_CHECKING:
     @(abc\.)?abstractmethod
     raise InternalError
+    typing.assert_never

.github/workflows/check-python.yaml

@@ -107,7 +107,7 @@ jobs:
 
       - name: Compile all
         shell: bash
-        run: poetry run python scripts/compile_all_examples.py --no-update-sizes
+        run: poetry run python scripts/compile_all_examples.py
         env:
           PYTHONUTF8: 1
 

.gitignore

@@ -11,6 +11,9 @@ examples/**/*.trace
 # ignore AWST from generated clients
 /examples/**/out/out/**/client_*.awst
 /test_cases/**/out/out/**/client_*.awst
+# ignore bytecode output
+/examples/**/out*/*.bin
+/test_cases/**/out*/*.bin
 # wheel output
 /dist
 /stubs/dist

docs/architecture-decisions/2024-05-22_teal-compilation.md

@@ -0,0 +1,83 @@
+# Compilation of Algorand TEAL Programs into AVM Bytecode
+
+- **Status**: Proposed
+- **Owner**: Daniel McGregor (MakerX)
+- **Deciders**: Rob Moore (MakerX), Adam Chidlow (MakerX), Alessandro Cappellato (Algorand Foundation)
+- **Date created**: 2024-05-22
+- **Date decided**: 
+- **Date updated**: 
+
+## Context
+
+The Puya compiler needs to assemble valid Algorand TEAL (Transaction Execution Approval Language) programs into AVM (Algorand Virtual Machine) bytecode 
+so the bytecode can be used within inner transactions that need to create applications. Additionally, the ability to produce compiled bytecode without
+an algod node is also desriable for other use cases, e.g. producing an [ARC-56 app description](https://github.com/algorandfoundation/ARCs/pull/258).
+
+There are several potential solutions to achieve this compilation, each with its own advantages and trade-offs.
+
+## Requirements
+
+* The solution works cross-platform (i.e. works on Windows, Mac and Linux)
+* The solution is fast
+* The solution is correct
+
+## Options
+
+### Option 1. Use Algod Compile Endpoint
+
+Utilize the Algorand algod `/v2/teal/compile` endpoint to compile TEAL programs into AVM bytecode.
+
+**Pros**
+
+* Ensures compatibility with the latest Algorand protocol updates.
+* Existing solution
+
+**Cons**
+
+* Requires configuration and connectivity to an Algorand algod node
+* A compiler requiring a network call to produce output is unusual, and may be problematic in environments adhering to the "principle of least privilege"
+* Potential latency due to network requests.
+
+### Option 2. Use Goal CLI to Compile
+
+Utilize the Algorand goal command-line interface to compile TEAL programs locally.
+
+**Pros**
+
+* Local compilation without the need for network connectivity.
+* Directly supported by Algorand, ensuring compatibility.
+
+**Cons**
+
+* Requires installation of the goal CLI, and configuration to locate it
+* No native windows binary of goal is available
+* Increased complexities due co-ordinating inputs and outputs between the compiler and goal
+
+### Option 3: Integrate TEAL to AVM Bytecode Assembly into Puya Compiler
+
+Extend the existing Puya compiler to directly convert TEAL programs into AVM bytecode.
+
+**Pros**
+
+* Full Control Over Compilation Process: By integrating the assembly within the Puya compiler, we maintain full control and can tailor the process to fit specific use cases and workflows.
+* No External Dependencies: This approach eliminates reliance on and configuration of external tools or services, ensuring the compilation process is self-contained and not affected by network issues or external updates.
+* Additional optimizations: We can introduce additional optimizations, if desired these could be upstreamed to the algod implementation
+* Simpler Implementation: Given that Puya already handles well-structured TEAL (as one of the IR layers in the compiler), extending it to assemble bytecode is relatively straightforward and does not require complex parsing. The conversion from a logical TEAL op to the underlying bytecode is based on information in the AVM langspec providing a high degree of confidence in the output.
+
+**Cons**
+
+* Potential for Bugs and Inconsistencies: There is a risk of introducing bugs or deviations from Algorand's official compiler, which could lead to compatibility issues. This risk can be offset by including tests that verify the compiler output matches the algod output for a suite of test programs. 
+* Ongoing Maintenance: We will need to continuously update our compiler to ensure it remains aligned with any changes or updates to the Algorand protocol. This is mitigated to a large degree by utilizing the AVM langspec as the source of truth for op code values.
+
+## Preferred option
+
+Option 3: Implement Assembly of TEAL to Bytecode in the Puya Compiler
+
+The preferred solution is to extend the existing Puya compiler to directly convert TEAL programs into AVM bytecode. 
+This approach offers several key advantages, including full control over the compilation process and independence from external tools or services. 
+
+To provide consistency with algod, we will introduce a flag that enables the generation of bytecode matching the results produced by algod. 
+Additionally, to mitigate any potential risks, we will implement comprehensive tests to verify that our output consistently 
+aligns with algod compilation results.
+
+## Selected option

docs/compiler.md

@@ -107,16 +107,62 @@ puyapy [-h] [--version] [-O {0,1,2}]
 ### Options
 
 | Option                                                   | Description                                                                                                                                                           | Default                 |
-| -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- |
+|----------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------| ----------------------- |
 | `-h`, `--help`                                           | Show the help message and exit                                                                                                                                        | N/A                     |
 | `--version`                                              | Show program's version number and exit                                                                                                                                | N/A                     |
 | `-O {0,1,2}` <br />`--optimization-level {0,1,2}`        | Set optimization level of output TEAL / AVM bytecode                                                                                                                  | `1`                     |
 | `--output-teal`, `--no-output-teal`                      | Output TEAL                                                                                                                                                           | `True`                  |
 | `--output-arc32`, `--no-output-arc32`                    | Output {contract}.arc32.json ARC-32 app spec file if the contract is an ARC-4 contract                                                                                | `True`                  |
 | `--output-client`, `--no-output-client`                  | Output Algorand Python contract client for typed ARC4 ABI calls                                                                                                       | `False`                 |
+| `--output-bytecode`, `--no-output-bytecode`              | Output AVM bytecode                                                                                                                                                   | `False`                 |
+| `--match-algod-bytecode` | When outputting bytecode  (via `--output-bytecode` or compiled programs), ensure bytecode matches algod output, by disabling additional optimizations | False   |
 | `--out-dir OUT_DIR`                                      | The path for outputting artefacts                                                                                                                                     | Same folder as contract |
 | `--log-level {notset,debug,info,warning,error,critical}` | Minimum level to log to console                                                                                                                                       | `info`                  |
 | `-g {0,1,2}`, `--debug-level {0,1,2}`                    | Output debug information level<br /> `0` = No debug annotations <br /> `1` = Output debug annotations <br /> `2` = Reserved for future use, currently the same as `1` | `1`                     |
+| `--template-var`         | Allows specifying template values. Can be used multiple times, see below for examples | N/A     |
+| `--template-vars-path`   | Path to a file containing template values                                             | N/A     |
+| `--template-vars-prefix` | Prefix to use for template variables                                                  | "TMPL_" |
+
+### Defining template values
+
+[Template Variables](#algopy.TemplateVar), can be replaced with literal values during compilation to bytecode using the `--template-var` option.
+Additionally, Algorand Python functions that provide AVM bytecode, such as [get_approval_program](#algopy.get_approval_program), 
+[get_clear_state_program](#algopy.get_clear_state_program), [get_logicsig_account](#algopy.get_logicsig_account) or [abi_call](#algopy.arc4.abi_call), will also utilize the specified values.
+
+#### Examples of Variable Definitions
+
+The table below illustrates how different variables and values can be defined:
+
+| Variable Type                     | Example Algorand Python                          | Value definition example |
+|--------------------------|-------------------------------------------|--------------------------|
+| [UInt64](#algopy.UInt64) | `algopy.TemplateVar[UInt64]("SOME_INT")`  | `SOME_INT=1234`          |
+| [Bytes](#algopy.Bytes)   | `algopy.TemplateVar[Bytes]("SOME_BYTES")` | `SOME_BYTES=0x1A2B`      |
+| [String](#algopy.String) | `algopy.TemplateVar[String]("SOME_STR")`  | `SOME_STR="hello"`       |
+
+
+
+
+These values can also be defined a file and referenced using the `--template-var-path` option.
+All template values specified via the command line or a template file are prefixed with "TMPL_". 
+This prefix can be modified using the `--template-vars-prefix` option or by defining the prefix within the template file 
+itself with `prefix=`.
+
+#### Example of a Template File
+
+Below is a complete example of a template file that can be used with the --template-vars-prefix option. Comments can be added using the `#` symbol.
+```
+# Template file example
+
+# Override default prefix
+prefix="TMPL_"
+
+# Define template variables, do not need to include the prefix
+SOME_INT=1234
+SOME_BYTES=0x1A2B
+
+# This is a comment explaining the next variable
+SOME_STR="hello"
+```
 
 ### Advanced options
 

docs/conf.py

@@ -23,7 +23,7 @@
 ]
 
 templates_path = ["_templates"]
-exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store", "architecture-decisions/*.md"]
 
 intersphinx_mapping = {
     "python": ("https://docs.python.org/3", None),