Support standalone expr.Var in QPY (Qiskit#11651)

* Support standalone `expr.Var` in QPY

This necessitates adding some extra definitions to the circuit header.
When standalone variables are used, we do not re-encode the entire
variable (which would take a miniumum of ~20 bytes per variable), but
instead just emit an index into the order that the variables were
defined at the top of the circuit.

At present, each control-flow scope will store the `Var` nodes anew, so
it's not a perfect only-store-once system.  This is consistent with how
registers are handled, though, and the QPY format isn't particularly
memory optimised as things stand.

* Add backwards compatibility tests

qiskit/qpy/__init__.py

@@ -79,6 +79,12 @@
 
 .. autoexception:: QpyError
 
+When a lower-than-maximum target QPY version is set for serialization, but the object to be
+serialized contains features that cannot be represented in that format, a subclass of
+:exc:`QpyError` is raised:
+
+.. autoexception:: UnsupportedFeatureForVersion
+
 Attributes:
     QPY_VERSION (int): The current QPY format version as of this release. This
         is the default value of the ``version`` keyword argument on
@@ -285,12 +291,97 @@
 The file header is immediately followed by the circuit payloads.
 Each individual circuit is composed of the following parts:
 
-``HEADER | METADATA | REGISTERS | CUSTOM_DEFINITIONS | INSTRUCTIONS``
+``HEADER | METADATA | REGISTERS | STANDALONE_VARS | CUSTOM_DEFINITIONS | INSTRUCTIONS``
+
+The ``STANDALONE_VARS`` are new in QPY version 12; before that, there was no data between
+``REGISTERS`` and ``CUSTOM_DEFINITIONS``.
 
 There is a circuit payload for each circuit (where the total number is dictated
 by ``num_circuits`` in the file header). There is no padding between the
 circuits in the data.
 
+.. _qpy_version_12:
+
+Version 12
+==========
+
+Version 12 adds support for:
+
+* circuits containing memory-owning :class:`.expr.Var` variables.
+
+Changes to HEADER
+-----------------
+
+The HEADER struct for an individual circuit has added three ``uint32_t`` counts of the input,
+captured and locally declared variables in the circuit.  The new form looks like:
+
+.. code-block:: c
+
+    struct {
+        uint16_t name_size;
+        char global_phase_type;
+        uint16_t global_phase_size;
+        uint32_t num_qubits;
+        uint32_t num_clbits;
+        uint64_t metadata_size;
+        uint32_t num_registers;
+        uint64_t num_instructions;
+        uint32_t num_vars;
+    } HEADER_V12;
+
+The ``HEADER_V12`` struct is followed immediately by the same name, global-phase, metadata
+and register information as the V2 version of the header.  Immediately following the registers is
+``num_vars`` instances of ``EXPR_VAR_STANDALONE`` that define the variables in this circuit.  After
+that, the data continues with custom definitions and instructions as in prior versions of QPY.
+
+
+EXPR_VAR_DECLARATION
+--------------------
+
+An ``EXPR_VAR_DECLARATION`` defines an :class:`.expr.Var` instance that is standalone; that is, it
+represents a self-owned memory location rather than wrapping a :class:`.Clbit` or
+:class:`.ClassicalRegister`.  The payload is a C struct:
+
+.. code-block:: c
+
+    struct {
+        char uuid_bytes[16];
+        char usage;
+        uint16_t name_size;
+    }
+
+which is immediately followed by an ``EXPR_TYPE`` payload and then ``name_size`` bytes of UTF-8
+encoding string data containing the name of the variable.
+
+The ``char`` usage type code takes the following values:
+
+=========  =========================================================================================
+Type code  Meaning
+=========  =========================================================================================
+``I``      An ``input`` variable to the circuit.
+
+``C``      A ``capture`` variable to the circuit.
+
+``L``      A locally declared variable to the circuit.
+=========  =========================================================================================
+
+
+Changes to EXPR_VAR
+-------------------
+
+The EXPR_VAR variable has gained a new type code and payload, in addition to the pre-existing ones:
+
+===========================  =========  ============================================================
+Python class                 Type code  Payload
+===========================  =========  ============================================================
+:class:`.UUID`               ``U``      One ``uint32_t`` index of the variable into the series of
+                                        ``EXPR_VAR_STANDALONE`` variables that were written
+                                        immediately after the circuit header.
+===========================  =========  ============================================================
+
+Notably, this new type-code indexes into pre-defined variables from the circuit header, rather than
+redefining the variable again in each location it is used.
+
 .. _qpy_version_11:
 
 Version 11
@@ -337,17 +428,18 @@
 Version 10
 ==========
 
-Version 10 adds support for symengine-native serialization for objects of type
-:class:`~.ParameterExpression` as well as symbolic expressions in Pulse schedule blocks. Version
-10 also adds support for new fields in the :class:`~.TranspileLayout` class added in the Qiskit
-0.45.0 release.
+Version 10 adds support for:
+
+* symengine-native serialization for objects of type :class:`~.ParameterExpression` as well as
+  symbolic expressions in Pulse schedule blocks.
+* new fields in the :class:`~.TranspileLayout` class added in the Qiskit 0.45.0 release.
 
 The symbolic_encoding field is added to the file header, and a new encoding type char
 is introduced, mapped to each symbolic library as follows: ``p`` refers to sympy
 encoding and ``e`` refers to symengine encoding.
 
-FILE_HEADER
------------
+Changes to FILE_HEADER
+----------------------
 
 The contents of FILE_HEADER after V10 are defined as a C struct as:
 
@@ -360,10 +452,10 @@
         uint8_t qiskit_patch_version;
         uint64_t num_circuits;
         char symbolic_encoding;
-    }
+    } FILE_HEADER_V10;
 
-LAYOUT
-------
+Changes to LAYOUT
+-----------------
 
 The ``LAYOUT`` struct is updated to have an additional ``input_qubit_count`` field.
 With version 10 the ``LAYOUT`` struct is now:
@@ -1522,7 +1614,7 @@ class if it's defined in Qiskit. Otherwise it falls back to the custom
 .. [#f3] https://docs.python.org/3/c-api/complex.html#c.Py_complex
 """
 
-from .exceptions import QpyError, QPYLoadingDeprecatedFeatureWarning
+from .exceptions import QpyError, UnsupportedFeatureForVersion, QPYLoadingDeprecatedFeatureWarning
 from .interface import dump, load
 
 # For backward compatibility. Provide, Runtime, Experiment call these private functions.