Various checkpointing fixes + QoL updates

Fixed and more robust ckpt script
Support changing PMPs in SpikeTile configs
New ckpt specific configs
Dedup + fix SpikeTile configs
Bump FireMarshal for HTIF-only serial console Linux version
Many docs + comments for clarification

docs/Advanced-Concepts/Architectural-Checkpoints.rst

@@ -9,16 +9,16 @@ The checkpoints include the contents of cacheable memory, core architectural reg
 RTL simulations of SoCs can resume execution from checkpoints after restoring the architectural state.
 
 .. note::
-   Currently, only checkpoints of single-core systems are supported
+   Currently, only checkpoints of single-core systems are supported.
 
 Generating Checkpoints
 ------------------------
 
-``scripts/generate-ckpt.sh`` is a script that runs spike with the right commands to generate an architectural checkpoint
+``scripts/generate-ckpt.sh`` is a script that runs Spike with the right commands to generate an architectural checkpoint.
 ``scripts/generate-ckpt.sh -h`` lists options for checkpoint generation.
 
 Example: run the ``hello.riscv`` binary for 1000 instructions before generating a checkpoint.
-This should produce a directory named ``hello.riscv.0x80000000.1000.loadarch``
+This should produce a directory named ``hello.riscv.*.loadarch``
 
 .. code::
 
@@ -29,10 +29,34 @@ Loading Checkpoints in RTL Simulation
 --------------------------------------
 
 Checkpoints can be loaded in RTL simulations with the ``LOADARCH`` flag.
-The target config **MUST** use dmi-based bringup (as opposed to the default TSI-based bringup), and support fast ``LOADMEM``.
-The target config should also match the architectural configuration of however spike was configured when generating the checkpoint.
+The target config needs the following properties:
+
+- **MUST** use DMI-based bringup (as opposed to the default TSI-based bringup)
+- **MUST** support fast ``LOADMEM``
+- Should match the architectural configuration of however Spike was configured when generating the checkpoint (i.e. same ISA, no PMPs, etc).
 
 .. code::
 
    cd sims/vcs
-   make CONFIG=dmiRocketConfig run-binary LOADARCH=../../hello.riscv.0x80000000.1000.loadarch
+   make CONFIG=dmiRocketConfig run-binary LOADARCH=../../hello.riscv.*.loadarch
+
+Checkpointing Linux Binaries
+----------------------------
+
+Checkpoints can be used to run Linux binaries with the following caveats:
+
+- The binary must only use the HTIF console and should be non-interactive (i.e no stdin available)
+- The target config must be built without a serial device (i.e. the Rocket Chip Blocks UART can't be used)
+- The binary must only use an initramfs (i.e. no block device)
+- The target config must be built without a block device (i.e. the IceBlk block device can't be used).
+- The binary size must be smaller than the size of the target configs memory region (for example if FireMarshal's ``rootfs-size`` is 1GB, and OpenSBI is 350KB then you must have at least 1G + 1KB of space)
+
+This means that you most likely need to do the following:
+
+- By default Spike has a default UART device that is used during most Linux boot's.
+  This can be bypassed by creating a DTS without a serial device then passing it to the ``generate-ckpt.sh`` script.
+  You can copy the DTS of the design you want to checkpoint into - located in Chipyards ``sims/<simulator>/generated-src/`` - and modify it to pass to the checkpointing script (needs to be stripped down of extra devices and nodes).
+  An example of a config made for checkpointing is ``dmiCheckpointingRocketConfig`` or ``dmiCheckpointingSpikeUltraFastConfig``.
+- Additionally, you need to change your Linux config in FireMarshal to default to only use HTIF during OpenSBI and force Linux to use the OpenSBI HTIF console.
+  This can be done by the following in the ``linux-config``: changing to ``CONFIG_CMDLINE="console=hvc0 earlycon=sbi"``, adding ``CONFIG_RISCV_SBI_V01=y``, adding ``CONFIG_HVC_RISCV_SBI=y``, and adding ``CONFIG_SERIAL_EARLYCON_RISCV_SBI=y``.
+  An example workload with these changes can be found at ``example-workloads/br-base-htif-only-serial.yaml``.

generators/chipyard/src/main/scala/SpikeTile.scala

@@ -16,7 +16,9 @@ import freechips.rocketchip.util._
 import freechips.rocketchip.tile._
 import freechips.rocketchip.prci._
 
-case class SpikeCoreParams() extends CoreParams {
+case class SpikeCoreParams(
+  nPMPs: Int = 16
+) extends CoreParams {
   val useVM = true
   val useHypervisor = false
   val useSupervisor = true
@@ -33,7 +35,6 @@ case class SpikeCoreParams() extends CoreParams {
   val nLocalInterrupts = 0
   val useNMI = false
   val nPTECacheEntries = 0
-  val nPMPs = 16
   val pmpGranularity = 4
   val nBreakpoints = 0
   val useBPWatch = false
@@ -115,6 +116,7 @@ class SpikeTile(
   val masterNode = visibilityNode
   val slaveNode = TLIdentityNode()
 
+  // Note: Rocket doesn't support zicntr but Spike does (err on the side of having Rocket's ISA)
   override def isaDTS = "rv64imafdcv_zicsr_zifencei_zihpm_zvl128b_zve64d"
 
   // Required entry of CPU device in the device tree for interrupt purpose

generators/chipyard/src/main/scala/clocking/ClockGroupCombiner.scala

@@ -51,9 +51,9 @@ class ClockGroupCombiner(implicit p: Parameters, v: ValName) extends LazyModule
       val names = g.map(_.name.getOrElse("unamed"))
       val takes = g.map(_.take).flatten
       require(takes.distinct.size <= 1,
-        s"Clock group $name has non-homogeneous requested ClockParameters ${names.zip(takes)}")
+        s"Clock group '$name' has non-homogeneous requested ClockParameters ${names.zip(takes)}")
       require(takes.size > 0,
-        s"Clock group $name has no inheritable frequencies")
+        s"Clock group '$name' has no inheritable frequencies")
       (grouped ++ Seq(ClockSinkParameters(take = takes.headOption, name = Some(name))), r)
     }
 

generators/chipyard/src/main/scala/config/PeripheralDeviceConfigs.scala

@@ -67,6 +67,10 @@ class dmiRocketConfig extends Config(
   new chipyard.config.AbstractConfig)
 // DOC include end: DmiRocket
 
+class dmiCheckpointingRocketConfig extends Config(
+  new chipyard.config.WithNPMPs(0) ++                            // remove PMPs (reduce non-core arch state)
+  new dmiRocketConfig)
+
 class ManyPeripheralsRocketConfig extends Config(
   new testchipip.iceblk.WithBlockDevice ++                   // add block-device module to peripherybus
   new testchipip.soc.WithOffchipBusClient(MBUS) ++           // OBUS provides backing memory to the MBUS

generators/chipyard/src/main/scala/config/SpikeConfigs.scala

@@ -18,40 +18,36 @@ class dmiSpikeConfig extends Config(
 // Avoids polling on the UART registers
 class SpikeFastUARTConfig extends Config(
   new chipyard.WithNSpikeCores(1) ++
-  new chipyard.config.WithUART(txEntries=128, rxEntries=128) ++ // Spike sim requires a larger UART FIFO buffer, 
-  new chipyard.config.WithNoUART() ++                           // so we overwrite the default one
+  new chipyard.config.WithUART(txEntries=128, rxEntries=128) ++   // Spike sim requires a larger UART FIFO buffer,
+  new chipyard.config.WithNoUART() ++                             // so we overwrite the default one
+  new chipyard.config.WithPeripheryBusFrequency(2) ++             // configured to be as fast as possible
   new chipyard.config.WithMemoryBusFrequency(2) ++
-  new chipyard.config.WithPeripheryBusFrequency(2) ++
+  new chipyard.config.WithControlBusFrequency(2) ++
+  new chipyard.config.WithSystemBusFrequency(2) ++
+  new chipyard.config.WithFrontBusFrequency(2) ++
+  new chipyard.config.WithOffchipBusFrequency(2) ++
   new chipyard.config.AbstractConfig)
 
-// Makes the UART fast, also builds no L2 and a ludicrous L1D
+// No L2 and a ludicrous L1D
 class SpikeUltraFastConfig extends Config(
+  new testchipip.soc.WithNoScratchpads ++
   new chipyard.WithSpikeTCM ++
-  new chipyard.WithNSpikeCores(1) ++
-  new chipyard.config.WithUART(txEntries=128, rxEntries=128) ++ // Spike sim requires a larger UART FIFO buffer, 
-  new chipyard.config.WithNoUART() ++                           // so we overwrite the default one
-  new chipyard.config.WithMemoryBusFrequency(2) ++
-  new chipyard.config.WithPeripheryBusFrequency(2) ++
   new chipyard.config.WithBroadcastManager ++
-  new chipyard.config.AbstractConfig)
+  new SpikeFastUARTConfig)
 
 class dmiSpikeUltraFastConfig extends Config(
   new chipyard.harness.WithSerialTLTiedOff ++                    // don't attach anything to serial-tilelink
   new chipyard.config.WithDMIDTM ++                              // have debug module expose a clocked DMI port
   new SpikeUltraFastConfig)
 
+class dmiCheckpointingSpikeUltraFastConfig extends Config(
+  new chipyard.config.WithNPMPs(0) ++                            // remove PMPs (reduce non-core arch state)
+  new dmiSpikeUltraFastConfig)
+
 // Add the default firechip devices
 class SpikeUltraFastDevicesConfig extends Config(
   new chipyard.harness.WithSimBlockDevice ++
   new chipyard.harness.WithLoopbackNIC ++
   new icenet.WithIceNIC ++
   new testchipip.iceblk.WithBlockDevice ++
-
-  new chipyard.WithSpikeTCM ++
-  new chipyard.WithNSpikeCores(1) ++
-  new chipyard.config.WithUART(txEntries=128, rxEntries=128) ++ // Spike sim requires a larger UART FIFO buffer, 
-  new chipyard.config.WithNoUART() ++                           // so we overwrite the default one
-  new chipyard.config.WithMemoryBusFrequency(2) ++
-  new chipyard.config.WithPeripheryBusFrequency(2) ++
-  new chipyard.config.WithBroadcastManager ++
-  new chipyard.config.AbstractConfig)
+  new SpikeUltraFastConfig)

generators/chipyard/src/main/scala/config/fragments/TileFragments.scala

@@ -71,6 +71,8 @@ class WithNPMPs(n: Int = 8) extends Config((site, here, up) => {
       core = tp.tileParams.core.copy(nPMPs = n)))
     case tp: boom.v4.common.BoomTileAttachParams => tp.copy(tileParams = tp.tileParams.copy(
       core = tp.tileParams.core.copy(nPMPs = n)))
+    case tp: chipyard.SpikeTileAttachParams => tp.copy(tileParams = tp.tileParams.copy(
+      core = tp.tileParams.core.copy(nPMPs = n)))
     case other => other
   }
 })

scripts/generate-ckpt.sh

@@ -2,30 +2,53 @@
 
 set -e
 
-usage() {
-    echo "Usage: $0 [OPTIONS]"
-    echo ""
-    echo "Options"
-    echo "  --help -h  : Display this message"
-    echo "  -n <n>     : Number of harts"
-    echo "  -b <elf>   : Binary to run in spike"
-    echo "  -p <pc>    : PC to take checkpoint at [default 0x80000000]"
-    echo "  -i <insns> : Instructions after PC to take checkpoint at [default 0]"
-    echo "  -m <isa>   : ISA to pass to spike for checkpoint generation [default rv64gc]"
-    echo "  -o <out>   : Output directory to store the checkpoint in. [default <elf>.<pc>.<insns>.loadarch]"
-    echo "  -r <mem>   : Memory regions to pass to spike. Passed to spike's '-m' flag. [default starting at 0x80000000 with 256MiB]"
-    echo "  -v         : Verbose"
-    exit "$1"
-}
+# assumes that only memory region is at 0x80000000. this might break with more regions (and/or at different locations)
+DEFAULT_MEM_START_ADDR=0x80000000
 
 NHARTS=1
 BINARY=""
-PC="0x80000000"
+PC="$DEFAULT_MEM_START_ADDR"
+INSN=
 INSNS=0
 ISA="rv64gc"
 OUTPATH=""
 MEMOVERRIDE=""
 VERBOSE=0
+DTB=
+DTS=
+TYPE="defaultspikedts"
+
+usage() {
+    echo "Usage: $0 [OPTIONS]"
+    echo ""
+    echo "Misc Options"
+    echo "  --help -h  : Display this message"
+    echo "  -o <out>   : Output directory to store the checkpoint in. [default <elf>.<pc>.<insn>.<insns>.<dtstype>.loadarch]"
+    echo "  -v         : Verbose"
+    echo ""
+    echo "Required Options"
+    echo "  -b <elf>   : Binary to run in spike"
+    echo ""
+    echo "Other Required Options (with defaults if excluded)"
+    echo "  -r <mem>   : Memory regions to pass to spike. Passed to spike's '-m' flag. [default starting at $DEFAULT_MEM_START_ADDR with 256MiB]"
+    echo ""
+    echo "Options Group (one is required)"
+    echo "  -p <pc>    : PC to take checkpoint at [default $PC]"
+    echo "  -t <insn>  : Instruction (in hex) to take checkpoint at [default is to use ${INSN:-none}]"
+    echo "  -i <insns> : Instructions after PC to take checkpoint at [default $INSNS]"
+    echo ""
+    echo "Mutually Exclusive Option Groups (each group is mutually exclusive with one another)"
+    echo "  Group: Use Spike default DTS with modifications (can choose multiple)"
+    echo "    -n <n>     : Number of harts [default $NHARTS]"
+    echo "    -m <isa>   : ISA to pass to spike for checkpoint generation [default $ISA]"
+    echo "  Group: Use custom DT{B,S} (choose one)"
+    echo "    Important Note: a dt{b,s} only affects the devices used, pmps, mmu."
+    echo "                    the isa string and num. of harts from the dt{b,s} is inferred from this script to pass to the spike --isa and -p flags, respectively."
+    echo "    -d <dtb>   : DTB file to use. Passed to spike's '--dtb' flag. [default is to use ${DTB:-none}]"
+    echo "    -s <dts>   : DTS file to use. Converted to a DTB then passed to spike's '--dtb' flag. [default is to use ${DTS:-none}]"
+    exit "$1"
+}
+
 while [ "$1" != "" ];
 do
     case $1 in
@@ -43,6 +66,9 @@ do
 	-i )
 	    shift
 	    INSNS=$1 ;;
+	-t )
+	    shift
+	    INSN=$1 ;;
         -m )
             shift
             ISA=$1 ;;
@@ -54,6 +80,14 @@ do
             MEMOVERRIDE=$1 ;;
         -v )
             VERBOSE=1 ;;
+	-d )
+	    shift
+	    TYPE="customdts"
+	    DTB=$1 ;;
+	-s )
+	    shift
+	    TYPE="customdts"
+	    DTS=$1 ;;
 	* )
 	    error "Invalid option $1"
 	    usage 1 ;;
@@ -65,30 +99,58 @@ if [[ $VERBOSE -eq 1 ]] ; then
     set -x
 fi
 
-if [ -z "$MEMOVERRIDE" ] ; then
-    BASEMEM="$((0x80000000)):$((0x10000000))"
-else
-    BASEMEM=$MEMOVERRIDE
-fi
-SPIKEFLAGS="-p$NHARTS --pmpregions=0 --isa=$ISA -m$BASEMEM"
 BASENAME=$(basename -- $BINARY)
 
 if [ -z "$OUTPATH" ] ; then
-    OUTPATH=$BASENAME.$PC.$INSNS.loadarch
+    OUTPATH=$BASENAME.$PC.${INSN:-unused}.$INSNS.$TYPE.loadarch
 fi
 
 echo "Generating loadarch directory $OUTPATH"
 rm -rf $OUTPATH
 mkdir -p $OUTPATH
 
+SPIKEFLAGS=""
+if [ -z "$MEMOVERRIDE" ] ; then
+    BASEMEM="$(($DEFAULT_MEM_START_ADDR)):$((0x10000000))"
+else
+    BASEMEM=$MEMOVERRIDE
+fi
+SPIKEFLAGS+=" -m$BASEMEM"
+
+if [ ! -z "$DTS" ] ; then
+    dtc -I dts -O dtb -o $OUTPATH/tmp.dtb $(readlink -f $DTS)
+    DTB=$OUTPATH/tmp.dtb
+fi
+
+if [ ! -z "$DTB" ]; then
+    SPIKEFLAGS+=" --dtb=$DTB"
+    # HACK: set the global spike isa string with the dtb value (ensure isa's are the same across harts)
+    all_isas=$(dtc -I dtb -O dts $(readlink -f $DTB) | grep "riscv,isa")
+    NHARTS=$(echo "$all_isas" | wc -l)
+    unique_isas=$(echo "$all_isas" | sort -u)
+    if [[ $(echo "$unique_isas" | wc -l) == "1" ]]; then
+	ISA=$(echo "$unique_isas" | sed 's/.*"\(.*\)".*/\1/')
+    else
+	echo "Unable to set ISA from DT{B,S}. Ensure all hart ISAs are equivalent."
+	exit 1
+    fi
+fi
+
+# pmpregions is overridden by the dt{b,s} so fine to include on CLI here
+SPIKEFLAGS+=" --pmpregions=0 --isa=$ISA -p$NHARTS"
+
 LOADARCH_FILE=$OUTPATH/loadarch
 RAWMEM_ELF=$OUTPATH/raw.elf
 LOADMEM_ELF=$OUTPATH/mem.elf
 CMDS_FILE=$OUTPATH/cmds_tmp.txt
 SPIKECMD_FILE=$OUTPATH/spikecmd.sh
 
 echo "Generating state capture spike interactive commands in $CMDS_FILE"
-echo "until pc 0 $PC" >> $CMDS_FILE
+if [ ! -z "$INSN" ]; then
+    echo "until insn 0 $INSN" >> $CMDS_FILE
+else
+    echo "until pc 0 $PC" >> $CMDS_FILE
+fi
 echo "rs $INSNS" >> $CMDS_FILE
 echo "dump" >> $CMDS_FILE
 for (( h=0; h<$NHARTS; h++ ))
@@ -146,15 +208,25 @@ echo $NHARTS > $LOADARCH_FILE
 spike -d --debug-cmd=$CMDS_FILE $SPIKEFLAGS $BINARY 2>> $LOADARCH_FILE
 
 
-echo "Finding tohost/fromhost in elf file"
-TOHOST=$(riscv64-unknown-elf-nm $BINARY | grep tohost | head -c 16)
-FROMHOST=$(riscv64-unknown-elf-nm $BINARY | grep fromhost | head -c 16)
+echo "Finding tohost/fromhost in elf file to inject in new elf"
+function get_symbol_value() {
+    echo $(riscv64-unknown-elf-nm $2 | grep " ${1}$" | head -c 16)
+}
+# ensure these symbols are not 'htif_*' versions
+TOHOST=$(get_symbol_value tohost $BINARY)
+FROMHOST=$(get_symbol_value fromhost $BINARY)
 
+MEM_DUMP=mem.${DEFAULT_MEM_START_ADDR}.bin
 echo "Compiling memory to elf"
-riscv64-unknown-elf-objcopy -I binary -O elf64-littleriscv mem.0x80000000.bin $RAWMEM_ELF
-rm -rf mem.0x80000000.bin
+du -sh $MEM_DUMP
+riscv64-unknown-elf-objcopy -I binary -O elf64-littleriscv $MEM_DUMP $RAWMEM_ELF
+rm -rf $MEM_DUMP
 
-riscv64-unknown-elf-ld -Tdata=0x80000000 -nmagic --defsym tohost=0x$TOHOST --defsym fromhost=0x$FROMHOST -o $LOADMEM_ELF $RAWMEM_ELF
+riscv64-unknown-elf-ld -Tdata=$DEFAULT_MEM_START_ADDR -nmagic --defsym tohost=0x$TOHOST --defsym fromhost=0x$FROMHOST -o $LOADMEM_ELF $RAWMEM_ELF
 rm -rf $RAWMEM_ELF
 
-echo "Ensure that at minimum you have memory regions corresponding to $BASEMEM in downstream RTL tooling"
+if [[ -z "$DTB" && -z "$DTS" ]] ; then
+    echo "Ensure that (at minimum) you have memory regions corresponding to $BASEMEM in downstream RTL tooling"
+fi
+
+echo "Loadarch directory $OUTPATH created"