Massive JSON blob in Stdout

[philipturner]

This was a bug affecting Swift-Colab 2.0 for quite a while, and I just recently found a workaround for it. Make a two-file Swift package with the following directory structure:

installed1
|-- Package.swift
|-- installed1.swift

In Package.swift, put the following:

// swift-tools-version:5.6
import PackageDescription
let package = Package(
  name: "installed1",
  products: [
    .library(
      name: "installed1",
      targets: ["installed1"]
    )
  ],
  dependencies: [],
  targets: [
    .target(
      name: "installed1",
      dependencies: [],
      path: ".",
      sources: ["installed1.swift"]
    )
  ]
)

In installed1.swift, put the following:

// intentionally blank
(this should be an empty line but GitHub doesn't want to show it)

Build it once by entering the installed1 directory and running swift build. Then, overwrite installed1.swift with its exact same contents using the command shown below. Run swift build once more and a massive JSON blob appears in stdout. This happens on both Linux (Google Colab, x86) and macOS (ARM64).

echo "// intentionally blank\n" > installed1.swift

Blob:

3291
{
  "command_arguments" : [
    "-frontend",
    "-emit-module",
    "-experimental-skip-non-inlinable-function-bodies-without-types",
    "\/Users\/philipturner\/Desktop\/installed1\/installed1.swift",
    "-target",
    "arm64-apple-macosx10.10",
    "-Xllvm",
    "-aarch64-use-tbi",
    "-enable-objc-interop",
    "-sdk",
    "\/Applications\/Xcode.app\/Contents\/Developer\/Platforms\/MacOSX.platform\/Developer\/SDKs\/MacOSX12.3.sdk",
    "-I",
    "\/Users\/philipturner\/Desktop\/installed1\/.build\/arm64-apple-macosx\/debug",
    "-I",
    "\/Applications\/Xcode.app\/Contents\/Developer\/Platforms\/MacOSX.platform\/Developer\/usr\/lib",
    "-F",
    "\/Applications\/Xcode.app\/Contents\/Developer\/Platforms\/MacOSX.platform\/Developer\/Library\/Frameworks",
    "-color-diagnostics",
    "-enable-testing",
    "-g",
    "-module-cache-path",
    "\/Users\/philipturner\/Desktop\/installed1\/.build\/arm64-apple-macosx\/debug\/ModuleCache",
    "-swift-version",
    "5",
    "-Onone",
    "-D",
    "SWIFT_PACKAGE",
    "-D",
    "DEBUG",
    "-new-driver-path",
    "\/Applications\/Xcode.app\/Contents\/Developer\/Toolchains\/XcodeDefault.xctoolchain\/usr\/bin\/swift-driver",
    "-resource-dir",
    "\/Applications\/Xcode.app\/Contents\/Developer\/Toolchains\/XcodeDefault.xctoolchain\/usr\/lib\/swift",
    "-enable-anonymous-context-mangled-names",
    "-module-name",
    "installed1",
    "-target-sdk-version",
    "12.3",
    "-emit-module-doc-path",
    "\/Users\/philipturner\/Desktop\/installed1\/.build\/arm64-apple-macosx\/debug\/installed1.swiftdoc",
    "-emit-module-source-info-path",
    "\/Users\/philipturner\/Desktop\/installed1\/.build\/arm64-apple-macosx\/debug\/installed1.swiftsourceinfo",
    "-emit-objc-header-path",
    "\/Users\/philipturner\/Desktop\/installed1\/.build\/arm64-apple-macosx\/debug\/installed1.build\/installed1-Swift.h",
    "-parse-as-library",
    "-o",
    "\/Users\/philipturner\/Desktop\/installed1\/.build\/arm64-apple-macosx\/debug\/installed1.swiftmodule",
    "-emit-abi-descriptor-path",
    "\/Users\/philipturner\/Desktop\/installed1\/.build\/arm64-apple-macosx\/debug\/installed1.abi.json"
  ],
  "command_executable" : "\/Applications\/Xcode.app\/Contents\/Developer\/Toolchains\/XcodeDefault.xctoolchain\/usr\/bin\/swift-frontend",
  "inputs" : [

  ],
  "kind" : "began",
  "name" : "emit-module",
  "outputs" : [
    {
      "path" : "\/Users\/philipturner\/Desktop\/installed1\/.build\/arm64-apple-macosx\/debug\/installed1.swiftmodule",
      "type" : "swiftmodule"
    },
    {
      "path" : "\/Users\/philipturner\/Desktop\/installed1\/.build\/arm64-apple-macosx\/debug\/installed1.swiftdoc",
      "type" : "swiftdoc"
    },
    {
      "path" : "\/Users\/philipturner\/Desktop\/installed1\/.build\/arm64-apple-macosx\/debug\/installed1.swiftsourceinfo",
      "type" : "swiftsourceinfo"
    },
    {
      "path" : "\/Users\/philipturner\/Desktop\/installed1\/.build\/arm64-apple-macosx\/debug\/installed1.build\/installed1-Swift.h",
      "type" : "objc-header"
    },
    {
      "path" : "\/Users\/philipturner\/Desktop\/installed1\/.build\/arm64-apple-macosx\/debug\/installed1.abi.json",
      "type" : "abi-baseline-json"
    }
  ],
  "pid" : 19044,
  "process" : {
    "real_pid" : 19044
  }
}
2885
{
  "command_arguments" : [
    "-frontend",
    "-c",
    "-primary-file",
    "\/Users\/philipturner\/Desktop\/installed1\/installed1.swift",
    "-emit-dependencies-path",
    "\/Users\/philipturner\/Desktop\/installed1\/.build\/arm64-apple-macosx\/debug\/installed1.build\/installed1.d",
    "-emit-reference-dependencies-path",
    "\/Users\/philipturner\/Desktop\/installed1\/.build\/arm64-apple-macosx\/debug\/installed1.build\/installed1.swiftdeps",
    "-target",
    "arm64-apple-macosx10.10",
    "-Xllvm",
    "-aarch64-use-tbi",
    "-enable-objc-interop",
    "-sdk",
    "\/Applications\/Xcode.app\/Contents\/Developer\/Platforms\/MacOSX.platform\/Developer\/SDKs\/MacOSX12.3.sdk",
    "-I",
    "\/Users\/philipturner\/Desktop\/installed1\/.build\/arm64-apple-macosx\/debug",
    "-I",
    "\/Applications\/Xcode.app\/Contents\/Developer\/Platforms\/MacOSX.platform\/Developer\/usr\/lib",
    "-F",
    "\/Applications\/Xcode.app\/Contents\/Developer\/Platforms\/MacOSX.platform\/Developer\/Library\/Frameworks",
    "-color-diagnostics",
    "-enable-testing",
    "-g",
    "-module-cache-path",
    "\/Users\/philipturner\/Desktop\/installed1\/.build\/arm64-apple-macosx\/debug\/ModuleCache",
    "-swift-version",
    "5",
    "-Onone",
    "-D",
    "SWIFT_PACKAGE",
    "-D",
    "DEBUG",
    "-new-driver-path",
    "\/Applications\/Xcode.app\/Contents\/Developer\/Toolchains\/XcodeDefault.xctoolchain\/usr\/bin\/swift-driver",
    "-resource-dir",
    "\/Applications\/Xcode.app\/Contents\/Developer\/Toolchains\/XcodeDefault.xctoolchain\/usr\/lib\/swift",
    "-enable-anonymous-context-mangled-names",
    "-module-name",
    "installed1",
    "-target-sdk-version",
    "12.3",
    "-parse-as-library",
    "-o",
    "\/Users\/philipturner\/Desktop\/installed1\/.build\/arm64-apple-macosx\/debug\/installed1.build\/installed1.swift.o",
    "-index-store-path",
    "\/Users\/philipturner\/Desktop\/installed1\/.build\/arm64-apple-macosx\/debug\/index\/store",
    "-index-system-modules"
  ],
  "command_executable" : "\/Applications\/Xcode.app\/Contents\/Developer\/Toolchains\/XcodeDefault.xctoolchain\/usr\/bin\/swift-frontend",
  "inputs" : [
    "\/Users\/philipturner\/Desktop\/installed1\/installed1.swift"
  ],
  "kind" : "began",
  "name" : "compile",
  "outputs" : [
    {
      "path" : "\/Users\/philipturner\/Desktop\/installed1\/.build\/arm64-apple-macosx\/debug\/installed1.build\/installed1.swift.o",
      "type" : "object"
    },
    {
      "path" : "\/Users\/philipturner\/Desktop\/installed1\/.build\/arm64-apple-macosx\/debug\/installed1.build\/installed1.d",
      "type" : "d"
    },
    {
      "path" : "\/Users\/philipturner\/Desktop\/installed1\/.build\/arm64-apple-macosx\/debug\/installed1.build\/installed1.swiftdeps",
      "type" : "swift-dependencies"
    }
  ],
  "pid" : 19045,
  "process" : {
    "real_pid" : 19045
  }
}
133
{
  "exit-status" : 0,
  "kind" : "finished",
  "name" : "emit-module",
  "pid" : 19044,
  "process" : {
    "real_pid" : 19044
  }
}
129
{
  "exit-status" : 0,
  "kind" : "finished",
  "name" : "compile",
  "pid" : 19045,
  "process" : {
    "real_pid" : 19045
  }
}

If you run swift build a third time without doing the echo command, then it won't output the blob during that third build. But it always outputs the blob if you modify a file before running a build command. Note that I do not see this bug when I overwrite Package.swift with the same contents as it previously had instead of installed1.swift.

[tomerd]

this sounds the same as #5482?

[philipturner]

I made a branch on my repo where I undid the workaround. I'm going to show you a way to easily reproduce and bisect this bug. I could technically do it myself, but there's a lot of Swift development snapshots to go back through and that's a lot of work to scan over. Plus, following these instructions should be doable for someone else. Go to my Swift-Colab repository and open the template Swift notebook. Replace all of its cells with just these two:

!curl "https://raw.githubusercontent.com/philipturner/swift-colab/spm5482/install_swift.sh" --output "install_swift.sh"
!bash "install_swift.sh" "5.5.2" --swift-colab-dev #// Replace 5.5.2 with newest Swift version.
#// After this command finishes, go to Runtime > Restart runtime.

%install '.package(url: "https://github.com/pvieito/PythonKit.git", .branch("master"))' PythonKit

Steps:

1. Run the first cell, and wait until it instructs you to restart the runtime. Then, restart the runtime using the "Runtime" tab in the GUI.
2. You will restart in Swift mode. Run the cell for installing PythonKit, all the way through so that it says Build complete!. Run it once more and the ugly JSON blob appears.
3. Restart the runtime again just like in Step 1. You're now back in Python mode and can run cell 1 with a different toolchain.

The --swift-colab-dev flag is something I use personally for modifying Swift-Colab. It is an undocumented feature. It alternates between the Swift and Python runtimes on every runtime restart, letting me quickly do things like swap out the toolchain.

4. It takes 24 seconds to download a 550 MB toolchain. Modify the "5.5.2" in the cell to become "5.5", and repeat steps 1 and 2.
5. Repeat step 4 with "2022-05-04" instead of "5.5". This activates different code in the install script that parses a development snapshot's date (May 4, 2022). You can trace the date back to something very early on.

If you go back far enough, Swift-Colab itself may not compile. I ran the above steps through with Swift 5.4, and Swift-Colab compiled successfully. I also saw the JSON blob there.

You might be able to modify the script to bypass the restriction on compiling Swift-Colab itself. In an open notebook after you did step 1 once, go to /content/install_swift.sh in the file viewer. You can change the script as it appears there, messing around with whether it force-recompiles Swift-Colab with the newly downloaded toolchain. Although I'm not sure this would work because it should corrupt the links in the JupyterKernel binary to depend on the old toolchain's standard library now.