Move packaging scripts from swiftwasm-package-sdk (#18)

Currently our CI scripts rely on validity of scripts in the [`swiftwasm-package-sdk`](https://github.com/swiftwasm/swiftwasm-package-sdk) repository. That can't always be the case, especialy as we don't checkout a specific commit in that repository. And even if we did, managing this isn't convenient. Adding a submodule has its own set of issues and I personally think that a monorepo approach is much simpler for a small set of scripts, at least at an early stage. In fact, it would make sense to have these scripts in the upstream [`swift`](https://github.com/apple/swift) repository at some point, similarly to [what Android people have previously done](https://github.com/apple/swift/tree/master/utils/android).

Thus, these scripts and a few small helper files are copied to `utils/webassembly` directory and are executed directly on CI.

After this PR is merged, I don't see a particular need for our [`swiftwasm-package-sdk`](https://github.com/swiftwasm/swiftwasm-package-sdk) and [`swiftwasm-sdk`](https://github.com/swiftwasm/swiftwasm-sdk) repos, those probably can be archived.

As a small cleanup addition, `.github/workflows/main.yml` file now has consistent indentation.

* Move packaging scripts from swiftwasm-package-sdk
* Rename `wasm` directory to `webassembly`
* Make all .sh scripts executable after download
* Make sdkroot/swiftwasm script executable
* Add newline to build-mac-package.sh
* Add newline to build-packages.sh
* Remove swift_start.o and swift_end.o

.github/workflows/main.yml

@@ -3,45 +3,50 @@ name: CI
 on:
   push:
     branches:
-    - swiftwasm
+      - swiftwasm
   pull_request:
     branches:
-    - swiftwasm
+      - swiftwasm
 
 jobs:
   linux_build:
     timeout-minutes: 0
     runs-on: ubuntu-18.04
 
     steps:
-    - uses: actions/checkout@v1
-    - name: Build Linux installable archive
-      run: ./ci-linux.sh
-    - name: Upload Linux installable archive
-      uses: actions/upload-artifact@v1
-      with:
-        name: linux-installable
-        path: ../swiftwasm-linux.tar.gz
+      - uses: actions/checkout@v1
+      - name: Build Linux installable archive
+        run: ./ci-linux.sh
+      - name: Upload Linux installable archive
+        uses: actions/upload-artifact@v1
+        with:
+          name: linux-installable
+          path: ../swiftwasm-linux.tar.gz
 
   macos_build:
     timeout-minutes: 0
     runs-on: macOS-10.14
 
     steps:
-    - uses: actions/checkout@v1
-    - name: Build macOS installable archive
-      run: ./ci-mac.sh
-    - name: Upload macOS installable archive
-      uses: actions/upload-artifact@v1
-      with:
-        name: macos-installable
-        path: ../swiftwasm-macos.tar.gz
+      - uses: actions/checkout@v1
+      - name: Build macOS installable archive
+        run: ./ci-mac.sh
+      - name: Upload macOS installable archive
+        uses: actions/upload-artifact@v1
+        with:
+          name: macos-installable
+          path: ../swiftwasm-macos.tar.gz
+      - name: Upload packaging scripts
+        uses: actions/upload-artifact@v1
+        with:
+          name: packaging-scripts
+          path: utils/webassembly
 
   package:
     name: Build SwiftWasm packages
     needs:
-    - linux_build
-    - macos_build
+      - linux_build
+      - macos_build
     runs-on: ubuntu-18.04
     steps:
       - name: Download installable Linux archive
@@ -52,11 +57,16 @@ jobs:
         uses: actions/download-artifact@v1
         with:
           name: macos-installable
+      - name: Download packaging scripts
+        uses: actions/download-artifact@v1
+        with:
+          name: packaging-scripts
       - name: Build the packages
         shell: bash
         run: |
-          git clone https://github.com/swiftwasm/swiftwasm-package-sdk.git
-          cd swiftwasm-package-sdk
+          cd packaging-scripts
+          find . -name '*.sh' -exec chmod +x {} \;
+          chmod +x sdkroot/swiftwasm
           ./download-prebuilts.sh
 
           cp ../linux-installable/swiftwasm-linux.tar.gz \
@@ -74,19 +84,19 @@ jobs:
         uses: actions/upload-artifact@v1
         with:
           name: macos-package
-          path: swiftwasm-package-sdk/output/swiftwasm-sdk-macos.tar.xz
+          path: packaging-scripts/output/swiftwasm-sdk-macos.tar.xz
 
       - name: Upload Linux package
         uses: actions/upload-artifact@v1
         with:
           name: linux-package
-          path: swiftwasm-package-sdk/output/swiftwasm-sdk-linux.tar.xz
+          path: packaging-scripts/output/swiftwasm-sdk-linux.tar.xz
 
       - name: Upload hello.wasm compiled with Linux package
         uses: actions/upload-artifact@v1
         with:
           name: linux-hello.wasm
-          path: swiftwasm-package-sdk/output/swiftwasm-sdk/hello.wasm
+          path: packaging-scripts/output/swiftwasm-sdk/hello.wasm
 
   macos_smoke_test:
     name: Compile hello.swift on macOS

utils/webassembly/.gitignore

@@ -0,0 +1,5 @@
+compiler
+swiftwasm-sdk
+prebuilt
+output
+tmpdir

utils/webassembly/README.md

@@ -0,0 +1,23 @@
+Creates packages containing everything needed to build WebAssembly programs with Swift.
+
+# Building
+
+```
+./download-prebuilts.sh
+./download-installable-prebuilts.sh
+./build-packages.sh
+```
+
+# Contents of package
+
+- Swift toolchain from [swiftwasm-sdk](https://github.com/swiftwasm/swiftwasm-sdk)
+- WASI modified sysroot from [wasi-sdk](https://github.com/swiftwasm/wasi-sdk)
+- libicu from [icu4c-wasi](https://github.com/swiftwasm/icu4c-wasi)
+- linking helpers from [swiftwasm-wasi-stubs](https://github.com/swiftwasm/swiftwasm-wasi-stubs)
+- wasi-ld, either from wasi-sdk (on Linux) or upstream LLVM 9.0 (on Mac)
+- build script for compiling a Swift file to a .wasm
+- a Getting Started guide
+
+# Notes
+
+This shares a lot with the [swiftwasm-compile-service](https://github.com/swiftwasm/swiftwasm-compile-service).

utils/webassembly/build-linux-package.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+echo "Unpacking Linux prebuilts"
+mkdir -p output
+cd linux
+./unpack-prebuilts.sh
+echo "Compressing"
+tar cJf ../output/swiftwasm-sdk-linux.tar.xz swiftwasm-sdk
+cd ..

utils/webassembly/build-mac-package.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+echo "Unpacking macOS prebuilts"
+mkdir -p output
+cd macos
+./unpack-prebuilts.sh
+echo "Compressing macOS package"
+tar cJf ../output/swiftwasm-sdk-macos.tar.xz swiftwasm-sdk
+cd ..

utils/webassembly/build-packages.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+set -e
+rm -rf output || true
+./build-linux-package.sh
+./build-mac-package.sh

utils/webassembly/copy-shared-files.sh

@@ -0,0 +1,4 @@
+#!/bin/bash
+set -e
+cp -r ../sdkroot/* compiler/
+cp ../linux/compiler/opt/swiftwasm-sdk/lib/swift/wasm/wasm32/glibc.modulemap compiler/extra_utils

utils/webassembly/download-installable-prebuilts.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+set -e
+mkdir -p prebuilt
+cd prebuilt
+wget -O swiftwasm-linux.tar.gz https://github.com/swiftwasm/swiftwasm-sdk/releases/download/20191112.1.linux/swiftwasm.tar.gz
+# Mac specific
+wget -O swiftwasm-macos.tar.gz https://github.com/swiftwasm/swiftwasm-sdk/releases/download/20191112.1.mac/swiftwasm-mac.tar.gz

utils/webassembly/download-prebuilts.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+set -e
+mkdir -p prebuilt
+cd prebuilt
+wget https://github.com/swiftwasm/wasi-sdk/releases/download/20191022.1/wasi-sdk-4.39g3025a5f47c04-linux.tar.gz
+wget https://github.com/swiftwasm/icu4c-wasi/releases/download/20190421.3/icu4c-wasi.tar.xz
+# Mac specific
+wget http://releases.llvm.org/9.0.0/clang+llvm-9.0.0-x86_64-darwin-apple.tar.xz

utils/webassembly/linux/unpack-prebuilts.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+set -e
+rm -rf swiftwasm-sdk compiler
+mkdir swiftwasm-sdk
+ln -s swiftwasm-sdk compiler
+cd compiler
+untar="../../prebuilt/wasi-sdk-"*"-linux.tar.gz
+../../prebuilt/swiftwasm-linux.tar.gz
+../../prebuilt/icu4c-wasi.tar.xz"
+for i in $untar
+do
+	echo $i
+	tar xf $i
+done
+cd ..
+mv "compiler/wasi-sdk-"* "compiler/wasi-sdk"
+mv compiler/wasi-sdk/share/wasi-sysroot compiler/wasi-sdk/share/sysroot
+../remove-swift-extra-files.sh || true
+../remove-wasi-extra-files.sh || true
+../copy-shared-files.sh || true

utils/webassembly/macos/unpack-prebuilts.sh

@@ -0,0 +1,30 @@
+#!/bin/bash
+set -e
+rm -rf swiftwasm-sdk compiler tmpdir
+mkdir swiftwasm-sdk tmpdir
+ln -s swiftwasm-sdk compiler
+cd compiler
+untar="../../prebuilt/wasi-sdk-"*"-linux.tar.gz
+../../prebuilt/swiftwasm-macos.tar.gz
+../../prebuilt/icu4c-wasi.tar.xz"
+for i in $untar
+do
+	echo $i
+	tar xf $i
+done
+# Mac: unpack the Linux one and copy stdlibs
+cd ..
+cd tmpdir
+tar xf ../../prebuilt/clang+llvm-*-x86_64-darwin-apple.tar.xz
+tar xf ../../prebuilt/swiftwasm-linux.tar.gz
+cd ..
+mv "compiler/wasi-sdk-"* "compiler/wasi-sdk"
+mv compiler/wasi-sdk/share/wasi-sysroot compiler/wasi-sdk/share/sysroot
+../remove-swift-extra-files.sh || true
+rm -r compiler/wasi-sdk/bin
+mkdir compiler/wasi-sdk/bin
+cp tmpdir/clang+llvm-*-x86_64-darwin-apple/bin/wasm-ld compiler/wasi-sdk/bin
+cp -a tmpdir/opt/swiftwasm-sdk/lib/swift/wasm compiler/opt/swiftwasm-sdk/lib/swift/wasm
+cp -a tmpdir/opt/swiftwasm-sdk/lib/swift_static compiler/opt/swiftwasm-sdk/lib/swift_static
+# ok, finally copy over the shared files
+../copy-shared-files.sh || true

utils/webassembly/remove-swift-extra-files.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+basepath="compiler/opt/swiftwasm-sdk"
+filestoremove="bin/sil-* 
+bin/lldb*
+bin/sourcekitd-*
+bin/swift-api-digester
+bin/swift-autolink-extract
+bin/swift-demangle
+bin/swift-demangle-yamldump
+bin/swift-format
+bin/swift-llvm-opt
+bin/swift-refactor
+bin/swift-reflection-dump
+bin/swift-*-test
+lib/libsourcekitdInProc.so
+lib/swift/clang/lib/linux/*
+lib/swift_static/linux/*
+lib/swift/linux/x86_64/*
+lib/swift/linux/*"
+for i in $filestoremove
+do
+	echo $basepath/$i
+	rm $basepath/$i
+done
+# Mac only
+rm -r $basepath/lib/swift/macosx $basepath/lib/sourcekitd.framework

utils/webassembly/remove-wasi-extra-files.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+basepath="compiler/wasi-sdk"
+filestoremove="bin/clang*
+bin/llvm*
+bin/llc"
+for i in $filestoremove
+do
+	echo $basepath/$i
+	rm $basepath/$i
+done

utils/webassembly/sdkroot/README.md

@@ -0,0 +1,87 @@
+SwiftWasm: Getting started
+==========================
+
+Thank you for trying SwiftWasm! Here's how to get started.
+
+Please visit our website at https://swiftwasm.org for the latest updates.
+
+
+Install dependencies
+====================
+
+Before running SwiftWasm, you will need to install some dependencies.
+
+Ubuntu:
+
+```
+sudo apt-get install libatomic1
+```
+
+macOS:
+
+(No dependencies needed.)
+
+Windows:
+
+Install Windows Subsystem for Linux, then follow the Ubuntu instructions.
+
+
+
+
+Compile SwiftWasm
+=================
+
+Run
+
+```
+./swiftwasm example/hello.swift hello.wasm
+```
+
+To compile example/hello.swift to hello.wasm.
+
+
+
+
+Running Wasm files
+==================
+
+To run the resulting hello.wasm file:
+
+- Visit https://swiftwasm.org/polyfill/
+- select "Browse", and choose the hello.wasm file
+- you should get output in the textbox.
+
+This polyfill should work in Firefox 66, Chrome 74, and Safari 12.1.
+
+You can also run the file outside a browser with:
+
+- Wasmtime https://github.com/CraneStation/wasmtime
+- Lucet https://github.com/swiftwasm/lucet/tree/swiftwasm
+- or any other WASI-compatible WebAssembly runtime.
+
+
+
+
+Questions and support
+=====================
+
+If you have any questions, please open an issue on
+
+https://github.com/swiftwasm/swift
+
+
+
+Third-party licenses
+====================
+
+This package contains components with their own license requirements.
+
+Swift compiler: https://github.com/apple/swift/blob/master/LICENSE.txt
+
+LLVM/Clang: https://github.com/llvm/llvm-project/blob/master/lld/LICENSE.TXT
+
+WASI sysroot: https://github.com/CraneStation/wasi-sysroot/blob/master/LICENSE
+
+ICU: https://github.com/unicode-org/icu/blob/master/icu4c/LICENSE
+
+WASI polyfill: https://github.com/CraneStation/wasmtime/blob/master/wasmtime-wasi/LICENSE

utils/webassembly/sdkroot/example/hello.swift

@@ -0,0 +1 @@
+print("Hello, 🌐!")

utils/webassembly/sdkroot/extra_utils/generateModulemap.sh

@@ -0,0 +1,2 @@
+#!/bin/sh
+exec sed -e "s@\"/include@\"$1/include@g" "$(dirname $0)/glibc.modulemap"