Distributed compilation support

[waruqi]

Features

• Cross-platform support
• Support msvc, clang, gcc and cross-compilation
• Support to build android, ios, linux, win, macOS programs
• No dependencies other than the compilation toolchain
• Support for load balancing scheduling
• Support for real-time large file compression transfer (use lz4)
• Almost zero configuration cost, no shared file system required, more convenient and secure

支持特性

• 跨平台支持
• 支持 msvc, clang, gcc 和交叉编译工具链
• 支持构建 android, ios, linux, win, macOS 程序
• 除了编译工具链，无任何其他依赖
• 支持编译服务器负载均衡调度
• 支持大文件实时压缩传输 (lz4)
• 几乎零配置成本，无需共享文件系统，更加方便和安全

Auto-generate server/client configuration.

$ xmake service
generating the config file to /Users/ruki/.xmake/service/server.conf ..
an token(590234653af52e91b9e438ed860f1a2b) is generated, we can use this token to connect service.
generating the config file to /Users/ruki/.xmake/service/client.conf ..
<remote_build_server>: listening 0.0.0.0:9691 ..
<distcc_build_server>: listening 0.0.0.0:9692 ..

$ cat ~/.xmake/service/server.conf 
{
    distcc_build = {
        listen = "0.0.0.0:9692",
        toolchains = {
            ndk = { }
        },
        workdir = "/Users/ruki/.xmake/service/server/distcc_build"
    },
    known_hosts = { },
    logfile = "/Users/ruki/.xmake/service/server/logs.txt",
    tokens = {
        "590234653af52e91b9e438ed860f1a2b"
    }
}

$cat ~/.xmake/service/client.conf 
{
    distcc_build = {
        hosts = {
            {
                connect = "127.0.0.1:9692",
                token = "590234653af52e91b9e438ed860f1a2b"
            }
        }
    }
}

Start service

$ xmake service
<distcc_build_server>: listening 0.0.0.0:9692 ..

Start server as daemon

$ xmake service --start

Connect project to server

$ cd projectdir
$ xmake service --connect --distcc
<client>: connect 127.0.0.1:9692 ..
<client>: 127.0.0.1:9692 connected!

Build project for distributed build

$ xmake
...
[ 93%]: ccache compiling.release src/demo/network/unix_echo_client.c         ----> local job
[ 93%]: ccache compiling.release src/demo/network/ipv6.c
[ 93%]: ccache compiling.release src/demo/network/ping.c
[ 93%]: distcc compiling.release src/demo/network/unix_echo_server.c.         ----> distcc job
[ 93%]: distcc compiling.release src/demo/network/http.c
[ 93%]: distcc compiling.release src/demo/network/unixaddr.c
[ 93%]: distcc compiling.release src/demo/network/ipv4.c
[ 94%]: distcc compiling.release src/demo/network/ipaddr.c
[ 94%]: distcc compiling.release src/demo/math/fixed.c
[ 94%]: distcc compiling.release src/demo/libm/float.c
[ 95%]: ccache compiling.release src/demo/libm/double.c
[ 95%]: ccache compiling.release src/demo/other/test.cpp
[ 98%]: archiving.release libtbox.a
[ 99%]: linking.release demo
[100%]: build ok!

Default jobs count

the default jobs count of cpu core

local default_njob = math.ceil(ncpu * 3 / 2)

the max jobs counts

local maxjobs = local_maxjobs + server_maxjobs (server count * default_njob)

Modify local jobs count

$ xmake -jN

Modify server jobs count

$cat ~/.xmake/service/client.conf 
{
    distcc_build = {
        hosts = {
            {
                connect = "127.0.0.1:9692",
                token = "590234653af52e91b9e438ed860f1a2b",
                njob = 8   <------- modify here
            },
            {
                connect = "192.168.01:9692",
                token = "590234653af52e91b9e438ed860f1a2b",
                njob = 4
            }
        }
    }
}

Build android project

We need set NDK root path in server

$ cat ~/.xmake/service/server.conf 
{
    distcc_build = {
        listen = "0.0.0.0:9692",
        toolchains = {
            ndk = {
                ndk = "~/files/android-ndk-r21e"   <------------ here
            }
        },
        workdir = "/Users/ruki/.xmake/service/server/distcc_build"
    },
    known_hosts = { },
    logfile = "/Users/ruki/.xmake/service/server/logs.txt",
    tokens = {
        "590234653af52e91b9e438ed860f1a2b"
    }
}

Build project

$ xmake f -p android --ndk=~/files/xxxx
$ xmake

Cross-compilation

We need set cross-compilation toolchain sdk path in server

$ cat ~/.xmake/service/server.conf 
{
    distcc_build = {
        listen = "0.0.0.0:9692",
        toolchains = {
            cross = {
                sdkdir = "~/files/xxxx/sdk"   <------------ here
            }
        },
        workdir = "/Users/ruki/.xmake/service/server/distcc_build"
    },
    known_hosts = { },
    logfile = "/Users/ruki/.xmake/service/server/logs.txt",
    tokens = {
        "590234653af52e91b9e438ed860f1a2b"
    }
}

Build project

$ xmake f -p cross --sdk=~/files/xxxx
$ xmake

we can also set other configurations, e.g. bindir, cross

cross = {
    sdkdir = "~/files/xxxx/sdk",
    bindir = "~/files/xxxx/sdk/bin",
    cross = "arm-linux-"
}

Build ios project

$ xmake f -p iphoneos
$ xmake

More

More service-related configuration notes are available at: #622

[waruqi]

I have supported it.