Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support remote compilation #622

Closed
5 tasks done
waruqi opened this issue Nov 19, 2019 · 11 comments
Closed
5 tasks done

Support remote compilation #622

waruqi opened this issue Nov 19, 2019 · 11 comments

Comments

@waruqi
Copy link
Member

waruqi commented Nov 19, 2019

Roadmap

  • socket io and coroutine scheduler
  • xmake connect xxx.xxx.xxx.xxx
  • xmake (remote build)
  • xmake config (remote configuration)
  • xmake run (remote run)

Refs

#165 (comment)
#593

@waruqi waruqi added this to the todo milestone Nov 19, 2019
@waruqi waruqi pinned this issue Nov 20, 2019
@waruqi waruqi modified the milestones: todo, v2.3.0 Dec 20, 2019
@waruqi waruqi modified the milestones: v2.3.0, v2.3.1 Feb 21, 2020
@waruqi waruqi modified the milestones: v2.3.2, v2.3.3 Mar 24, 2020
@waruqi waruqi modified the milestones: v2.3.3, v2.3.4 Apr 17, 2020
@waruqi waruqi modified the milestones: v2.3.4, todo May 13, 2020
@waruqi waruqi unpinned this issue May 15, 2020
@arkena00
Copy link

arkena00 commented Feb 15, 2022

+1 for this

I was thinking about something like this :

xmake build --host host.com --user me --pass security

or using a custom toolchain maybe

xmake build --toolchain=gcc@host

toolchain("gcc@host")
    set_kind("remote")
    set_user("me")
    ...

I'm not sure if it's possible to make it simple to integrate in an IDE if you have to run xmake connect first

@waruqi
Copy link
Member Author

waruqi commented Feb 15, 2022

This feature need take a lot of time and work, and maybe I'll start supporting it again in the future, but not now.

Of course, if more people request this feature, I'll also consider it when I have time.

@SongZihui-sudo
Copy link

做一个插件,可不可以用git来同步远程与本地的源文件,之后在远程建一个socket服务器来接受编译的命令,编译完成,在本地用pull同步结果。

Make a plugin,Maybe we can use 'git' to synchronize remote and local source files, and then run a 'socket server' remotely to accept compiled commands, etc., compile finish ,then we can synchronize the results with 'pull' locally.

@waruqi
Copy link
Member Author

waruqi commented Apr 8, 2022

做一个插件,可不可以用git来同步远程与本地的源文件,之后在远程建一个socket服务器来接受编译的命令,编译完成,在本地用pull同步结果。

Make a plugin,Maybe we can use 'git' to synchronize remote and local source files, and then run a 'socket server' remotely to accept compiled commands, etc., compile finish ,then we can synchronize the results with 'pull' locally.

现在的问题不是怎么做,是我没时间精力做

@waruqi
Copy link
Member Author

waruqi commented Apr 15, 2022

Start service

$ xmake service
<remote_build_server>: listening 0.0.0.0:90091 ..

with verbose logs

$ xmake service -vD
<remote_build_server>: listening 0.0.0.0:90091 ..

Start and stop service with daemon mode

$ xmake service --start
$ xmake service --restart
$ xmake service --stop

Configure service on server side

run xmake service will generate a default service.conf file in ~/.xmake/service.conf

edit it and modify server user name for using git push to sync files.

{
    logfile = "/Users/ruki/.xmake/service/logs.txt",
    remote_build = {
        server = {
            listen = "0.0.0.0:90091"
        }
    }
}

Configure service on client side

~/.xmake/service.conf

{
    logfile = "/Users/ruki/.xmake/service/logs.txt",
    remote_build = {
        client = {
            connect = "192.168.56.101:90091",
        }
    }
}

Import the given configuration file

$ xmake service --config=/tmp/service.conf

Connect remote build service on client side

enter project directory

$ xmake create test
$ cd test
$ xmake service --connect 
<remote_build_client>: connect 192.168.56.110:90091 ..
<remote_build_client>: connected!
<remote_build_client>: sync files in 192.168.56.110:90091 ..
Scanning files ..
Comparing 3 files ..
    [+]: src/main.cpp
    [+]: .gitignore
    [+]: xmake.lua
3 files has been changed!
Archiving files ..
Uploading files with 1372 bytes ..
<remote_build_client>: sync files ok!

Build project with remote mode

$ xmake
<remote_build_client>: run xmake in 192.168.56.110:90091 ..
checking for platform ... macosx
checking for architecture ... x86_64
checking for Xcode directory ... /Applications/Xcode.app
checking for Codesign Identity of Xcode ... Apple Development: waruqi@gmail.com (T3NA4MRVPU)
checking for SDK version of Xcode for macosx (x86_64) ... 11.3
checking for Minimal target version of Xcode for macosx (x86_64) ... 11.4
[ 25%]: ccache compiling.release src/main.cpp
[ 50%]: linking.release test
[100%]: build ok!
<remote_build_client>: run command ok!

Run target with remote mode

$ xmake run
<remote_build_client>: run xmake run in 192.168.56.110:90091 ..
hello world!
<remote_build_client>: run command ok!

Rebuild target with remote mode

$ xmake -rv
<remote_build_client>: run xmake -rv in 192.168.56.110:90091 ..
[ 25%]: ccache compiling.release src/main.cpp
/usr/local/bin/ccache /usr/bin/xcrun -sdk macosx clang -c -Qunused-arguments -arch x86_64 -mmacosx-version-min=11.4 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG -o build/.objs/test/macosx/x86_64/release/src/main.cpp.o src/main.cpp
[ 50%]: linking.release test
"/usr/bin/xcrun -sdk macosx clang++" -o build/macosx/x86_64/release/test build/.objs/test/macosx/x86_64/release/src/main.cpp.o -arch x86_64 -mmacosx-version-min=11.4 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk -stdlib=libc++ -Wl,-x -lz
[100%]: build ok!
<remote_build_client>: run command ok!

Configure project with remote mode

$ xmake f --xxx --yy

Sync files to remote server

use git to commit code and run sync

$ git add .
$ git commit -a -m "..."
$ xmake service --sync 
<remote_build_client>: sync files in 192.168.56.110:90091 ..
Scanning files ..
Comparing 3 files ..
    [+]: src/main.cpp
    [+]: .gitignore
    [+]: xmake.lua
3 files has been changed!
Archiving files ..
Uploading files with 1372 bytes ..
<remote_build_client>: sync files ok!

Disconnect remote service for the current project

$ xmake service --disconnect
<remote_build_client>: disconnect 192.168.56.110:90091 ..
<remote_build_client>: disconnected!

View service logs

$ xmake service --logs

Clean all built and cache files for the current project on server

$ cd projectdir
$ xmake service --clean

TODO

  • Pull built target files to local host

@waruqi
Copy link
Member Author

waruqi commented Apr 15, 2022

I have removed git/ssh dependences

Some issues related to git/OpenSSH server on windows

https://github.com/PowerShell/Win32-OpenSSH/wiki/Setting-up-a-Git-server-on-Windows-using-Git-for-Windows-and-Win32_OpenSSH

Set powershell as default shell

Fix git push ruki@host:C:/xx/test failed, it will cause that sync files failure.

New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -PropertyType String -Force
New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShellCommandOption -Value "/c" -PropertyType String -Force

Set git/mingw64/bin to %PATH%

to solve git-receive-pack not found!

$gitPath = Join-Path -Path $env:ProgramFiles -ChildPath "git\mingw64\bin"
$machinePath = [Environment]::GetEnvironmentVariable('Path', 'MACHINE')
[Environment]::SetEnvironmentVariable('Path', "$gitPath;$machinePath", 'Machine')

FIXME

it still be failure when we run git push to windows server. : (

Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 8 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 1.25 KiB | 638.00 KiB/s, done.
Total 6 (delta 0), reused 0 (delta 0), pack-reused 0
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly

@waruqi
Copy link
Member Author

waruqi commented Apr 16, 2022

I have supported it.

@waruqi waruqi closed this as completed Apr 16, 2022
@xq114
Copy link
Contributor

xq114 commented Apr 18, 2022

Can I build the project on the remote machine, but run it on my local machine? How can I install the built binaries on localhost?

@waruqi
Copy link
Member Author

waruqi commented Apr 18, 2022

Can I build the project on the remote machine, but run it on my local machine? How can I install the built binaries on localhost?

Build project and run target on the remote machine. And we can also pull built target to the local machine (but I have not supported it yet.)

@waruqi
Copy link
Member Author

waruqi commented Apr 30, 2022

User authorization

At the moment I don't have time to do perfect encryption and authentication based on ssl, but I have implemented some basic user authorization authentication. On a LAN it should also avoid most of the security problems.

We can support two authentication modes

  • User password mode
  • Token mode

Token mode

Generate default configuration in server

We run xmake service to generate a default configuraion. It will generate an initial token.

$ xmake service
generating the config file to /Users/ruki/.xmake/service.conf ..
an token(4b928c7563a0cba10ff4c3f5ca0c8e24) is generated, we can use this token to connect servi
ce.
$ cat /Users/ruki/.xmake/service.conf
{
    logfile = "/Users/ruki/.xmake/service/logs.txt",
    remote_build = {
        client = {
            connect = "root@127.0.0.1:9691",
            token = "4b928c7563a0cba10ff4c3f5ca0c8e24"
        },
        server = {
            listen = "0.0.0.0:9691",
            workdir = "/Users/ruki/.xmake/service/remote_build"
        }
    },
    server = {
        tokens = {
            "4b928c7563a0cba10ff4c3f5ca0c8e24"
        },
        known_hosts = { }
    }
}

Add token in client

Then we add this token in the client configuraion.

$ cat /Users/ruki/.xmake/service.conf
{
    logfile = "/Users/ruki/.xmake/service/logs.txt",
    remote_build = {
        client = {
            connect = "root@127.0.0.1:9691",
            token = "4b928c7563a0cba10ff4c3f5ca0c8e24"
        }
    }
}

Connect to server

Now we can connect to server to build project.

$ xmake service --connect
<remote_build_client>: connect 127.0.0.1:9691 ..
<remote_build_client>: connected!
<remote_build_client>: sync files in 127.0.0.1:9691 ..
Scanning files ..
Comparing 3 files ..
    [+]: xmake.lua
    [+]: .gitignore
    [+]: src/main.cpp
3 files has been changed!
Archiving files ..
Uploading files with 1591 bytes ..
<remote_build_client>: sync files ok!

Generate a new token

We can also generate a new token for new client.

$ xmake service --gen-token
New token a7b9fc2d3bfca1472aabc38bb5f5d612 is generated!

Then we can copy and save this token to new client configuraion file.

User password mode

Add user in server

$ xmake service --add-user=ruki
Please input user ruki password:
123456
Add user ruki ok!
$ cat /Users/ruki/.xmake/service.conf
{
    logfile = "/Users/ruki/.xmake/service/logs.txt",
    remote_build = {
        client = {
            connect = "root@127.0.0.1:9691"
        },
        server = {
            listen = "0.0.0.0:9691",
            workdir = "/Users/ruki/.xmake/service/remote_build"
        }
    },
    server = {
        tokens = {
            "4b928c7563a0cba10ff4c3f5ca0c8e24",
            "7889e25402413e93fd37395a636bf942"
        },
        known_hosts = { }
    }
}

Remove user in server

$xmake service --rm-user=ruki
Please input user ruki password:
123456
Remove user ruki ok!

Configurate user in client

We can add user name to the connect address. eg. user@address:port

{
    remote_build = {
        client = {
            connect = "root@127.0.0.1:9691"
        }
  }
}

Connect to server

We need input user password before connecting server.

$ xmake service --connect
Please input user root password:
000000
<remote_build_client>: connect 127.0.0.1:9691 ..
<remote_build_client>: connected!
<remote_build_client>: sync files in 127.0.0.1:9691 ..
Scanning files ..
Comparing 3 files ..
    [+]: xmake.lua
    [+]: .gitignore
    [+]: src/main.cpp
3 files has been changed!
Archiving files ..
Uploading files with 1591 bytes ..
<remote_build_client>: sync files ok!

If password is incorrect, it will raise errors.

 xmake service --connect
Please input user root password:
123
<remote_build_client>: connect 127.0.0.1:9691 ..
<remote_build_client>: connect 127.0.0.1:9691 failed, user and password are incorrect!

Configure known hosts in server

We can also add known hosts in server to guarantee the connection of trusted hosts.

We can add them to the known_hosts = {} in service.conf

$ cat ~/.xmake/service.conf 
{
    logfile = "/Users/ruki/.xmake/service/logs.txt",
    server = {
        tokens = {
            "4b928c7563a0cba10ff4c3f5ca0c8e24"
        },
        known_hosts = { "127.0.0.1", "xx.xx.xx.xx"}
    }
}

If it is configured, then only hosts included in this list will be able to successfully connect to the server.

@waruqi
Copy link
Member Author

waruqi commented May 15, 2022

To better manage the configuration, I split service.conf into two configuration files, server.conf and client.conf

$ cat ~/.xmake/service/server.conf 
{
    known_hosts = { },
    logfile = "/Users/ruki/.xmake/service/server/logs.txt",
    remote_build = {
        listen = "0.0.0.0:9691",
        workdir = "/Users/ruki/.xmake/service/server/remote_build"
    },
    tokens = {
        "e438d816c95958667747c318f1532c0f"
    }
}
$ cat ~/.xmake/service/client.conf 
{
    remote_build = {
        connect = "127.0.0.1:9691",
        token = "e438d816c95958667747c318f1532c0f"
    }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants