The security aspects are separated into three different levels of trust, with a distinction in the protocol implemented for solidity bounty contracts with solidity solutions only, or allowing all languages.
Since the Ethereum blockchain is fully deterministic, there is no randomness possible that a malicious bounty hunter could not predict. For example, when setting a random seed, the malicious bounty hunter could download the Ethereum blockchain and simulate it locally to determine what the inputs are that will be generated by the randomness. By knowing what the inputs are going to be, the malicious bounty hunter can hardcode the expected answers without solving the challenge and cash the bounty. Possibly a random seed based on block time could be difficult to predict by the bounty hunter but no concrete solution for this problem was found/investigated thoroughly, hence this option is not implemented.
See concern above.
An example of this protocol is included in the main git, e.g. the pair of NQueens contracts. The bounty contract and sponsor contract are both soldidity. The randomness that generates the random inputs can be queries using VRF of Chainlink, this means the bounty hunter cannot predict the inputs and hence must actually solve the challenge. Concerns raised (and accompanying mitigation) are:
0.a But then the bounty hunter can just send a fake solution, get the randomness and hardcode it to resubmit.
0.b The sponsor contract is terminated after it is evaluated regardless of payout, hence the sponsor would have to create a new smart contract that queries new VFR.
1.a But based on 0.a, the bounty hunter could give the sponsor unwanted costs to make the sponsor have to set up the bounty contract over and over.
1.b That is why the bounty hunter must pay the fee to set up and evaluate the smartcontract of the bounty hunter to submit a solution (this can be compensated if the bounty is payed out succesfully).
2.a What if the sponsor sets up fake tests that do not pass on valid input to simply read of the solution in the smart contract of the bounty hunter without having to pay out?
2.b It is the responsibility of the bounty hunter to verify whether the unit tests are correct, if they are not, the bounty hunter should not work on it and not send a solution (if it is rational).
3.a So what stops the sponsor from writing out very difficult to understand contracts making it very difficult for bounty hunters to determine if the tests are right or not?
3.b Nothing, but it costs the sponsor money to deploy a bounty contract, and bounty hunters can simply not work on it if it is not well written.
This could be done with Ewasm and/or parity. The directly above concerns are valid.
Here, an oracle is used for randomnes, and oracles are used to read out a CI status of a github repository, and to read out file content or file checksums of files in github repositories. The concept is described here. Either the bounty hunter could pay for the billing of the CI (which grants the bounty hunter access to the CI). Best would be if the smart contract of the sponsor would be secretly able to generate a random password (e.g. through Chainlink VFR) to generate a temporary account at some CI, e.g. Travis-CI and use its own bounty/gas/money/ETH to pay for the billing. The latter option would eliminate the possibibility of the sponsor or bounty hunter to attack through tampering with the CI settings.
0.a One risk identified in advance is that the bounty hunter could possibly modify the settings within the shielded/private CI by setting evaluation time limit to 0 seconds or withdrawing all CI funds. 0.b But this is not a real risk, because the sponsor smart contract has funds and is its own agent, so it (theoretically) could set up the CI on its own, open source, deterministic and transparent.
1.a A malicious sponsor might want to disable the CI tests. 1.b The sponsor does not have access to the CI, so the employer can't hack the CI.
2.a The bounty hunter has access to the CI and can hence hack it, 2.b But only to break the test (which is not in the interest of the bounty hunter).
3.a E.g. when talking about Travis CI, the bounty hunter could trigger a malicious custom build that always passes the tests 3.b This build is not propagated to the pass flag in the GitHub pull request.
4.a The bounty hunter could maliciously modify the files in the pull request to make the tests falsely pass. 4.b Then the hash code/checksum of the smart contract of the employer would detect this attack and not pay out.
5.a Though I'm sure in the complex system of Travis-CI in a hacker could break it in practice. 5.b But that is because Travis is not defending against this. I think if they were (to facilitate this market), those attacks could be (relatively easily) mitigated by modifying Travis API for this market. So for a proof of concept I think this is fine.