[#1172] Mutation testing support for Solidity

[ameten]

No description provided.

[chriseth]

Thanks for the pull request! Unfortunately, this is a quite invasive change and I'm not sure how to best integrate it into the compiler.

[ameten]

@chriseth, yes, the change looks big, but the options other than "mutate" work as before, and it is possible to review per commit. Each commit should compile and keep compiler functional.

[ameten]

@chriseth, no chance for mutations being intergrated?

[axic]

@ameten I wonder how possible would it be doing this based on the JSON AST output?

[ameten]

@axis, do you mean to create another utility totally separated from solc?

[chriseth]

Yes, I think this is what he means. It would be great if this could be done in a more "plugin-ish" way.

[axic]

@ameten trying to understand if the outputs generated by Solidity (the JSON AST for example) would enable writing this tool outside of the compiler

[ameten]

@axic, yes, JSON AST can be used for mutation assuming that "src" defines the location of an operator. It is a possible approach but it has its drawbacks, though.

[chriseth]

I'm sorry, I know this was quite some work, but I have to close this now. If you want to talk about how te integrate these changes in another way, please don't hesitate to schedule a meeting.

[pieterhartel]

What is the status of the mutation project for solidity?

• Is there a prototype tool?
• documentation?
• plans?

I am asking because I have written a mutation tool and would like to share ideas and results.

--pieter

[chriseth]

We plan to add ast-import to the compiler, i.e. compile bytecode starting from an AST instead of starting from source code. That's all we have and plan. Please share your ideas and results! Maybe you could open a new issue to discuss? Maybe the discussion would also be better suited with some external testing tool.

[pieterhartel]

I'm not sure how to open a new issue, but I'm happy to share some issues (below). All of these apply to the LegacyAST. I have found work-arounds for all of the issues, but the resulting code is not very pretty.

1. The program text is assumed to be a long string and the src property of legacyAST nodes contains the beginning and end position of the substring that corresponds to the node. This makes it difficult to transform a program, because where would one put new text?
2. Semantic information is largely missing. For example whether a variable is used as an L-value, or whether an expression is a rational expression.
3. The legacyAST really only works for ASCII, and not for UTF8.
4. Predefined identifiers are not included in the AST. Since these vary, depending on the compiler version this is not convenient.
5. Comments are hidden in the src property of legacyAST nodes, hence it is not feasible to use the legacyAST for a comment stripper.
6. Older versions of Solidity may generate incomplete legacyAST. For example some attributes do not have the scope property.
7. The src property of some legacyAST nodes (e.g. return) contains a trailing semicolon, but this is not done consistently.

[chriseth]

Sorry for the late reply, let me answer on your individual points.

1. this is indeed a problem for us, but I think @djudjuu is working on a solution.
2. is this also the case for the current AST format? Can you maybe elaborate in a separate issue? https://github.com/ethereum/solidity/issues/new/choose
3. Same as above
4. I don't really have a solution to that.
5. A comment stripper should be a very easy to implement separate tool, isn't it?
6. Again, no solution to that.
7. Can you open a new issue with some example code, please?

[djudjuu]

@pieterhartel

I am working on the importAST feaure in this PR.
I think it is in a good-enough state to play aorund with, as it works for most simple contracts (imported contracts and inline-assembly still cause trouble). I'd be glad to help you out if you have questions.

re: legacyAST vs compactAST

If you haven't yet, I'd recommend that you have a look at that format. It is a lot more readable, and also complete for all kinds of programs. Also, the '''importAST''' function will only be available for the new version of the AST.
You get the new one like this:

solc --combined-json ast,compact-format --pretty-json

re: comment stripper

in the compact AST, comments are in their own "documentation" field, so a stripper should be easy to do.

[pieterhartel]

@chriseth @djudjuu Thanks for your efforts on compact AST import.

May I suggest to make a clear difference between a minimal set of properties that the compiler requires on AST input and a maximal set of properties that the compiler produces as AST output? A program transformation tool would then be able to use the rich output to generate new code, without actually having to do any analysis on the generated code, because the compiler can do this better. It would be convenient if the rich output could easily be stripped.

I will let you know when the paper on our mutation tool is finished. Our current tool works only on the legacy AST, so we will have to rewrite it for the compact AST.

[chriseth]

Yep, that is the idea, export happens after analysis, but import re-starts the compiler just after parsing, i.e. before analysis, and it ignores any analysis data present in the import.