diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000000..c7672f85e2 --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,58 @@ +# Community Code of Conduct + +In the IOTA community, participants from all over the world come together to create. This is made possible by the support, hard work and enthusiasm of thousands of people, including those who create and use the IOTA technology. + +This document offers some guidance to ensure IOTA participants can cooperate effectively in a positive and inspiring atmosphere, and to explain how together we can strengthen and support each other. + +This Code of Conduct is shared by all contributors and users who engage with the IOTA Foundation team and its community services. + +## Overview + +This Code of Conduct presents a summary of the shared values and “common sense” thinking in our community. The basic social ingredients that hold our project together include: + +- Being considerate +- Being respectful +- Being collaborative +- Being pragmatic +- Supporting others in the community +- Getting support from others in the community + +This Code of Conduct reflects the agreed standards of behavior for members of the IOTA community, in any social media platform, forum, mailing list, wiki, web site, discord channel, public meeting or private correspondence within the context of the IOTA Foundation team and the IOTA Tangle technology. The community acts according to the standards written down in this Code of Conduct and will defend these standards for the benefit of the community. Leaders of any group, such as moderators of social media groups, mailing lists, discord channels, forums, etc., will exercise the right to suspend access to any person who persistently breaks our shared Code of Conduct. + +## Be considerate + +Your actions and work will affect and be used by other people and you, in turn, will depend on the work and actions of others. Any decision you take will affect other community members, and we expect you to take those consequences into account when making decisions. + +As a user, remember that community members work hard on their part of IOTA and take great pride in it. + +## Be respectful + +In order for the IOTA community to stay healthy, its members must feel comfortable and accepted. Treating one another with respect is absolutely necessary for this. In a disagreement, in the first instance, assume that people mean well. + +We do not tolerate personal attacks, racism, sexism or any other form of discrimination. Disagreement is inevitable, from time to time, but respect for the views of others will go a long way to winning respect for your own view. Respecting other people, their work, their contributions and assuming well-meaning motivation will make community members feel comfortable and safe and will result in motivation and productivity. + +We expect members of our community to be respectful when dealing with other contributors, users, and communities. Remember that IOTA is an international project and that you may be unaware of important aspects of other cultures. + +## Be collaborative + +Your feedback is important, as is its form. Poorly thought out comments can cause pain and the demotivation of other community members, but considerate discussion of problems can bring positive results. An encouraging word works wonders. + +## Be pragmatic + +The IOTA community is pragmatic and fair. We value tangible results over having the last word in a discussion. We defend our core values like freedom and respectful collaboration, but we don’t let arguments about minor issues get in the way of achieving more important results. We are open to suggestions and welcome solutions regardless of their origin. When in doubt support a solution which helps to get things done over one which has theoretical merits, but isn’t being worked on. Use the tools and methods which help to get the job done. Let decisions be taken by those who do the work. + +## Support others in the community + +The IOTA community is made strong by mutual respect, collaboration and pragmatic, responsible behavior. Sometimes there are situations where this has to be defended and other community members need help. + +If you witness others being attacked, think first about how you can offer them personal support. If you feel that the situation is beyond your ability to help individually, go privately to the victim and ask if some form of official intervention is needed. + +When problems do arise, consider respectfully reminding those involved of our shared Code of Conduct as a first action. Leaders are defined by their actions and can help set a good example by working to resolve issues in the spirit of this Code of Conduct before they escalate. + +## Get support from others in the community + +Disagreements, both political and technical, happen all the time. Our community is no exception to the rule. The goal is not to avoid disagreements or differing views but to resolve them constructively. You should turn to the community to seek advice and to resolve disagreements and where possible consult the team most directly involved. + +Think deeply before turning a disagreement into a public dispute. If necessary, request mediation, and try to resolve differences in a less emotional medium. If you do feel that you or your work is being attacked, take your time to think things through before writing heated replies. Consider a 24-hour moratorium if emotional language is being used – a cooling-off period is sometimes all that is needed. If you really want to go a different way, then we encourage you to publish your ideas and your work, so that it can be tried and tested. + +This work, "IOTA Community Guidelines", is a derivative of the [Community code of conduct by ownCloud](https://owncloud.org/community/code-of-conduct/), used under [CC BY-SA 3.0](https://creativecommons.org/licenses/by-sa/3.0/). "IOTA Community Guidelines" is licensed under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/) by IOTA Foundation. \ No newline at end of file diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 70f4435bb3..8c06f1c68c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,40 +1,118 @@ -## How to contribute to IRI +# Contribute to IRI -#### **How should a Pull Request look like**? -Before creating a PR make sure you have read the entire contributing guideline. Ensure the PR description clearly describes the problem and solution by following the template. It is best to discuss your changes in an issue before creating the PR. A PR should address a **specific concern**. If it addresses more than one concern it **should be split**. Small PRs get merged in faster. Very large PRs are difficult to review and test. This makes them less likely to be merged. +This document describes how to contribute to IRI. -#### **Did you find a bug?** +We encourage everyone with knowledge of IOTA technology to contribute. -* **Do not open a GitHub issue if the bug is a security vulnerability - in IRI**, and instead, please contact us via [security@iota.org](mailto:security@iota.org). +Thanks! :heart: -* **Ensure the bug was not already reported** by searching on GitHub under [Issues](https://github.com/iotaledger/iri/issues). You can also look up related issues by a label. For example, if your issue is database-related, filter based on the `C-DB` label and look for related issues. If it's API related, use `C-API` and so on. `C` stands for component. Make sure you skim through the labels to find your category. +## Do you have a question? -* If you're unable to find an open issue addressing the problem, [open a new one](https://github.com/iotaledger/iri/issues/new). Be sure to include a **title and clear description**, and as much relevant information as possible. Make sure you follow our new issue template. The first part of the issue template is dedicated to reporting bugs, delete the second part. Steps to reproduce are particularly important. Add any relevant log output and screenshots. The easier it will be to reproduce your issue, the more likely it's getting fixed. +If you have a general or technical question, you can use one of the following resources instead of submitting an issue: -#### **Did you write a patch that fixes a bug?** +- [**Developer documentation:**](https://docs.iota.org/) For official information about developing with IOTA technology +- [**Discord:**](https://discord.iota.org/) For real-time chats with the developers and community members +- [**IOTA cafe:**](https://iota.cafe/) For technical discussions with the Research and Development Department at the IOTA Foundation +- [**StackExchange:**](https://iota.stackexchange.com/) For technical and troubleshooting questions -* [New issues](https://github.com/iotaledger/iri/issues/new) should be used for reporting issues. If you already wrote a patch for an issue that you or someone else reported, make sure you link it to the open issue. You can do this by mentioning the issue number in the comment or description on the PR, for example `#123` to link issue "123". +## Ways to contribute -#### **Do you intend to add a new feature or change an existing one?** +To contribute to IRI on GitHub, you can: -* Suggest your change as a [new issues](https://github.com/iotaledger/iri/issues/new). Use the second part of the issue template, dedicated to new feature requests and delete the first part. If you plan on implementing the change yourself, start writing the code and submit a PR. Note that the fact that the feature is "code ready" doesn't mean it will get merged. It is advisable to gather feedback for the change first. You are encouraged to start and drive a discussion on [Discord](https://discord.iota.org/). The IRI team also monitors the repo and will provide feedback where relevant. +- Report a bug +- Suggest a new feature +- Build a new feature +- Contribute to the documentation -#### **Want to write a regression test for your feature? Or to any other feature that needs a test?** +## Report a bug -* Please refer to our current [regression tests](https://github.com/iotaledger/iri/tree/dev/python-regression). +This section guides you through reporting a bug. Following these guidelines helps maintainers and the community understand the bug, reproduce the behavior, and find related bugs. +### Before reporting a bug -#### **Do you have questions about the functioning of IRI nodes, the network or anything IOTA related?** +Please check the following list: -* Use our [Discord](https://discordapp.com/invite/fNGZXvh) to join the discussion and ask questions. +- **Do not open a GitHub issue for [security vulnerabilities](SECURITY.MD)**, instead, please contact us at [security@iota.org](mailto:security@iota.org). -#### **Do you want to contribute to the IRI documentation?** +- **Ensure the bug was not already reported** by searching on GitHub under [**Issues**](https://github.com/iotaledger/iri/issues). If the bug has already been reported **and the issue is still open**, add a comment to the existing issue instead of opening a new one. You can also find related issues by their [label](https://github.com/iotaledger/iri/labels?page=1&sort=name-asc). For example, if your issue is database related, filter issues based on the `C-DB` label to look for related ones. `C` stands for component. -The IOTA documentation is based in the [docs](https://github.com/iotaledger/docs) repo. You are welcome to contribute. Make sure you follow the instructions on adding new docs. +**Note:** If you find a **Closed** issue that seems similar to what you're experiencing, open a new issue and include a link to the original issue in the body of your new one. -Thanks! :heart: :heart: :heart: +### Submitting A Bug Report -The contribution guidelines are inspired by the Ruby on Rails contribution guidelines. +To report a bug, [open a new issue](https://github.com/iotaledger/iri/issues/new), and be sure to include as many details as possible, using the template. -IRI Team +**Note:** Minor changes such as fixing a typo can but do not need an open issue. + +If you also want to fix the bug, submit a [pull request](#pull-requests) and reference the issue. + +## Suggest a new feature + +This section guides you through suggesting a new feature. Following these guidelines helps maintainers and the community collaborate to find the best possible way forward with your suggestion. + +### Before suggesting a new feature + +**Ensure the feature has not already been suggested** by searching on GitHub under [**Issues**](https://github.com/iotaledger/iri/issues). + +### Suggesting a new feature + +To suggest a new feature, talk to the IOTA community and IOTA Foundation members in the #iri-discussion channel on [Discord](https://discord.iota.org/). + +If the IRI team approves your feature, the team will create an issue for it. + +## Build a new feature + +This section guides you through building a new feature. Following these guidelines helps give your feature the best chance of being approved and merged. + +### Before building a new feature + +Make sure to discuss the feature in the #iri-discussion channel on [Discord](https://discord.iota.org/). + +Otherwise, your feature may not be approved at all. + +### Building a new feature + +To build a new feature, check out a new branch based on the `dev` branch, and be sure to consider the following: + +- If the feature has a public facing API, make sure to document it, using Javadoc code comments + +- Where necessary, please write regression tests for your feature. Refer to our current [regression tests](https://github.com/iotaledger/iri/tree/dev/python-regression) for guidance. + +## Contribute to the IRI documentation + +The IRI documentation is hosted on https://docs.iota.org, which is built from content in the [documentation](https://github.com/iotaledger/documentation) repository. + +Please see the [guidelines](https://github.com/iotaledger/documentation/CONTRIBUTING.md) on the documentation repository for information on how to contribute to the documentation. + +## Pull requests + +This section guides you through submitting a pull request (PR). Following these guidelines helps give your PR the best chance of being approved and merged. + +### Before submitting a pull request + +When creating a pull request, please follow these steps to have your contribution considered by the maintainers: + +- A pull request should have exactly one concern (for example one feature or one bug). If a PR address more than one concern, it should be split into two or more PRs. + +- A pull request can be merged only if it references an open issue + + **Note:** Minor changes such as fixing a typo can but do not need an open issue. + +- All code should be well tested and follow the [code styleguide](STYLEGUIDE.md) + +### Submitting a pull request + +The following is a typical workflow for submitting a new pull request: + +1. Fork this repository +2. Create a new branch based on your fork +3. Commit changes and push them to your fork +4. Create a pull request against the `dev` branch + +If all [status checks](https://help.github.com/articles/about-status-checks/) pass, and the maintainer approves the PR, it will be merged. + +**Note:** Reviewers may ask you to complete additional work, tests, or other changes before your pull request can be approved and merged. + +## Code of Conduct + +This project and everyone participating in it is governed by the [IOTA Code of Conduct](CODE_OF_CONDUCT.md). \ No newline at end of file diff --git a/python-regression/README.md b/python-regression/README.md index ed5e49d535..b9d2e96fc7 100644 --- a/python-regression/README.md +++ b/python-regression/README.md @@ -19,37 +19,39 @@ pip install -e . ``` ### Available Tests -Machine 1 - API Tests: This machine uses 2 nodes and tests each of the api calls, and ensures that the responses are -the expected values +Machine 1 - Local Snapshotting Tests: This machine uses 4 nodes. The first node contains the snapshot `meta` and `state` +files, the `spent-addresses-db` and `testnetdb` of a synced node. The second only contains the database, and the third +only contains the snapshot files. All three of these nodes are tested to ensure that they solidify to the same point, +and that the proper information is contained in the snapshot files and databases. The fourth node has a larger database +that contains more milestones than the `local-snapshots-pruning-depth`. This node is checked to make sure that after +starting, transactions that should be pruned from the database have been pruned correctly. This machine also includes +tests for spent addresses including a test for exporting and merging spent addresses using IXI +modules. -Machine 2 - Transaction Tests: This machine uses 2 nodes. Several zero value transactions are sent to the first node, +Machine 2 - Blowball Tests: This machine uses 6 nodes by default, but can be customized to be performed on as many/few +as desired. 1000 `getTransactionsToApprove` calls are made across these nodes, and the responses checked to make sure +that less than 5% of the results are milestone transactions. If the responses are over this threshold, then that means +blowballs are occurring. + + +Machine 3 - Transaction Tests: This machine uses 2 nodes. Several zero value transactions are sent to the first node, as well as a milestone transaction. Then node two is checked to make sure the transactions are all confirmed. After these transactions are resolved, the same approach is used to ensure that value transactions are also being confirmed correctly. -Machine 3 - Blowball Tests: This machine uses 6 nodes by default, but can be customized to be performed on as many/few -as desired. 1000 `getTransactionsToApprove` calls are made across these nodes, and the responses checked to make sure -that less than 5% of the results are milestone transactions. If the responses are over this threshold, then that means -blowballs are occurring. +Machine 4 - API Tests: This machine uses 2 nodes and tests each of the api calls, and ensures that the responses are +the expected values -Machine 4 - Stitching Tests: This machine uses 1 node. The node is loaded with a db containing a large side tangle. A +Machine 5 - Stitching Tests: This machine uses 1 node. The node is loaded with a db containing a large side tangle. A stitching transaction is issued, and another transaction referencing that one is issued. After these transactions are issued, making `getTransactionsToApprove` calls should not crash the node. -Machine 5 - Milestone Validation Tests: This machine uses 2 nodes. Both nodes are loaded with the same db. The db +Machine 6 - Milestone Validation Tests: This machine uses 2 nodes. Both nodes are loaded with the same db. The db contains several signed milestone transactions, and several unsigned transactions. One node is set to validate the testnet coordinator signature, while the other is not. The one that requires validation should solidify to one point, while the other should solidify further. -Machine 6 - Local Snapshotting Tests: This machine uses 4 nodes. The first node contains the snapshot `meta` and `state` -files, the `spent-addresses-db` and `testnetdb` of a synced node. The second only contains the database, and the third -only contains the snapshot files. All three of these nodes are tested to ensure that they solidify to the same point, -and that the proper information is contained in the snapshot files and databases. The fourth node has a larger database -that contains more milestones than the `local-snapshots-pruning-depth`. This node is checked to make sure that after -starting, transactions that should be pruned from the database have been pruned correctly. This machine also includes -tests for spent addresses including a test for exporting and merging spent addresses using IXI -modules. - +__*Note:*__ _The db's used for these tests have been documented below for reference_ ### Running Tests Locally @@ -137,7 +139,7 @@ iri --/tests ---/features ----/machine1 [Same structure for other machines] ------/1_api_tests.feature +-----/4_api_tests.feature -----/config.yml -----/output.yml ``` @@ -151,7 +153,7 @@ From the `iri/python-regression` directory, a test can be run using the followin ``` i.e. For the api tests: ``` -aloe 1_api_tests.feature -w ./tests/features/machine1/ -v --nologcapture +aloe 4_api_tests.feature -w ./tests/features/machine1/ -v --nologcapture ``` @@ -168,12 +170,64 @@ When running the aloe command, you can add the `-a` flag to register that you wo the given attribute. Inversely you can also run all tests that do not contain that flag by using `!`. This is shown below: ``` -aloe 1_api_tests.feature -w ./tests/features/machine1 -a getNodeInfo +aloe 4_api_tests.feature -w ./tests/features/machine1 -a getNodeInfo ``` or to not run the flagged tests: ``` -aloe 1_api_tests.feature -w ./tests/features/machine1 -a '!getNodeInfo' +aloe 4_api_tests.feature -w ./tests/features/machine1 -a '!getNodeInfo' ``` _Note: To negate running the tests using the flag requires the `!` and flag to be wrapped in parentheses as shown above_ The same flag can be used for several scenarios, and they will all either be included or negated by this flag. + + +### _*DB Descriptions:*_ +##### _Machine 1_ +https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/LS_Test_Db_With_LS_Db.tar.gz - Full LocalSnapshot test db +synced to milestone 10321 and a local snapshot at 10220. Contains mostly 0 value transactions with a few spends early on +to generate the localsnapshot-db as well as to provide transactions for pruning reference + +https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/LS_Test_DB_and_Snapshot.tar - Full LocalSnapshot test db +synced to milestone 10321 without any localsnapshots-db. + +https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/LS_Test_LS_Db.tar.gz - No LocalSnapshot test db provided, +instead the localsnapshot-db from index 10220 is provided instead. + +https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/PruningTestDB.tar - A full db synced to milestone 15000 +containing a mix of value and 0 value transactions that will be pruned once the node takes its snapshot. This is used +to ensure that any transaction below the threshold is pruned correctly. + +https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/SpentAddressesTestDB.tar - A full db synced to milestone +10100 with a mix of value and 0 value transactions that will be pruned once the node takes its snapshot. This is used +to ensure that spent addresses are persisted in the local snapshot data after pruning occurs. + + +##### _Machine 2_ +https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/Blowball_Tests_db.tar - A full db synced to milestone 27. +There are several tips surrounding the last milestone, and the test is used to ensure that new transactions aren't +attaching en masse to the last milestone present. + + +##### _Machine 3_ +https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/Transactions_Tests_db.tar - A small db synced to milestone +50 with a snapshot file containing a list of addresses preset for value spending. This DB is used to test value and non +value transactions and their inclusion in the next milestone. + + +##### _Machine 4_ +https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/testnet_files.tgz - A full db synced to milestone 8412 used +for testing basic api commands. + + +##### _Machine 5_ +https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/Stitching_tests_db.tar - A full db synced to milestone 37 +with a large subtangle that is building beside the main tangle. This db is used to test the success or failure of +stitching this subtangle back into the original. + + +##### _Machine 6_ +https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/Validation_tests_db.tar - A full db containing 2 separate +synchronisation points. The first point is hit at milestone 37, where the last milestone issued with valid signatures is +present. The db contains several more milestones that have been attached without a valid signature. The test will sync +to 37 if a valid signature is required and 45 if not. + diff --git a/python-regression/tests/features/machine6/6_local_snapshots_tests.feature b/python-regression/tests/features/machine1/1_local_snapshots_tests.feature similarity index 70% rename from python-regression/tests/features/machine6/6_local_snapshots_tests.feature rename to python-regression/tests/features/machine1/1_local_snapshots_tests.feature index 11ea4a12d1..31a6236f1e 100644 --- a/python-regression/tests/features/machine6/6_local_snapshots_tests.feature +++ b/python-regression/tests/features/machine1/1_local_snapshots_tests.feature @@ -10,13 +10,13 @@ Feature: Test Bootstrapping With LS Check that the permanode has been started correctly and is synced. #First make sure nodes are neighbored - Given "nodeA-m6" and "nodeB-m6" are neighbors - And "nodeA-m6" and "nodeC-m6" are neighbors + Given "nodeA-m1" and "nodeB-m1" are neighbors + And "nodeA-m1" and "nodeC-m1" are neighbors #Default for test is to issue 10322 - When milestone 10322 is issued on "nodeA-m6" + When milestone 10322 is issued on "nodeA-m1" And we wait "30" second/seconds - Then "nodeA-m6" is synced up to milestone 10322 + Then "nodeA-m1" is synced up to milestone 10322 Scenario: DB node is synced, and files contain expected values @@ -24,21 +24,21 @@ Feature: Test Bootstrapping With LS stored correctly. #First make sure nodes are neighbored - Given "nodeB-m6" and "nodeA-m6" are neighbors - And "nodeB-m6" and "nodeC-m6" are neighbors + Given "nodeB-m1" and "nodeA-m1" are neighbors + And "nodeB-m1" and "nodeC-m1" are neighbors # Default for test is to issue 10323 - When milestone 10323 is issued on "nodeA-m6" + When milestone 10323 is issued on "nodeA-m1" #Give the node time to finish syncing properly, then make sure that the node is synced to the latest milestone. And we wait "30" second/seconds - Then "nodeB-m6" is synced up to milestone 10323 - And A local snapshot was taken on "nodeB-m6" at index 10220 + Then "nodeB-m1" is synced up to milestone 10323 + And A local snapshot was taken on "nodeB-m1" at index 10220 - When reading the local snapshot state on "nodeB-m6" returns with: + When reading the local snapshot state on "nodeB-m1" returns with: |keys |values |type | |address |LS_TEST_STATE_ADDRESSES |staticValue | - And reading the local snapshot metadata on "nodeB-m6" returns with: + And reading the local snapshot metadata on "nodeB-m1" returns with: |keys |values |type | |hashes |LS_TEST_MILESTONE_HASHES |staticValue | @@ -47,14 +47,14 @@ Feature: Test Bootstrapping With LS Check that the node started with just a LS DB is synced correctly. #First make sure nodes are neighbored - Given "nodeC-m6" and "nodeA-m6" are neighbors - And "nodeC-m6" and "nodeB-m6" are neighbors + Given "nodeC-m1" and "nodeA-m1" are neighbors + And "nodeC-m1" and "nodeB-m1" are neighbors #Default for test is to issue 10324 - When milestone 10324 is issued on "nodeA-m6" + When milestone 10324 is issued on "nodeA-m1" #Give the node time to finish syncing properly, then make sure that the node is synced to the latest milestone. And we wait "120" second/seconds - Then "nodeC-m6" is synced up to milestone 10324 + Then "nodeC-m1" is synced up to milestone 10324 Scenario: Check DB for milestone hashes @@ -62,12 +62,12 @@ Feature: Test Bootstrapping With LS are present in the new node. #First make sure nodes are neighbored - Given "nodeC-m6" and "nodeA-m6" are neighbors + Given "nodeC-m1" and "nodeA-m1" are neighbors #Default for test is to issue 10325 - When milestone 10325 is issued on "nodeA-m6" + When milestone 10325 is issued on "nodeA-m1" And we wait "30" second/seconds - Then "checkConsistency" is called on "nodeC-m6" with: + Then "checkConsistency" is called on "nodeC-m1" with: |keys |values |type | |tails |LS_TEST_MILESTONE_HASHES |staticValue | @@ -80,23 +80,39 @@ Feature: Test Bootstrapping With LS Takes a node with a large db and transaction pruning enabled, and checks to make sure that the transactions below the pruning depth are no longer present. - Given "checkConsistency" is called on "nodeD-m6" with: + Given "getInclusionStates" is called on "nodeD-m1" with: |keys |values |type | - |tails |LS_PRUNED_TRANSACTIONS |staticValue | + |transactions |LS_PRUNED_TRANSACTIONS |staticValue | + |tips |LS_PRUNING_TIP |staticList | - Then the response for "checkConsistency" should return null + And the response for "getInclusionStates" should return with: + |keys |values |type | + |states |True |boolList | + + # Trigger pruning and wait for it to finish + When the next 10 milestones are issued + And we wait "15" second/seconds + + And "getInclusionStates" is called on "nodeD-m1" with: + |keys |values |type | + |transactions |LS_PRUNED_TRANSACTIONS |staticValue | + |tips |LS_PRUNING_TIP |staticList | + + Then the response for "getInclusionStates" should return with: + |keys |values |type | + |states |False |boolList | Scenario: Check unconfirmed transaction is spent from Issues a value transaction that will be unconfirmed, and check that the address was spent from. - Given a transaction is generated and attached on "nodeE-m6" with: + Given a transaction is generated and attached on "nodeE-m1" with: |keys |values |type | |address |TEST_ADDRESS |staticValue | |value |10 |int | |seed |UNCONFIRMED_TEST_SEED |staticValue | - When "wereAddressesSpentFrom" is called on "nodeE-m6" with: + When "wereAddressesSpentFrom" is called on "nodeE-m1" with: |keys |values |type | |addresses |UNCONFIRMED_TEST_ADDRESS |staticValue | @@ -110,7 +126,7 @@ Feature: Test Bootstrapping With LS transaction has been pruned from the DB. # Check that addresses were spent from before pruning - Given "wereAddressesSpentFrom" is called on "nodeE-m6" with: + Given "wereAddressesSpentFrom" is called on "nodeE-m1" with: |keys |values |type | |addresses |LS_SPENT_ADDRESSES |staticValue | @@ -122,7 +138,7 @@ Feature: Test Bootstrapping With LS When the next 30 milestones are issued # Check that addresses were spent after transaction have been pruned - And "wereAddressesSpentFrom" is called on "nodeE-m6" with: + And "wereAddressesSpentFrom" is called on "nodeE-m1" with: |keys |values |type | |addresses |LS_SPENT_ADDRESSES |staticValue | @@ -131,7 +147,7 @@ Feature: Test Bootstrapping With LS |addresses |True |boolList | # Check that transactions from those addresses were pruned - And "getTrytes" is called on "nodeE-m6" with: + And "getTrytes" is called on "nodeE-m1" with: |keys |values |type | |hashes |LS_SPENT_TRANSACTIONS |staticValue | diff --git a/python-regression/tests/features/machine1/config.yml b/python-regression/tests/features/machine1/config.yml index 029b6ce6a8..0afa883fc4 100644 --- a/python-regression/tests/features/machine1/config.yml +++ b/python-regression/tests/features/machine1/config.yml @@ -1,26 +1,71 @@ -defaults: &api_tests_config_files - db: https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/testnet_files.tgz - db_checksum: 6eaa06d5442416b7b8139e337a1598d2bae6a7f55c2d9d01f8c5dac69c004f75 - iri_args: ['--testnet-coordinator', - 'BTCAAFIH9CJIVIMWFMIHKFNWRTLJRKSTMRCVRE9CIP9AEDTOULVFRHQZT9QAQBZXXAZGBNMVOOKTKAXTB', - '--milestone-start', - '0', - '--snapshot', - './snapshot.txt', - '--testnet-no-coo-validation', - 'true', - '--testnet', - 'true' +default_args: &args + ['--testnet-coordinator', + 'EFPNKGPCBXXXLIBYFGIGYBYTFFPIOQVNNVVWTTIYZO9NFREQGVGDQQHUUQ9CLWAEMXVDFSSMOTGAHVIBH', + '--mwm', + '1', + '--milestone-start', + '0', + '--testnet-no-coo-validation', + 'true', + '--testnet', + 'true', + '--snapshot', + './snapshot.txt', + '--local-snapshots-pruning-enabled', + 'true', + '--local-snapshots-pruning-delay', + '10000', + '--remote', + 'true', + '--remote-limit-api', + '""' ] - java_options: -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=n -javaagent:/opt/jacoco/lib/jacocoagent.jar=destfile=/iri/jacoco.exec,output=file,append=true,dumponexit=true -seeds: # For internal use by the regression system. - - SEED - - SIID +default_ixi: &ixi + ['IXI/LocalSnapshots.ixi'] + +java_options: -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=n -javaagent:/opt/jacoco/lib/jacocoagent.jar=destfile=/iri/jacoco.exec,output=file,append=true,dumponexit=true + +defaults: &db_full + db: https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/LS_Test_Db_With_LS_Db.tar.gz + db_checksum: 2055406bf312136d7cd0efa21248bd8cc9c407ab14ef0d18b921cf18c72c5270 + iri_args: *args + ixis: *ixi + +db_with_snapshot: &db_with_snapshot + db: https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/LS_Test_DB_and_Snapshot.tar + db_checksum: eabb81b0570a20e8d1c65c3d29e4b4e723de537ebca0eada536e3155d5a96972 + iri_args: *args + ixis: *ixi + +db_with_ls_db: &db_with_ls_db + db: https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/LS_Test_LS_Db.tar.gz + db_checksum: d217729fd5efb0432d179ec59472f283cd61e8ad4ca9aab32e5c1f82632a1a29 + iri_args: *args + ixis: *ixi + +db_for_pruning: &db_for_pruning + db: https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/PruningTestDB.tar + db_checksum: 15122ba80c0a03dc5b6b4186e5d880d0a1a15b5a6de48bafe4002c4c9b682221 + iri_args: *args + +db_for_spent_addresses: &db_for_spent_addresses + db: https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/SpentAddressesTestDB.tar + db_checksum: 7e15b2cbc76585d6483668cb1709201daa71314e7d488d9e7d71d7052479e73e + iri_args: *args nodes: nodeA-m1: #name - <<: *api_tests_config_files - + <<: *db_full + nodeB-m1: - <<: *api_tests_config_files + <<: *db_with_snapshot + + nodeC-m1: + <<: *db_with_ls_db + + nodeD-m1: + <<: *db_for_pruning + + nodeE-m1: + <<: *db_for_spent_addresses diff --git a/python-regression/tests/features/machine1/output.yml b/python-regression/tests/features/machine1/output.yml index e84b1543bc..27aaa66771 100644 --- a/python-regression/tests/features/machine1/output.yml +++ b/python-regression/tests/features/machine1/output.yml @@ -1,5 +1,5 @@ nodes: - nodeA: + nodeA-m1: host: localhost podip: localhost ports: @@ -10,7 +10,7 @@ nodes: api: 14265 gossip-tcp: 15600 zmq-feed: 5556 - nodeB: + nodeB-m1: host: localhost podip: localhost ports: @@ -20,4 +20,26 @@ nodes: clusterip_ports: api: 15265 gossip-tcp: 15605 - zmq-feed: 6556 \ No newline at end of file + zmq-feed: 6556 + nodeC-m1: + host: localhost + podip: localhost + ports: + api: 14265 + gossip-tcp: 15600 + zmq-feed: 5556 + clusterip_ports: + api: 14265 + gossip-tcp: 15600 + zmq-feed: 5556 + nodeD-m1: + host: localhost + podip: localhost + ports: + api: 14265 + gossip-tcp: 15600 + zmq-feed: 5556 + clusterip_ports: + api: 14265 + gossip-tcp: 15600 + zmq-feed: 5556 diff --git a/python-regression/tests/features/machine3/3_blowball_tests.feature b/python-regression/tests/features/machine2/2_blowball_tests.feature similarity index 93% rename from python-regression/tests/features/machine3/3_blowball_tests.feature rename to python-regression/tests/features/machine2/2_blowball_tests.feature index 2b6185b631..403194743a 100644 --- a/python-regression/tests/features/machine3/3_blowball_tests.feature +++ b/python-regression/tests/features/machine2/2_blowball_tests.feature @@ -10,7 +10,7 @@ Feature: Test GTTA for blowballs |keys |values |type | |depth |3 |int | - And "findTransactions" is called on "nodeA-m3" with: + And "findTransactions" is called on "nodeA-m2" with: |keys |values |type | |addresses |TEST_BLOWBALL_COO |staticList | diff --git a/python-regression/tests/features/machine2/config.yml b/python-regression/tests/features/machine2/config.yml index 8eda2fe8a4..dcd4171e0a 100644 --- a/python-regression/tests/features/machine2/config.yml +++ b/python-regression/tests/features/machine2/config.yml @@ -1,19 +1,39 @@ -defaults: &transaction_tests_config_files - db: https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/Transactions_Tests_db.tar - db_checksum: 756237276479da4b01deaa0c1211ca65a4c8ec6f081452ea7e8153648c53bd67 +defaults: &blowball_tests_config_files + db: https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/Blowball_Tests_db.tar + db_checksum: cf4da9fef58f74d8721eee7e2726f3321ecc2257ef111e6b8b2f4c348b40c980 iri_args: ['--testnet-coordinator', 'EFPNKGPCBXXXLIBYFGIGYBYTFFPIOQVNNVVWTTIYZO9NFREQGVGDQQHUUQ9CLWAEMXVDFSSMOTGAHVIBH', + '--milestone-keys', + '18', '--milestone-start', - '0', + '2', '--testnet-no-coo-validation', 'true', '--testnet', - 'true', - '--snapshot', - './snapshot.txt' + 'true' ] java_options: -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=n -javaagent:/opt/jacoco/lib/jacocoagent.jar=destfile=/iri/jacoco.exec,output=file,append=true,dumponexit=true +seeds: # For internal use by the regression system. + - SEED + - SIID + nodes: nodeA-m2: #name - <<: *transaction_tests_config_files + <<: *blowball_tests_config_files + + nodeB-m2: + <<: *blowball_tests_config_files + + nodeC-m2: + <<: *blowball_tests_config_files + + nodeD-m2: + <<: *blowball_tests_config_files + + nodeE-m2: + <<: *blowball_tests_config_files + + nodeF-m2: + <<: *blowball_tests_config_files + diff --git a/python-regression/tests/features/machine2/output.yml b/python-regression/tests/features/machine2/output.yml index 4afb135f97..b15b42d3c4 100644 --- a/python-regression/tests/features/machine2/output.yml +++ b/python-regression/tests/features/machine2/output.yml @@ -11,6 +11,50 @@ nodes: gossip-tcp: 15600 zmq-feed: 5556 nodeB-m2: + host: localhost + podip: localhost + ports: + api: 15265 + gossip-tcp: 15605 + zmq-feed: 6556 + clusterip_ports: + api: 15265 + gossip-tcp: 15605 + zmq-feed: 6556 + nodeC-m2: + host: localhost + podip: localhost + ports: + api: 15265 + gossip-tcp: 15605 + zmq-feed: 6556 + clusterip_ports: + api: 15265 + gossip-tcp: 15605 + zmq-feed: 6556 + nodeD-m2: + host: localhost + podip: localhost + ports: + api: 15265 + gossip-tcp: 15605 + zmq-feed: 6556 + clusterip_ports: + api: 15265 + gossip-tcp: 15605 + zmq-feed: 6556 + nodeE-m2: + host: localhost + podip: localhost + ports: + api: 15265 + gossip-tcp: 15605 + zmq-feed: 6556 + clusterip_ports: + api: 15265 + gossip-tcp: 15605 + zmq-feed: 6556 + nodeF-m2: host: localhost podip: localhost ports: diff --git a/python-regression/tests/features/machine2/2_transaction_tests.feature b/python-regression/tests/features/machine3/3_transaction_tests.feature similarity index 90% rename from python-regression/tests/features/machine2/2_transaction_tests.feature rename to python-regression/tests/features/machine3/3_transaction_tests.feature index 18e275280f..2d1f7dc4bc 100644 --- a/python-regression/tests/features/machine2/2_transaction_tests.feature +++ b/python-regression/tests/features/machine3/3_transaction_tests.feature @@ -5,7 +5,7 @@ Feature: Test transaction confirmation A milestone will be issued that references these transactions, and this should confirm the transations. - Given "10" transactions are issued on "nodeA-m2" with: + Given "10" transactions are issued on "nodeA-m3" with: |keys |values |type | |address |TEST_ADDRESS |staticValue | |value |0 |int | @@ -19,7 +19,7 @@ Feature: Test transaction confirmation #Give the node 10 seconds to solidify the milestone And we wait "15" second/seconds - Then "getInclusionStates" is called on "nodeA-m2" with: + Then "getInclusionStates" is called on "nodeA-m3" with: |keys |values |type | |transactions |evaluate_and_send |responseValue | |tips |latestMilestone |configValue | @@ -29,12 +29,12 @@ Feature: Test transaction confirmation | states | True True True True True True True True True True | boolListMixed | - When a transaction is generated and attached on "nodeA-m2" with: + When a transaction is generated and attached on "nodeA-m3" with: | keys | values | type | | address | TEST_ADDRESS | staticValue | | value | 0 | int | - And "getInclusionStates" is called on "nodeA-m2" with: + And "getInclusionStates" is called on "nodeA-m3" with: | keys | values | type | | transactions | TEST_STORE_ADDRESS | staticList | | tips | latestMilestone | configValue | @@ -49,7 +49,7 @@ Feature: Test transaction confirmation A milestone will be issued that references these transactions, and this should confirm the transations. - Given "10" transactions are issued on "nodeA-m2" with: + Given "10" transactions are issued on "nodeA-m3" with: |keys |values |type | |seed |THE_BANK |staticList | |address |TEST_ADDRESS |staticValue | @@ -64,21 +64,21 @@ Feature: Test transaction confirmation #Give the node time to solidify the milestone And we wait "15" second/seconds - Then "getInclusionStates" is called on "nodeA-m2" with: + Then "getInclusionStates" is called on "nodeA-m3" with: |keys |values |type | |transactions |evaluate_and_send |responseValue | |tips |latestMilestone |configValue | And the response for "getInclusionStates" should return with: |keys |values |type | - | states | True True True True True True True True True False | boolListMixed | + | states | True True True True True True True True True True | boolListMixed | - When a transaction is generated and attached on "nodeA-m2" with: + When a transaction is generated and attached on "nodeA-m3" with: | keys | values | type | | address | TEST_ADDRESS | staticValue | | value | 0 | int | - And "getInclusionStates" is called on "nodeA-m2" with: + And "getInclusionStates" is called on "nodeA-m3" with: | keys | values | type | | transactions | TEST_STORE_ADDRESS | staticList | | tips | latestMilestone | configValue | diff --git a/python-regression/tests/features/machine3/config.yml b/python-regression/tests/features/machine3/config.yml index 0b5c3f5823..194bc1b2b0 100644 --- a/python-regression/tests/features/machine3/config.yml +++ b/python-regression/tests/features/machine3/config.yml @@ -1,39 +1,19 @@ -defaults: &blowball_tests_config_files - db: https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/Blowball_Tests_db.tar - db_checksum: cf4da9fef58f74d8721eee7e2726f3321ecc2257ef111e6b8b2f4c348b40c980 +defaults: &transaction_tests_config_files + db: https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/Transactions_Tests_db.tar + db_checksum: 756237276479da4b01deaa0c1211ca65a4c8ec6f081452ea7e8153648c53bd67 iri_args: ['--testnet-coordinator', 'EFPNKGPCBXXXLIBYFGIGYBYTFFPIOQVNNVVWTTIYZO9NFREQGVGDQQHUUQ9CLWAEMXVDFSSMOTGAHVIBH', - '--milestone-keys', - '18', '--milestone-start', - '2', + '0', '--testnet-no-coo-validation', 'true', '--testnet', - 'true' + 'true', + '--snapshot', + './snapshot.txt' ] java_options: -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=n -javaagent:/opt/jacoco/lib/jacocoagent.jar=destfile=/iri/jacoco.exec,output=file,append=true,dumponexit=true -seeds: # For internal use by the regression system. - - SEED - - SIID - nodes: nodeA-m3: #name - <<: *blowball_tests_config_files - - nodeB-m3: - <<: *blowball_tests_config_files - - nodeC-m3: - <<: *blowball_tests_config_files - - nodeD-m3: - <<: *blowball_tests_config_files - - nodeE-m3: - <<: *blowball_tests_config_files - - nodeF-m3: - <<: *blowball_tests_config_files - + <<: *transaction_tests_config_files diff --git a/python-regression/tests/features/machine3/output.yml b/python-regression/tests/features/machine3/output.yml index 6e8258dd8c..95f32d9408 100644 --- a/python-regression/tests/features/machine3/output.yml +++ b/python-regression/tests/features/machine3/output.yml @@ -1,5 +1,5 @@ nodes: - nodeA: + nodeA-m3: host: localhost podip: localhost ports: @@ -10,51 +10,7 @@ nodes: api: 14265 gossip-tcp: 15600 zmq-feed: 5556 - nodeB: - host: localhost - podip: localhost - ports: - api: 15265 - gossip-tcp: 15605 - zmq-feed: 6556 - clusterip_ports: - api: 15265 - gossip-tcp: 15605 - zmq-feed: 6556 - nodeC: - host: localhost - podip: localhost - ports: - api: 15265 - gossip-tcp: 15605 - zmq-feed: 6556 - clusterip_ports: - api: 15265 - gossip-tcp: 15605 - zmq-feed: 6556 - nodeD: - host: localhost - podip: localhost - ports: - api: 15265 - gossip-tcp: 15605 - zmq-feed: 6556 - clusterip_ports: - api: 15265 - gossip-tcp: 15605 - zmq-feed: 6556 - nodeE: - host: localhost - podip: localhost - ports: - api: 15265 - gossip-tcp: 15605 - zmq-feed: 6556 - clusterip_ports: - api: 15265 - gossip-tcp: 15605 - zmq-feed: 6556 - nodeF: + nodeB-m3: host: localhost podip: localhost ports: diff --git a/python-regression/tests/features/machine1/1_api_tests.feature b/python-regression/tests/features/machine4/4_api_tests.feature similarity index 83% rename from python-regression/tests/features/machine1/1_api_tests.feature rename to python-regression/tests/features/machine4/4_api_tests.feature index 15f98f2eb9..3674917cb6 100644 --- a/python-regression/tests/features/machine1/1_api_tests.feature +++ b/python-regression/tests/features/machine4/4_api_tests.feature @@ -14,7 +14,7 @@ Feature: Test API calls on Machine 1 #See tests/features/steps/api_test_steps.py for further details # - Given "getNodeInfo" is called on "nodeA-m1" with: + Given "getNodeInfo" is called on "nodeA-m4" with: |keys |values |type | Then a response with the following is returned: @@ -45,11 +45,11 @@ Feature: Test API calls on Machine 1 Scenario: GetNeighbors is called - Given "addNeighbors" is called on "nodeA-m1" with: + Given "addNeighbors" is called on "nodeA-m4" with: |keys |values |type | - |uris |nodeB-m1 |nodeAddress | + |uris |nodeB-m4 |nodeAddress | - And "getNeighbors" is called on "nodeA-m1" with: + And "getNeighbors" is called on "nodeA-m4" with: |keys |values |type | Then a response with the following is returned: @@ -67,11 +67,11 @@ Feature: Test API calls on Machine 1 Scenario: Add and Remove Neighbors - Adds nodeB-m1 as a neighbor to nodeA-m1, and then removes it. + Adds nodeB-m4 as a neighbor to nodeA-m4, and then removes it. - Given "addNeighbors" is called on "nodeA-m1" with: + Given "addNeighbors" is called on "nodeA-m4" with: |keys |values |type | - |uris |nodeB-m1 |nodeAddress | + |uris |nodeB-m4 |nodeAddress | Then a response with the following is returned: |keys | @@ -79,9 +79,9 @@ Feature: Test API calls on Machine 1 |duration | - When "removeNeighbors" is called on "nodeA-m1" with: + When "removeNeighbors" is called on "nodeA-m4" with: |keys |values |type | - |uris |nodeB-m1 |nodeAddress | + |uris |nodeB-m4 |nodeAddress | Then a response with the following is returned: @@ -91,7 +91,7 @@ Feature: Test API calls on Machine 1 #Values can be found in util/static_vals.py Scenario: GetTrytes is called - Given "getTrytes" is called on "nodeA-m1" with: + Given "getTrytes" is called on "nodeA-m4" with: |keys |values |type | |hashes |TEST_HASH |staticList | @@ -102,7 +102,7 @@ Feature: Test API calls on Machine 1 Scenario: GetTransactionsToApprove is called - Given "getTransactionsToApprove" is called on "nodeA-m1" with: + Given "getTransactionsToApprove" is called on "nodeA-m4" with: |keys |values |type | |depth |3 |int | @@ -114,7 +114,7 @@ Feature: Test API calls on Machine 1 Scenario: CheckConsistency is called - Given "checkConsistency" is called on "nodeA-m1" with: + Given "checkConsistency" is called on "nodeA-m4" with: |keys |values |type | |tails |TEST_HASH |staticList | @@ -122,9 +122,9 @@ Feature: Test API calls on Machine 1 |keys |values |type | |state |True |bool | - When an inconsistent transaction is generated on "nodeA-m1" + When an inconsistent transaction is generated on "nodeA-m4" - And "checkConsistency" is called on "nodeA-m1" with: + And "checkConsistency" is called on "nodeA-m4" with: |keys |values |type | |tails |inconsistentTransactions |responseList | @@ -136,7 +136,7 @@ Feature: Test API calls on Machine 1 #Values can be found in util/static_vals.py Scenario: GetInclusionStates is called - Given "getInclusionStates" is called on "nodeA-m1" with: + Given "getInclusionStates" is called on "nodeA-m4" with: |keys |values |type | |transactions |TEST_HASH |staticList | |tips |TEST_TIP_LIST |staticValue | @@ -147,7 +147,7 @@ Feature: Test API calls on Machine 1 #Values can be found in util/static_vals.py Scenario: GetInclusionStates is called with transaction list - Given "getInclusionStates" is called on "nodeA-m1" with: + Given "getInclusionStates" is called on "nodeA-m4" with: | keys | values | type | | transactions | TEST_HASH_LIST | staticValue | | tips | TEST_TIP_LIST | staticValue | @@ -159,7 +159,7 @@ Feature: Test API calls on Machine 1 #Address can be found in util/static_vals.py Scenario: GetBalances is called - Given "getBalances" is called on "nodeA-m1" with: + Given "getBalances" is called on "nodeA-m4" with: |keys |values |type | |addresses |TEST_EMPTY_ADDRESS |staticList | |threshold |100 |int | @@ -173,7 +173,7 @@ Feature: Test API calls on Machine 1 Begins attaching a transaction to the tangle with a high MWM, then issues an interrupt to the node If the interrupt is successful, the attachToTangle response will return a null tryte list - Given "attachToTangle" is called in parallel on "nodeA-m1" with: + Given "attachToTangle" is called in parallel on "nodeA-m4" with: |keys |values |type | |trytes |EMPTY_TRANSACTION_TRYTES|staticList | |trunk_transaction |TEST_HASH |staticValue | @@ -181,7 +181,7 @@ Feature: Test API calls on Machine 1 |min_weight_magnitude |50 |int | And we wait "1" second/seconds - Then "interruptAttachingToTangle" is called in parallel on "nodeA-m1" with: + Then "interruptAttachingToTangle" is called in parallel on "nodeA-m4" with: |keys |values |type | # Do not include duration in the return expectations as it will always return a variable amount @@ -192,7 +192,7 @@ Feature: Test API calls on Machine 1 Scenario: WereAddressesSpentFrom is called - Given "wereAddressesSpentFrom" is called on "nodeA-m1" with: + Given "wereAddressesSpentFrom" is called on "nodeA-m4" with: |keys |values |type | |addresses |TEST_EMPTY_ADDRESS |staticList | @@ -207,7 +207,7 @@ Feature: Test API calls on Machine 1 Generate a transaction, attach it to the tangle, and store it locally. Then find that transaction via its address. - Given a transaction is generated and attached on "nodeA-m1" with: + Given a transaction is generated and attached on "nodeA-m4" with: |keys |values |type | |address |TEST_STORE_ADDRESS |staticValue | |value |0 |int | @@ -216,11 +216,11 @@ Feature: Test API calls on Machine 1 |keys | |trytes | - When "storeTransactions" is called on "nodeA-m1" with: + When "storeTransactions" is called on "nodeA-m4" with: |keys |values |type | |trytes |TEST_STORE_TRANSACTION |staticValue | - And "findTransactions" is called on "nodeA-m1" with: + And "findTransactions" is called on "nodeA-m4" with: |keys |values |type | |addresses |TEST_STORE_ADDRESS |staticList | @@ -234,8 +234,8 @@ Feature: Test API calls on Machine 1 Send a test transaction from one node in a machine with a unique tag, and find that transaction through a different node in the same machine - Given "nodeA-m1" and "nodeB-m1" are neighbors - When a transaction is generated and attached on "nodeA-m1" with: + Given "nodeA-m4" and "nodeB-m4" are neighbors + When a transaction is generated and attached on "nodeA-m4" with: |keys |values |type | |address |TEST_ADDRESS |staticValue | |tag |TEST9TAG9ONE |string | @@ -244,7 +244,7 @@ Feature: Test API calls on Machine 1 #Give the transaction time to propagate And we wait "3" second/seconds - And "findTransactions" is called on "nodeB-m1" with: + And "findTransactions" is called on "nodeB-m4" with: |keys |values |type | |tags |TEST9TAG9ONE |list | diff --git a/python-regression/tests/features/machine4/config.yml b/python-regression/tests/features/machine4/config.yml index 21e69e895f..1b5f17b529 100644 --- a/python-regression/tests/features/machine4/config.yml +++ b/python-regression/tests/features/machine4/config.yml @@ -1,23 +1,26 @@ -defaults: &stitching_tests_config_files - db: https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/Stitching_tests_db.tar - db_checksum: ac987dc8e61e37d6420e78da18c4cf94671f51e8014b4dd593f36bbd4fb0cfc1 +defaults: &api_tests_config_files + db: https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/testnet_files.tgz + db_checksum: 6eaa06d5442416b7b8139e337a1598d2bae6a7f55c2d9d01f8c5dac69c004f75 iri_args: ['--testnet-coordinator', - 'EFPNKGPCBXXXLIBYFGIGYBYTFFPIOQVNNVVWTTIYZO9NFREQGVGDQQHUUQ9CLWAEMXVDFSSMOTGAHVIBH', - '--milestone-keys', - '18', - '--milestone-start', - '2', + 'BTCAAFIH9CJIVIMWFMIHKFNWRTLJRKSTMRCVRE9CIP9AEDTOULVFRHQZT9QAQBZXXAZGBNMVOOKTKAXTB', + '--milestone-start', + '0', + '--snapshot', + './snapshot.txt', '--testnet-no-coo-validation', 'true', '--testnet', 'true' ] java_options: -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=n -javaagent:/opt/jacoco/lib/jacocoagent.jar=destfile=/iri/jacoco.exec,output=file,append=true,dumponexit=true - + seeds: # For internal use by the regression system. - SEED - SIID nodes: nodeA-m4: #name - <<: *stitching_tests_config_files + <<: *api_tests_config_files + + nodeB-m4: + <<: *api_tests_config_files diff --git a/python-regression/tests/features/machine4/output.yml b/python-regression/tests/features/machine4/output.yml index e84b1543bc..a9a0d80060 100644 --- a/python-regression/tests/features/machine4/output.yml +++ b/python-regression/tests/features/machine4/output.yml @@ -1,5 +1,5 @@ nodes: - nodeA: + nodeA-m4: host: localhost podip: localhost ports: @@ -10,7 +10,7 @@ nodes: api: 14265 gossip-tcp: 15600 zmq-feed: 5556 - nodeB: + nodeB-m4: host: localhost podip: localhost ports: diff --git a/python-regression/tests/features/machine4/4_stitching.feature b/python-regression/tests/features/machine5/5_stitching.feature similarity index 87% rename from python-regression/tests/features/machine4/4_stitching.feature rename to python-regression/tests/features/machine5/5_stitching.feature index 7b5c9a000c..796258ac6f 100644 --- a/python-regression/tests/features/machine4/4_stitching.feature +++ b/python-regression/tests/features/machine5/5_stitching.feature @@ -8,8 +8,8 @@ Feature: Ensure node reliability while stitching a side tangle Scenario: Check consistency on a stitching transaction responds - Given a stitching transaction is issued on "nodeA-m4" with the tag "STITCHING" - And "checkConsistency" is called in parallel on "nodeA-m4" with: + Given a stitching transaction is issued on "nodeA-m5" with the tag "STITCHING" + And "checkConsistency" is called in parallel on "nodeA-m5" with: |keys |values |type | |tails |previousTransaction |responseList | @@ -19,7 +19,7 @@ Feature: Ensure node reliability while stitching a side tangle When a transaction is issued referencing the previous transaction - And "getTransactionsToApprove" is called on "nodeA-m4" with: + And "getTransactionsToApprove" is called on "nodeA-m5" with: |keys |values |type | |depth |3 |int | diff --git a/python-regression/tests/features/machine5/config.yml b/python-regression/tests/features/machine5/config.yml index 766079162e..6055f30f00 100644 --- a/python-regression/tests/features/machine5/config.yml +++ b/python-regression/tests/features/machine5/config.yml @@ -1,6 +1,6 @@ -defaults: &no_validation_tests_config_files - db: https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/Validation_tests_db.tar - db_checksum: 40874a129db910be2466244b3cdeea2dde9f9601221cb94b837f55ac2be384f5 +defaults: &stitching_tests_config_files + db: https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/Stitching_tests_db.tar + db_checksum: ac987dc8e61e37d6420e78da18c4cf94671f51e8014b4dd593f36bbd4fb0cfc1 iri_args: ['--testnet-coordinator', 'EFPNKGPCBXXXLIBYFGIGYBYTFFPIOQVNNVVWTTIYZO9NFREQGVGDQQHUUQ9CLWAEMXVDFSSMOTGAHVIBH', '--milestone-keys', @@ -12,30 +12,12 @@ defaults: &no_validation_tests_config_files '--testnet', 'true' ] - java_options: -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=n -javaagent:/opt/jacoco/lib/jacocoagent.jar=destfile=/workdir/jacoco.exec,output=file,append=true,dumponexit=true - -db_with_validation: &validation_tests_config_files - db: https://s3.amazonaws.com/iota-db-files/dbs/machine4/validation_test_db.tar - db_checksum: 40874a129db910be2466244b3cdeea2dde9f9601221cb94b837f55ac2be384f5 - iri_args: ['--testnet-coordinator', - 'EFPNKGPCBXXXLIBYFGIGYBYTFFPIOQVNNVVWTTIYZO9NFREQGVGDQQHUUQ9CLWAEMXVDFSSMOTGAHVIBH', - '--milestone-keys', - '18', - '--milestone-start', - '2', - '--testnet', - 'true' - ] - java_options: -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=n -javaagent:/opt/jacoco/lib/jacocoagent.jar=destfile=/workdir/jacoco.exec,output=file,append=true,dumponexit=true - - + java_options: -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=n -javaagent:/opt/jacoco/lib/jacocoagent.jar=destfile=/iri/jacoco.exec,output=file,append=true,dumponexit=true + seeds: # For internal use by the regression system. - SEED - SIID nodes: nodeA-m5: #name - <<: *no_validation_tests_config_files - - nodeB-m5: - <<: *validation_tests_config_files + <<: *stitching_tests_config_files diff --git a/python-regression/tests/features/machine5/output.yml b/python-regression/tests/features/machine5/output.yml index e84b1543bc..0f8081d76f 100644 --- a/python-regression/tests/features/machine5/output.yml +++ b/python-regression/tests/features/machine5/output.yml @@ -1,5 +1,5 @@ nodes: - nodeA: + nodeA-m5: host: localhost podip: localhost ports: @@ -10,14 +10,3 @@ nodes: api: 14265 gossip-tcp: 15600 zmq-feed: 5556 - nodeB: - host: localhost - podip: localhost - ports: - api: 15265 - gossip-tcp: 15605 - zmq-feed: 6556 - clusterip_ports: - api: 15265 - gossip-tcp: 15605 - zmq-feed: 6556 \ No newline at end of file diff --git a/python-regression/tests/features/machine5/5_milestone_validation.feature b/python-regression/tests/features/machine6/6_milestone_validation.feature similarity index 87% rename from python-regression/tests/features/machine5/5_milestone_validation.feature rename to python-regression/tests/features/machine6/6_milestone_validation.feature index 6b74c619e9..e2b807a8ee 100644 --- a/python-regression/tests/features/machine5/5_milestone_validation.feature +++ b/python-regression/tests/features/machine6/6_milestone_validation.feature @@ -5,14 +5,14 @@ Feature: Test milestone validation Scenario: Verify current milestone index - Given "getNodeInfo" is called on "nodeA-m5" with: + Given "getNodeInfo" is called on "nodeA-m6" with: |keys |values |type | Then the response for "getNodeInfo" should return with: |keys |values |type | |latestMilestoneIndex |45 |int | - And "getNodeInfo" is called on "nodeB-m5" with: + And "getNodeInfo" is called on "nodeB-m6" with: |keys |values |type | Then the response for "getNodeInfo" should return with: diff --git a/python-regression/tests/features/machine6/__init__.py b/python-regression/tests/features/machine6/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/python-regression/tests/features/machine6/config.yml b/python-regression/tests/features/machine6/config.yml index 1be1ae6bc9..465558b49e 100644 --- a/python-regression/tests/features/machine6/config.yml +++ b/python-regression/tests/features/machine6/config.yml @@ -1,71 +1,41 @@ -default_args: &args - ['--testnet-coordinator', - 'EFPNKGPCBXXXLIBYFGIGYBYTFFPIOQVNNVVWTTIYZO9NFREQGVGDQQHUUQ9CLWAEMXVDFSSMOTGAHVIBH', - '--mwm', - '1', - '--milestone-start', - '0', - '--testnet-no-coo-validation', - 'true', - '--testnet', - 'true', - '--snapshot', - './snapshot.txt', - '--local-snapshots-pruning-enabled', - 'true', - '--local-snapshots-pruning-delay', - '10000', - '--remote', - 'true', - '--remote-limit-api', - '""' +defaults: &no_validation_tests_config_files + db: https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/Validation_tests_db.tar + db_checksum: 40874a129db910be2466244b3cdeea2dde9f9601221cb94b837f55ac2be384f5 + iri_args: ['--testnet-coordinator', + 'EFPNKGPCBXXXLIBYFGIGYBYTFFPIOQVNNVVWTTIYZO9NFREQGVGDQQHUUQ9CLWAEMXVDFSSMOTGAHVIBH', + '--milestone-keys', + '18', + '--milestone-start', + '2', + '--testnet-no-coo-validation', + 'true', + '--testnet', + 'true' ] + java_options: -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=n -javaagent:/opt/jacoco/lib/jacocoagent.jar=destfile=/workdir/jacoco.exec,output=file,append=true,dumponexit=true + +db_with_validation: &validation_tests_config_files + db: https://s3.amazonaws.com/iota-db-files/dbs/machine4/validation_test_db.tar + db_checksum: 40874a129db910be2466244b3cdeea2dde9f9601221cb94b837f55ac2be384f5 + iri_args: ['--testnet-coordinator', + 'EFPNKGPCBXXXLIBYFGIGYBYTFFPIOQVNNVVWTTIYZO9NFREQGVGDQQHUUQ9CLWAEMXVDFSSMOTGAHVIBH', + '--milestone-keys', + '18', + '--milestone-start', + '2', + '--testnet', + 'true' + ] + java_options: -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=n -javaagent:/opt/jacoco/lib/jacocoagent.jar=destfile=/workdir/jacoco.exec,output=file,append=true,dumponexit=true -default_ixi: &ixi - ['IXI/LocalSnapshots.ixi'] - -java_options: -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=n -javaagent:/opt/jacoco/lib/jacocoagent.jar=destfile=/iri/jacoco.exec,output=file,append=true,dumponexit=true - -defaults: &db_full - db: https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/LS_Test_Db_With_LS_Db.tar.gz - db_checksum: 2055406bf312136d7cd0efa21248bd8cc9c407ab14ef0d18b921cf18c72c5270 - iri_args: *args - ixis: *ixi - -db_with_snapshot: &db_with_snapshot - db: https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/LS_Test_DB_and_Snapshot.tar - db_checksum: eabb81b0570a20e8d1c65c3d29e4b4e723de537ebca0eada536e3155d5a96972 - iri_args: *args - ixis: *ixi - -db_with_ls_db: &db_with_ls_db - db: https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/LS_Test_LS_Db.tar.gz - db_checksum: d217729fd5efb0432d179ec59472f283cd61e8ad4ca9aab32e5c1f82632a1a29 - iri_args: *args - ixis: *ixi - -db_for_pruning: &db_for_pruning - db: https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/PruningTestDB.tar - db_checksum: 15122ba80c0a03dc5b6b4186e5d880d0a1a15b5a6de48bafe4002c4c9b682221 - iri_args: *args -db_for_spent_addresses: &db_for_spent_addresses - db: https://s3.eu-central-1.amazonaws.com/iotaledger-dbfiles/dev/SpentAddressesTestDB.tar - db_checksum: 7e15b2cbc76585d6483668cb1709201daa71314e7d488d9e7d71d7052479e73e - iri_args: *args +seeds: # For internal use by the regression system. + - SEED + - SIID nodes: nodeA-m6: #name - <<: *db_full - + <<: *no_validation_tests_config_files + nodeB-m6: - <<: *db_with_snapshot - - nodeC-m6: - <<: *db_with_ls_db - - nodeD-m6: - <<: *db_for_pruning - - nodeE-m6: - <<: *db_for_spent_addresses + <<: *validation_tests_config_files diff --git a/python-regression/tests/features/machine6/output.yml b/python-regression/tests/features/machine6/output.yml index 2b2fe945c9..7dd9f6e952 100644 --- a/python-regression/tests/features/machine6/output.yml +++ b/python-regression/tests/features/machine6/output.yml @@ -1,5 +1,5 @@ nodes: - nodeA: + nodeA-m6: host: localhost podip: localhost ports: @@ -10,7 +10,7 @@ nodes: api: 14265 gossip-tcp: 15600 zmq-feed: 5556 - nodeB: + nodeB-m6: host: localhost podip: localhost ports: @@ -20,26 +20,4 @@ nodes: clusterip_ports: api: 15265 gossip-tcp: 15605 - zmq-feed: 6556 - nodeC: - host: localhost - podip: localhost - ports: - api: 14265 - gossip-tcp: 15600 - zmq-feed: 5556 - clusterip_ports: - api: 14265 - gossip-tcp: 15600 - zmq-feed: 5556 - nodeD: - host: localhost - podip: localhost - ports: - api: 14265 - gossip-tcp: 15600 - zmq-feed: 5556 - clusterip_ports: - api: 14265 - gossip-tcp: 15600 - zmq-feed: 5556 + zmq-feed: 6556 \ No newline at end of file diff --git a/python-regression/tests/features/steps/api_test_steps.py b/python-regression/tests/features/steps/api_test_steps.py index 886a4eea7b..9afdc7a8d5 100644 --- a/python-regression/tests/features/steps/api_test_steps.py +++ b/python-regression/tests/features/steps/api_test_steps.py @@ -23,7 +23,7 @@ def api_method_is_called(step, api_call, node_name): :param api_call: The api call that will be requested :param node_name: The name identifying the node you would like to make this request on :param step.hashes: A gherkin table outlining any arguments needed for the call - (See tests/features/machine1/1_api_tests.feature for examples) + (See tests/features/machine1/4_api_tests.feature for examples) The table parameter is unique in that there are several input types available depending on the call being made. diff --git a/python-regression/tests/features/steps/response_handling_steps.py b/python-regression/tests/features/steps/response_handling_steps.py index 87d372814b..c0c52584a1 100644 --- a/python-regression/tests/features/steps/response_handling_steps.py +++ b/python-regression/tests/features/steps/response_handling_steps.py @@ -1,4 +1,3 @@ -import logging from aloe import world, step from util.response_logic import response_handling as response_handling from util.test_logic import api_test_logic as api_utils diff --git a/python-regression/util/static_vals.py b/python-regression/util/static_vals.py index a4d332bd50..035ea262ae 100644 --- a/python-regression/util/static_vals.py +++ b/python-regression/util/static_vals.py @@ -118,12 +118,13 @@ ] LS_PRUNED_TRANSACTIONS = [ - "FKGOUSFWFWSPQXPGEVBXQRYFRGFJWBXRYCXJDZTQWQBQKUJOGEY9JCIVYYSHGNFVIJBIXMZIVDUPBUIYA", - "EVQEKYWYYDSPHYDVPHATRUHBMKANTTIRA9FXXUYSVRBSEMESKWDBJBWBDARWMGBCFWGI9CFOEI9MXGMYX", - "HVGYYBVCBWKSMQFTFURLQWWRTNBKYXMXQ9QP9SHSNZJQQ9P9HL9IGNWZBTNMOOLV9GVOBPIBEUCXWNFAY", - "KESYDCSVJ9TJJIOIMHJDMD9BSPWHQVUIKEBSMYLBTUZZTKKFALDTMATCQWAKIQHNHHZLP9NJFZJMFNEMC", - "TJRXEMZZMORIHQSGOTRUFATRMMQF9NCGVVCKIDJFIMEZNXFGC9HZQQWZXTMIWILGYBBBFZURU99G9YRLY" -] + "ZUGEWWWKGETGGVVVTXYFSMAP9KLRYQJSVHNTYWEXQONNNHTCMAMQANEPEFKNAJYRFJINTSHXTGFOWYWO9", + "UXBNYASZOCIGHTDTGHLJWRMUHXABNDSTDFVJLBIA9SMGYEFYZLKQETFIAERHSY9YZVUEJUMCTNQE9NIMY", + "XJZMRLGVXLIBAYPYWMZGXUZZWJEFAXRPGAAYIALSI9UBRSYDACGO9UEQJEOVROIBQMAMIFGZJHGHKUNNB", + "HDXVLKMJNKQNPTTSBETLNRY9WNWXILCTNICEEWEUKHKFFAHKA9D99XQCPQRHOVOBEYTLARJORYGMNHUOZ" + ] + +LS_PRUNING_TIP = "HO9YADXIDEUJUKOZYNWVRS9PWDXCFXNVM9LEGPZMBJSJKKQAZBFPXYMH9VABBG9NLNIDQPDLVWDKOEAOW" UNCONFIRMED_TEST_ADDRESS = [ "OXKSHMEQYDMJMDPJX9FIEJVWOKCROTCBRPCDCCSBPMZ9EEQMHTEDSDKIKAMZTYNYWIIMZBFY9IXBUGTZD" diff --git a/python-regression/util/test_logic/api_test_logic.py b/python-regression/util/test_logic/api_test_logic.py index fbe2e4c204..f6fe3f858d 100644 --- a/python-regression/util/test_logic/api_test_logic.py +++ b/python-regression/util/test_logic/api_test_logic.py @@ -1,5 +1,4 @@ import json -import logging import urllib3 from aloe import world from iota import Iota, Address, Tag, TryteString diff --git a/python-regression/util/test_logic/value_fetch_logic.py b/python-regression/util/test_logic/value_fetch_logic.py index 3dede4a655..9f662cb37c 100644 --- a/python-regression/util/test_logic/value_fetch_logic.py +++ b/python-regression/util/test_logic/value_fetch_logic.py @@ -121,6 +121,8 @@ def fetch_bool_list(value): node = world.config['nodeId'] response = world.responses[api_call][node] + keys = list(response.keys()) + response = response[keys[0]] if value == "False": return [False] * len(response) diff --git a/src/main/java/com/iota/iri/Iota.java b/src/main/java/com/iota/iri/Iota.java index c901fe8fff..420cc51c71 100644 --- a/src/main/java/com/iota/iri/Iota.java +++ b/src/main/java/com/iota/iri/Iota.java @@ -22,6 +22,8 @@ import com.iota.iri.service.transactionpruning.DepthPruningCondition; import com.iota.iri.service.transactionpruning.SizePruningCondition; import com.iota.iri.service.transactionpruning.TransactionPruner; +import com.iota.iri.service.validation.TransactionSolidifier; +import com.iota.iri.service.validation.TransactionValidator; import com.iota.iri.storage.*; import com.iota.iri.storage.rocksDB.RocksDBPersistenceProvider; import com.iota.iri.utils.Pair; @@ -95,6 +97,8 @@ public class Iota { public final MilestoneSolidifier milestoneSolidifier; + public final TransactionSolidifier transactionSolidifier; + public final BundleValidator bundleValidator; public final Tangle tangle; @@ -126,7 +130,7 @@ public Iota(IotaConfig configuration, SpentAddressesProvider spentAddressesProvi TransactionRequester transactionRequester, NeighborRouter neighborRouter, TransactionProcessingPipeline transactionProcessingPipeline, TipsRequester tipsRequester, TipsViewModel tipsViewModel, TipSelector tipsSelector, LocalSnapshotsPersistenceProvider localSnapshotsDb, - CacheManager cacheManager) { + CacheManager cacheManager, TransactionSolidifier transactionSolidifier) { this.configuration = configuration; this.ledgerService = ledgerService; @@ -144,9 +148,9 @@ public Iota(IotaConfig configuration, SpentAddressesProvider spentAddressesProvi this.neighborRouter = neighborRouter; this.txPipeline = transactionProcessingPipeline; this.tipsRequester = tipsRequester; + this.transactionSolidifier = transactionSolidifier; this.localSnapshotsDb = localSnapshotsDb; - // legacy classes this.bundleValidator = bundleValidator; this.tangle = tangle; @@ -199,8 +203,6 @@ public void init() throws Exception { tangle.clearMetadata(com.iota.iri.model.persistables.Transaction.class); } - transactionValidator.init(); - txPipeline.start(); neighborRouter.start(); tipsRequester.start(); @@ -209,6 +211,7 @@ public void init() throws Exception { latestSolidMilestoneTracker.start(); seenMilestonesRetriever.start(); milestoneSolidifier.start(); + transactionSolidifier.start(); if (localSnapshotManager != null) { localSnapshotManager.addSnapshotCondition(new SnapshotDepthCondition(configuration, snapshotProvider)); @@ -255,6 +258,7 @@ private void rescanDb() throws Exception { public void shutdown() throws Exception { // shutdown in reverse starting order (to not break any dependencies) milestoneSolidifier.shutdown(); + transactionSolidifier.shutdown(); seenMilestonesRetriever.shutdown(); latestSolidMilestoneTracker.shutdown(); latestMilestoneTracker.shutdown(); @@ -269,7 +273,6 @@ public void shutdown() throws Exception { tipsRequester.shutdown(); txPipeline.shutdown(); neighborRouter.shutdown(); - transactionValidator.shutdown(); localSnapshotsDb.shutdown(); tangle.shutdown(); diff --git a/src/main/java/com/iota/iri/MainInjectionConfiguration.java b/src/main/java/com/iota/iri/MainInjectionConfiguration.java index f512d3f7bc..ba01af69d2 100644 --- a/src/main/java/com/iota/iri/MainInjectionConfiguration.java +++ b/src/main/java/com/iota/iri/MainInjectionConfiguration.java @@ -4,7 +4,6 @@ import com.iota.iri.cache.impl.CacheManagerImpl; import com.iota.iri.conf.IotaConfig; import com.iota.iri.controllers.TipsViewModel; -import com.iota.iri.model.persistables.Transaction; import com.iota.iri.network.NeighborRouter; import com.iota.iri.network.TipsRequester; import com.iota.iri.network.TransactionRequester; @@ -28,6 +27,9 @@ import com.iota.iri.service.tipselection.impl.*; import com.iota.iri.service.transactionpruning.TransactionPruner; import com.iota.iri.service.transactionpruning.async.AsyncTransactionPruner; +import com.iota.iri.service.validation.TransactionSolidifier; +import com.iota.iri.service.validation.TransactionValidator; +import com.iota.iri.service.validation.impl.TransactionSolidifierImpl; import com.iota.iri.storage.LocalSnapshotsPersistenceProvider; import com.iota.iri.storage.Tangle; import com.iota.iri.storage.rocksDB.RocksDBPersistenceProvider; @@ -116,8 +118,8 @@ SeenMilestonesRetriever provideSeenMilestonesRetriever(Tangle tangle, SnapshotPr @Singleton @Provides - MilestoneSolidifier provideMilestoneSolidifier(SnapshotProvider snapshotProvider, TransactionValidator transactionValidator) { - return new MilestoneSolidifierImpl(snapshotProvider, transactionValidator); + MilestoneSolidifier provideMilestoneSolidifier(SnapshotProvider snapshotProvider, TransactionSolidifier transactionSolidifier) { + return new MilestoneSolidifierImpl(snapshotProvider, transactionSolidifier); } @Singleton @@ -138,8 +140,14 @@ LocalSnapshotManager provideLocalSnapshotManager(SnapshotProvider snapshotProvid @Singleton @Provides - TransactionValidator provideTransactionValidator(Tangle tangle, SnapshotProvider snapshotProvider, TipsViewModel tipsViewModel, TransactionRequester transactionRequester) { - return new TransactionValidator(tangle, snapshotProvider, tipsViewModel, transactionRequester, configuration); + TransactionValidator provideTransactionValidator(SnapshotProvider snapshotProvider, TransactionRequester transactionRequester) { + return new TransactionValidator(snapshotProvider, transactionRequester, configuration); + } + + @Singleton + @Provides + TransactionSolidifier provideTransactionSolidifier(Tangle tangle, SnapshotProvider snapshotProvider, TransactionRequester transactionRequester, TipsViewModel tipsViewModel){ + return new TransactionSolidifierImpl(tangle, snapshotProvider, transactionRequester, tipsViewModel); } @Singleton @@ -174,12 +182,12 @@ Iota provideIota(SpentAddressesProvider spentAddressesProvider, SpentAddressesSe TransactionRequester transactionRequester, NeighborRouter neighborRouter, TransactionProcessingPipeline transactionProcessingPipeline, TipsRequester tipsRequester, TipsViewModel tipsViewModel, TipSelector tipsSelector, LocalSnapshotsPersistenceProvider localSnapshotsDb, - CacheManager cacheManager) { + CacheManager cacheManager, TransactionSolidifier transactionSolidifier) { return new Iota(configuration, spentAddressesProvider, spentAddressesService, snapshotProvider, snapshotService, localSnapshotManager, milestoneService, latestMilestoneTracker, latestSolidMilestoneTracker, seenMilestonesRetriever, ledgerService, transactionPruner, milestoneSolidifier, bundleValidator, tangle, transactionValidator, transactionRequester, neighborRouter, transactionProcessingPipeline, - tipsRequester, tipsViewModel, tipsSelector, localSnapshotsDb, cacheManager); + tipsRequester, tipsViewModel, tipsSelector, localSnapshotsDb, cacheManager, transactionSolidifier); } @Singleton @@ -191,8 +199,8 @@ IXI provideIxi(Iota iota) { @Singleton @Provides @Named("AsyncTipSelSolidifier") - TipSelSolidifier provideAsyncTipSelSolidifier (TransactionValidator transactionValidator) { - return new AsyncTipSelSolidifier(transactionValidator); + TipSelSolidifier provideAsyncTipSelSolidifier (TransactionSolidifier transactionSolidifier) { + return new AsyncTipSelSolidifier(transactionSolidifier); } @Singleton @@ -204,15 +212,14 @@ TipSelSolidifier provideDummyTipSelSolidifier () { @Singleton @Provides - API provideApi(IXI ixi, TransactionRequester transactionRequester, SpentAddressesService spentAddressesService, - Tangle tangle, BundleValidator bundleValidator, SnapshotProvider snapshotProvider, - LedgerService ledgerService, NeighborRouter neighborRouter, TipSelector tipsSelector, - TipsViewModel tipsViewModel, TransactionValidator transactionValidator, - LatestMilestoneTracker latestMilestoneTracker, TransactionProcessingPipeline txPipeline, - @Named("DummyTipSelSolidifier") TipSelSolidifier checkConsistencySolidifier) { + API provideApi(IXI ixi, TransactionRequester transactionRequester, + SpentAddressesService spentAddressesService, Tangle tangle, BundleValidator bundleValidator, + SnapshotProvider snapshotProvider, LedgerService ledgerService, NeighborRouter neighborRouter, TipSelector tipsSelector, + TipsViewModel tipsViewModel, TransactionValidator transactionValidator, + LatestMilestoneTracker latestMilestoneTracker, TransactionProcessingPipeline txPipeline, TransactionSolidifier transactionSolidifier, @Named("DummyTipSelSolidifier") TipSelSolidifier checkConsistencySolidifier) { return new API(configuration, ixi, transactionRequester, spentAddressesService, tangle, bundleValidator, snapshotProvider, ledgerService, neighborRouter, tipsSelector, tipsViewModel, transactionValidator, - latestMilestoneTracker, txPipeline, checkConsistencySolidifier); + latestMilestoneTracker, txPipeline,transactionSolidifier, checkConsistencySolidifier); } @Singleton diff --git a/src/main/java/com/iota/iri/TransactionValidator.java b/src/main/java/com/iota/iri/TransactionValidator.java deleted file mode 100644 index 23d1a77692..0000000000 --- a/src/main/java/com/iota/iri/TransactionValidator.java +++ /dev/null @@ -1,464 +0,0 @@ -package com.iota.iri; - -import com.google.common.annotations.VisibleForTesting; -import com.iota.iri.conf.ProtocolConfig; -import com.iota.iri.controllers.TipsViewModel; -import com.iota.iri.controllers.TransactionViewModel; -import com.iota.iri.crypto.Curl; -import com.iota.iri.crypto.Sponge; -import com.iota.iri.crypto.SpongeFactory; -import com.iota.iri.model.Hash; -import com.iota.iri.model.TransactionHash; -import com.iota.iri.network.TransactionRequester; -import com.iota.iri.service.snapshot.SnapshotProvider; -import com.iota.iri.storage.Tangle; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.util.*; -import java.util.concurrent.atomic.AtomicBoolean; - -import static com.iota.iri.controllers.TransactionViewModel.*; - -public class TransactionValidator { - private static final Logger log = LoggerFactory.getLogger(TransactionValidator.class); - private static final int TESTNET_MWM_CAP = 13; - public static final int SOLID_SLEEP_TIME = 500; - - private final Tangle tangle; - private final SnapshotProvider snapshotProvider; - private final TipsViewModel tipsViewModel; - private final TransactionRequester transactionRequester; - private int minWeightMagnitude = 81; - private static final long MAX_TIMESTAMP_FUTURE = 2L * 60L * 60L; - private static final long MAX_TIMESTAMP_FUTURE_MS = MAX_TIMESTAMP_FUTURE * 1_000L; - - - /////////////////////////////////fields for solidification thread////////////////////////////////////// - - private Thread newSolidThread; - - /** - * If true use {@link #newSolidTransactionsOne} while solidifying. Else use {@link #newSolidTransactionsTwo}. - */ - private final AtomicBoolean useFirst = new AtomicBoolean(true); - /** - * Is {@link #newSolidThread} shutting down - */ - private final AtomicBoolean shuttingDown = new AtomicBoolean(false); - /** - * mutex for solidification - */ - private final Object cascadeSync = new Object(); - private final Set newSolidTransactionsOne = new LinkedHashSet<>(); - private final Set newSolidTransactionsTwo = new LinkedHashSet<>(); - - /** - * Constructor for Tangle Validator - * - * @param tangle relays tangle data to and from the persistence layer - * @param snapshotProvider data provider for the snapshots that are relevant for the node - * @param tipsViewModel container that gets updated with the latest tips (transactions with no children) - * @param transactionRequester used to request missing transactions from neighbors - * @param protocolConfig used for checking if we are in testnet and mwm. testnet true if we are in testnet - * mode, this caps {@code mwm} to {@value #TESTNET_MWM_CAP} regardless of parameter input. - * minimum weight magnitude: the minimal number of 9s that ought to appear at the end of the - * transaction hash - */ - TransactionValidator(Tangle tangle, SnapshotProvider snapshotProvider, TipsViewModel tipsViewModel, TransactionRequester transactionRequester, ProtocolConfig protocolConfig) { - this.tangle = tangle; - this.snapshotProvider = snapshotProvider; - this.tipsViewModel = tipsViewModel; - this.transactionRequester = transactionRequester; - this.newSolidThread = new Thread(spawnSolidTransactionsPropagation(), "Solid TX cascader"); - setMwm(protocolConfig.isTestnet(), protocolConfig.getMwm()); - } - - /** - * Does two things: - *
    - *
  1. Sets the minimum weight magnitude (MWM). POW on a transaction is validated by counting a certain - * number of consecutive 9s in the end of the transaction hash. The number of 9s is the MWM.
    2. - *
  2. Starts the transaction solidification thread.
    3. - *
- * - * - * @see #spawnSolidTransactionsPropagation() - */ - public void init() { - newSolidThread.start(); - } - - @VisibleForTesting - void setMwm(boolean testnet, int mwm) { - minWeightMagnitude = mwm; - - //lowest allowed MWM encoded in 46 bytes. - if (!testnet){ - minWeightMagnitude = Math.max(minWeightMagnitude, TESTNET_MWM_CAP); - } - } - - /** - * Shutdown roots to tip solidification thread - * @throws InterruptedException - * @see #spawnSolidTransactionsPropagation() - */ - public void shutdown() throws InterruptedException { - shuttingDown.set(true); - newSolidThread.join(); - } - - /** - * @return the minimal number of trailing 9s that have to be present at the end of the transaction hash - * in order to validate that sufficient proof of work has been done - */ - public int getMinWeightMagnitude() { - return minWeightMagnitude; - } - - /** - * Checks that the timestamp of the transaction is below the last global snapshot time - * or more than {@value #MAX_TIMESTAMP_FUTURE} seconds in the future, and thus invalid. - * - *

- * First the attachment timestamp (set after performing POW) is checked, and if not available - * the regular timestamp is checked. Genesis transaction will always be valid. - *

- * @param transactionViewModel transaction under test - * @return true if timestamp is not in valid bounds and {@code transactionViewModel} is not genesis. - * Else returns false. - */ - private boolean hasInvalidTimestamp(TransactionViewModel transactionViewModel) { - // ignore invalid timestamps for transactions that were requested by our node while solidifying a milestone - if(transactionRequester.wasTransactionRecentlyRequested(transactionViewModel.getHash())) { - return false; - } - - if (transactionViewModel.getAttachmentTimestamp() == 0) { - return transactionViewModel.getTimestamp() < snapshotProvider.getInitialSnapshot().getTimestamp() && !snapshotProvider.getInitialSnapshot().hasSolidEntryPoint(transactionViewModel.getHash()) - || transactionViewModel.getTimestamp() > (System.currentTimeMillis() / 1000) + MAX_TIMESTAMP_FUTURE; - } - return transactionViewModel.getAttachmentTimestamp() < (snapshotProvider.getInitialSnapshot().getTimestamp() * 1000L) - || transactionViewModel.getAttachmentTimestamp() > System.currentTimeMillis() + MAX_TIMESTAMP_FUTURE_MS; - } - - /** - * Runs the following validation checks on a transaction: - *
    - *
  1. {@link #hasInvalidTimestamp} check.
    2. - *
  2. Check that no value trits are set beyond the usable index, otherwise we will have values larger - * than max supply.
    3. - *
  3. Check that sufficient POW was performed.
    4. - *
  4. In value transactions, we check that the address has 0 set as the last trit. This must be because of the - * conversion between bytes to trits.
    5. - *
- *Exception is thrown upon failure. - * - * @param transactionViewModel transaction that should be validated - * @param minWeightMagnitude the minimal number of trailing 9s at the end of the transaction hash - * @throws StaleTimestampException if timestamp check fails - * @throws IllegalStateException if any of the other checks fail - */ - public void runValidation(TransactionViewModel transactionViewModel, final int minWeightMagnitude) { - transactionViewModel.setMetadata(); - transactionViewModel.setAttachmentData(); - if(hasInvalidTimestamp(transactionViewModel)) { - throw new StaleTimestampException("Invalid transaction timestamp."); - } - for (int i = VALUE_TRINARY_OFFSET + VALUE_USABLE_TRINARY_SIZE; i < VALUE_TRINARY_OFFSET + VALUE_TRINARY_SIZE; i++) { - if (transactionViewModel.trits()[i] != 0) { - throw new IllegalStateException("Invalid transaction value"); - } - } - - int weightMagnitude = transactionViewModel.weightMagnitude; - if(weightMagnitude < minWeightMagnitude) { - throw new IllegalStateException("Invalid weight magnitude"); - } - - if (transactionViewModel.value() != 0 && transactionViewModel.getAddressHash().trits()[Curl.HASH_LENGTH - 1] != 0) { - throw new IllegalStateException("Invalid transaction address"); - } - } - - /** - * Creates a new transaction from {@code trits} and validates it with {@link #runValidation}. - * - * @param trits raw transaction trits - * @param minWeightMagnitude minimal number of trailing 9s in transaction for POW validation - * @return the transaction resulting from the raw trits if valid. - * @throws RuntimeException if validation fails - */ - public TransactionViewModel validateTrits(final byte[] trits, int minWeightMagnitude) { - TransactionViewModel transactionViewModel = new TransactionViewModel(trits, TransactionHash.calculate(trits, 0, trits.length, SpongeFactory.create(SpongeFactory.Mode.CURLP81))); - runValidation(transactionViewModel, minWeightMagnitude); - return transactionViewModel; - } - - /** - * Creates a new transaction from {@code bytes} and validates it with {@link #runValidation}. - * - * @param bytes raw transaction bytes - * @param minWeightMagnitude minimal number of trailing 9s in transaction for POW validation - * @return the transaction resulting from the raw bytes if valid - * @throws RuntimeException if validation fails - */ - public TransactionViewModel validateBytes(final byte[] bytes, int minWeightMagnitude, Sponge curl) { - TransactionViewModel transactionViewModel = new TransactionViewModel(bytes, TransactionHash.calculate(bytes, TRINARY_SIZE, curl)); - runValidation(transactionViewModel, minWeightMagnitude); - return transactionViewModel; - } - - /** - * This method does the same as {@link #checkSolidity(Hash, int)} but defaults to an unlimited amount - * of transactions that are allowed to be traversed. - * - * @param hash hash of the transactions that shall get checked - * @return true if the transaction is solid and false otherwise - * @throws Exception if anything goes wrong while trying to solidify the transaction - */ - public boolean checkSolidity(Hash hash) throws Exception { - return checkSolidity(hash, Integer.MAX_VALUE); - } - - /** - * This method checks transactions for solidity and marks them accordingly if they are found to be solid. - * - * It iterates through all approved transactions until it finds one that is missing in the database or until it - * reached solid transactions on all traversed subtangles. In case of a missing transactions it issues a transaction - * request and returns false. If no missing transaction is found, it marks the processed transactions as solid in - * the database and returns true. - * - * Since this operation can potentially take a long time to terminate if it would have to traverse big parts of the - * tangle, it is possible to limit the amount of transactions that are allowed to be processed, while looking for - * unsolid / missing approvees. This can be useful when trying to "interrupt" the solidification of one transaction - * (if it takes too many steps) to give another one the chance to be solidified instead (i.e. prevent blocks in the - * solidification threads). - * - * @param hash hash of the transactions that shall get checked - * @param maxProcessedTransactions the maximum amount of transactions that are allowed to be traversed - * @return true if the transaction is solid and false otherwise - * @throws Exception if anything goes wrong while trying to solidify the transaction - */ - public boolean checkSolidity(Hash hash, int maxProcessedTransactions) throws Exception { - if(fromHash(tangle, hash).isSolid()) { - return true; - } - LinkedHashSet analyzedHashes = new LinkedHashSet<>(snapshotProvider.getInitialSnapshot().getSolidEntryPoints().keySet()); - if(maxProcessedTransactions != Integer.MAX_VALUE) { - maxProcessedTransactions += analyzedHashes.size(); - } - boolean solid = true; - final Queue nonAnalyzedTransactions = new LinkedList<>(Collections.singleton(hash)); - Hash hashPointer; - while ((hashPointer = nonAnalyzedTransactions.poll()) != null) { - if (!analyzedHashes.add(hashPointer)) { - continue; - } - - if (analyzedHashes.size() >= maxProcessedTransactions) { - return false; - } - - TransactionViewModel transaction = fromHash(tangle, hashPointer); - if (!transaction.isSolid() && !snapshotProvider.getInitialSnapshot().hasSolidEntryPoint(hashPointer)) { - if (transaction.getType() == PREFILLED_SLOT) { - solid = false; - - if (!transactionRequester.isTransactionRequested(hashPointer)) { - transactionRequester.requestTransaction(hashPointer); - continue; - } - } else { - nonAnalyzedTransactions.offer(transaction.getTrunkTransactionHash()); - nonAnalyzedTransactions.offer(transaction.getBranchTransactionHash()); - } - } - } - if (solid) { - updateSolidTransactions(tangle, snapshotProvider.getInitialSnapshot(), analyzedHashes); - analyzedHashes.forEach(this::addSolidTransaction); - } - analyzedHashes.clear(); - return solid; - } - - public void addSolidTransaction(Hash hash) { - synchronized (cascadeSync) { - if (useFirst.get()) { - newSolidTransactionsOne.add(hash); - } else { - newSolidTransactionsTwo.add(hash); - } - } - } - - /** - * Creates a runnable that runs {@link #propagateSolidTransactions()} in a loop every {@value #SOLID_SLEEP_TIME} ms - * @return runnable that is not started - */ - private Runnable spawnSolidTransactionsPropagation() { - return () -> { - while(!shuttingDown.get()) { - propagateSolidTransactions(); - try { - Thread.sleep(SOLID_SLEEP_TIME); - } catch (InterruptedException e) { - // Ignoring InterruptedException. Do not use Thread.currentThread().interrupt() here. - log.error("Thread was interrupted: ", e); - } - } - }; - } - - /** - * Iterates over all currently known solid transactions. For each solid transaction, we find - * its children (approvers) and try to quickly solidify them with {@link #quietQuickSetSolid}. - * If we manage to solidify the transactions, we add them to the solidification queue for a traversal by a later run. - */ - @VisibleForTesting - void propagateSolidTransactions() { - Set newSolidHashes = new HashSet<>(); - useFirst.set(!useFirst.get()); - //synchronized to make sure no one is changing the newSolidTransactions collections during addAll - synchronized (cascadeSync) { - //We are using a collection that doesn't get updated by other threads - if (useFirst.get()) { - newSolidHashes.addAll(newSolidTransactionsTwo); - newSolidTransactionsTwo.clear(); - } else { - newSolidHashes.addAll(newSolidTransactionsOne); - newSolidTransactionsOne.clear(); - } - } - Iterator cascadeIterator = newSolidHashes.iterator(); - while(cascadeIterator.hasNext() && !shuttingDown.get()) { - try { - Hash hash = cascadeIterator.next(); - TransactionViewModel transaction = fromHash(tangle, hash); - Set approvers = transaction.getApprovers(tangle).getHashes(); - for(Hash h: approvers) { - TransactionViewModel tx = fromHash(tangle, h); - if(quietQuickSetSolid(tx)) { - tipsViewModel.setSolid(h); - addSolidTransaction(h); - } - } - } catch (Exception e) { - log.error("Error while propagating solidity upwards", e); - } - } - } - - - /** - * Updates a transaction after it was stored in the tangle. Tells the node to not request the transaction anymore, - * to update the live tips accordingly, and attempts to quickly solidify the transaction. - * - *

- * Performs the following operations: - * - *

    - *
  1. Removes {@code transactionViewModel}'s hash from the the request queue since we already found it.
    2. - *
  2. If {@code transactionViewModel} has no children (approvers), we add it to the node's active tip list.
    3. - *
  3. Removes {@code transactionViewModel}'s parents (branch & trunk) from the node's tip list - * (if they're present there).
    4. - *
  4. Attempts to quickly solidify {@code transactionViewModel} by checking whether its direct parents - * are solid. If solid we add it to the queue transaction solidification thread to help it propagate the - * solidification to the approving child transactions.
    5. - *
  5. Requests missing direct parent (trunk & branch) transactions that are needed to solidify - * {@code transactionViewModel}.
    6. - *
- * @param transactionViewModel received transaction that is being updated - * @throws Exception if an error occurred while trying to solidify - * @see TipsViewModel - */ - //Not part of the validation process. This should be moved to a component in charge of - //what transaction we gossip. - public void updateStatus(TransactionViewModel transactionViewModel) throws Exception { - transactionRequester.clearTransactionRequest(transactionViewModel.getHash()); - if(transactionViewModel.getApprovers(tangle).size() == 0) { - tipsViewModel.addTipHash(transactionViewModel.getHash()); - } - tipsViewModel.removeTipHash(transactionViewModel.getTrunkTransactionHash()); - tipsViewModel.removeTipHash(transactionViewModel.getBranchTransactionHash()); - - if(quickSetSolid(transactionViewModel)) { - tipsViewModel.setSolid(transactionViewModel.getHash()); - addSolidTransaction(transactionViewModel.getHash()); - } - } - - /** - * Perform a {@link #quickSetSolid} while capturing and logging errors - * @param transactionViewModel transaction we try to solidify. - * @return true if we managed to solidify, else false. - */ - private boolean quietQuickSetSolid(TransactionViewModel transactionViewModel) { - try { - return quickSetSolid(transactionViewModel); - } catch (Exception e) { - log.error(e.getMessage(), e); - return false; - } - } - - /** - * Tries to solidify the transactions quickly by performing {@link #checkApproovee} on both parents (trunk and - * branch). If the parents are solid, mark the transactions as solid. - * @param transactionViewModel transaction to solidify - * @return true if we made the transaction solid, else false. - * @throws Exception - */ - private boolean quickSetSolid(final TransactionViewModel transactionViewModel) throws Exception { - if(!transactionViewModel.isSolid()) { - boolean solid = true; - if (!checkApproovee(transactionViewModel.getTrunkTransaction(tangle))) { - solid = false; - } - if (!checkApproovee(transactionViewModel.getBranchTransaction(tangle))) { - solid = false; - } - if(solid) { - transactionViewModel.updateSolid(tangle, snapshotProvider.getInitialSnapshot(), true); - transactionViewModel.updateHeights(tangle, snapshotProvider.getInitialSnapshot()); - return true; - } - } - return false; - } - - /** - * If the the {@code approvee} is missing, request it from a neighbor. - * @param approovee transaction we check. - * @return true if {@code approvee} is solid. - * @throws Exception if we encounter an error while requesting a transaction - */ - private boolean checkApproovee(TransactionViewModel approovee) throws Exception { - if(snapshotProvider.getInitialSnapshot().hasSolidEntryPoint(approovee.getHash())) { - return true; - } - if(approovee.getType() == PREFILLED_SLOT) { - // don't solidify from the bottom until cuckoo filters can identify where we deleted -> otherwise we will - // continue requesting old transactions forever - //transactionRequester.requestTransaction(approovee.getHash(), false); - return false; - } - return approovee.isSolid(); - } - - @VisibleForTesting - boolean isNewSolidTxSetsEmpty () { - return newSolidTransactionsOne.isEmpty() && newSolidTransactionsTwo.isEmpty(); - } - - /** - * Thrown if transaction fails {@link #hasInvalidTimestamp} check. - */ - public static class StaleTimestampException extends RuntimeException { - StaleTimestampException (String message) { - super(message); - } - } -} diff --git a/src/main/java/com/iota/iri/controllers/TransactionViewModel.java b/src/main/java/com/iota/iri/controllers/TransactionViewModel.java index 32e99df332..9c3da45ea6 100644 --- a/src/main/java/com/iota/iri/controllers/TransactionViewModel.java +++ b/src/main/java/com/iota/iri/controllers/TransactionViewModel.java @@ -4,6 +4,7 @@ import com.iota.iri.model.*; import com.iota.iri.model.persistables.*; import com.iota.iri.service.snapshot.Snapshot; +import com.iota.iri.service.validation.TransactionValidator; import com.iota.iri.storage.Indexable; import com.iota.iri.storage.Persistable; import com.iota.iri.storage.Tangle; @@ -512,9 +513,10 @@ public final int getType() { /** * Sets the {@link Transaction#arrivalTime}. * - * @param tangle The tangle reference for the database. - * @param initialSnapshot snapshot that acts as genesis - * @param time The time to be set in the {@link Transaction} + * @param tangle The tangle reference for the database. + * @param initialSnapshot snapshot that acts as genesis + * @param time The time to be set in the {@link Transaction} + * @throws Exception Exception */ public void setArrivalTime(Tangle tangle, Snapshot initialSnapshot, long time) throws Exception { transaction.arrivalTime = time; @@ -778,36 +780,15 @@ public void setMetadata() { : TransactionViewModel.FILLED_SLOT); } - /** - * Update solid transactions - * @param tangle Tangle - * @param initialSnapshot Initial snapshot - * @param analyzedHashes analyzed hashes - * @throws Exception Exception - */ - public static void updateSolidTransactions(Tangle tangle, Snapshot initialSnapshot, - final Set analyzedHashes) throws Exception { - Object[] hashes = analyzedHashes.toArray(); - TransactionViewModel transactionViewModel; - for (int i = hashes.length - 1; i >= 0; i--) { - transactionViewModel = TransactionViewModel.fromHash(tangle, (Hash) hashes[i]); - - transactionViewModel.updateHeights(tangle, initialSnapshot); - - if (!transactionViewModel.isSolid()) { - transactionViewModel.updateSolid(tangle, initialSnapshot,true); - } - } - } /** * Updates the {@link Transaction#solid} value of the referenced {@link Transaction} object. * - * Used by the {@link com.iota.iri.TransactionValidator} to quickly set the solidity of a {@link Transaction} set. + * Used by the {@link TransactionValidator} to quickly set the solidity of a {@link Transaction} set. * - * @param tangle The tangle reference for the database. - * @param initialSnapshot snapshot that acts as genesis - * @param solid The solidity of the transaction in the database + * @param tangle The tangle reference for the database. + * @param initialSnapshot snapshot that acts as genesis + * @param solid The solidity of the transaction in the database * @return True if the {@link Transaction#solid} has been updated, False if not. */ public boolean updateSolid(Tangle tangle, Snapshot initialSnapshot, boolean solid) throws Exception { @@ -929,13 +910,14 @@ public void updateHeights(Tangle tangle, Snapshot initialSnapshot) throws Except /** * Updates the {@link Transaction#sender}. * - * @param tangle The tangle reference for the database. - * @param initialSnapshot snapshot that acts as genesis - * @param sender The sender of the {@link Transaction} + * @param tangle The tangle reference for the database. + * @param initialSnapshot snapshot that acts as genesis + * @param sender The sender of the {@link Transaction} + * @throws Exception Exception */ public void updateSender(Tangle tangle, Snapshot initialSnapshot, String sender) throws Exception { transaction.sender.set(sender); - update(tangle, initialSnapshot, "sender"); + this.update(tangle, initialSnapshot, "sender"); } /** @return The {@link Transaction#sender} */ diff --git a/src/main/java/com/iota/iri/network/NetworkInjectionConfiguration.java b/src/main/java/com/iota/iri/network/NetworkInjectionConfiguration.java index 032715736d..d4caa618e5 100644 --- a/src/main/java/com/iota/iri/network/NetworkInjectionConfiguration.java +++ b/src/main/java/com/iota/iri/network/NetworkInjectionConfiguration.java @@ -3,7 +3,8 @@ import com.google.inject.AbstractModule; import com.google.inject.Provides; import com.google.inject.Singleton; -import com.iota.iri.TransactionValidator; +import com.iota.iri.service.validation.TransactionSolidifier; +import com.iota.iri.service.validation.TransactionValidator; import com.iota.iri.conf.IotaConfig; import com.iota.iri.controllers.TipsViewModel; import com.iota.iri.network.impl.TipsRequesterImpl; @@ -46,9 +47,9 @@ TipsRequester provideTipsRequester(NeighborRouter neighborRouter, Tangle tangle, TransactionProcessingPipeline provideTransactionProcessingPipeline(NeighborRouter neighborRouter, TransactionValidator txValidator, Tangle tangle, SnapshotProvider snapshotProvider, TipsViewModel tipsViewModel, LatestMilestoneTracker latestMilestoneTracker, - TransactionRequester transactionRequester) { + TransactionRequester transactionRequester, TransactionSolidifier transactionSolidifier) { return new TransactionProcessingPipelineImpl(neighborRouter, configuration, txValidator, tangle, - snapshotProvider, tipsViewModel, latestMilestoneTracker, transactionRequester); + snapshotProvider, tipsViewModel, latestMilestoneTracker, transactionRequester, transactionSolidifier); } @Singleton diff --git a/src/main/java/com/iota/iri/network/pipeline/BroadcastStage.java b/src/main/java/com/iota/iri/network/pipeline/BroadcastStage.java index 4f26765362..8d8848f34c 100644 --- a/src/main/java/com/iota/iri/network/pipeline/BroadcastStage.java +++ b/src/main/java/com/iota/iri/network/pipeline/BroadcastStage.java @@ -3,6 +3,7 @@ import com.iota.iri.controllers.TransactionViewModel; import com.iota.iri.network.NeighborRouter; import com.iota.iri.network.neighbor.Neighbor; +import com.iota.iri.service.validation.TransactionSolidifier; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -18,13 +19,16 @@ public class BroadcastStage implements Stage { private NeighborRouter neighborRouter; + private TransactionSolidifier transactionSolidifier; + /** * Creates a new {@link BroadcastStage}. * * @param neighborRouter The {@link NeighborRouter} instance to use to broadcast */ - public BroadcastStage(NeighborRouter neighborRouter) { + public BroadcastStage(NeighborRouter neighborRouter, TransactionSolidifier transactionSolidifier) { this.neighborRouter = neighborRouter; + this.transactionSolidifier = transactionSolidifier; } /** @@ -54,6 +58,15 @@ public ProcessingContext process(ProcessingContext ctx) { } } + // Check the transaction solidifier to see if there are solid transactions that need to be broadcast. + // If so, forward them to the BroadcastStageQueue to be processed. + TransactionViewModel transactionToBroadcast; + if((transactionToBroadcast = transactionSolidifier.getNextTxInBroadcastQueue()) != null){ + ctx.setNextStage(TransactionProcessingPipeline.Stage.BROADCAST); + ctx.setPayload(new BroadcastPayload(payload.getOriginNeighbor(), transactionToBroadcast)); + return ctx; + } + ctx.setNextStage(TransactionProcessingPipeline.Stage.FINISH); return ctx; } diff --git a/src/main/java/com/iota/iri/network/pipeline/ReceivedStage.java b/src/main/java/com/iota/iri/network/pipeline/ReceivedStage.java index 86a0d3b02b..2a30dd4d5a 100644 --- a/src/main/java/com/iota/iri/network/pipeline/ReceivedStage.java +++ b/src/main/java/com/iota/iri/network/pipeline/ReceivedStage.java @@ -1,6 +1,6 @@ package com.iota.iri.network.pipeline; -import com.iota.iri.TransactionValidator; +import com.iota.iri.service.validation.TransactionSolidifier; import com.iota.iri.controllers.TransactionViewModel; import com.iota.iri.network.TransactionRequester; import com.iota.iri.network.neighbor.Neighbor; @@ -19,19 +19,19 @@ public class ReceivedStage implements Stage { private Tangle tangle; private TransactionRequester transactionRequester; - private TransactionValidator txValidator; + private TransactionSolidifier txSolidifier; private SnapshotProvider snapshotProvider; /** * Creates a new {@link ReceivedStage}. - * + * * @param tangle The {@link Tangle} database used to store/update the transaction - * @param txValidator The {@link TransactionValidator} used to store/update the transaction + * @param txSolidifier The {@link TransactionSolidifier} used to store/update the transaction * @param snapshotProvider The {@link SnapshotProvider} used to store/update the transaction */ - public ReceivedStage(Tangle tangle, TransactionValidator txValidator, SnapshotProvider snapshotProvider, + public ReceivedStage(Tangle tangle, TransactionSolidifier txSolidifier, SnapshotProvider snapshotProvider, TransactionRequester transactionRequester) { - this.txValidator = txValidator; + this.txSolidifier = txSolidifier; this.tangle = tangle; this.snapshotProvider = snapshotProvider; this.transactionRequester = transactionRequester; @@ -39,8 +39,8 @@ public ReceivedStage(Tangle tangle, TransactionValidator txValidator, SnapshotPr /** * Stores the given transaction in the database, updates it status - * ({@link TransactionValidator#updateStatus(TransactionViewModel)}) and updates the sender. - * + * ({@link TransactionSolidifier#updateStatus(TransactionViewModel)}) and updates the sender. + * * @param ctx the received stage {@link ProcessingContext} * @return a {@link ProcessingContext} which redirects to the {@link BroadcastStage} */ @@ -65,7 +65,7 @@ public ProcessingContext process(ProcessingContext ctx) { if (stored) { try { tvm.setArrivalTime(tangle, snapshotProvider.getInitialSnapshot(), System.currentTimeMillis()); - txValidator.updateStatus(tvm); + txSolidifier.updateStatus(tvm); // free up the recently requested transaction set if(transactionRequester.removeRecentlyRequestedTransaction(tvm.getHash())){ @@ -90,8 +90,8 @@ public ProcessingContext process(ProcessingContext ctx) { } // broadcast the newly saved tx to the other neighbors - ctx.setNextStage(TransactionProcessingPipeline.Stage.BROADCAST); - ctx.setPayload(new BroadcastPayload(originNeighbor, tvm)); + ctx.setNextStage(TransactionProcessingPipeline.Stage.SOLIDIFY); + ctx.setPayload(new SolidifyPayload(originNeighbor, tvm)); return ctx; } -} +} \ No newline at end of file diff --git a/src/main/java/com/iota/iri/network/pipeline/SolidifyPayload.java b/src/main/java/com/iota/iri/network/pipeline/SolidifyPayload.java new file mode 100644 index 0000000000..6bd84029ef --- /dev/null +++ b/src/main/java/com/iota/iri/network/pipeline/SolidifyPayload.java @@ -0,0 +1,39 @@ +package com.iota.iri.network.pipeline; + +import com.iota.iri.controllers.TransactionViewModel; +import com.iota.iri.network.neighbor.Neighbor; + +/** + * Defines a payload which gets submitted to the {@link SolidifyStage}. + */ +public class SolidifyPayload extends Payload { + private Neighbor originNeighbor; + private TransactionViewModel tvm; + + /** + * Constructor for solidification payload. + * + * @param originNeighbor The originating point of a received transaction + * @param tvm The transaction that needs to be solidified + */ + public SolidifyPayload(Neighbor originNeighbor, TransactionViewModel tvm){ + this.originNeighbor = originNeighbor; + this.tvm = tvm; + } + + /** + * {@inheritDoc} + */ + @Override + public Neighbor getOriginNeighbor(){ + return originNeighbor; + } + + /** + * Fetches the transaction from the payload. + * @return The transaction stored in the payload. + */ + public TransactionViewModel getTransaction(){ + return tvm; + } +} \ No newline at end of file diff --git a/src/main/java/com/iota/iri/network/pipeline/SolidifyStage.java b/src/main/java/com/iota/iri/network/pipeline/SolidifyStage.java new file mode 100644 index 0000000000..e3d5bd6d72 --- /dev/null +++ b/src/main/java/com/iota/iri/network/pipeline/SolidifyStage.java @@ -0,0 +1,88 @@ +package com.iota.iri.network.pipeline; + +import com.iota.iri.controllers.TipsViewModel; +import com.iota.iri.controllers.TransactionViewModel; +import com.iota.iri.model.Hash; +import com.iota.iri.service.validation.TransactionSolidifier; +import com.iota.iri.storage.Tangle; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import static com.iota.iri.controllers.TransactionViewModel.fromHash; + +/** + * The {@link SolidifyStage} is used to process newly received transaction for solidity. Once a transaction has been + * passed from the {@link ReceivedStage} it will be placed into this stage to have the {@link TransactionSolidifier} + * check the solidity of the transaction. If the transaction is found to be solid, it will be passed forward to the + * {@link BroadcastStage}. If it is found to be unsolid, it is put through the solidity check so that missing reference + * transactions get requested. If the transaction is unsolid, a random solid tip is broadcast instead to keep the + * requests transmitting to neighbors. + */ +public class SolidifyStage implements Stage { + private static final Logger log = LoggerFactory.getLogger(SolidifyStage.class); + + private TransactionSolidifier txSolidifier; + private TipsViewModel tipsViewModel; + private Tangle tangle; + + /** + * Constructor for the {@link SolidifyStage}. + * + * @param txSolidifier Transaction solidifier implementation for determining the validity of a transaction + * @param tipsViewModel Used for broadcasting random solid tips if the subject transaction is unsolid + * @param tangle A reference to the nodes DB + */ + public SolidifyStage(TransactionSolidifier txSolidifier, TipsViewModel tipsViewModel, Tangle tangle){ + this.txSolidifier = txSolidifier; + this.tipsViewModel = tipsViewModel; + this.tangle = tangle; + } + + /** + * Processes the payload of the {@link ProcessingContext} as a {@link SolidifyPayload}. First the transaction will + * be checked for solidity and validity. If the transaction is already solid or can be set solid quickly by the + * transaction solidifier, the transaction is passed to the {@link BroadcastStage}. If not, a random solid tip is + * pulled form the {@link TipsViewModel} to be broadcast instead. + * + * @param ctx The context to process + * @return The output context, in most cases a {@link BroadcastPayload}. + */ + @Override + public ProcessingContext process(ProcessingContext ctx){ + try { + SolidifyPayload payload = (SolidifyPayload) ctx.getPayload(); + TransactionViewModel tvm = payload.getTransaction(); + + if (tvm.isSolid() || txSolidifier.quickSetSolid(tvm)) { + // If the transaction is in the solidifier broadcast queue, remove it as it will be broadcast now + txSolidifier.clearFromBroadcastQueue(tvm); + ctx.setNextStage(TransactionProcessingPipeline.Stage.BROADCAST); + ctx.setPayload(new BroadcastPayload(payload.getOriginNeighbor(), payload.getTransaction())); + return ctx; + } + + return broadcastTip(ctx, payload); + }catch (Exception e){ + log.error("Failed to process transaction for solidification", e); + ctx.setNextStage(TransactionProcessingPipeline.Stage.ABORT); + return ctx; + } + + } + + private ProcessingContext broadcastTip(ProcessingContext ctx, SolidifyPayload payload) throws Exception{ + Hash tipHash = tipsViewModel.getRandomSolidTipHash(); + + if (tipHash == null) { + ctx.setNextStage(TransactionProcessingPipeline.Stage.FINISH); + return ctx; + } + + TransactionViewModel tip = fromHash(tangle, tipHash); + + ctx.setNextStage(TransactionProcessingPipeline.Stage.BROADCAST); + ctx.setPayload(new BroadcastPayload(payload.getOriginNeighbor(), tip)); + + return ctx; + } +} diff --git a/src/main/java/com/iota/iri/network/pipeline/TransactionProcessingPipeline.java b/src/main/java/com/iota/iri/network/pipeline/TransactionProcessingPipeline.java index 1f19e4248c..bf45d44a98 100644 --- a/src/main/java/com/iota/iri/network/pipeline/TransactionProcessingPipeline.java +++ b/src/main/java/com/iota/iri/network/pipeline/TransactionProcessingPipeline.java @@ -14,7 +14,7 @@ public interface TransactionProcessingPipeline { * Defines the different stages of the {@link TransactionProcessingPipelineImpl}. */ enum Stage { - PRE_PROCESS, HASHING, VALIDATION, REPLY, RECEIVED, BROADCAST, MULTIPLE, ABORT, FINISH, + PRE_PROCESS, HASHING, VALIDATION, REPLY, RECEIVED, BROADCAST, MULTIPLE, ABORT, FINISH, SOLIDIFY, } /** @@ -111,4 +111,11 @@ enum Stage { * @param hashingStage the {@link HashingStage} to use */ void setHashingStage(HashingStage hashingStage); -} + + /** + * Sets the solidify stage. This method should only be used for injecting mocked objects. + * + * @param solidifyStage the {@link SolidifyStage} to use + */ + void setSolidifyStage(SolidifyStage solidifyStage); +} \ No newline at end of file diff --git a/src/main/java/com/iota/iri/network/pipeline/TransactionProcessingPipelineImpl.java b/src/main/java/com/iota/iri/network/pipeline/TransactionProcessingPipelineImpl.java index ccdc1ccce7..02f4d1dfa7 100644 --- a/src/main/java/com/iota/iri/network/pipeline/TransactionProcessingPipelineImpl.java +++ b/src/main/java/com/iota/iri/network/pipeline/TransactionProcessingPipelineImpl.java @@ -1,6 +1,7 @@ package com.iota.iri.network.pipeline; -import com.iota.iri.TransactionValidator; +import com.iota.iri.service.validation.TransactionSolidifier; +import com.iota.iri.service.validation.TransactionValidator; import com.iota.iri.conf.NodeConfig; import com.iota.iri.controllers.TipsViewModel; import com.iota.iri.crypto.batched.BatchedHasher; @@ -66,12 +67,14 @@ public class TransactionProcessingPipelineImpl implements TransactionProcessingP private BroadcastStage broadcastStage; private BatchedHasher batchedHasher; private HashingStage hashingStage; + private SolidifyStage solidifyStage; private BlockingQueue preProcessStageQueue = new LinkedBlockingQueue<>(); private BlockingQueue validationStageQueue = new LinkedBlockingQueue<>(); private BlockingQueue receivedStageQueue = new LinkedBlockingQueue<>(); private BlockingQueue broadcastStageQueue = new LinkedBlockingQueue<>(); private BlockingQueue replyStageQueue = new LinkedBlockingQueue<>(); + private BlockingQueue solidifyStageQueue = new LinkedBlockingQueue<>(); /** * Creates a {@link TransactionProcessingPipeline}. @@ -88,16 +91,17 @@ public class TransactionProcessingPipelineImpl implements TransactionProcessingP public TransactionProcessingPipelineImpl(NeighborRouter neighborRouter, NodeConfig config, TransactionValidator txValidator, Tangle tangle, SnapshotProvider snapshotProvider, TipsViewModel tipsViewModel, LatestMilestoneTracker latestMilestoneTracker, - TransactionRequester transactionRequester) { + TransactionRequester transactionRequester, TransactionSolidifier txSolidifier) { FIFOCache recentlySeenBytesCache = new FIFOCache<>(config.getCacheSizeBytes()); this.preProcessStage = new PreProcessStage(recentlySeenBytesCache); this.replyStage = new ReplyStage(neighborRouter, config, tangle, tipsViewModel, latestMilestoneTracker, snapshotProvider, recentlySeenBytesCache); - this.broadcastStage = new BroadcastStage(neighborRouter); + this.broadcastStage = new BroadcastStage(neighborRouter, txSolidifier); this.validationStage = new ValidationStage(txValidator, recentlySeenBytesCache); - this.receivedStage = new ReceivedStage(tangle, txValidator, snapshotProvider, transactionRequester); + this.receivedStage = new ReceivedStage(tangle, txSolidifier, snapshotProvider, transactionRequester); this.batchedHasher = BatchedHasherFactory.create(BatchedHasherFactory.Type.BCTCURL81, 20); this.hashingStage = new HashingStage(batchedHasher); + this.solidifyStage = new SolidifyStage(txSolidifier, tipsViewModel, tangle); } @Override @@ -108,6 +112,7 @@ public void start() { addStage("reply", replyStageQueue, replyStage); addStage("received", receivedStageQueue, receivedStage); addStage("broadcast", broadcastStageQueue, broadcastStage); + addStage("solidify", solidifyStageQueue, solidifyStage); } /** @@ -123,6 +128,7 @@ private void addStage(String name, BlockingQueue queue, try { while (!Thread.currentThread().isInterrupted()) { ProcessingContext ctx = stage.process(queue.take()); + switch (ctx.getNextStage()) { case REPLY: replyStageQueue.put(ctx); @@ -141,6 +147,9 @@ private void addStage(String name, BlockingQueue queue, case BROADCAST: broadcastStageQueue.put(ctx); break; + case SOLIDIFY: + solidifyStageQueue.put(ctx); + break; case ABORT: break; case FINISH: @@ -252,4 +261,9 @@ public void setBroadcastStage(BroadcastStage broadcastStage) { public void setHashingStage(HashingStage hashingStage) { this.hashingStage = hashingStage; } + + @Override + public void setSolidifyStage(SolidifyStage solidifyStage){ + this.solidifyStage = solidifyStage; + } } diff --git a/src/main/java/com/iota/iri/network/pipeline/ValidationStage.java b/src/main/java/com/iota/iri/network/pipeline/ValidationStage.java index a139210eb4..47c41fa3a9 100644 --- a/src/main/java/com/iota/iri/network/pipeline/ValidationStage.java +++ b/src/main/java/com/iota/iri/network/pipeline/ValidationStage.java @@ -1,6 +1,6 @@ package com.iota.iri.network.pipeline; -import com.iota.iri.TransactionValidator; +import com.iota.iri.service.validation.TransactionValidator; import com.iota.iri.controllers.TransactionViewModel; import com.iota.iri.model.Hash; import com.iota.iri.model.HashFactory; diff --git a/src/main/java/com/iota/iri/service/API.java b/src/main/java/com/iota/iri/service/API.java index b7787c516d..f7b76889da 100644 --- a/src/main/java/com/iota/iri/service/API.java +++ b/src/main/java/com/iota/iri/service/API.java @@ -3,7 +3,8 @@ import com.iota.iri.BundleValidator; import com.iota.iri.IRI; import com.iota.iri.IXI; -import com.iota.iri.TransactionValidator; +import com.iota.iri.service.validation.TransactionValidator; +import com.iota.iri.service.validation.TransactionSolidifier; import com.iota.iri.conf.APIConfig; import com.iota.iri.conf.IotaConfig; import com.iota.iri.controllers.AddressViewModel; @@ -146,6 +147,7 @@ public class API { private final TipSelector tipsSelector; private final TipsViewModel tipsViewModel; private final TransactionValidator transactionValidator; + private final TransactionSolidifier transactionSolidifier; private final LatestMilestoneTracker latestMilestoneTracker; private final TipSelSolidifier dummySolidifier; @@ -187,14 +189,16 @@ public class API { * @param tipsViewModel Contains the current tips of this node * @param transactionValidator Validates transactions * @param latestMilestoneTracker Service that tracks the latest milestone + * @param transactionSolidifier Holds transaction pipeline, including broadcast transactions * @param dummySolidifier Solidifies transactions. A dummy object is used by default as a placeholder. + * */ public API(IotaConfig configuration, IXI ixi, TransactionRequester transactionRequester, SpentAddressesService spentAddressesService, Tangle tangle, BundleValidator bundleValidator, SnapshotProvider snapshotProvider, LedgerService ledgerService, NeighborRouter neighborRouter, TipSelector tipsSelector, TipsViewModel tipsViewModel, TransactionValidator transactionValidator, LatestMilestoneTracker latestMilestoneTracker, TransactionProcessingPipeline txPipeline, - TipSelSolidifier dummySolidifier) { + TransactionSolidifier transactionSolidifier, TipSelSolidifier dummySolidifier) { this.configuration = configuration; this.ixi = ixi; @@ -209,6 +213,7 @@ public API(IotaConfig configuration, IXI ixi, TransactionRequester transactionRe this.tipsSelector = tipsSelector; this.tipsViewModel = tipsViewModel; this.transactionValidator = transactionValidator; + this.transactionSolidifier = transactionSolidifier; this.latestMilestoneTracker = latestMilestoneTracker; this.dummySolidifier = dummySolidifier; @@ -718,8 +723,8 @@ public AbstractResponse storeTransactionsStatement(List trytes) throws E //store transactions if(transactionViewModel.store(tangle, snapshotProvider.getInitialSnapshot())) { transactionViewModel.setArrivalTime(tangle, snapshotProvider.getInitialSnapshot(), System.currentTimeMillis()); - transactionValidator.updateStatus(transactionViewModel); - transactionViewModel.updateSender(tangle, snapshotProvider.getInitialSnapshot(),"local"); + transactionSolidifier.updateStatus(transactionViewModel); + transactionViewModel.updateSender(tangle, snapshotProvider.getInitialSnapshot(), "local"); } } return AbstractResponse.createEmptyResponse(); diff --git a/src/main/java/com/iota/iri/service/milestone/MilestoneSolidifier.java b/src/main/java/com/iota/iri/service/milestone/MilestoneSolidifier.java index a88a900b52..f17c7172c7 100644 --- a/src/main/java/com/iota/iri/service/milestone/MilestoneSolidifier.java +++ b/src/main/java/com/iota/iri/service/milestone/MilestoneSolidifier.java @@ -1,7 +1,7 @@ package com.iota.iri.service.milestone; -import com.iota.iri.TransactionValidator; import com.iota.iri.model.Hash; +import com.iota.iri.service.validation.TransactionSolidifier; /** * This interface defines the contract for a manager that tries to solidify unsolid milestones by incorporating a @@ -17,7 +17,7 @@ public interface MilestoneSolidifier { *

*

* Note: We want to find the next previous milestone and not get stuck somewhere at the end of the tangle with a - * long running {@link TransactionValidator#checkSolidity(Hash)} call. + * long running {@link TransactionSolidifier#checkSolidity(Hash)} call. *

*/ int SOLIDIFICATION_TRANSACTIONS_LIMIT = 50000; diff --git a/src/main/java/com/iota/iri/service/milestone/impl/MilestoneSolidifierImpl.java b/src/main/java/com/iota/iri/service/milestone/impl/MilestoneSolidifierImpl.java index a7677373ce..9db1887a42 100644 --- a/src/main/java/com/iota/iri/service/milestone/impl/MilestoneSolidifierImpl.java +++ b/src/main/java/com/iota/iri/service/milestone/impl/MilestoneSolidifierImpl.java @@ -1,6 +1,6 @@ package com.iota.iri.service.milestone.impl; -import com.iota.iri.TransactionValidator; +import com.iota.iri.service.validation.TransactionSolidifier; import com.iota.iri.model.Hash; import com.iota.iri.service.milestone.MilestoneSolidifier; import com.iota.iri.service.snapshot.SnapshotProvider; @@ -35,7 +35,19 @@ public class MilestoneSolidifierImpl implements MilestoneSolidifier { /** * Defines the interval in which solidity checks are issued (in milliseconds). */ - private static final int SOLIDIFICATION_INTERVAL = 500; + private static final int SOLIDIFICATION_INTERVAL = 100; + + /** + *

+ * Defines the maximum amount of transactions that are allowed to get processed while trying to solidify a + * milestone. + *

+ *

+ * Note: We want to find the next previous milestone and not get stuck somewhere at the end of the tangle with a + * long running {@link TransactionSolidifier#checkSolidity(Hash)} call. + *

+ */ + private static final int SOLIDIFICATION_TRANSACTIONS_LIMIT = 50000; /** * Logger for this class allowing us to dump debug and status messages. @@ -48,9 +60,9 @@ public class MilestoneSolidifierImpl implements MilestoneSolidifier { private final SnapshotProvider snapshotProvider; /** - * Holds a reference to the TransactionValidator which allows us to issue solidity checks. + * Holds a reference to the transactionSolidifier which allows us to issue solidity checks. */ - private final TransactionValidator transactionValidator; + private final TransactionSolidifier transactionSolidifier; /** * Holds a reference to the manager of the background worker. @@ -93,16 +105,16 @@ public class MilestoneSolidifierImpl implements MilestoneSolidifier { /** * @param snapshotProvider snapshot provider which gives us access to the relevant snapshots - * @param transactionValidator TransactionValidator instance that is used by the node + * @param transactionSolidifier transactionSolidifier instance that is used by the node */ - public MilestoneSolidifierImpl(SnapshotProvider snapshotProvider, TransactionValidator transactionValidator) { + public MilestoneSolidifierImpl(SnapshotProvider snapshotProvider, TransactionSolidifier transactionSolidifier) { this.snapshotProvider = snapshotProvider; - this.transactionValidator = transactionValidator; + this.transactionSolidifier = transactionSolidifier; } /** * {@inheritDoc} - * + * *

* Since this method might be called from a performance critical context, we simply add the milestone to a temporary * pool, that gets examined later by the background process. This doesn't just speed up the addition of new jobs but @@ -194,7 +206,7 @@ private void milestoneSolidificationThread() { */ private void processNewlyAddedMilestones() { for (Iterator> iterator = newlyAddedMilestones.entrySet().iterator(); - !Thread.currentThread().isInterrupted() && iterator.hasNext();) { + !Thread.currentThread().isInterrupted() && iterator.hasNext();) { Map.Entry currentEntry = iterator.next(); @@ -220,7 +232,7 @@ private void processNewlyAddedMilestones() { */ private void processSolidificationQueue() { for (Iterator> iterator = milestonesToSolidify.entrySet().iterator(); - !Thread.currentThread().isInterrupted() && iterator.hasNext();) { + !Thread.currentThread().isInterrupted() && iterator.hasNext();) { Map.Entry currentEntry = iterator.next(); @@ -309,7 +321,7 @@ private Map.Entry getNextSolidificationCandidate() { *

*

* It first dumps a log message to keep the node operator informed about the progress of solidification, and then - * issues the {@link TransactionValidator#checkSolidity(Hash, int)} call that starts the solidification + * issues the {@link TransactionSolidifier#checkSolidity(Hash, int)} call that starts the solidification * process. *

*

@@ -329,11 +341,11 @@ private boolean isSolid(Map.Entry currentEntry) { } try { - return transactionValidator.checkSolidity(currentEntry.getKey(), SOLIDIFICATION_TRANSACTIONS_LIMIT); + return transactionSolidifier.checkSolidity(currentEntry.getKey(), SOLIDIFICATION_TRANSACTIONS_LIMIT); } catch (Exception e) { log.error("Error while solidifying milestone #" + currentEntry.getValue(), e); return false; } } -} +} \ No newline at end of file diff --git a/src/main/java/com/iota/iri/service/tipselection/impl/AsyncTipSelSolidifier.java b/src/main/java/com/iota/iri/service/tipselection/impl/AsyncTipSelSolidifier.java index cdff07a95f..b10b160e3f 100644 --- a/src/main/java/com/iota/iri/service/tipselection/impl/AsyncTipSelSolidifier.java +++ b/src/main/java/com/iota/iri/service/tipselection/impl/AsyncTipSelSolidifier.java @@ -1,6 +1,5 @@ package com.iota.iri.service.tipselection.impl; -import com.iota.iri.TransactionValidator; import com.iota.iri.model.Hash; import com.iota.iri.service.milestone.MilestoneSolidifier; import com.iota.iri.service.tipselection.TipSelSolidifier; @@ -12,6 +11,7 @@ import java.util.concurrent.ThreadFactory; import com.google.common.util.concurrent.ThreadFactoryBuilder; +import com.iota.iri.service.validation.TransactionSolidifier; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -20,12 +20,12 @@ public class AsyncTipSelSolidifier implements TipSelSolidifier { private static final Logger log = LoggerFactory.getLogger(AsyncTipSelSolidifier.class); - private final TransactionValidator transactionValidator; + private final TransactionSolidifier transactionSolidifier; private final Set transactionToSolidify = ConcurrentHashMap.newKeySet(); private final ExecutorService solidExecutor; - public AsyncTipSelSolidifier(TransactionValidator transactionValidator) { - this.transactionValidator = transactionValidator; + public AsyncTipSelSolidifier(TransactionSolidifier transactionSolidifier) { + this.transactionSolidifier = transactionSolidifier; ThreadFactory threadFactory = new ThreadFactoryBuilder() .setNameFormat(this.getClass().getSimpleName() + " %d") .build(); @@ -41,7 +41,7 @@ public void solidify(Hash transactionHash) { solidExecutor.submit(() -> { try { log.debug("attempting to solidify transaction {}", transactionHash); - transactionValidator.checkSolidity(transactionHash, MilestoneSolidifier.SOLIDIFICATION_TRANSACTIONS_LIMIT); + transactionSolidifier.checkSolidity(transactionHash, MilestoneSolidifier.SOLIDIFICATION_TRANSACTIONS_LIMIT); } catch (Exception e) { log.error("Failed to solidify transaction during a walk", e); } diff --git a/src/main/java/com/iota/iri/service/validation/TransactionSolidifier.java b/src/main/java/com/iota/iri/service/validation/TransactionSolidifier.java new file mode 100644 index 0000000000..37c57fef42 --- /dev/null +++ b/src/main/java/com/iota/iri/service/validation/TransactionSolidifier.java @@ -0,0 +1,101 @@ +package com.iota.iri.service.validation; + +import com.iota.iri.controllers.TipsViewModel; +import com.iota.iri.controllers.TransactionViewModel; +import com.iota.iri.model.Hash; +import com.iota.iri.service.validation.impl.TransactionSolidifierImpl; +import com.iota.iri.network.TransactionRequester; + +/** + * Solidification tool. Transactions placed into the solidification queue will be checked for solidity. Any missing + * reference transactions will be placed into the {@link TransactionRequester}. If a transaction is found to be solid + * it is updated as such and placed into the BroadcastQueue to be sent off to the node's neighbours. + */ +public interface TransactionSolidifier { + + /** + * Initialize the executor service. Start processing transactions to solidify. + */ + void start(); + + /** + * Interrupt thread processes and shut down the executor service. + */ + void shutdown(); + + /** + * Fetch the next transaction in the transactionsToBroadcast set. + * @return A {@link TransactionViewModel} object to be broadcast. + */ + TransactionViewModel getNextTxInBroadcastQueue(); + + /** + * Remove a broadcasted transaction from the transactionsToBroadcast set + * @param transactionsBroadcasted A {@link TransactionViewModel} object to remove from the set. + */ + void clearFromBroadcastQueue(TransactionViewModel transactionsBroadcasted); + + /** + * This method does the same as {@link #checkSolidity(Hash, int)} but defaults to an unlimited amount + * of transactions that are allowed to be traversed. + * + * @param hash hash of the transactions that shall get checked + * @return true if the transaction is solid and false otherwise + * @throws Exception if anything goes wrong while trying to solidify the transaction + */ + boolean checkSolidity(Hash hash) throws Exception; + + /** + * This method checks transactions for solidity and marks them accordingly if they are found to be solid. + * + * It iterates through all approved transactions until it finds one that is missing in the database or until it + * reached solid transactions on all traversed subtangles. In case of a missing transactions it issues a transaction + * request and returns false. If no missing transaction is found, it marks the processed transactions as solid in + * the database and returns true. + * + * Since this operation can potentially take a long time to terminate if it would have to traverse big parts of the + * tangle, it is possible to limit the amount of transactions that are allowed to be processed, while looking for + * unsolid / missing approvees. This can be useful when trying to "interrupt" the solidification of one transaction + * (if it takes too many steps) to give another one the chance to be solidified instead (i.e. prevent blocks in the + * solidification threads). + * + * @param hash hash of the transactions that shall get checked + * @param maxProcessedTransactions the maximum amount of transactions that are allowed to be traversed + * @return true if the transaction is solid and false otherwise + * @throws Exception if anything goes wrong while trying to solidify the transaction + */ + boolean checkSolidity(Hash hash, int maxProcessedTransactions) throws Exception; + + /** + * Updates a transaction after it was stored in the tangle. Tells the node to not request the transaction anymore, + * to update the live tips accordingly, and attempts to quickly solidify the transaction. + * + *

+ * Performs the following operations: + * + *

    + *
  1. Removes {@code transactionViewModel}'s hash from the the request queue since we already found it.
    2. + *
  2. If {@code transactionViewModel} has no children (approvers), we add it to the node's active tip list.
    3. + *
  3. Removes {@code transactionViewModel}'s parents (branch & trunk) from the node's tip list + * (if they're present there).
    4. + *
  4. Attempts to quickly solidify {@code transactionViewModel} by checking whether its direct parents + * are solid. If solid we add it to the queue transaction solidification thread to help it propagate the + * solidification to the approving child transactions.
    5. + *
  5. Requests missing direct parent (trunk & branch) transactions that are needed to solidify + * {@code transactionViewModel}.
    6. + *
+ * @param transactionViewModel received transaction that is being updated + * @throws Exception if an error occurred while trying to solidify + * @see TipsViewModel + */ + void updateStatus(TransactionViewModel transactionViewModel) throws Exception; + + /** + * Tries to solidify the transactions quickly by performing {@link TransactionSolidifierImpl#checkApproovee} on + * both parents (trunk and branch). If the parents are solid, mark the transactions as solid. + * @param transactionViewModel transaction to solidify + * @return true if we made the transaction solid, else false. + * @throws Exception + */ + boolean quickSetSolid(TransactionViewModel transactionViewModel) throws Exception; +} \ No newline at end of file diff --git a/src/main/java/com/iota/iri/service/validation/TransactionValidator.java b/src/main/java/com/iota/iri/service/validation/TransactionValidator.java new file mode 100644 index 0000000000..dd4ea75028 --- /dev/null +++ b/src/main/java/com/iota/iri/service/validation/TransactionValidator.java @@ -0,0 +1,170 @@ +package com.iota.iri.service.validation; + +import com.google.common.annotations.VisibleForTesting; +import com.iota.iri.conf.ProtocolConfig; +import com.iota.iri.controllers.TransactionViewModel; +import com.iota.iri.crypto.Curl; +import com.iota.iri.crypto.Sponge; +import com.iota.iri.crypto.SpongeFactory; +import com.iota.iri.model.TransactionHash; +import com.iota.iri.network.TransactionRequester; +import com.iota.iri.service.snapshot.SnapshotProvider; + +/** + * Tool for determining validity of a transaction via a {@link TransactionViewModel}, tryte array or byte array. + */ +public class TransactionValidator { + private static final int TESTNET_MWM_CAP = 13; + + private final SnapshotProvider snapshotProvider; + private final TransactionRequester transactionRequester; + private int minWeightMagnitude = 81; + private static final long MAX_TIMESTAMP_FUTURE = 2L * 60L * 60L; + private static final long MAX_TIMESTAMP_FUTURE_MS = MAX_TIMESTAMP_FUTURE * 1_000L; + + + /** + * Constructor for Tangle Validator + * + * @param snapshotProvider data provider for the snapshots that are relevant for the node + * @param transactionRequester used to request missing transactions from neighbors + * @param protocolConfig used for checking if we are in testnet and mwm. testnet true if we are in testnet + * mode, this caps {@code mwm} to {@value #TESTNET_MWM_CAP} regardless of parameter input. + * minimum weight magnitude: the minimal number of 9s that ought to appear at the end of the + * transaction hash + */ + public TransactionValidator(SnapshotProvider snapshotProvider, TransactionRequester transactionRequester, ProtocolConfig protocolConfig) { + this.snapshotProvider = snapshotProvider; + this.transactionRequester = transactionRequester; + setMwm(protocolConfig.isTestnet(), protocolConfig.getMwm()); + } + + /** + * Set the Minimum Weight Magnitude for validation checks. + */ + @VisibleForTesting + void setMwm(boolean testnet, int mwm) { + minWeightMagnitude = mwm; + + //lowest allowed MWM encoded in 46 bytes. + if (!testnet){ + minWeightMagnitude = Math.max(minWeightMagnitude, TESTNET_MWM_CAP); + } + } + + /** + * @return the minimal number of trailing 9s that have to be present at the end of the transaction hash + * in order to validate that sufficient proof of work has been done + */ + public int getMinWeightMagnitude() { + return minWeightMagnitude; + } + + /** + * Checks that the timestamp of the transaction is below the last global snapshot time + * or more than {@value #MAX_TIMESTAMP_FUTURE} seconds in the future, and thus invalid. + * + *

+ * First the attachment timestamp (set after performing POW) is checked, and if not available + * the regular timestamp is checked. Genesis transaction will always be valid. + *

+ * @param transactionViewModel transaction under test + * @return true if timestamp is not in valid bounds and {@code transactionViewModel} is not genesis. + * Else returns false. + */ + private boolean hasInvalidTimestamp(TransactionViewModel transactionViewModel) { + // ignore invalid timestamps for transactions that were requested by our node while solidifying a milestone + if(transactionRequester.wasTransactionRecentlyRequested(transactionViewModel.getHash())) { + return false; + } + + if (transactionViewModel.getAttachmentTimestamp() == 0) { + return transactionViewModel.getTimestamp() < snapshotProvider.getInitialSnapshot().getTimestamp() && !snapshotProvider.getInitialSnapshot().hasSolidEntryPoint(transactionViewModel.getHash()) + || transactionViewModel.getTimestamp() > (System.currentTimeMillis() / 1000) + MAX_TIMESTAMP_FUTURE; + } + return transactionViewModel.getAttachmentTimestamp() < (snapshotProvider.getInitialSnapshot().getTimestamp() * 1000L) + || transactionViewModel.getAttachmentTimestamp() > System.currentTimeMillis() + MAX_TIMESTAMP_FUTURE_MS; + } + + /** + * Runs the following validation checks on a transaction: + *
    + *
  1. {@link #hasInvalidTimestamp} check.
    2. + *
  2. Check that no value trits are set beyond the usable index, otherwise we will have values larger + * than max supply.
    3. + *
  3. Check that sufficient POW was performed.
    4. + *
  4. In value transactions, we check that the address has 0 set as the last trit. This must be because of the + * conversion between bytes to trits.
    5. + *
+ *Exception is thrown upon failure. + * + * @param transactionViewModel transaction that should be validated + * @param minWeightMagnitude the minimal number of trailing 9s at the end of the transaction hash + * @throws StaleTimestampException if timestamp check fails + * @throws IllegalStateException if any of the other checks fail + */ + public void runValidation(TransactionViewModel transactionViewModel, final int minWeightMagnitude) { + transactionViewModel.setMetadata(); + transactionViewModel.setAttachmentData(); + if (hasInvalidTimestamp(transactionViewModel)) { + throw new StaleTimestampException("Invalid transaction timestamp."); + } + + confirmValidTransactionValues(transactionViewModel); + int weightMagnitude = transactionViewModel.weightMagnitude; + if (weightMagnitude < minWeightMagnitude) { + throw new IllegalStateException("Invalid transaction hash"); + } + + if (transactionViewModel.value() != 0 && transactionViewModel.getAddressHash().trits()[Curl.HASH_LENGTH - 1] != 0) { + throw new IllegalStateException("Invalid transaction address"); + } + } + + private void confirmValidTransactionValues(TransactionViewModel transactionViewModel) throws IllegalStateException { + for (int i = TransactionViewModel.VALUE_TRINARY_OFFSET + TransactionViewModel.VALUE_USABLE_TRINARY_SIZE; + i < TransactionViewModel.VALUE_TRINARY_OFFSET + TransactionViewModel.VALUE_TRINARY_SIZE; i++) { + if (transactionViewModel.trits()[i] != 0) { + throw new IllegalStateException("Invalid transaction value"); + } + } + } + + /** + * Creates a new transaction from {@code trits} and validates it with {@link #runValidation}. + * + * @param trits raw transaction trits + * @param minWeightMagnitude minimal number of trailing 9s in transaction for POW validation + * @return the transaction resulting from the raw trits if valid. + * @throws RuntimeException if validation fails + */ + public TransactionViewModel validateTrits(final byte[] trits, int minWeightMagnitude) { + TransactionViewModel transactionViewModel = new TransactionViewModel(trits, TransactionHash.calculate(trits, 0, trits.length, SpongeFactory.create(SpongeFactory.Mode.CURLP81))); + runValidation(transactionViewModel, minWeightMagnitude); + return transactionViewModel; + } + + /** + * Creates a new transaction from {@code bytes} and validates it with {@link #runValidation}. + * + * @param bytes raw transaction bytes + * @param minWeightMagnitude minimal number of trailing 9s in transaction for POW validation + * @return the transaction resulting from the raw bytes if valid + * @throws RuntimeException if validation fails + */ + public TransactionViewModel validateBytes(final byte[] bytes, int minWeightMagnitude, Sponge curl) { + TransactionViewModel transactionViewModel = new TransactionViewModel(bytes, TransactionHash.calculate(bytes, + TransactionViewModel.TRINARY_SIZE, curl)); + runValidation(transactionViewModel, minWeightMagnitude); + return transactionViewModel; + } + + /** + * Thrown if transaction fails {@link #hasInvalidTimestamp} check. + */ + public static class StaleTimestampException extends RuntimeException { + StaleTimestampException (String message) { + super(message); + } + } +} diff --git a/src/main/java/com/iota/iri/service/validation/impl/TransactionSolidifierImpl.java b/src/main/java/com/iota/iri/service/validation/impl/TransactionSolidifierImpl.java new file mode 100644 index 0000000000..9c5c9f7363 --- /dev/null +++ b/src/main/java/com/iota/iri/service/validation/impl/TransactionSolidifierImpl.java @@ -0,0 +1,334 @@ +package com.iota.iri.service.validation.impl; + +import com.google.common.annotations.VisibleForTesting; +import com.iota.iri.controllers.TipsViewModel; +import com.iota.iri.controllers.TransactionViewModel; +import com.iota.iri.model.Hash; +import com.iota.iri.network.pipeline.TransactionProcessingPipeline; +import com.iota.iri.network.TransactionRequester; +import com.iota.iri.service.snapshot.SnapshotProvider; +import com.iota.iri.service.validation.TransactionSolidifier; +import com.iota.iri.storage.Tangle; +import com.iota.iri.utils.log.interval.IntervalLogger; +import com.iota.iri.utils.thread.DedicatedScheduledExecutorService; +import com.iota.iri.utils.thread.SilentScheduledExecutorService; + +import java.util.*; +import java.util.concurrent.*; + +import static com.iota.iri.controllers.TransactionViewModel.PREFILLED_SLOT; +import static com.iota.iri.controllers.TransactionViewModel.fromHash; + +/** + * A solidifier class for processing transactions. Transactions are checked for solidity, and missing transactions are + * subsequently requested. Once a transaction is solidified correctly it is placed into a broadcasting set to be sent to + * neighboring nodes. + */ +public class TransactionSolidifierImpl implements TransactionSolidifier { + + private Tangle tangle; + private SnapshotProvider snapshotProvider; + private TransactionRequester transactionRequester; + + /** + * Max size for all queues. + */ + private static final int MAX_SIZE= 10000; + + private static final int SOLIDIFICATION_INTERVAL = 100; + + private static final IntervalLogger log = new IntervalLogger(TransactionSolidifier.class); + + /** + * Executor service for running the {@link #processTransactionsToSolidify()}. + */ + private SilentScheduledExecutorService executorService = new DedicatedScheduledExecutorService( + "Transaction Solidifier", log.delegate()); + + /** + * A set of transactions that will be called by the {@link TransactionProcessingPipeline} to be broadcast to + * neighboring nodes. + */ + private BlockingQueue transactionsToBroadcast = new ArrayBlockingQueue<>(MAX_SIZE); + + private TipsViewModel tipsViewModel; + + private TransactionPropagator transactionPropagator; + + /** + * Constructor for the solidifier. + * @param tangle The DB reference + * @param snapshotProvider For fetching entry points for solidity checks + * @param transactionRequester A requester for missing transactions + */ + public TransactionSolidifierImpl(Tangle tangle, SnapshotProvider snapshotProvider, TransactionRequester transactionRequester, + TipsViewModel tipsViewModel){ + this.tangle = tangle; + this.snapshotProvider = snapshotProvider; + this.transactionRequester = transactionRequester; + this.tipsViewModel = tipsViewModel; + this.transactionPropagator = new TransactionPropagator(); + } + + /** + *{@inheritDoc} + */ + @Override + public void start(){ + executorService.silentScheduleWithFixedDelay(this::processTransactionsToSolidify, 0, + SOLIDIFICATION_INTERVAL, TimeUnit.MILLISECONDS); + } + + /** + *{@inheritDoc} + */ + @Override + public void shutdown() { + executorService.shutdownNow(); + } + + /** + *{@inheritDoc} + */ + @Override + public TransactionViewModel getNextTxInBroadcastQueue(){ + return transactionsToBroadcast.poll(); + } + + /** + *{@inheritDoc} + */ + @Override + public void clearFromBroadcastQueue(TransactionViewModel transaction){ + transactionsToBroadcast.remove(transaction); + } + + + /** + * Process any solid transactions present in the {@link TransactionPropagator}. + */ + private void processTransactionsToSolidify(){ + transactionPropagator.propagateSolidTransactions(); + } + + /** + *{@inheritDoc} + */ + @Override + public boolean checkSolidity(Hash hash) throws Exception { + return checkSolidity(hash, 50000); + } + + /** + *{@inheritDoc} + */ + @Override + public boolean checkSolidity(Hash hash, int maxProcessedTransactions) throws Exception { + if(fromHash(tangle, hash).isSolid()) { + return true; + } + LinkedHashSet analyzedHashes = new LinkedHashSet<>(snapshotProvider.getInitialSnapshot().getSolidEntryPoints().keySet()); + maxProcessedTransactions = Math.addExact(maxProcessedTransactions, analyzedHashes.size()); + boolean solid = true; + final Deque nonAnalyzedTransactions = new ArrayDeque<>(Collections.singleton(hash)); + Hash hashPointer; + while ((hashPointer = nonAnalyzedTransactions.poll()) != null) { + if (!analyzedHashes.add(hashPointer)) { + continue; + } + + if (analyzedHashes.size() >= maxProcessedTransactions) { + return false; + } + + TransactionViewModel transaction = fromHash(tangle, hashPointer); + if (isUnsolidWithoutEntryPoint(transaction, hashPointer)) { + if (transaction.getType() == PREFILLED_SLOT) { + solid = false; + checkRequester(hashPointer); + } else { + nonAnalyzedTransactions.offer(transaction.getTrunkTransactionHash()); + nonAnalyzedTransactions.offer(transaction.getBranchTransactionHash()); + } + } + } + if (solid) { + updateTransactions(analyzedHashes); + } + analyzedHashes.clear(); + return solid; + } + + + /** + * Check if a transaction is present in the {@link #transactionRequester}, if not, it is added. + * @param hashPointer The hash of the transaction to request + */ + private void checkRequester(Hash hashPointer){ + if (!transactionRequester.isTransactionRequested(hashPointer)) { + transactionRequester.requestTransaction(hashPointer); + } + } + + /** + * Iterate through analyzed hashes and place them in the {@link #transactionsToBroadcast} queue + * @param hashes Analyzed hashes from the {@link #checkSolidity(Hash)} call + */ + private void updateTransactions(Set hashes) { + hashes.forEach(hash -> { + try { + TransactionViewModel tvm = fromHash(tangle, hash); + tvm.updateHeights(tangle, snapshotProvider.getInitialSnapshot()); + + if(!tvm.isSolid()){ + tvm.updateSolid(tangle, snapshotProvider.getInitialSnapshot(), true); + } + addToBroadcastQueue(tvm); + transactionPropagator.addToPropagationQueue(tvm.getHash()); + } catch (Exception e) { + log.info(e.getMessage()); + } + }); + } + + /** + * Returns true if transaction is not solid and there are no solid entry points from the initial snapshot. + */ + private boolean isUnsolidWithoutEntryPoint(TransactionViewModel transaction, Hash hashPointer) throws Exception{ + if(!transaction.isSolid() && !snapshotProvider.getInitialSnapshot().hasSolidEntryPoint(hashPointer)){ + return true; + } + transactionPropagator.addToPropagationQueue(hashPointer); + return false; + } + + + private void addToBroadcastQueue(TransactionViewModel tvm) { + try { + if (transactionsToBroadcast.size() >= MAX_SIZE) { + transactionsToBroadcast.remove(); + } + + transactionsToBroadcast.put(tvm); + } catch(Exception e){ + log.info("Error placing transaction into broadcast queue: " + e.getMessage()); + } + } + + @Override + public void updateStatus(TransactionViewModel transactionViewModel) throws Exception { + transactionRequester.clearTransactionRequest(transactionViewModel.getHash()); + if(transactionViewModel.getApprovers(tangle).size() == 0) { + tipsViewModel.addTipHash(transactionViewModel.getHash()); + } + tipsViewModel.removeTipHash(transactionViewModel.getTrunkTransactionHash()); + tipsViewModel.removeTipHash(transactionViewModel.getBranchTransactionHash()); + + if(quickSetSolid(transactionViewModel)) { + transactionViewModel.update(tangle, snapshotProvider.getInitialSnapshot(), "solid|height"); + tipsViewModel.setSolid(transactionViewModel.getHash()); + transactionPropagator.addToPropagationQueue(transactionViewModel.getHash()); + } + } + + + @Override + public boolean quickSetSolid(final TransactionViewModel transactionViewModel) throws Exception { + if(!transactionViewModel.isSolid()) { + boolean solid = true; + if (!checkApproovee(transactionViewModel.getTrunkTransaction(tangle))) { + solid = false; + } + if (!checkApproovee(transactionViewModel.getBranchTransaction(tangle))) { + solid = false; + } + if(solid) { + transactionViewModel.updateSolid(tangle, snapshotProvider.getInitialSnapshot(), true); + transactionViewModel.updateHeights(tangle, snapshotProvider.getInitialSnapshot()); + transactionPropagator.addToPropagationQueue(transactionViewModel.getHash()); + addToBroadcastQueue(transactionViewModel); + return true; + } + } + return false; + } + + /** + * If the the {@code approvee} is missing, request it from a neighbor. + * @param approovee transaction we check. + * @return true if {@code approvee} is solid. + * @throws Exception if we encounter an error while requesting a transaction + */ + private boolean checkApproovee(TransactionViewModel approovee) throws Exception { + if(snapshotProvider.getInitialSnapshot().hasSolidEntryPoint(approovee.getHash())) { + return true; + } + if(approovee.getType() == PREFILLED_SLOT) { + // don't solidify from the bottom until cuckoo filters can identify where we deleted -> otherwise we will + // continue requesting old transactions forever + //transactionRequester.requestTransaction(approovee.getHash(), false); + return false; + } + return approovee.isSolid(); + } + + /** + * A transaction solidification service that propagates upwards through transactions approving a solid transaction + * and performs {@link #quickSetSolid} checks to determine their solidity as well + */ + private class TransactionPropagator { + /** + * A queue for processing transactions with the {@link #propagateSolidTransactions()} call. This will check + * approving transactions with {@link #quickSetSolid(TransactionViewModel)}. + */ + private BlockingQueue solidTransactions = new ArrayBlockingQueue<>(MAX_SIZE); + + /** + * Add to the propagation queue where it will be processed to help solidify approving transactions faster + * @param hash The transaction hash to be removed + * @throws Exception + */ + public void addToPropagationQueue(Hash hash) throws Exception{ + if(!solidTransactions.contains(hash)) { + if (solidTransactions.size() >= MAX_SIZE) { + solidTransactions.poll(); + } + solidTransactions.put(hash); + } + } + + @VisibleForTesting + void propagateSolidTransactions() { + while(!Thread.currentThread().isInterrupted() && solidTransactions.peek() != null) { + try { + Hash hash = solidTransactions.poll(); + TransactionViewModel transaction = fromHash(tangle, hash); + Set approvers = transaction.getApprovers(tangle).getHashes(); + for(Hash h: approvers) { + TransactionViewModel tx = fromHash(tangle, h); + if (quietQuickSetSolid(tx)) { + tx.update(tangle, snapshotProvider.getInitialSnapshot(), "solid|height"); + tipsViewModel.setSolid(h); + } + } + } catch (Exception e) { + log.error("Error while propagating solidity upwards", e); + } + } + } + + /** + * Perform a {@link #quickSetSolid} while capturing and logging errors + * @param transactionViewModel transaction we try to solidify. + * @return true if we managed to solidify, else false. + */ + private boolean quietQuickSetSolid(TransactionViewModel transactionViewModel) { + try { + return quickSetSolid(transactionViewModel); + } catch (Exception e) { + log.error(e.getMessage(), e); + return false; + } + } + } +} \ No newline at end of file diff --git a/src/main/java/com/iota/iri/zmq/README.md b/src/main/java/com/iota/iri/zmq/README.md index b32c232d65..4b8fe53e59 100644 --- a/src/main/java/com/iota/iri/zmq/README.md +++ b/src/main/java/com/iota/iri/zmq/README.md @@ -28,7 +28,7 @@ Topics currently found in the latest code are * `sn` for newly confirmed transactions ( by solid milestone children measurement ) * `sn_trytes` for raw trytes, transaction hash, and snapshot index of confirmed transactions. * `tx` for newly seen transactions -* `tx_trytes` trytes of newly seen transactions +* `trytes` trytes of newly seen transactions * `
` to watch activity on an address All topic must be lowercase (to not clash with `
` containing the topic title - like `TXCR9...` & `TX`) diff --git a/src/main/java/com/iota/iri/zmq/ZmqMessageQueueProvider.java b/src/main/java/com/iota/iri/zmq/ZmqMessageQueueProvider.java index 289bce9a6a..fd167224b9 100644 --- a/src/main/java/com/iota/iri/zmq/ZmqMessageQueueProvider.java +++ b/src/main/java/com/iota/iri/zmq/ZmqMessageQueueProvider.java @@ -40,7 +40,7 @@ public boolean publishTransaction(Persistable model, Indexable index, String ite TransactionViewModel transactionViewModel = new TransactionViewModel(transaction, (Hash)index); publishTx(transactionViewModel); - publishTxTrytes(transactionViewModel); + publishTrytes(transactionViewModel); return true; } @@ -70,18 +70,18 @@ private void publishTx(TransactionViewModel transactionViewModel) { } } - private void publishTxTrytes(TransactionViewModel transactionViewModel) { - StringBuilder txTrytesStringBuilder = new StringBuilder(TransactionViewModel.TRINARY_SIZE/3); + private void publishTrytes(TransactionViewModel transactionViewModel) { + StringBuilder trytesStringBuilder = new StringBuilder(TransactionViewModel.TRINARY_SIZE/3); try { - txTrytesStringBuilder.append("tx_trytes "); - txTrytesStringBuilder.append(Converter.trytes(transactionViewModel.trits())); txTrytesStringBuilder.append(" "); - txTrytesStringBuilder.append(transactionViewModel.getHash()); + trytesStringBuilder.append("trytes "); + trytesStringBuilder.append(Converter.trytes(transactionViewModel.trits())); trytesStringBuilder.append(" "); + trytesStringBuilder.append(transactionViewModel.getHash()); - messageQ.publish(txTrytesStringBuilder.toString()); + messageQ.publish(trytesStringBuilder.toString()); } catch (Exception e) { - log.error(txTrytesStringBuilder.toString()); - log.error("Error publishing tx_trytes to zmq.", e); + log.error(trytesStringBuilder.toString()); + log.error("Error publishing trytes to zmq.", e); } } diff --git a/src/test/java/com/iota/iri/MainInjectionConfigurationTest.java b/src/test/java/com/iota/iri/MainInjectionConfigurationTest.java index 3695f6cdba..be16d5df2f 100644 --- a/src/test/java/com/iota/iri/MainInjectionConfigurationTest.java +++ b/src/test/java/com/iota/iri/MainInjectionConfigurationTest.java @@ -23,6 +23,7 @@ import com.iota.iri.service.spentaddresses.SpentAddressesProvider; import com.iota.iri.service.spentaddresses.SpentAddressesService; import com.iota.iri.service.transactionpruning.TransactionPruner; +import com.iota.iri.service.validation.TransactionValidator; import com.iota.iri.storage.LocalSnapshotsPersistenceProvider; import com.iota.iri.storage.Tangle; import org.junit.Test; diff --git a/src/test/java/com/iota/iri/TransactionValidatorTest.java b/src/test/java/com/iota/iri/TransactionValidatorTest.java deleted file mode 100644 index 4f84b7aad5..0000000000 --- a/src/test/java/com/iota/iri/TransactionValidatorTest.java +++ /dev/null @@ -1,224 +0,0 @@ -package com.iota.iri; - -import com.iota.iri.conf.MainnetConfig; -import com.iota.iri.conf.ProtocolConfig; - -import com.iota.iri.controllers.TipsViewModel; -import com.iota.iri.controllers.TransactionViewModel; -import com.iota.iri.crypto.SpongeFactory; -import com.iota.iri.model.TransactionHash; -import com.iota.iri.network.TransactionRequester; -import com.iota.iri.service.snapshot.SnapshotProvider; - -import com.iota.iri.service.snapshot.impl.SnapshotMockUtils; -import com.iota.iri.storage.Tangle; -import com.iota.iri.storage.rocksDB.RocksDBPersistenceProvider; -import com.iota.iri.utils.Converter; - -import org.junit.*; -import org.junit.rules.TemporaryFolder; -import org.mockito.Mock; -import org.mockito.junit.MockitoJUnit; -import org.mockito.junit.MockitoRule; - -import static com.iota.iri.TransactionTestUtils.getTransactionHash; -import static com.iota.iri.TransactionTestUtils.getTransactionTrits; -import static com.iota.iri.TransactionTestUtils.getTransactionTritsWithTrunkAndBranch; -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertTrue; -import static org.mockito.Mockito.mock; -import static org.mockito.Mockito.when; - -public class TransactionValidatorTest { - - private static final int MAINNET_MWM = 14; - private static final TemporaryFolder dbFolder = new TemporaryFolder(); - private static final TemporaryFolder logFolder = new TemporaryFolder(); - private static Tangle tangle; - private static TransactionValidator txValidator; - - @Rule - public MockitoRule mockitoRule = MockitoJUnit.rule(); - - @Mock - private static SnapshotProvider snapshotProvider; - - @BeforeClass - public static void setUp() throws Exception { - dbFolder.create(); - logFolder.create(); - tangle = new Tangle(); - tangle.addPersistenceProvider( - new RocksDBPersistenceProvider( - dbFolder.getRoot().getAbsolutePath(), logFolder.getRoot().getAbsolutePath(),1000, Tangle.COLUMN_FAMILIES, Tangle.METADATA_COLUMN_FAMILY)); - tangle.init(); - } - - @AfterClass - public static void tearDown() throws Exception { - tangle.shutdown(); - dbFolder.delete(); - logFolder.delete(); - } - - @Before - public void setUpEach() { - when(snapshotProvider.getInitialSnapshot()).thenReturn(SnapshotMockUtils.createSnapshot()); - TipsViewModel tipsViewModel = new TipsViewModel(); - TransactionRequester txRequester = new TransactionRequester(tangle, snapshotProvider); - txValidator = new TransactionValidator(tangle, snapshotProvider, tipsViewModel, txRequester, new MainnetConfig()); - txValidator.setMwm(false, MAINNET_MWM); - } - - @Test - public void testMinMwm() { - ProtocolConfig protocolConfig = mock(ProtocolConfig.class); - when(protocolConfig.getMwm()).thenReturn(5); - TransactionValidator transactionValidator = new TransactionValidator(null, null, null, null, protocolConfig); - assertEquals("Expected testnet minimum minWeightMagnitude", 13, transactionValidator.getMinWeightMagnitude()); - } - - @Test - public void validateTrits() { - byte[] trits = getTransactionTrits(); - Converter.copyTrits(0, trits, 0, trits.length); - txValidator.validateTrits(trits, MAINNET_MWM); - } - - @Test(expected = RuntimeException.class) - public void validateTritsWithInvalidMetadata() { - byte[] trits = getTransactionTrits(); - txValidator.validateTrits(trits, MAINNET_MWM); - } - - @Test - public void validateBytesWithNewCurl() { - byte[] trits = getTransactionTrits(); - Converter.copyTrits(0, trits, 0, trits.length); - byte[] bytes = Converter.allocateBytesForTrits(trits.length); - Converter.bytes(trits, 0, bytes, 0, trits.length); - txValidator.validateBytes(bytes, txValidator.getMinWeightMagnitude(), SpongeFactory.create(SpongeFactory.Mode.CURLP81)); - } - - @Test - public void verifyTxIsSolid() throws Exception { - TransactionViewModel tx = getTxWithBranchAndTrunk(); - assertTrue(txValidator.checkSolidity(tx.getHash())); - assertTrue(txValidator.checkSolidity(tx.getHash())); - } - - @Test - public void verifyTxIsNotSolid() throws Exception { - TransactionViewModel tx = getTxWithoutBranchAndTrunk(); - assertFalse(txValidator.checkSolidity(tx.getHash())); - assertFalse(txValidator.checkSolidity(tx.getHash())); - } - - @Test - public void addSolidTransactionWithoutErrors() { - byte[] trits = getTransactionTrits(); - Converter.copyTrits(0, trits, 0, trits.length); - txValidator.addSolidTransaction(TransactionHash.calculate(SpongeFactory.Mode.CURLP81, trits)); - } - - private TransactionViewModel getTxWithBranchAndTrunk() throws Exception { - TransactionViewModel tx, trunkTx, branchTx; - String trytes = "999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999CFDEZBLZQYA9999999999999999999999999999999999999999999ZZWQHWD99C99999999C99999999CKWWDBWSCLMQULCTAAJGXDEMFJXPMGMAQIHDGHRBGEMUYNNCOK9YPHKEEFLFCZUSPMCJHAKLCIBQSGWAS999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999"; - - byte[] trits = Converter.allocateTritsForTrytes(trytes.length()); - Converter.trits(trytes, trits, 0); - trunkTx = new TransactionViewModel(trits, TransactionHash.calculate(SpongeFactory.Mode.CURLP81, trits)); - branchTx = new TransactionViewModel(trits, TransactionHash.calculate(SpongeFactory.Mode.CURLP81, trits)); - - byte[] childTx = getTransactionTrits(); - System.arraycopy(trunkTx.getHash().trits(), 0, childTx, TransactionViewModel.TRUNK_TRANSACTION_TRINARY_OFFSET, TransactionViewModel.TRUNK_TRANSACTION_TRINARY_SIZE); - System.arraycopy(branchTx.getHash().trits(), 0, childTx, TransactionViewModel.BRANCH_TRANSACTION_TRINARY_OFFSET, TransactionViewModel.BRANCH_TRANSACTION_TRINARY_SIZE); - tx = new TransactionViewModel(childTx, TransactionHash.calculate(SpongeFactory.Mode.CURLP81, childTx)); - - trunkTx.store(tangle, snapshotProvider.getInitialSnapshot()); - branchTx.store(tangle, snapshotProvider.getInitialSnapshot()); - tx.store(tangle, snapshotProvider.getInitialSnapshot()); - - return tx; - } - - @Test - public void testTransactionPropagation() throws Exception { - TransactionViewModel leftChildLeaf = TransactionTestUtils.createTransactionWithTrytes("CHILDTX"); - leftChildLeaf.updateSolid(tangle, snapshotProvider.getInitialSnapshot(), true); - leftChildLeaf.store(tangle, snapshotProvider.getInitialSnapshot()); - - TransactionViewModel rightChildLeaf = TransactionTestUtils.createTransactionWithTrytes("CHILDTWOTX"); - rightChildLeaf.updateSolid(tangle, snapshotProvider.getInitialSnapshot(), true); - rightChildLeaf.store(tangle, snapshotProvider.getInitialSnapshot()); - - TransactionViewModel parent = TransactionTestUtils.createTransactionWithTrunkAndBranch("PARENT", - leftChildLeaf.getHash(), rightChildLeaf.getHash()); - parent.updateSolid(tangle, snapshotProvider.getInitialSnapshot(), false); - parent.store(tangle, snapshotProvider.getInitialSnapshot()); - - TransactionViewModel parentSibling = TransactionTestUtils.createTransactionWithTrytes("PARENTLEAF"); - parentSibling.updateSolid(tangle, snapshotProvider.getInitialSnapshot(), true); - parentSibling.store(tangle, snapshotProvider.getInitialSnapshot()); - - TransactionViewModel grandParent = TransactionTestUtils.createTransactionWithTrunkAndBranch("GRANDPARENT", parent.getHash(), - parentSibling.getHash()); - grandParent.updateSolid(tangle, snapshotProvider.getInitialSnapshot(), false); - grandParent.store(tangle, snapshotProvider.getInitialSnapshot()); - - txValidator.addSolidTransaction(leftChildLeaf.getHash()); - while (!txValidator.isNewSolidTxSetsEmpty()) { - txValidator.propagateSolidTransactions(); - } - - parent = TransactionViewModel.fromHash(tangle, parent.getHash()); - assertTrue("Parent tx was expected to be solid", parent.isSolid()); - grandParent = TransactionViewModel.fromHash(tangle, grandParent.getHash()); - assertTrue("Grandparent was expected to be solid", grandParent.isSolid()); - } - - @Test - public void testTransactionPropagationFailure() throws Exception { - TransactionViewModel leftChildLeaf = new TransactionViewModel(getTransactionTrits(), getTransactionHash()); - leftChildLeaf.updateSolid(tangle, snapshotProvider.getInitialSnapshot(), true); - leftChildLeaf.store(tangle, snapshotProvider.getInitialSnapshot()); - - TransactionViewModel rightChildLeaf = new TransactionViewModel(getTransactionTrits(), getTransactionHash()); - rightChildLeaf.updateSolid(tangle, snapshotProvider.getInitialSnapshot(), true); - rightChildLeaf.store(tangle, snapshotProvider.getInitialSnapshot()); - - TransactionViewModel parent = new TransactionViewModel(getTransactionTritsWithTrunkAndBranch(leftChildLeaf.getHash(), - rightChildLeaf.getHash()), getTransactionHash()); - parent.updateSolid(tangle, snapshotProvider.getInitialSnapshot(), false); - parent.store(tangle, snapshotProvider.getInitialSnapshot()); - - TransactionViewModel parentSibling = new TransactionViewModel(getTransactionTrits(), getTransactionHash()); - parentSibling.updateSolid(tangle, snapshotProvider.getInitialSnapshot(), false); - parentSibling.store(tangle, snapshotProvider.getInitialSnapshot()); - - TransactionViewModel grandParent = new TransactionViewModel(getTransactionTritsWithTrunkAndBranch(parent.getHash(), - parentSibling.getHash()), getTransactionHash()); - grandParent.updateSolid(tangle, snapshotProvider.getInitialSnapshot(), false); - grandParent.store(tangle, snapshotProvider.getInitialSnapshot()); - - txValidator.addSolidTransaction(leftChildLeaf.getHash()); - while (!txValidator.isNewSolidTxSetsEmpty()) { - txValidator.propagateSolidTransactions(); - } - - parent = TransactionViewModel.fromHash(tangle, parent.getHash()); - assertTrue("Parent tx was expected to be solid", parent.isSolid()); - grandParent = TransactionViewModel.fromHash(tangle, grandParent.getHash()); - assertFalse("GrandParent tx was expected to be not solid", grandParent.isSolid()); - } - - private TransactionViewModel getTxWithoutBranchAndTrunk() throws Exception { - byte[] trits = getTransactionTrits(); - TransactionViewModel tx = new TransactionViewModel(trits, TransactionHash.calculate(SpongeFactory.Mode.CURLP81, trits)); - - tx.store(tangle, snapshotProvider.getInitialSnapshot()); - - return tx; - } -} diff --git a/src/test/java/com/iota/iri/controllers/TransactionViewModelTest.java b/src/test/java/com/iota/iri/controllers/TransactionViewModelTest.java index 080674882b..7866285e90 100644 --- a/src/test/java/com/iota/iri/controllers/TransactionViewModelTest.java +++ b/src/test/java/com/iota/iri/controllers/TransactionViewModelTest.java @@ -445,13 +445,13 @@ public void updateTxShouldBeSavedBeforeRelease() throws Exception { Hash hash0 = tvms[0].getHash(); TransactionViewModel tvm0 = new TransactionViewModel((Transaction) tangle.load(Transaction.class, hash0), hash0); - assertTrue("TVM should be a milestone", tvm0.isMilestone()); + Assert.assertTrue("TVM should be a milestone", tvm0.isMilestone()); Hash hash1 = tvms[1].getHash(); TransactionViewModel tvm1 = new TransactionViewModel((Transaction) tangle.load(Transaction.class, hash1), hash1); - assertTrue("TVM should be solid", tvm1.isSolid()); + Assert.assertTrue("TVM should be solid", tvm1.isSolid()); Hash hash2 = tvms[2].getHash(); TransactionViewModel tvm2 = new TransactionViewModel((Transaction) tangle.load(Transaction.class, hash2), hash2); - assertEquals("TVM sender should be equal", "sender", tvm2.getSender()); + Assert.assertEquals("TVM sender should be equal", "sender", tvm2.getSender()); } @Test @@ -477,12 +477,12 @@ public void updatedTxShouldBeSavedToDbBeforeTangleShutdown() throws Exception { for (int i : tvmIndicesToUpdate) { TransactionViewModel tvm = cache.get(tvms[i].getHash()); if (tvm != null) { - if(i%2 == 0){ - tvm.isMilestone(tangle, snapshot, true); - }else{ - //just another test - tvm.updateSolid(tangle, snapshot, true); - } + if(i%2 == 0){ + tvm.isMilestone(tangle, snapshot, true); + }else{ + //just another test + tvm.updateSolid(tangle, snapshot, true); + } } } @@ -494,9 +494,9 @@ public void updatedTxShouldBeSavedToDbBeforeTangleShutdown() throws Exception { TransactionViewModel tvm = new TransactionViewModel((Transaction) tangle.load(Transaction.class, hash), hash); if(i%2 == 0){ - assertTrue("TVM should be a milestone", tvm.isMilestone()); + Assert.assertTrue("TVM should be a milestone", tvm.isMilestone()); }else{ - assertTrue("TVM should be solid", tvm.isSolid()); + Assert.assertTrue("TVM should be solid", tvm.isSolid()); } } } diff --git a/src/test/java/com/iota/iri/network/NetworkInjectionConfigurationTest.java b/src/test/java/com/iota/iri/network/NetworkInjectionConfigurationTest.java index c02749a893..5affbf062e 100644 --- a/src/test/java/com/iota/iri/network/NetworkInjectionConfigurationTest.java +++ b/src/test/java/com/iota/iri/network/NetworkInjectionConfigurationTest.java @@ -3,7 +3,8 @@ import com.google.inject.AbstractModule; import com.google.inject.Guice; import com.google.inject.Injector; -import com.iota.iri.TransactionValidator; +import com.iota.iri.service.validation.TransactionSolidifier; +import com.iota.iri.service.validation.TransactionValidator; import com.iota.iri.conf.BaseIotaConfig; import com.iota.iri.conf.IotaConfig; import com.iota.iri.network.pipeline.TransactionProcessingPipeline; @@ -37,6 +38,11 @@ public void provideTransactionProcessingPipeline() { assertNotNull("instance creation did not work", testInjector().getInstance(TransactionProcessingPipeline.class)); } + @Test + public void provideTransactionSolidifier(){ + assertNotNull("instance creation did not work", testInjector().getInstance(TransactionSolidifier.class)); + } + private Injector testInjector() { IotaConfig config = mock(IotaConfig.class); when(config.getCoordinator()).thenReturn(BaseIotaConfig.Defaults.COORDINATOR); @@ -50,6 +56,7 @@ protected void configure() { bind(LatestMilestoneTracker.class).toInstance(mock(LatestMilestoneTracker.class)); bind(SnapshotProvider.class).toInstance(mock(SnapshotProvider.class)); bind(TransactionValidator.class).toInstance(mock(TransactionValidator.class)); + bind(TransactionSolidifier.class).toInstance(mock(TransactionSolidifier.class)); } } diff --git a/src/test/java/com/iota/iri/network/pipeline/BroadcastStageTest.java b/src/test/java/com/iota/iri/network/pipeline/BroadcastStageTest.java index ba8678e9a9..ce1b8fdc85 100644 --- a/src/test/java/com/iota/iri/network/pipeline/BroadcastStageTest.java +++ b/src/test/java/com/iota/iri/network/pipeline/BroadcastStageTest.java @@ -9,6 +9,7 @@ import java.util.HashMap; import java.util.Map; +import com.iota.iri.service.validation.TransactionSolidifier; import org.junit.Rule; import org.junit.Test; import org.mockito.Mock; @@ -21,6 +22,9 @@ public class BroadcastStageTest { @Rule public MockitoRule mockitoRule = MockitoJUnit.rule(); + @Mock + private TransactionSolidifier transactionSolidifier; + @Mock private NeighborRouter neighborRouter; @@ -38,7 +42,7 @@ public class BroadcastStageTest { public void doesntGossipToOriginNeighbor() { Mockito.when(neighborRouter.getConnectedNeighbors()).thenReturn(neighbors); - BroadcastStage broadcastStage = new BroadcastStage(neighborRouter); + BroadcastStage broadcastStage = new BroadcastStage(neighborRouter, transactionSolidifier); TransactionViewModel tvm = new TransactionViewModel(new Transaction(), null); BroadcastPayload broadcastPayload = new BroadcastPayload(neighborA, tvm); ProcessingContext ctx = new ProcessingContext(null, broadcastPayload); @@ -58,7 +62,7 @@ public void doesntGossipToOriginNeighbor() { public void gossipsToAllIfNoOriginNeighbor() { Mockito.when(neighborRouter.getConnectedNeighbors()).thenReturn(neighbors); - BroadcastStage broadcastStage = new BroadcastStage(neighborRouter); + BroadcastStage broadcastStage = new BroadcastStage(neighborRouter, transactionSolidifier); TransactionViewModel tvm = new TransactionViewModel(new Transaction(), null); BroadcastPayload broadcastPayload = new BroadcastPayload(null, tvm); ProcessingContext ctx = new ProcessingContext(null, broadcastPayload); diff --git a/src/test/java/com/iota/iri/network/pipeline/ReceivedStageTest.java b/src/test/java/com/iota/iri/network/pipeline/ReceivedStageTest.java index 965a7c3999..23518b7ae3 100644 --- a/src/test/java/com/iota/iri/network/pipeline/ReceivedStageTest.java +++ b/src/test/java/com/iota/iri/network/pipeline/ReceivedStageTest.java @@ -1,6 +1,6 @@ package com.iota.iri.network.pipeline; -import com.iota.iri.TransactionValidator; +import com.iota.iri.service.validation.TransactionSolidifier; import com.iota.iri.controllers.TransactionViewModel; import com.iota.iri.network.TransactionRequester; import com.iota.iri.network.neighbor.Neighbor; @@ -26,7 +26,7 @@ public class ReceivedStageTest { private Tangle tangle; @Mock - private TransactionValidator transactionValidator; + private TransactionSolidifier transactionSolidifier; @Mock private SnapshotProvider snapshotProvider; @@ -49,7 +49,7 @@ public void newlyStoredTransactionUpdatesAlsoArrivalTimeAndSender() throws Excep Mockito.when(neighbor.getMetrics()).thenReturn(neighborMetrics); Mockito.when(transactionRequester.removeRecentlyRequestedTransaction(Mockito.any())).thenReturn(true); - ReceivedStage stage = new ReceivedStage(tangle, transactionValidator, snapshotProvider, transactionRequester); + ReceivedStage stage = new ReceivedStage(tangle, transactionSolidifier, snapshotProvider, transactionRequester); ReceivedPayload receivedPayload = new ReceivedPayload(neighbor, tvm); ProcessingContext ctx = new ProcessingContext(null, receivedPayload); stage.process(ctx); @@ -58,11 +58,11 @@ public void newlyStoredTransactionUpdatesAlsoArrivalTimeAndSender() throws Excep Mockito.verify(tvm).updateSender(Mockito.any(), Mockito.any(), Mockito.any()); Mockito.verify(transactionRequester).removeRecentlyRequestedTransaction(Mockito.any()); Mockito.verify(transactionRequester).requestTrunkAndBranch(Mockito.any()); - assertEquals("should submit to broadcast stage next", TransactionProcessingPipeline.Stage.BROADCAST, + assertEquals("should submit to broadcast stage next", TransactionProcessingPipeline.Stage.SOLIDIFY, ctx.getNextStage()); - BroadcastPayload broadcastPayload = (BroadcastPayload) ctx.getPayload(); - assertEquals("neighbor is still the same", neighbor, broadcastPayload.getOriginNeighbor()); - assertEquals("tvm is still the same", tvm, broadcastPayload.getTransactionViewModel()); + SolidifyPayload solidifyPayload = (SolidifyPayload) ctx.getPayload(); + assertEquals("neighbor is still the same", neighbor, solidifyPayload.getOriginNeighbor()); + assertEquals("tvm is still the same", tvm, solidifyPayload.getTransaction()); } @Test @@ -70,7 +70,7 @@ public void alreadyStoredTransactionDoesNoUpdates() throws Exception { Mockito.when(tvm.store(tangle, snapshotProvider.getInitialSnapshot())).thenReturn(false); Mockito.when(neighbor.getMetrics()).thenReturn(neighborMetrics); - ReceivedStage stage = new ReceivedStage(tangle, transactionValidator, snapshotProvider, transactionRequester); + ReceivedStage stage = new ReceivedStage(tangle, transactionSolidifier, snapshotProvider, transactionRequester); ReceivedPayload receivedPayload = new ReceivedPayload(neighbor, tvm); ProcessingContext ctx = new ProcessingContext(null, receivedPayload); stage.process(ctx); @@ -79,11 +79,11 @@ public void alreadyStoredTransactionDoesNoUpdates() throws Exception { Mockito.verify(tvm, Mockito.never()).update(Mockito.any(), Mockito.any(), Mockito.any()); Mockito.verify(transactionRequester).removeRecentlyRequestedTransaction(Mockito.any()); Mockito.verify(transactionRequester, Mockito.never()).requestTrunkAndBranch(Mockito.any()); - assertEquals("should submit to broadcast stage next", TransactionProcessingPipeline.Stage.BROADCAST, + assertEquals("should submit to broadcast stage next", TransactionProcessingPipeline.Stage.SOLIDIFY, ctx.getNextStage()); - BroadcastPayload broadcastPayload = (BroadcastPayload) ctx.getPayload(); - assertEquals("neighbor should still be the same", neighbor, broadcastPayload.getOriginNeighbor()); - assertEquals("tvm should still be the same", tvm, broadcastPayload.getTransactionViewModel()); + SolidifyPayload solidifyPayload = (SolidifyPayload) ctx.getPayload(); + assertEquals("neighbor should still be the same", neighbor, solidifyPayload.getOriginNeighbor()); + assertEquals("tvm should still be the same", tvm, solidifyPayload.getTransaction()); } } \ No newline at end of file diff --git a/src/test/java/com/iota/iri/network/pipeline/SolidifyStageTest.java b/src/test/java/com/iota/iri/network/pipeline/SolidifyStageTest.java new file mode 100644 index 0000000000..1c309adfe7 --- /dev/null +++ b/src/test/java/com/iota/iri/network/pipeline/SolidifyStageTest.java @@ -0,0 +1,113 @@ +package com.iota.iri.network.pipeline; + +import com.iota.iri.controllers.TipsViewModel; +import com.iota.iri.controllers.TransactionViewModel; +import com.iota.iri.model.Hash; +import com.iota.iri.service.validation.TransactionSolidifier; +import com.iota.iri.storage.Tangle; +import org.junit.Rule; +import org.junit.Test; +import org.mockito.Mock; +import org.mockito.Mockito; +import org.mockito.junit.MockitoJUnit; +import org.mockito.junit.MockitoRule; + +import static org.junit.Assert.assertEquals; + +public class SolidifyStageTest { + @Rule + public MockitoRule mockitoRule = MockitoJUnit.rule(); + + @Mock + private Tangle tangle; + + @Mock + private TipsViewModel tipsViewModel; + + @Mock + private TransactionViewModel tvm; + + @Mock + private Hash originalHash; + + @Mock + private Hash tipHash; + + @Mock + private TransactionSolidifier transactionSolidifier; + + @Test + public void solidTransactionIsBroadcast() throws Exception{ + Mockito.when(tvm.isSolid()).thenReturn(true); + Mockito.when(tvm.getHash()).thenReturn(originalHash); + + SolidifyStage solidifyStage = new SolidifyStage(transactionSolidifier, tipsViewModel, tangle); + SolidifyPayload solidifyPayload = new SolidifyPayload(null, tvm); + ProcessingContext ctx = new ProcessingContext(solidifyPayload); + + solidifyStage.process(ctx); + Thread.sleep(100); + + assertEquals("Expected next stage to be broadcast", ctx.getNextStage(), + TransactionProcessingPipeline.Stage.BROADCAST); + BroadcastPayload broadcastPayload = (BroadcastPayload) ctx.getPayload(); + assertEquals("Expected payload hash to equal the original transaction hash", + broadcastPayload.getTransactionViewModel().getHash(), originalHash); + } + + @Test + public void quickSetSolidTransactionIsBroadcast() throws Exception{ + Mockito.when(transactionSolidifier.quickSetSolid(tvm)).thenReturn(true); + Mockito.when(tvm.getHash()).thenReturn(originalHash); + + SolidifyStage solidifyStage = new SolidifyStage(transactionSolidifier, tipsViewModel, tangle); + SolidifyPayload solidifyPayload = new SolidifyPayload(null, tvm); + ProcessingContext ctx = new ProcessingContext(solidifyPayload); + + solidifyStage.process(ctx); + Thread.sleep(100); + + assertEquals("Expected next stage to be broadcast", ctx.getNextStage(), + TransactionProcessingPipeline.Stage.BROADCAST); + BroadcastPayload broadcastPayload = (BroadcastPayload) ctx.getPayload(); + assertEquals("Expected payload hash to equal the original transaction hash", + broadcastPayload.getTransactionViewModel().getHash(), originalHash); + } + + @Test + public void unsolidTransactionBroadcastsRandomSolidTip() throws Exception{ + Mockito.when(tvm.isSolid()).thenReturn(false); + Mockito.when(transactionSolidifier.quickSetSolid(tvm)).thenReturn(false); + Mockito.when(tipsViewModel.getRandomSolidTipHash()).thenReturn(tipHash); + + SolidifyStage solidifyStage = new SolidifyStage(transactionSolidifier, tipsViewModel, tangle); + SolidifyPayload solidifyPayload = new SolidifyPayload(null, tvm); + ProcessingContext ctx = new ProcessingContext(solidifyPayload); + + solidifyStage.process(ctx); + Thread.sleep(100); + + assertEquals("Expected next stage to be broadcast", ctx.getNextStage(), + TransactionProcessingPipeline.Stage.BROADCAST); + BroadcastPayload broadcastPayload = (BroadcastPayload) ctx.getPayload(); + assertEquals("Expected payload hash to equal random tip hash", + broadcastPayload.getTransactionViewModel().getHash(), tipHash); + } + + @Test + public void unsolidWithNoRandomTipsAborts() throws Exception{ + Mockito.when(tvm.isSolid()).thenReturn(false); + Mockito.when(transactionSolidifier.quickSetSolid(tvm)).thenReturn(false); + Mockito.when(tipsViewModel.getRandomSolidTipHash()).thenReturn(null); + + SolidifyStage solidifyStage = new SolidifyStage(transactionSolidifier, tipsViewModel, tangle); + SolidifyPayload solidifyPayload = new SolidifyPayload(null, tvm); + ProcessingContext ctx = new ProcessingContext(solidifyPayload); + + solidifyStage.process(ctx); + Thread.sleep(100); + + assertEquals("Expected next stage to be broadcast", ctx.getNextStage(), + TransactionProcessingPipeline.Stage.FINISH); + } +} \ No newline at end of file diff --git a/src/test/java/com/iota/iri/network/pipeline/TransactionProcessingPipelineTest.java b/src/test/java/com/iota/iri/network/pipeline/TransactionProcessingPipelineTest.java index b9b8c4ab8f..576f418bc0 100644 --- a/src/test/java/com/iota/iri/network/pipeline/TransactionProcessingPipelineTest.java +++ b/src/test/java/com/iota/iri/network/pipeline/TransactionProcessingPipelineTest.java @@ -1,6 +1,7 @@ package com.iota.iri.network.pipeline; -import com.iota.iri.TransactionValidator; +import com.iota.iri.service.validation.TransactionSolidifier; +import com.iota.iri.service.validation.TransactionValidator; import com.iota.iri.conf.NodeConfig; import com.iota.iri.controllers.TipsViewModel; import com.iota.iri.network.NeighborRouter; @@ -18,6 +19,7 @@ import org.mockito.junit.MockitoJUnit; import org.mockito.junit.MockitoRule; + public class TransactionProcessingPipelineTest { @Rule @@ -74,6 +76,15 @@ public class TransactionProcessingPipelineTest { @Mock private HashingPayload hashingPayload; + @Mock + private BroadcastPayload broadcastPayload; + + @Mock + private SolidifyStage solidifyStage; + + @Mock + private SolidifyPayload solidifyPayload; + @Mock private ProcessingContext validationCtx; @@ -89,9 +100,15 @@ public class TransactionProcessingPipelineTest { @Mock private ProcessingContext broadcastCtx; + @Mock + private ProcessingContext solidifyCtx; + @Mock private ProcessingContext abortCtx; + @Mock + private TransactionSolidifier transactionSolidifier; + private void mockHashingStage(TransactionProcessingPipeline pipeline) { Mockito.when(hashingPayload.getTxTrits()).thenReturn(null); Mockito.doAnswer(invocation -> { @@ -107,6 +124,7 @@ private void injectMockedStagesIntoPipeline(TransactionProcessingPipeline pipeli pipeline.setHashingStage(hashingStage); pipeline.setReplyStage(replyStage); pipeline.setValidationStage(validationStage); + pipeline.setSolidifyStage(solidifyStage); } @Test @@ -114,7 +132,7 @@ public void processingAValidNewTransactionFlowsThroughTheEntirePipeline() throws TransactionProcessingPipeline pipeline = new TransactionProcessingPipelineImpl(neighborRouter, nodeConfig, transactionValidator, tangle, snapshotProvider, tipsViewModel, latestMilestoneTracker, - transactionRequester); + transactionRequester, transactionSolidifier); // inject mocks injectMockedStagesIntoPipeline(pipeline); @@ -136,7 +154,13 @@ public void processingAValidNewTransactionFlowsThroughTheEntirePipeline() throws // mock received Mockito.when(broadcastCtx.getNextStage()).thenReturn(TransactionProcessingPipeline.Stage.BROADCAST); - Mockito.when(receivedStage.process(receivedCtx)).thenReturn(broadcastCtx); + Mockito.when(broadcastCtx.getPayload()).thenReturn(broadcastPayload); + Mockito.when(receivedStage.process(receivedCtx)).thenReturn(solidifyCtx); + + // mock solidify + Mockito.when(solidifyCtx.getPayload()).thenReturn(solidifyPayload); + Mockito.when(solidifyCtx.getNextStage()).thenReturn(TransactionProcessingPipeline.Stage.SOLIDIFY); + Mockito.when(solidifyStage.process(solidifyCtx)).thenReturn(broadcastCtx); pipeline.start(); @@ -152,6 +176,7 @@ public void processingAValidNewTransactionFlowsThroughTheEntirePipeline() throws Mockito.verify(validationStage).process(Mockito.any()); Mockito.verify(receivedStage).process(Mockito.any()); Mockito.verify(replyStage).process(Mockito.any()); + Mockito.verify(solidifyStage).process(Mockito.any()); Mockito.verify(broadcastStage).process(Mockito.any()); } @@ -159,7 +184,7 @@ public void processingAValidNewTransactionFlowsThroughTheEntirePipeline() throws public void processingAKnownTransactionOnlyFlowsToTheReplyStage() throws InterruptedException { TransactionProcessingPipeline pipeline = new TransactionProcessingPipelineImpl(neighborRouter, nodeConfig, transactionValidator, tangle, snapshotProvider, tipsViewModel, latestMilestoneTracker, - transactionRequester); + transactionRequester, transactionSolidifier); // inject mocks pipeline.setPreProcessStage(preProcessStage); @@ -181,6 +206,7 @@ public void processingAKnownTransactionOnlyFlowsToTheReplyStage() throws Interru Mockito.verify(hashingStage, Mockito.never()).process(Mockito.any()); Mockito.verify(validationStage, Mockito.never()).process(Mockito.any()); Mockito.verify(receivedStage, Mockito.never()).process(Mockito.any()); + Mockito.verify(solidifyStage, Mockito.never()).process(Mockito.any()); Mockito.verify(broadcastStage, Mockito.never()).process(Mockito.any()); // should have called @@ -193,7 +219,7 @@ public void processingAValidNewTransactionNotOriginatingFromANeighborFlowsThroug throws InterruptedException { TransactionProcessingPipeline pipeline = new TransactionProcessingPipelineImpl(neighborRouter, nodeConfig, transactionValidator, tangle, snapshotProvider, tipsViewModel, latestMilestoneTracker, - transactionRequester); + transactionRequester, transactionSolidifier); // inject mocks injectMockedStagesIntoPipeline(pipeline); @@ -202,7 +228,6 @@ public void processingAValidNewTransactionNotOriginatingFromANeighborFlowsThroug Mockito.when(hashingCtx.getNextStage()).thenReturn(TransactionProcessingPipeline.Stage.HASHING); Mockito.when(hashingCtx.getPayload()).thenReturn(hashingPayload); - // mock hashing context/stage // mock hashing context/stage mockHashingStage(pipeline); @@ -212,7 +237,12 @@ public void processingAValidNewTransactionNotOriginatingFromANeighborFlowsThroug // mock received Mockito.when(broadcastCtx.getNextStage()).thenReturn(TransactionProcessingPipeline.Stage.BROADCAST); - Mockito.when(receivedStage.process(receivedCtx)).thenReturn(broadcastCtx); + Mockito.when(receivedStage.process(receivedCtx)).thenReturn(solidifyCtx); + + // mock solidify + Mockito.when(solidifyCtx.getPayload()).thenReturn(solidifyPayload); + Mockito.when(solidifyCtx.getNextStage()).thenReturn(TransactionProcessingPipeline.Stage.SOLIDIFY); + Mockito.when(solidifyStage.process(solidifyCtx)).thenReturn(broadcastCtx); pipeline.start(); @@ -231,6 +261,7 @@ public void processingAValidNewTransactionNotOriginatingFromANeighborFlowsThroug Mockito.verify(hashingStage).process(Mockito.any()); Mockito.verify(validationStage).process(Mockito.any()); Mockito.verify(receivedStage).process(Mockito.any()); + Mockito.verify(solidifyStage).process(Mockito.any()); Mockito.verify(broadcastStage).process(Mockito.any()); } @@ -238,7 +269,7 @@ public void processingAValidNewTransactionNotOriginatingFromANeighborFlowsThroug public void anInvalidNewTransactionStopsBeingProcessedAfterTheValidationStage() throws InterruptedException { TransactionProcessingPipeline pipeline = new TransactionProcessingPipelineImpl(neighborRouter, nodeConfig, transactionValidator, tangle, snapshotProvider, tipsViewModel, latestMilestoneTracker, - transactionRequester); + transactionRequester, transactionSolidifier); // inject mocks injectMockedStagesIntoPipeline(pipeline); @@ -269,6 +300,7 @@ public void anInvalidNewTransactionStopsBeingProcessedAfterTheValidationStage() Mockito.verify(preProcessStage, Mockito.never()).process(Mockito.any()); Mockito.verify(broadcastStage, Mockito.never()).process(Mockito.any()); Mockito.verify(receivedStage, Mockito.never()).process(Mockito.any()); + Mockito.verify(solidifyStage, Mockito.never()).process(Mockito.any()); // should have called Mockito.verify(hashingStage).process(Mockito.any()); diff --git a/src/test/java/com/iota/iri/network/pipeline/ValidationStageTest.java b/src/test/java/com/iota/iri/network/pipeline/ValidationStageTest.java index 5bd0eecdff..c02d57d3b4 100644 --- a/src/test/java/com/iota/iri/network/pipeline/ValidationStageTest.java +++ b/src/test/java/com/iota/iri/network/pipeline/ValidationStageTest.java @@ -1,6 +1,6 @@ package com.iota.iri.network.pipeline; -import com.iota.iri.TransactionValidator; +import com.iota.iri.service.validation.TransactionValidator; import com.iota.iri.controllers.TransactionViewModel; import com.iota.iri.model.Hash; import com.iota.iri.network.FIFOCache; diff --git a/src/test/java/com/iota/iri/service/APITest.java b/src/test/java/com/iota/iri/service/APITest.java index d456cb3ab3..a19e58f511 100644 --- a/src/test/java/com/iota/iri/service/APITest.java +++ b/src/test/java/com/iota/iri/service/APITest.java @@ -1,6 +1,7 @@ package com.iota.iri.service; -import com.iota.iri.TransactionValidator; +import com.iota.iri.service.validation.TransactionSolidifier; +import com.iota.iri.service.validation.TransactionValidator; import com.iota.iri.conf.IotaConfig; import com.iota.iri.controllers.TransactionViewModel; import com.iota.iri.service.snapshot.SnapshotProvider; @@ -29,6 +30,9 @@ public class APITest { @Mock(answer = Answers.RETURNS_SMART_NULLS) private TransactionValidator transactionValidator; + @Mock + private TransactionSolidifier transactionSolidifier; + @Mock private SnapshotProvider snapshotProvider; @@ -44,7 +48,7 @@ public void whenStoreTransactionsStatementThenSetArrivalTimeToCurrentMillis() th API api = new API(config, null, null, null, null, null, snapshotProvider, null, null, null, null, - transactionValidator, null, null, null); + transactionValidator, null, null, transactionSolidifier, null); api.storeTransactionsStatement(Collections.singletonList("FOO")); diff --git a/src/test/java/com/iota/iri/service/ApiCallTest.java b/src/test/java/com/iota/iri/service/ApiCallTest.java index d2ca7dc9b0..5a8f3e74b8 100644 --- a/src/test/java/com/iota/iri/service/ApiCallTest.java +++ b/src/test/java/com/iota/iri/service/ApiCallTest.java @@ -14,7 +14,7 @@ public class ApiCallTest { @Before public void setUp() { IotaConfig configuration = Mockito.mock(IotaConfig.class); - api = new API(configuration, null, null, null, null, null, null, null, null, null, null, null, null, null, null); + api = new API(configuration, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null); } @Test diff --git a/src/test/java/com/iota/iri/service/validation/TransactionValidatorTest.java b/src/test/java/com/iota/iri/service/validation/TransactionValidatorTest.java new file mode 100644 index 0000000000..4e7b11552f --- /dev/null +++ b/src/test/java/com/iota/iri/service/validation/TransactionValidatorTest.java @@ -0,0 +1,100 @@ +package com.iota.iri.service.validation; + +import com.iota.iri.conf.MainnetConfig; +import com.iota.iri.conf.ProtocolConfig; +import com.iota.iri.crypto.SpongeFactory; +import com.iota.iri.network.TransactionRequester; +import com.iota.iri.service.snapshot.SnapshotProvider; +import com.iota.iri.service.snapshot.impl.SnapshotMockUtils; +import com.iota.iri.storage.Tangle; +import com.iota.iri.storage.rocksDB.RocksDBPersistenceProvider; +import com.iota.iri.utils.Converter; + +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.Rule; +import org.junit.AfterClass; +import org.junit.rules.TemporaryFolder; +import org.mockito.Mock; +import org.mockito.junit.MockitoJUnit; +import org.mockito.junit.MockitoRule; + +import static com.iota.iri.TransactionTestUtils.getTransactionTrits; +import static org.junit.Assert.assertEquals; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +public class TransactionValidatorTest { + + private static final int MAINNET_MWM = 14; + private static final TemporaryFolder dbFolder = new TemporaryFolder(); + private static final TemporaryFolder logFolder = new TemporaryFolder(); + private static Tangle tangle; + private static TransactionValidator txValidator; + + @Rule + public MockitoRule mockitoRule = MockitoJUnit.rule(); + + @Mock + private static SnapshotProvider snapshotProvider; + + @Mock + private static TransactionRequester txRequester; + + @BeforeClass + public static void setUp() throws Exception { + dbFolder.create(); + logFolder.create(); + tangle = new Tangle(); + tangle.addPersistenceProvider( + new RocksDBPersistenceProvider( + dbFolder.getRoot().getAbsolutePath(), logFolder.getRoot().getAbsolutePath(),1000, Tangle.COLUMN_FAMILIES, Tangle.METADATA_COLUMN_FAMILY)); + tangle.init(); + } + + @AfterClass + public static void tearDown() throws Exception { + tangle.shutdown(); + dbFolder.delete(); + logFolder.delete(); + } + + @Before + public void setUpEach() { + when(snapshotProvider.getInitialSnapshot()).thenReturn(SnapshotMockUtils.createSnapshot()); + txRequester = new TransactionRequester(tangle, snapshotProvider); + txValidator = new TransactionValidator(snapshotProvider, txRequester, new MainnetConfig()); + txValidator.setMwm(false, MAINNET_MWM); + } + + @Test + public void testMinMwm() { + ProtocolConfig protocolConfig = mock(ProtocolConfig.class); + when(protocolConfig.getMwm()).thenReturn(5); + TransactionValidator transactionValidator = new TransactionValidator(null, null, protocolConfig); + assertEquals("Expected testnet minimum minWeightMagnitude", 13, transactionValidator.getMinWeightMagnitude()); + } + + @Test + public void validateTrits() { + byte[] trits = getTransactionTrits(); + Converter.copyTrits(0, trits, 0, trits.length); + txValidator.validateTrits(trits, MAINNET_MWM); + } + + @Test(expected = RuntimeException.class) + public void validateTritsWithInvalidMetadata() { + byte[] trits = getTransactionTrits(); + txValidator.validateTrits(trits, MAINNET_MWM); + } + + @Test + public void validateBytesWithNewCurl() { + byte[] trits = getTransactionTrits(); + Converter.copyTrits(0, trits, 0, trits.length); + byte[] bytes = Converter.allocateBytesForTrits(trits.length); + Converter.bytes(trits, 0, bytes, 0, trits.length); + txValidator.validateBytes(bytes, txValidator.getMinWeightMagnitude(), SpongeFactory.create(SpongeFactory.Mode.CURLP81)); + } +} diff --git a/src/test/java/com/iota/iri/service/validation/impl/TransactionSolidifierImplTest.java b/src/test/java/com/iota/iri/service/validation/impl/TransactionSolidifierImplTest.java new file mode 100644 index 0000000000..b9353cd72b --- /dev/null +++ b/src/test/java/com/iota/iri/service/validation/impl/TransactionSolidifierImplTest.java @@ -0,0 +1,169 @@ +package com.iota.iri.service.validation.impl; + +import com.iota.iri.controllers.TipsViewModel; +import com.iota.iri.controllers.TransactionViewModel; +import com.iota.iri.crypto.SpongeFactory; +import com.iota.iri.model.TransactionHash; +import com.iota.iri.network.TransactionRequester; +import com.iota.iri.service.snapshot.SnapshotProvider; +import com.iota.iri.service.snapshot.impl.SnapshotMockUtils; +import com.iota.iri.storage.Tangle; +import com.iota.iri.storage.rocksDB.RocksDBPersistenceProvider; +import com.iota.iri.utils.Converter; +import org.junit.Rule; +import org.junit.BeforeClass; +import org.junit.Before; +import org.junit.AfterClass; +import org.junit.After; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; +import org.mockito.Mock; +import org.mockito.junit.MockitoJUnit; +import org.mockito.junit.MockitoRule; + +import java.util.ArrayList; +import java.util.List; + +import static com.iota.iri.TransactionTestUtils.getTransactionTrits; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; +import static org.mockito.Mockito.when; + +public class TransactionSolidifierImplTest { + private static final TemporaryFolder dbFolder = new TemporaryFolder(); + private static final TemporaryFolder logFolder = new TemporaryFolder(); + private static Tangle tangle; + + private List broadcastTransactions = new ArrayList<>(); + private TransactionViewModel transactionForBroadcast; + + @Rule + public MockitoRule mockitoRule = MockitoJUnit.rule(); + + @Mock + private static SnapshotProvider snapshotProvider; + + @Mock + private static TransactionSolidifierImpl txSolidifier; + + @Mock + private static TipsViewModel tipsViewModel; + + @Mock + private static TransactionRequester txRequester; + + @BeforeClass + public static void setUp() throws Exception { + dbFolder.create(); + logFolder.create(); + tangle = new Tangle(); + tangle.addPersistenceProvider( + new RocksDBPersistenceProvider( + dbFolder.getRoot().getAbsolutePath(), logFolder.getRoot().getAbsolutePath(),1000, Tangle.COLUMN_FAMILIES, Tangle.METADATA_COLUMN_FAMILY)); + tangle.init(); + } + + @AfterClass + public static void tearDown() throws Exception { + tangle.shutdown(); + dbFolder.delete(); + logFolder.delete(); + } + + @Before + public void setUpEach() { + when(snapshotProvider.getInitialSnapshot()).thenReturn(SnapshotMockUtils.createSnapshot()); + txRequester = new TransactionRequester(tangle, snapshotProvider); + txSolidifier = new TransactionSolidifierImpl(tangle, snapshotProvider, txRequester, tipsViewModel); + txSolidifier.start(); + } + + @After + public void tearDownEach(){ + txSolidifier.shutdown(); + } + + + @Test + public void verifyTxIsSolid() throws Exception { + TransactionViewModel tx = getTxWithBranchAndTrunk(); + assertTrue("Expected transaction to be solid", txSolidifier.checkSolidity(tx.getHash())); + assertTrue("Expected transaction to be solid", txSolidifier.checkSolidity(tx.getHash())); + } + + @Test + public void verifyTxIsNotSolid() throws Exception { + TransactionViewModel tx = getTxWithoutBranchAndTrunk(); + assertFalse("Expected transaction to fail solidity check", txSolidifier.checkSolidity(tx.getHash())); + assertFalse("Expected transaction to fail solidity check", txSolidifier.checkSolidity(tx.getHash())); + } + + + @Test + public void verifyTransactionIsProcessedFully() throws Exception { + TransactionViewModel tx = getTxWithBranchAndTrunk(); + txSolidifier.checkSolidity(tx.getHash()); + + //Time to process through the steps + Thread.sleep(1000); + + while((transactionForBroadcast = txSolidifier.getNextTxInBroadcastQueue()) != null){ + broadcastTransactions.add(transactionForBroadcast); + } + + assertTrue("Expected transaction to be present in the broadcast queue", + broadcastTransactions.contains(tx)); + + broadcastTransactions.clear(); + } + + + @Test + public void verifyInconsistentTransactionIsNotProcessedFully() throws Exception { + TransactionViewModel tx = getTxWithoutBranchAndTrunk(); + txSolidifier.checkSolidity(tx.getHash()); + + //Time to process through the steps + Thread.sleep(1000); + + while((transactionForBroadcast = txSolidifier.getNextTxInBroadcastQueue()) != null){ + broadcastTransactions.add(transactionForBroadcast); + } + + assertFalse("Expected transaction not to be present in the broadcast queue", + broadcastTransactions.contains(tx)); + + broadcastTransactions.clear(); + } + + private TransactionViewModel getTxWithBranchAndTrunk() throws Exception { + TransactionViewModel tx, trunkTx, branchTx; + String trytes = "999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999CFDEZBLZQYA9999999999999999999999999999999999999999999ZZWQHWD99C99999999C99999999CKWWDBWSCLMQULCTAAJGXDEMFJXPMGMAQIHDGHRBGEMUYNNCOK9YPHKEEFLFCZUSPMCJHAKLCIBQSGWAS999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999"; + + byte[] trits = Converter.allocateTritsForTrytes(trytes.length()); + Converter.trits(trytes, trits, 0); + trunkTx = new TransactionViewModel(trits, TransactionHash.calculate(SpongeFactory.Mode.CURLP81, trits)); + branchTx = new TransactionViewModel(trits, TransactionHash.calculate(SpongeFactory.Mode.CURLP81, trits)); + + byte[] childTx = getTransactionTrits(); + System.arraycopy(trunkTx.getHash().trits(), 0, childTx, TransactionViewModel.TRUNK_TRANSACTION_TRINARY_OFFSET, TransactionViewModel.TRUNK_TRANSACTION_TRINARY_SIZE); + System.arraycopy(branchTx.getHash().trits(), 0, childTx, TransactionViewModel.BRANCH_TRANSACTION_TRINARY_OFFSET, TransactionViewModel.BRANCH_TRANSACTION_TRINARY_SIZE); + tx = new TransactionViewModel(childTx, TransactionHash.calculate(SpongeFactory.Mode.CURLP81, childTx)); + + trunkTx.store(tangle, snapshotProvider.getInitialSnapshot()); + branchTx.store(tangle, snapshotProvider.getInitialSnapshot()); + tx.store(tangle, snapshotProvider.getInitialSnapshot()); + + return tx; + } + + private TransactionViewModel getTxWithoutBranchAndTrunk() throws Exception { + byte[] trits = getTransactionTrits(); + TransactionViewModel tx = new TransactionViewModel(trits, TransactionHash.calculate(SpongeFactory.Mode.CURLP81, trits)); + + tx.store(tangle, snapshotProvider.getInitialSnapshot()); + + return tx; + } + +} \ No newline at end of file