This repository has been archived by the owner on Jun 11, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 457
Refactor rounds logic - Closes #543 #597
Merged
Isabello
merged 98 commits into
LiskArchive:1.0.0
from
4miners:543_refactor-rounds-logic
Aug 8, 2017
Merged
Changes from 38 commits
Commits
Show all changes
98 commits
Select commit
Hold shift + click to select a range
ddb983f
Merge with development
4miners 742445e
Add missing bracket
4miners d676cdd
Drop not needed 'async.series'
4miners 7d90dc8
Replace table 'rounds_fees' with 'rounds_rewards', add support for ex…
4miners 3ae9b59
Create SQL function for validating memory balances, validate at node …
4miners 897c5ad
Improve SQL query formatting
4miners ad0b94c
Create SQL function generateDelegatesList, added tests
4miners fd6527a
Merge branch 'development' into 543_refactor-rounds-logic
4miners dd7c450
Port exceptions for rounds to db layer on node start
4miners 26b01be
Check if rounds exceptions match database instead of force-update
4miners a7a8789
Initial implementation of rounds management in postgres
4miners d50f559
Improve comments for SQL stuff
4miners 638ccb9
Adjust delegates endpoint to use new table, add rank update trigger
4miners 796aa2f
Merge branch 'development' into 543_refactor-rounds-logic
4miners b8dbf90
Merge remote-tracking branch 'LiskHQ/development' into 543_refactor-r…
4miners 1614640
Fix error from merge
4miners 9fcf8e0
Not use destructuring assignment
4miners ea5479d
Fix intend, add 'grunt release' test
4miners 652730b
Delete deprecated migration
4miners 768b1db
Remove test for duplicated delegates
4miners a564b1c
Initial implementation for postgres LISTEN/NOTIFY feature
4miners b94582f
Fix intend and missing bracket in Jenkinsfile
4miners 6a72081
Adjust DelegatesListSQL tests to new deps versions
4miners d1db361
Notify from postgres when round changes
4miners ab64820
Replaced 'round' notification with 'round-closed' and 'round-reopened'
4miners 036107b
Listen/unlisten to all supported channels properly
4miners 4ecff42
Update 'pg-promise' dependency to latest varsion
4miners 5eba226
Change log level for logging fail of initial connection
4miners 8aafed5
Add 'init' tests for pg-notify helper
4miners 8ddfeeb
Merge branch 'development' into 543_refactor-rounds-logic
a6ea671
Refactor visibility, don't call process.exit on test env
4miners 8bbf0a2
Added more tests for pg-notify helper
4miners 2c4d001
Better error handling for pg-notify helper
4miners d257664
Added/refactored tests for pg-notify helper
4miners e1f45b0
Fix eslint errors
4miners 2770d93
Improved error handling, remove isTestEnv function
4miners df30d35
Add triggerNotify SQL query for pg-notify tests
4miners 33be545
Add tests for 'onNotification', removed tests for 'isTestEnv'
4miners a153365
Upgrade 'pg-promise' dependency to 6.3.5
4miners 9c80dcd
Fix indexes bug in pg-notify tests
4miners b548cc8
Removed unused property, improve comment
4miners 96ae6eb
Merge remote-tracking branch 'LiskHQ/development' into 543_refactor-r…
4miners 1000078
Merge remote-tracking branch 'LiskHQ/development' into 543_refactor-r…
4miners e398391
Fix bug in jobsQueue, added unit tests for it
4miners f47acf9
Merge remote-tracking branch 'LiskHQ/development' into 543_refactor-r…
4miners 19cf5e5
Move variables and functions to proper scope for jobsQueue test
4miners c929941
Reverted jobsQueue to exec functions immediatelly when added to queue
4miners 471a579
Chenged expectations for jobsQueue tests
4miners 2ba556e
Create SQL function getDelegatesList
4miners b4fa22a
Trigger delegates_update_on_block also on height 1
4miners fac2661
Maintain delegates list in memory, changed finishRound event to round…
4miners 95ce3b1
Remove old rounds logic
4miners 65ab982
Removed rounds unit test from Jenkinsfile
4miners af41657
Remove dapp name and link from unconfirmed tracking objects when apply
4miners a846cc3
Allow pg-notify helper to receive JSON in notifications
4miners 6c3086b
Changes expectations for pg-notify unit test, add test cases
4miners 34a0c94
Return round number and delegates list via notifications
4miners 269c880
Support new data fromat in onRoundChanged events
4miners 300e55e
Added note about pgcrypto extension to readme
4miners 726feaf
Added test for SQL function getDelegatesList
4miners 6ac28a5
Reorganise pg-notify tests to prevent interrupt on working node
4miners bfb0e70
Fix DApps module initialisation
4miners 6150efb
Fix spelling and code standards
4miners 5af4103
Change vote_insert_delete trigger to deffered
4miners 11eecab
Add tests for rounds-related things - genesisBlock
4miners 29eb954
Improve code readability, standards in tests
4miners e77c4cb
Add manual forging control for rounds-related tests
4miners e48dd79
Fix wrong trigger name
4miners 70974e0
Do not calculate rewards for forger of genesis block
4miners 2011949
Add first round test to rounds-related tests
4miners 24bfe2c
Merge remote-tracking branch 'LiskHQ/1.0.0' into 543_refactor-rounds-…
4miners a89f367
Run SQL tests in sequence
4miners 4e60dd6
Reorder unit tests
4miners 7d02046
Fix transaction unit test
4miners 216e4e3
Init webSocket for rounds-related tests
4miners b431008
Wait 1 sec for network initialisation in rounds-related tests
4miners 0fb92c4
Fix/improve comments, add test for last block of round deletion (type 0)
4miners 22b6107
Use generateBlock instead of forge to speed up forging process
4miners 196ad48
Base voters calculations on block height instead of round
4miners b33de41
Add tests for rollback when forger of last block of round is unvoted
4miners 3898dc3
Exec validateMemBalances after every tests
4miners b9e694d
Add test for round rollback replace last block forger at last block
4miners 19ff1f5
Replace setTimeout with Promise.delay, wait 20 ms for pg-notify
4miners d8add19
Create functions for maintain outsiders (missed blocks)
4miners 02d4894
Added tests for outsiders
4miners 9f8fe9e
Add tests for round rewards consistency
4miners 8e873e3
Merge remote-tracking branch 'LiskHQ/1.0.0' into 543_refactor-rounds-…
4miners 52562c1
Remove unused modules var and bindModules event
4miners 0c90a8b
Fix standards
4miners f011aaa
Rename migrations so they can run after last applied one
4miners ab3e08a
Improve descriptions for tests
4miners 45e6ec8
Better structure for delegatesList SQL tests
4miners dff4b7a
Remove unused bindModules call
4miners 452f892
Remove unused bindModules call
4miners 559ab05
Improve description in tests
4miners c13d919
Fix code standards in pg-notify helper
4miners 82e03cb
Added TODO/FIXME comments for things that need to be done
4miners 0b2365a
Remove unused rounds module references in tests
4miners File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,155 @@ | ||
'use strict'; | ||
|
||
var Promise = require('bluebird'); | ||
|
||
// Init global module variables | ||
var db, bus, logger; | ||
// Global connection for permanent event listeners | ||
var connection; | ||
// Map channels to bus.message events | ||
var channels = { | ||
'round-closed': 'finishRound', | ||
'round-reopened': 'finishRound' | ||
}; | ||
|
||
function onNotification (data) { | ||
logger.debug('pg-notify: Notification received', {channel: data.channel, data: data.payload}); | ||
|
||
if (!channels[data.channel]) { | ||
// Channel is invalid - should never happen | ||
logger.error('pg-notify: Invalid channel', data.channel); | ||
return; | ||
} | ||
|
||
// Process round-releated things | ||
if (data.channel === 'round-closed') { | ||
data.payload = parseInt(data.payload); | ||
logger.info('pg-notify: Round closed', data.payload); | ||
// Set new round | ||
data.payload += 1; | ||
} else if (data.channel === 'round-reopened') { | ||
data.payload = parseInt(data.payload); | ||
logger.warn('pg-notify: Round reopened', data.payload); | ||
} else { | ||
// Channel is not supported - should never happen | ||
logger.error('pg-notify: Channel not supported', data.channel); | ||
return; | ||
} | ||
|
||
// Broadcast notify via events | ||
bus.message(channels[data.channel], data.payload); | ||
} | ||
|
||
// Generate list of queries for listen to every supported channels | ||
function listenQueries (t) { | ||
var queries = []; | ||
Object.keys(channels).forEach(function (channel) { | ||
queries.push(t.none('LISTEN $1~', channel)); | ||
}); | ||
return t.batch(queries); | ||
} | ||
|
||
function setListeners (client, cb) { | ||
client.on('notification', onNotification); | ||
|
||
connection.task(listenQueries) | ||
.then(function () { | ||
return setImmediate(cb); | ||
}) | ||
.catch(function (err) { | ||
logger.error('pg-notify: Failed to execute LISTEN queries', err); | ||
return setImmediate(cb, err); | ||
}); | ||
} | ||
|
||
// Generate list of queries for unlisten to every supported channels | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
function unlistenQueries (t) { | ||
var queries = []; | ||
Object.keys(channels).forEach(function (channel) { | ||
queries.push(t.none('UNLISTEN $1~', channel)); | ||
}); | ||
return t.batch(queries); | ||
} | ||
|
||
function removeListeners (client, cb) { | ||
client.removeListener('notification', onNotification); | ||
|
||
if (connection) { | ||
connection.task(unlistenQueries) | ||
.then(function () { | ||
return setImmediate(cb); | ||
}) | ||
.catch(function (err) { | ||
logger.error('pg-notify: Failed to execute UNLISTEN queries', err); | ||
return setImmediate(cb, err); | ||
}); | ||
} else { | ||
return setImmediate(cb); | ||
} | ||
} | ||
|
||
function onConnectionLost (err, e) { | ||
logger.error('pg-notify: Connection lost', err); | ||
// Prevent use of the connection | ||
connection = null; | ||
// We don't care about error here, so passing empty function as callback | ||
removeListeners(e.client, function () {}); | ||
// Try to re-establish connection 10 times, every 5 seconds | ||
reconnect(5000, 10) | ||
.then(function (obj) { | ||
logger.info('pg-notify: Reconnected successfully'); | ||
}) | ||
.catch(function () { | ||
// Failed after 10 attempts | ||
logger.error('pg-notify: Failed to reconnect - connection lost'); | ||
process.exit(); | ||
}); | ||
} | ||
|
||
function reconnect (delay, maxAttempts) { | ||
delay = delay > 0 ? delay : 0; | ||
maxAttempts = maxAttempts > 0 ? maxAttempts : 1; | ||
return new Promise(function (resolve, reject) { | ||
setTimeout(function () { | ||
db.connect({direct: true, onLost: onConnectionLost}) | ||
.then(function (obj) { | ||
// Global connection is now available | ||
connection = obj; | ||
setListeners(obj.client, function (err) { | ||
if (err) { | ||
reject(err); | ||
} else { | ||
resolve(obj); | ||
} | ||
}); | ||
}) | ||
.catch(function (err) { | ||
logger.error('pg-notify: Error connecting', err); | ||
if (--maxAttempts) { | ||
reconnect(delay, maxAttempts) | ||
.then(resolve) | ||
.catch(reject); | ||
} else { | ||
reject(err); | ||
} | ||
}); | ||
}, delay); | ||
}); | ||
} | ||
|
||
module.exports.init = function (_db, _bus, _logger, cb) { | ||
db = _db; | ||
bus = _bus; | ||
logger = _logger; | ||
|
||
reconnect () | ||
.then(function (obj) { | ||
logger.info('pg-notify: Initial connection estabilished'); | ||
return setImmediate(cb); | ||
}) | ||
.catch(function (err) { | ||
logger.error('pg-notify: Initial connection failed', err); | ||
// Error is passed to callback here, so node will not start in that case | ||
return setImmediate(cb, err); | ||
}); | ||
}; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -275,9 +275,7 @@ Chain.prototype.applyGenesisBlock = function (block, cb) { | |
} else { | ||
// Set genesis block as last block | ||
modules.blocks.lastBlock.set(block); | ||
// Tick round | ||
// WARNING: DB_WRITE | ||
modules.rounds.tick(block, cb); | ||
return cb(); | ||
} | ||
}); | ||
}; | ||
|
@@ -460,14 +458,11 @@ Chain.prototype.applyBlock = function (block, broadcast, cb, saveBlock) { | |
library.logger.debug('Block applied correctly with ' + block.transactions.length + ' transactions'); | ||
library.bus.message('newBlock', block, broadcast); | ||
|
||
// DATABASE write. Update delegates accounts | ||
modules.rounds.tick(block, seriesCb); | ||
return seriesCb(); | ||
}); | ||
} else { | ||
library.bus.message('newBlock', block, broadcast); | ||
|
||
// DATABASE write. Update delegates accounts | ||
modules.rounds.tick(block, seriesCb); | ||
return seriesCb(); | ||
} | ||
}, | ||
// Push back unconfirmed transactions list (minus the one that were on the block if applied correctly). | ||
|
@@ -549,28 +544,17 @@ __private.popLastBlock = function (oldLastBlock, cb) { | |
return process.exit(0); | ||
} | ||
|
||
// Perform backward tick on rounds | ||
// WARNING: DB_WRITE | ||
modules.rounds.backwardTick(oldLastBlock, previousBlock, function (err) { | ||
// Delete last block from blockchain | ||
// WARNING: Db_WRITE | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
self.deleteBlock(oldLastBlock.id, function (err) { | ||
if (err) { | ||
// Fatal error, memory tables will be inconsistent | ||
library.logger.error('Failed to perform backwards tick', err); | ||
library.logger.error('Failed to delete block', err); | ||
|
||
return process.exit(0); | ||
} | ||
|
||
// Delete last block from blockchain | ||
// WARNING: Db_WRITE | ||
self.deleteBlock(oldLastBlock.id, function (err) { | ||
if (err) { | ||
// Fatal error, memory tables will be inconsistent | ||
library.logger.error('Failed to delete block', err); | ||
|
||
return process.exit(0); | ||
} | ||
|
||
return setImmediate(cb, null, previousBlock); | ||
}); | ||
return setImmediate(cb, null, previousBlock); | ||
}); | ||
}); | ||
}); | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Process round-related things