-
Notifications
You must be signed in to change notification settings - Fork 458
Conversation
Codecov Report
@@ Coverage Diff @@
## release/6.0.0 #9058 +/- ##
=================================================
- Coverage 83.52% 83.45% -0.07%
=================================================
Files 603 603
Lines 22671 22770 +99
Branches 3348 3360 +12
=================================================
+ Hits 18935 19003 +68
- Misses 3736 3767 +31
|
framework/src/engine/legacy/codec.ts
Outdated
if (!blockSchema) { | ||
throw new Error(`Legacy block version ${blockHeader.version} is not registered.`); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This part is repeating (4 times) in this file, can move to a function
const getBlockSchema = (version: number): LegacyBlockSchema => {
const blockSchema = blockSchemaMap[version];
if (!blockSchema) {
throw new Error(`Legacy block version ${version} is not registered.`);
}
return blockSchema;
};
@@ -77,4 +83,24 @@ export class LegacyEndpoint { | |||
|
|||
return decodeBlockJSON(await this.storage.getBlockByHeight(height)).block; | |||
} | |||
|
|||
public async getLegacyBrackets(_context: RequestContext): Promise<LegacyChainBracketInfo[]> { | |||
const legacyBracketInfos = []; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Do you mean you couldn't find any utility of this endpoint for users or you are saying there is no usage in legacy? coz if its latter then we endpoints usually don't have any usage within sdk in other places, its used directly by users. If its former, then it would be nice to show the bracket info to the user as they can see what they configured and how much they have synced based on last block. Related to test, i'm adding it.
- imo, Array is singular but it holds multiple bracketInfo which can be clearly seen from
LegacyChainBracketInfo[]
. The google search that you made is in the case when you call something asLegacyChainBracketInfoArray
but instead you call itLegacyChainBracketInfoArrays
, that is incorrect
Buffer.from(bracket.snapshotBlockID, 'hex'), | ||
); | ||
|
||
const decodedBracketInfo = codec.decode<LegacyChainBracketInfo>( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After decode
, variable should have same name as it's type (LegacyChainBracketInfo
) hence, decodedBracketInfo
=> legacyChainBracketInfo
@@ -54,16 +54,50 @@ export class LegacyChainHandler { | |||
|
|||
for (const bracket of this._legacyConfig.brackets) { | |||
try { | |||
await this._storage.getLegacyChainBracketInfo(Buffer.from(bracket.snapshotBlockID, 'hex')); | |||
const encodedBracketInfo = await this._storage.getLegacyChainBracketInfo( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As it has type Buffer
, can simply call it bracketInfo
await this._storage.setLegacyChainBracketInfo(Buffer.from(bracket.snapshotBlockID, 'hex'), { | ||
...decodedBracketInfo, | ||
startHeight: bracket.startHeight, | ||
snapshotBlockHeight: bracket.snapshotHeight, | ||
// if start block already exists then assign to lastBlockHeight | ||
lastBlockHeight: startBlock ? bracket.startHeight : bracket.snapshotHeight, | ||
}); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- This code is saving
decodedBracketInfo
while at the same time overriding all of it's properties - So there is no point to prepare
decodedBracketInfo
variable at all
let startBlock; | ||
try { | ||
startBlock = await this._storage.getBlockByHeight(bracket.startHeight); | ||
} catch (error) { | ||
if (!(error instanceof NotFoundError)) { | ||
throw error; | ||
} | ||
} | ||
await this._storage.setLegacyChainBracketInfo(Buffer.from(bracket.snapshotBlockID, 'hex'), { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- This is doing exactly the same thing as
try
block (where only change is to first preparedecodedBracketInfo
object, which is not needed, since all of it's properties are then overridden) - Hence, this whole
try/catch
block could be simplified like this
for (const bracket of this._legacyConfig.brackets) {
// Update the existing legacyInfo
let startBlock;
try {
startBlock = await this._storage.getBlockByHeight(bracket.startHeight);
} catch (error) {
if (!(error instanceof NotFoundError)) {
throw error;
}
}
await this._storage.setLegacyChainBracketInfo(Buffer.from(bracket.snapshotBlockID, 'hex'), {
startHeight: bracket.startHeight,
snapshotBlockHeight: bracket.snapshotHeight,
// if start block already exists then assign to lastBlockHeight
lastBlockHeight: startBlock ? bracket.startHeight : bracket.snapshotHeight,
});
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this whole block is rewritten now
@@ -354,18 +354,23 @@ export const createFakeLegacyBlockHeaderV2 = ( | |||
* @params start: Start height of the block range going backwards | |||
* @params numberOfBlocks: Number of blocks to be generated with decreasing height | |||
*/ | |||
export const getLegacyBlocksRangeV2 = (startHeight: number, numberOfBlocks: number): Buffer[] => { | |||
const blocks: LegacyBlockWithID[] = []; | |||
export const getLegacyBlockHeadersRangeV2 = ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- It's purely a function (not testing data), so could be moved to a separate file
fixtureUtils.ts
- If we do it, then
createFakeLegacyBlockHeaderV2
could be moved outside also - Could have a separate
fixtures/
folder for these files
d15f589
to
be5141f
Compare
framework/src/engine/legacy/codec.ts
Outdated
@@ -43,17 +44,22 @@ export const blockSchemaMap: Record<number, LegacyBlockSchema> = { | |||
}, | |||
}; | |||
|
|||
export const getBlockSchemaByVersion = (version: number) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ByVersion
could be removed as param name is descriptive- & we don't have any other similar method
getBlockSchemaByXxx
to differentiate from - If we look at usages,
readVersion()
is used from inside of the called functions, so we can refactor it like this
export const getBlockSchemaByVersion = () => {
const version = readVersion();
const blockSchema = blockSchemaMap[version];
if (!blockSchema) {
throw new Error(`Legacy block version ${version} is not registered.`);
}
return blockSchema;
};
framework/src/engine/legacy/codec.ts
Outdated
@@ -68,6 +74,17 @@ export const decodeBlock = ( | |||
}; | |||
}; | |||
|
|||
export const decodeBlockHeader = (blockHeaderBytes: Buffer): LegacyBlockHeaderWithID => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As expected type is Buffer
, Bytes
postfix could be removed from param
|
||
export const FAILED_SYNC_RETRY_TIMEOUT = 120000; // When no peer was found then resyncing after 2 minutes | ||
export const SUCCESS_SYNC_RETRY_TIMEOUT = 5000; // To avoid syncing with the same peer frequently and get banned due to RPC limit | ||
export const MAX_NUMBER_OF_FAILED_ATTEMPTS = 10; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can simplify to MAX_FAILED_ATTEMPTS
legacyConfig: LegacyConfig; | ||
} | ||
|
||
interface GetBracketInfoEndpointResponse extends LegacyChainBracketInfo { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- All endpoints in fact return some type, so
EndpointResponse
postfix could be removed - Can rename to
LegacyChainBracketInfoWithSnapshotBlockID
- Also this interface could be moved to
types.ts
_context: RequestContext, | ||
): Promise<GetBracketInfoEndpointResponse[]> { | ||
const legacyBracketInfos: GetBracketInfoEndpointResponse[] = []; | ||
for (const bracket of this._legacyConfig.brackets) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Since it converts one array into another, can use
map
- Also this code is sequentially executing. For parallel execution, preferable option is
Promise.all
-https://www.typescriptlang.org/play?pretty=true#code/IYZwngdgxgBAZgV2gFwJYHsIxAGwKZ4AOAFALYgBcMECpARngE4CUMA3gFAwyN7IKMsEPAHcYABUbpSqEHmLFeIdDgBueVgF4AfOy7dsfACqpSedAmSK8ytXgA0McswDc+gL6uO7jhyiYQZEMARwQ8CDRgHAARPFJ0GE0YUEhYYi1dTm5-CED4ROTwaBhiCCoaeiYMvQNkkWBUINwCEgBmAAZOr1refkFqN253N318IMYCgG0AXUH4dAniHLysdDgYSYBGRwAmadYsg2WVPAA6HHQAc1Lug0ZTwgQQAAtiYHrG+Bvbn2yAk-OV2IACI5KFwpEYnF0FRgTAANQ8LzDXxgsIRVBRWLxdIjZZBQjARhRfBQ+IFFLFdKJTIcACQ+PgmwpRTS1U4dM5DP++EB12BqFyqAAJngmcCvFy6e8Gk18ERiDtOu1JVzegIsJs3FyUfTOYy4DsWakSuy9fqeWcLvzBSARWLDRLtVyZZ9mgqOl1nZz1f0dt7dVzGRMkq6gpJpLIziTiJM4Jt0o5Del9t7jrzrSDCcScKTsTCYHDESw3CjsyS8GT0LjfEA
public async getLegacyBrackets(
_context: RequestContext,
): Promise<LegacyChainBracketInfoWithSnapshotBlockID[]> {
let legacyBracketInfos: LegacyChainBracketInfoWithSnapshotBlockID[] = [];
try {
legacyBracketInfos = await Promise.all(
this._legacyConfig.brackets.map(async (bracket) => {
const bracketInfo = await this.storage.getLegacyChainBracketInfo(
Buffer.from(bracket.snapshotBlockID, 'hex'),
);
return {
...bracketInfo,
snapshotBlockID: bracket.snapshotBlockID,
}
}));
} catch (error) {
if (!(error instanceof NotFoundError)) {
throw error;
}
}
return legacyBracketInfos;
}
@@ -116,6 +133,20 @@ export class Storage { | |||
); | |||
} | |||
|
|||
public async legacyChainBracketInfoExist(snapshotBlockID: Buffer): Promise<boolean> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
public async legacyChainBracketInfoExist(snapshotBlockID: Buffer): Promise<boolean> { | |
public async hasBracketInfo(snapshotBlockID: Buffer): Promise<boolean> { |
|
||
export const buildBlockIDDbKey = (id: Buffer): Buffer => Buffer.concat([DB_KEY_BLOCKS_ID, id]); | ||
export const buildBlockIDDbKey = (id: Buffer): Buffer => | ||
Buffer.from(`${DB_KEY_BLOCKS_ID}:${id.toString('binary')}`); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
], | ||
}; | ||
|
||
const legacyStorage = new Storage(new InMemoryDatabase() as any); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
const legacyStorage = new Storage(new InMemoryDatabase() as any); | |
const storage = new Storage(new InMemoryDatabase() as any); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm aligning it with legacyDB
@@ -123,5 +143,43 @@ describe('Legacy endpoint', () => { | |||
expect(transactions).toBeArray(); | |||
matchTxExpectations(transactions[0], tx, txId); | |||
}); | |||
|
|||
it('getLegacyBrackets', async () => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since blockFixtures[0].header
keeps repeating
const { header } = blockFixtures[0];
export const encodeLegacyChainBracketInfo = (data: LegacyChainBracketInfo): Buffer => | ||
codec.encode(legacyChainBracketInfoSchema, data); | ||
|
||
export const decodeLegacyChainBracketInfo = (data: Buffer): LegacyChainBracketInfo => |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
encodeLegacyChainBracketInfo
& decodeLegacyChainBracketInfo
I see only one usage for each of these 2 functions , so probably could be removed & codec.xxx
calls could be inlined
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It doesn't matter how many times its being called, the codec related parts can simply go in codec file and also the encodeLegacyChainBracketInfo was already there
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some points are optional, but other changes could be applied
86a6bc9
to
dec2c6d
Compare
@@ -102,8 +117,10 @@ export class Storage { | |||
await this._db.write(batch); | |||
} | |||
|
|||
public async getLegacyChainBracketInfo(snapshotBlockID: Buffer): Promise<Buffer> { | |||
return this._db.get(buildLegacyBracketDBKey(snapshotBlockID)); | |||
public async getLegacyChainBracketInfo(snapshotBlockID: Buffer): Promise<LegacyChainBracketInfo> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To simplify: can remove LegacyChain
from all such methods here
- getLegacyChainBracketInfo
public async getLegacyChainBracketInfo(snapshotBlockID: Buffer): Promise<LegacyChainBracketInfo>
=>
public async getBracketInfo(snapshotBlockID: Buffer): Promise<BracketInfo> {
- setLegacyChainBracketInfo
public async setLegacyChainBracketInfo(
snapshotBlockID: Buffer,
bracketInfo: LegacyChainBracketInfo,
): Promise<void>
=>
public async setBracketInfo(
snapshotBlockID: Buffer,
bracketInfo: BracketInfo,
): Promise<void>
@@ -68,6 +74,17 @@ export const decodeBlock = ( | |||
}; | |||
}; | |||
|
|||
export const decodeBlockHeader = (blockHeaderBytes: Buffer): LegacyBlockHeaderWithID => { | |||
const version = readVersion(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
readVersion()
is used from inside of decodeBlockHeader
, so can go inside getBlockSchemaByVersion
instead
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
but we want getBlockSchemaByVersion to get schema by version when in future we will have multiple schemas
).block; | ||
let lastBlockID; | ||
try { | ||
const lastBlock = decodeBlock( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can rename decodeBlock
=> toBlockWithIDAndSchema
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess decodeBlock is already very clear
|
||
// After successful sync of a bracket, communicate to the network | ||
this._syncedBrackets.push(Buffer.from(bracket.snapshotBlockID, 'hex')); | ||
this._network.applyNodeInfo({ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For better IDE type hint: applyNodeInfo
has a data
param, which could be renamed to options
await this._updateBracketInfo(lastBlock, bracket); | ||
await this.syncBlocks(bracket, lastBlock); | ||
clearTimeout(this._syncTimeout); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
private async _updateBracketInfo(lastBlock: LegacyBlock, bracket: LegacyBlockBracket) {
await this._storage.setBracketInfo(Buffer.from(bracket.snapshotBlockID, 'hex'), {
startHeight: bracket.startHeight,
lastBlockHeight: lastBlock?.header.height,
snapshotBlockHeight: bracket.snapshotHeight,
});
}
?
could be removed, since lastBlock
is not optional
@@ -215,6 +300,11 @@ export class LegacyChainHandler { | |||
}); | |||
} | |||
|
|||
private applyPenaltyAndRetrySync(msg: string, peerId: string) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since the sync logic is taken out, can rename to warnAndApplyPenalty
if (peersWithLegacyInfo.length === 0) { | ||
const errorMessage = 'No peer found with legacy info.'; | ||
this._logger.warn({ engineModule: 'legacy', method: 'syncBlocks' }, errorMessage); | ||
throw new FailAndAttemptSyncError(errorMessage); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
vs (line 221)
if (syncRetryCounter > MAX_NUMBER_OF_FAILED_ATTEMPTS) { // ***
const errorMessage = `Failed ${MAX_NUMBER_OF_FAILED_ATTEMPTS} times to request from peer.`;
this._logger.warn(
{ engineModule: 'legacy', peerId, method: 'requestFromPeer' },
errorMessage,
);
throw new FailAndAttemptSyncError(errorMessage); / ***
}
SeemsFailAndAttemptSyncError
was created keeping MAX_NUMBER_OF_FAILED_ATTEMPTS
in mind, which probably is not the case for if (peersWithLegacyInfo.length === 0)
|
||
const bufferToHex = (b: Buffer) => Buffer.from(b).toString('hex'); | ||
const randomSnapshotBlockID = utils.getRandomBytes(20); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can move this bufferToHex
to utils.ts
as there are many such Buffer.from(xxx, 'hex')
calls in legacy_chain_handler.ts
also
ac57123
to
da495aa
Compare
- Add snapshotBlockID in requestSchema - Use bracketInfo to respond with legacyBlocks - Add legacyChainBracketInfoExist storage api - Fix tests and add test for getLegacyBrackets endpoint
- Fix storage bug related to getTransactionsByBlockID - Handle timeouts - Handler errors
da495aa
to
999d3b6
Compare
legacyBlock16270316.header.id as Buffer, | ||
0, | ||
), | ||
).rejects.toThrow('No peer found with legacy info.: Attempting to sync again after 12000 ms'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
as above
legacyBlock16270316.header.id as Buffer, | ||
0, | ||
), | ||
).rejects.toThrow('No peer found with legacy info.: Attempting to sync again after 12000 ms'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
export const FAILED_SYNC_RETRY_TIMEOUT = 12000;
export const DB_KEY_LEGACY_BRACKET = Buffer.from([2]); | ||
|
||
// When no peer was found then resyncing after 12 seconds, 1000 * 12 ms | ||
export const FAILED_SYNC_RETRY_TIMEOUT = 12000; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can rename to FAILED_SYNC_RETRY_TIMEOUT_MS
What was the problem?
This PR resolves #9033
How was it solved?
How was it tested?
Add legacy data to legacy.db under data path and call all the legacy endpoints