Skip to content
This repository has been archived by the owner on Jun 11, 2024. It is now read-only.

Legacy endpoints issue #9058

Merged
merged 10 commits into from
Oct 6, 2023
Merged

Legacy endpoints issue #9058

merged 10 commits into from
Oct 6, 2023

Conversation

ishantiw
Copy link
Contributor

What was the problem?

This PR resolves #9033

How was it solved?

  • Align DB keys with legacy DB
  • Fix encode/decode of blocks
  • Add endpoint to get legacyBracketInfo
  • Update existing legacyBracketInfo from config on startup

How was it tested?

Add legacy data to legacy.db under data path and call all the legacy endpoints

@ishantiw ishantiw self-assigned this Sep 29, 2023
@codecov
Copy link

codecov bot commented Sep 29, 2023

Codecov Report

Merging #9058 (999d3b6) into release/6.0.0 (da598fb) will decrease coverage by 0.07%.
The diff coverage is 70.76%.

Impacted file tree graph

@@                Coverage Diff                @@
##           release/6.0.0    #9058      +/-   ##
=================================================
- Coverage          83.52%   83.45%   -0.07%     
=================================================
  Files                603      603              
  Lines              22671    22770      +99     
  Branches            3348     3360      +12     
=================================================
+ Hits               18935    19003      +68     
- Misses              3736     3767      +31     
Files Coverage Δ
framework/src/engine/engine.ts 70.73% <100.00%> (+0.17%) ⬆️
framework/src/engine/legacy/constants.ts 100.00% <100.00%> (ø)
framework/src/engine/legacy/endpoint.ts 88.23% <100.00%> (+2.52%) ⬆️
framework/src/engine/legacy/errors.ts 100.00% <100.00%> (+75.00%) ⬆️
framework/src/engine/legacy/schemas.ts 100.00% <100.00%> (ø)
framework/src/engine/legacy/utils.ts 100.00% <100.00%> (ø)
framework/src/engine/legacy/codec.ts 97.67% <94.11%> (+3.92%) ⬆️
framework/src/engine/legacy/storage.ts 91.66% <81.81%> (-4.77%) ⬇️
framework/src/engine/legacy/network_endpoint.ts 80.00% <70.83%> (-11.43%) ⬇️
...ramework/src/engine/legacy/legacy_chain_handler.ts 67.46% <53.65%> (-12.54%) ⬇️

Comment on lines 132 to 134
if (!blockSchema) {
throw new Error(`Legacy block version ${blockHeader.version} is not registered.`);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part is repeating (4 times) in this file, can move to a function

const getBlockSchema = (version: number): LegacyBlockSchema => {
    const blockSchema = blockSchemaMap[version];
    if (!blockSchema) {
        throw new Error(`Legacy block version ${version} is not registered.`);
    }
    return blockSchema;
};

@@ -77,4 +83,24 @@ export class LegacyEndpoint {

return decodeBlockJSON(await this.storage.getBlockByHeight(height)).block;
}

public async getLegacyBrackets(_context: RequestContext): Promise<LegacyChainBracketInfo[]> {
const legacyBracketInfos = [];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • I can't find any usage of this endpoint. No test case either ?
  • As per Google
Screenshot 2023-10-02 at 12 11 21 PM
  • Hence, const legacyBracketInfo: LegacyChainBracketInfo[] = [];

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Do you mean you couldn't find any utility of this endpoint for users or you are saying there is no usage in legacy? coz if its latter then we endpoints usually don't have any usage within sdk in other places, its used directly by users. If its former, then it would be nice to show the bracket info to the user as they can see what they configured and how much they have synced based on last block. Related to test, i'm adding it.
  • imo, Array is singular but it holds multiple bracketInfo which can be clearly seen from LegacyChainBracketInfo[]. The google search that you made is in the case when you call something as LegacyChainBracketInfoArray but instead you call it LegacyChainBracketInfoArrays, that is incorrect

Buffer.from(bracket.snapshotBlockID, 'hex'),
);

const decodedBracketInfo = codec.decode<LegacyChainBracketInfo>(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After decode, variable should have same name as it's type (LegacyChainBracketInfo) hence, decodedBracketInfo => legacyChainBracketInfo

@@ -54,16 +54,50 @@ export class LegacyChainHandler {

for (const bracket of this._legacyConfig.brackets) {
try {
await this._storage.getLegacyChainBracketInfo(Buffer.from(bracket.snapshotBlockID, 'hex'));
const encodedBracketInfo = await this._storage.getLegacyChainBracketInfo(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As it has type Buffer, can simply call it bracketInfo

Comment on lines 76 to 95
await this._storage.setLegacyChainBracketInfo(Buffer.from(bracket.snapshotBlockID, 'hex'), {
...decodedBracketInfo,
startHeight: bracket.startHeight,
snapshotBlockHeight: bracket.snapshotHeight,
// if start block already exists then assign to lastBlockHeight
lastBlockHeight: startBlock ? bracket.startHeight : bracket.snapshotHeight,
});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • This code is saving decodedBracketInfo while at the same time overriding all of it's properties
  • So there is no point to prepare decodedBracketInfo variable at all

Comment on lines 88 to 96
let startBlock;
try {
startBlock = await this._storage.getBlockByHeight(bracket.startHeight);
} catch (error) {
if (!(error instanceof NotFoundError)) {
throw error;
}
}
await this._storage.setLegacyChainBracketInfo(Buffer.from(bracket.snapshotBlockID, 'hex'), {
Copy link
Contributor

@sitetester sitetester Oct 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • This is doing exactly the same thing as try block (where only change is to first prepare decodedBracketInfo object, which is not needed, since all of it's properties are then overridden)
  • Hence, this whole try/catch block could be simplified like this
for (const bracket of this._legacyConfig.brackets) {
	// Update the existing legacyInfo
	let startBlock;
	try {
		startBlock = await this._storage.getBlockByHeight(bracket.startHeight);
	} catch (error) {
		if (!(error instanceof NotFoundError)) {
			throw error;
		}
	}

	await this._storage.setLegacyChainBracketInfo(Buffer.from(bracket.snapshotBlockID, 'hex'), {
		startHeight: bracket.startHeight,
		snapshotBlockHeight: bracket.snapshotHeight,
		// if start block already exists then assign to lastBlockHeight
		lastBlockHeight: startBlock ? bracket.startHeight : bracket.snapshotHeight,
	});
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this whole block is rewritten now

@@ -354,18 +354,23 @@ export const createFakeLegacyBlockHeaderV2 = (
* @params start: Start height of the block range going backwards
* @params numberOfBlocks: Number of blocks to be generated with decreasing height
*/
export const getLegacyBlocksRangeV2 = (startHeight: number, numberOfBlocks: number): Buffer[] => {
const blocks: LegacyBlockWithID[] = [];
export const getLegacyBlockHeadersRangeV2 = (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • It's purely a function (not testing data), so could be moved to a separate file fixtureUtils.ts
  • If we do it, then createFakeLegacyBlockHeaderV2 could be moved outside also
  • Could have a separate fixtures/ folder for these files

framework/src/engine/legacy/legacy_chain_handler.ts Outdated Show resolved Hide resolved
@ishantiw ishantiw requested review from shuse2 and sitetester October 2, 2023 15:39
@ishantiw ishantiw force-pushed the 9033-legacy-endpoints-issue branch 4 times, most recently from d15f589 to be5141f Compare October 4, 2023 15:40
@@ -43,17 +44,22 @@ export const blockSchemaMap: Record<number, LegacyBlockSchema> = {
},
};

export const getBlockSchemaByVersion = (version: number) => {
Copy link
Contributor

@sitetester sitetester Oct 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • ByVersion could be removed as param name is descriptive
  • & we don't have any other similar method getBlockSchemaByXxx to differentiate from
  • If we look at usages, readVersion() is used from inside of the called functions, so we can refactor it like this
export const getBlockSchemaByVersion = () => {
	const version = readVersion();
	const blockSchema = blockSchemaMap[version];
	if (!blockSchema) {
		throw new Error(`Legacy block version ${version} is not registered.`);
	}
	return blockSchema;
};

@@ -68,6 +74,17 @@ export const decodeBlock = (
};
};

export const decodeBlockHeader = (blockHeaderBytes: Buffer): LegacyBlockHeaderWithID => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As expected type is Buffer, Bytes postfix could be removed from param


export const FAILED_SYNC_RETRY_TIMEOUT = 120000; // When no peer was found then resyncing after 2 minutes
export const SUCCESS_SYNC_RETRY_TIMEOUT = 5000; // To avoid syncing with the same peer frequently and get banned due to RPC limit
export const MAX_NUMBER_OF_FAILED_ATTEMPTS = 10;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can simplify to MAX_FAILED_ATTEMPTS

legacyConfig: LegacyConfig;
}

interface GetBracketInfoEndpointResponse extends LegacyChainBracketInfo {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • All endpoints in fact return some type, so EndpointResponse postfix could be removed
  • Can rename to LegacyChainBracketInfoWithSnapshotBlockID
  • Also this interface could be moved to types.ts

_context: RequestContext,
): Promise<GetBracketInfoEndpointResponse[]> {
const legacyBracketInfos: GetBracketInfoEndpointResponse[] = [];
for (const bracket of this._legacyConfig.brackets) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

public async getLegacyBrackets(
		_context: RequestContext,
	): Promise<LegacyChainBracketInfoWithSnapshotBlockID[]> {
		let legacyBracketInfos: LegacyChainBracketInfoWithSnapshotBlockID[] = [];
		try {
			legacyBracketInfos = await Promise.all(
				this._legacyConfig.brackets.map(async (bracket) => {
					const bracketInfo = await this.storage.getLegacyChainBracketInfo(
						Buffer.from(bracket.snapshotBlockID, 'hex'),
					);
					return {
						...bracketInfo,
						snapshotBlockID: bracket.snapshotBlockID,
					}
				}));
		} catch (error) {
			if (!(error instanceof NotFoundError)) {
				throw error;
			}
		}

		return legacyBracketInfos;
	}

@@ -116,6 +133,20 @@ export class Storage {
);
}

public async legacyChainBracketInfoExist(snapshotBlockID: Buffer): Promise<boolean> {
Copy link
Contributor

@sitetester sitetester Oct 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
public async legacyChainBracketInfoExist(snapshotBlockID: Buffer): Promise<boolean> {
public async hasBracketInfo(snapshotBlockID: Buffer): Promise<boolean> {


export const buildBlockIDDbKey = (id: Buffer): Buffer => Buffer.concat([DB_KEY_BLOCKS_ID, id]);
export const buildBlockIDDbKey = (id: Buffer): Buffer =>
Buffer.from(`${DB_KEY_BLOCKS_ID}:${id.toString('binary')}`);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can add a line space before buildTxsBlockIDDbKey to make it more readable & consistent

Screenshot 2023-10-05 at 04 53 24 PM

],
};

const legacyStorage = new Storage(new InMemoryDatabase() as any);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const legacyStorage = new Storage(new InMemoryDatabase() as any);
const storage = new Storage(new InMemoryDatabase() as any);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm aligning it with legacyDB

@@ -123,5 +143,43 @@ describe('Legacy endpoint', () => {
expect(transactions).toBeArray();
matchTxExpectations(transactions[0], tx, txId);
});

it('getLegacyBrackets', async () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since blockFixtures[0].header keeps repeating

const { header } = blockFixtures[0];

export const encodeLegacyChainBracketInfo = (data: LegacyChainBracketInfo): Buffer =>
codec.encode(legacyChainBracketInfoSchema, data);

export const decodeLegacyChainBracketInfo = (data: Buffer): LegacyChainBracketInfo =>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

encodeLegacyChainBracketInfo & decodeLegacyChainBracketInfo
I see only one usage for each of these 2 functions , so probably could be removed & codec.xxx calls could be inlined

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't matter how many times its being called, the codec related parts can simply go in codec file and also the encodeLegacyChainBracketInfo was already there

Copy link
Contributor

@sitetester sitetester left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some points are optional, but other changes could be applied

@ishantiw ishantiw force-pushed the 9033-legacy-endpoints-issue branch from 86a6bc9 to dec2c6d Compare October 5, 2023 17:20
@@ -102,8 +117,10 @@ export class Storage {
await this._db.write(batch);
}

public async getLegacyChainBracketInfo(snapshotBlockID: Buffer): Promise<Buffer> {
return this._db.get(buildLegacyBracketDBKey(snapshotBlockID));
public async getLegacyChainBracketInfo(snapshotBlockID: Buffer): Promise<LegacyChainBracketInfo> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To simplify: can remove LegacyChain from all such methods here

  • getLegacyChainBracketInfo
public async getLegacyChainBracketInfo(snapshotBlockID: Buffer): Promise<LegacyChainBracketInfo>
=>
public async getBracketInfo(snapshotBlockID: Buffer): Promise<BracketInfo> {
  • setLegacyChainBracketInfo
public async setLegacyChainBracketInfo(
		snapshotBlockID: Buffer,
		bracketInfo: LegacyChainBracketInfo,
	): Promise<void>
=>
public async setBracketInfo(
		snapshotBlockID: Buffer,
		bracketInfo: BracketInfo,
	): Promise<void>

@@ -68,6 +74,17 @@ export const decodeBlock = (
};
};

export const decodeBlockHeader = (blockHeaderBytes: Buffer): LegacyBlockHeaderWithID => {
const version = readVersion();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

readVersion() is used from inside of decodeBlockHeader, so can go inside getBlockSchemaByVersion instead

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but we want getBlockSchemaByVersion to get schema by version when in future we will have multiple schemas

).block;
let lastBlockID;
try {
const lastBlock = decodeBlock(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can rename decodeBlock => toBlockWithIDAndSchema

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess decodeBlock is already very clear


// After successful sync of a bracket, communicate to the network
this._syncedBrackets.push(Buffer.from(bracket.snapshotBlockID, 'hex'));
this._network.applyNodeInfo({
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For better IDE type hint: applyNodeInfo has a data param, which could be renamed to options

await this._updateBracketInfo(lastBlock, bracket);
await this.syncBlocks(bracket, lastBlock);
clearTimeout(this._syncTimeout);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

private async _updateBracketInfo(lastBlock: LegacyBlock, bracket: LegacyBlockBracket) {
	await this._storage.setBracketInfo(Buffer.from(bracket.snapshotBlockID, 'hex'), {
		startHeight: bracket.startHeight,
		lastBlockHeight: lastBlock?.header.height,
		snapshotBlockHeight: bracket.snapshotHeight,
	});
}

? could be removed, since lastBlock is not optional

@@ -215,6 +300,11 @@ export class LegacyChainHandler {
});
}

private applyPenaltyAndRetrySync(msg: string, peerId: string) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the sync logic is taken out, can rename to warnAndApplyPenalty

Comment on lines 195 to 198
if (peersWithLegacyInfo.length === 0) {
const errorMessage = 'No peer found with legacy info.';
this._logger.warn({ engineModule: 'legacy', method: 'syncBlocks' }, errorMessage);
throw new FailAndAttemptSyncError(errorMessage);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vs (line 221)

if (syncRetryCounter > MAX_NUMBER_OF_FAILED_ATTEMPTS) { // ***
	const errorMessage = `Failed ${MAX_NUMBER_OF_FAILED_ATTEMPTS} times to request from peer.`;
	this._logger.warn(
		{ engineModule: 'legacy', peerId, method: 'requestFromPeer' },
		errorMessage,
	);
	throw new FailAndAttemptSyncError(errorMessage); / ***
}

SeemsFailAndAttemptSyncError was created keeping MAX_NUMBER_OF_FAILED_ATTEMPTS in mind, which probably is not the case for if (peersWithLegacyInfo.length === 0)

@ishantiw ishantiw requested review from sitetester and shuse2 October 6, 2023 11:28

const bufferToHex = (b: Buffer) => Buffer.from(b).toString('hex');
const randomSnapshotBlockID = utils.getRandomBytes(20);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can move this bufferToHex to utils.ts as there are many such Buffer.from(xxx, 'hex') calls in legacy_chain_handler.ts also

@ishantiw ishantiw force-pushed the 9033-legacy-endpoints-issue branch from ac57123 to da495aa Compare October 6, 2023 15:10
@ishantiw ishantiw requested a review from sitetester October 6, 2023 15:13
- Add snapshotBlockID in requestSchema
- Use bracketInfo to respond with legacyBlocks
- Add legacyChainBracketInfoExist storage api
- Fix tests and add test for getLegacyBrackets endpoint
@ishantiw ishantiw force-pushed the 9033-legacy-endpoints-issue branch from da495aa to 999d3b6 Compare October 6, 2023 15:14
legacyBlock16270316.header.id as Buffer,
0,
),
).rejects.toThrow('No peer found with legacy info.: Attempting to sync again after 12000 ms');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as above

legacyBlock16270316.header.id as Buffer,
0,
),
).rejects.toThrow('No peer found with legacy info.: Attempting to sync again after 12000 ms');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

export const FAILED_SYNC_RETRY_TIMEOUT = 12000;

export const DB_KEY_LEGACY_BRACKET = Buffer.from([2]);

// When no peer was found then resyncing after 12 seconds, 1000 * 12 ms
export const FAILED_SYNC_RETRY_TIMEOUT = 12000;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can rename to FAILED_SYNC_RETRY_TIMEOUT_MS

@ishantiw ishantiw merged commit c589442 into release/6.0.0 Oct 6, 2023
@ishantiw ishantiw deleted the 9033-legacy-endpoints-issue branch October 6, 2023 16:12
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants