Sepolia static peers dead or congested

[keithchew]

Using devp2p, I tested the peers in SepoliaStaticPeers defined in bootnodes.go. For all the peers above the last 5 entries, the command hangs, indicating the endpoint is either blocking the connection or does not exists. For the last 5 peers, they often respond with too many peers connected, indicating that these peers are online but just overloaded!

This is probably not the only place I should be reporting this, apart from erigon removing the dead peers in that list. I will also try to report this to geth to see if they have any comments on the overloaded peers.

[Giulio2002]

Can you cite this PR in geth?

[Giulio2002]

Hey, the sepolia bootnodes seems fine. you should not set them as StaticPeers though. static peers will be used to request data. bootnodes have a different function

[keithchew]

PRs from Erigon:
#6433
#7314

The 2nd PR brings across the last 5 nodes from geth's PR:
ethereum/go-ethereum#27099

You can see from geth's PR at that time, the "devp2p rlpx ping" works fine for the 5 peers. As stated in my initial description, I executed the command against all the peers in Erigon's SepoliaStaticPeers , and they seem dead except for the last 5, which often give "too many peers" replies.

[keithchew]

Well, in geth they are boot nodes, but in Erigon, they are coded to be static peers, as in setStaticPeers() function in flag.go.

[keithchew]

Example reply from these nodes:

devp2p rlpx ping "enode://9e9492e2e8836114cc75f5b929784f4f46c324ad01daf87d956f98b3b6c5fcba95524d6e5cf9861dc96a2c8a171e
a7105bb554a197455058de185fa870970c7c@138.68.123.152:30303"
received disconnect message: too many peers

So, yes they are fine as in online, but hardly usable and stable.

[Giulio2002]

mh, but then it does not make a difference. why is this a bug? what are you trying to accomplish?

[Giulio2002]

oh var SepoliaStaticPeers = []string{ // from https://github.com/ledgerwatch/erigon/issues/6134#issuecomment-1354923418 "enode://8ae4559db1b1e160be8cc46018d7db123ed6d03fbbfe481da5ec05f71f0aa4d5f4b02ad059127096aa994568706a0d02933984083b87c5e1e3de2b7692444d37@35.161.233.158:46855", "enode://d0b3b290422f35ec3e68356f3a4cdf9c661f71a868110670e31441a5021d7abd0440ae8dfb9360aafdd0198f177863361e3a7a7eb5e1a3e26575bf1ac3ef4ab3@162.19.136.65:48264", "enode://d64624bda3cdb65d542c90757a4a661cfe9dddf8328bdb1ea97a8d70fad287c360f0101c492d8fd6ab30d79160a3bf148cacfd68f5d2e47eab0b709516419304@51.195.63.10:30040", "enode://c7df835939e027325c6bba926220fae5912a33c83d96b3eef8ef445c98083f3191788581c9a0e8f74cadb0b13229b847f5c1ebd315b22bcf11faf6468020eb48@54.163.51.157:30303", "enode://da0609bad3afcab9b93175a41a2d621d07aa7ff6c134a00792d4541f0ce8d30d8f3c51bb37a47573508a0bf18865b04066af2a661edf1d3a3d8d133fc1031aa0@88.151.101.14:45192", "enode://7a4534d392c59369eae6befa56ac670476d9edc16597cf53c92bbefa6e741b6b0b9e6822cab12afb09123e03ca1131026fbef145adec429fe2e50182dfb650a5@94.130.18.108:31312", "enode://db6fa13b63a885440de581ee3fc8df9c6a590326b39fc5ccba7991707ee0cebac306211f7eca5270a350201a3132511f2338481edd81f3dc819c2a1c60419cf2@65.21.89.157:30303", "enode://fcf03e9404cace34c60e4eed374ef9a779471014319b3346352fbc2f992a399af6517486e8e65a4ab55f4645fe55420bbea1cddc13a4af4df63b0f731915c6a6@13.125.238.49:46173", "enode://8b973816278fdd56966709e4794c7ccce1f256eaa9165a6b013b991a9bdf3886a8f2d23af50ee723a5614a9fe9d197252b803b4455a87ab2468e128f7b06e0ca@172.104.107.145:30303", "enode://5a1fb15f826a213d3ef4adb9be47ab58b2240ea05df0d760a244f04762b0847dcb08276b1284f726c22eea30fce0c601cf121b81bac0c151f1b3b4ad00d1482a@34.159.55.147:51262", "enode://560928dd14819f88113586726e452b16bbc694ed4144ddadd6290053e7f3fc66bfad13add6889f7d8f37e0c21ccbb6948eb8899c8b30743f4b45a3081f1efed8@34.138.254.5:29888", "enode://69a13b575b8c5278431409e9f7db36e7218667ae286bfb65a72dfec9201b2c5bbbe2797a1babbdf17a7bf7ca68fa3fbe1554612637eb1b2425fa975e1bccb54c@35.223.41.3:30303", "enode://66158b31eecff939f220b291d2b448edbfe94f1d4c992d9395b5d476e55e54b5abd11d3ee44daf1e18ee27b910ef99cdf6f19775eb4820ebe4f77d7aa948e3b6@51.195.63.10:55198", "enode://bf94acbd51170bf075cacb9f149b21ff46354d659ab434a0d40688f776e1e1556bc62be2dc2867ba513844268c0dc8240099a6b60efe1713fbc25da7fdeb6ff1@3.82.105.139:30303", "enode://41329e5ceb51cdddbe6a475db00b682505768b71ff8ee37d2d3500ca1b78918f9fad57d6006dd9f79cd418437dbcf87ec2fd58d60710f925cb17da05a51197cf@65.21.34.60:30303", "enode://4e5e92199ee224a01932a377160aa432f31d0b351f84ab413a8e0a42f4f36476f8fb1cbe914af0d9aef0d51665c214cf653c651c4bbd9d5550a934f241f1682b@138.197.51.181:30303", "enode://143e11fb766781d22d92a2e33f8f104cddae4411a122295ed1fdb6638de96a6ce65f5b7c964ba3763bba27961738fef7d3ecc739268f3e5e771fb4c87b6234ba@146.190.1.103:30303", "enode://8b61dc2d06c3f96fddcbebb0efb29d60d3598650275dc469c22229d3e5620369b0d3dedafd929835fe7f489618f19f456fe7c0df572bf2d914a9f4e006f783a9@170.64.250.88:30303", "enode://10d62eff032205fcef19497f35ca8477bea0eadfff6d769a147e895d8b2b8f8ae6341630c645c30f5df6e67547c03494ced3d9c5764e8622a26587b083b028e8@139.59.49.206:30303", "enode://9e9492e2e8836114cc75f5b929784f4f46c324ad01daf87d956f98b3b6c5fcba95524d6e5cf9861dc96a2c8a171ea7105bb554a197455058de185fa870970c7c@138.68.123.152:30303", }

these are not bootnodes FYI, they are nodes for faster startup time. if they are congested, this should have no effect.

[keithchew]

See here:

func setStaticPeers(ctx *cli.Context, cfg *p2p.Config) {
	var urls []string
	if ctx.IsSet(StaticPeersFlag.Name) {
		urls = libcommon.CliString2Array(ctx.String(StaticPeersFlag.Name))
	} else {
		chain := ctx.String(ChainFlag.Name)
		urls = params.StaticPeerURLsOfChain(chain)
	}

	nodes, err := ParseNodesFromURLs(urls)
	if err != nil {
		Fatalf("Option %s: %v", StaticPeersFlag.Name, err)
	}

	cfg.StaticNodes = nodes
}

It will set them as static peers. What I am trying to achieve is a stable Erigon on sepolia... mainnet is rock solid, but sepolia is quite unstable. Once GoodPeers goes to 0, it never recovers and multiple reboots are required and if lucky, it will continue to sync. Not the best environment for development and testing of dapps. This is with or without specifying static peers.

[Giulio2002]

Then if you could, could you open a different issue pointing to this one saying that sepolia nodes go to 0? sepolia static peers congested is probably not then what you are trying to fix

[keithchew]

In my issue description, I stated that all the peers above the last 5 appear to be dead (using devp2p to check), can you confirm that any of them are alive and responding? If they are dead, then they should be removed from the list. For the last 5, congestion is the root cause of Erigon's instability, which is a better description than GoodPeers is 0, as 0 can be caused by different reasons, eg dead nodes, bug in code, congested nodes. If you consider this information not helpful, then please ignore this report. I will just have to wait to see if the condition improves over time, just thought Erigon's dev team might have contacts to the Ethereum development team that you can help pass on the information.

[Giulio2002]

In my issue description, I stated that all the peers above the last 5 appear to be dead (using devp2p to check), can you confirm that any of them are alive and responding? If they are dead, then they should be removed from the list. For the last 5, congestion is the root cause of Erigon's instability, which is a better description than GoodPeers is 0, as 0 can be caused by different reasons, eg dead nodes, bug in code, congested nodes. If you consider this information not helpful, then please ignore this report. I will just have to wait to see if the condition improves over time, just thought Erigon's dev team might have contacts to the Ethereum development team that you can help pass on the information.

Ok, can you open the ticket and add this as a consideration? also i personally doubt it is related to the static peers because if the issue is: can connect and after a while, you lose all peers, it makes me believe it has little to do with hardcoded bootstrap nodes and static peers

[keithchew]

Hi @Giulio2002

My apologies, you are correct. After reviewing the infrastructure configuration, we found the sepolia node was not configured correctly in the firewall (incorrect port). Once fixed, GoodPeers are now looking healthier, with 60+ peers!

So, looks like even with potentially congested bootnodes, if your firewall and network is setup correctly, it should be enough to get good peers.