Gateway - getting price query failed error after restarting gateway connection #164

nikspz · 2023-07-19T15:26:24Z

Gateway - after restart of the gateway Client failed to get access to the token list for amm_data_feed_example.py script

Steps To Reproduce

Clone and install Source client + gateway
gateway generate certs
./gateway-setup.sh
yarn start --passphrase=password, make sure gateway is ONLINE
connect quickswap and uniswap polygon mainnet
start the script amm data_feed example
check status and review script working
stop the gateway
yarn start --passphrase=password again and make sure gateway is ONLINE
start the same script again

Actual:
review that bot Failed to get data from AmmDataFeed

Expected:
Client are able to start the script successfully again after restart of the gateway

Loom Video:
https://www.loom.com/share/c456e9e18ff04e47af9f5153b1393118

logs_amm_data_feed_example.log
logs_gateway_app.log


2023-07-19 15:01:18,451 - 439205 - hummingbot.data_feed.amm_gateway_data_feed - ERROR - Error getting data from AmmDataFeed[uniswap_polygon_mainnet]Check network connection. Error: Error on POST https://localhost:15888/amm/price Error: {'message': 'Price query failed: Token not supported: WMATIC', 'httpErrorCode': 500, 'errorCode': 1013, 'stack': 'Error: Price query failed: Token not supported: WMATIC\n    at /home/nikita/gwstaging/dist/src/connectors/uniswap/uniswap.controllers.js:74:23\n    at Generator.throw (<anonymous>)\n    at rejected (/home/nikita/gwstaging/dist/src/connectors/uniswap/uniswap.controllers.js:6:65)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)'}
2023-07-19 15:01:18,452 - 439205 - hummingbot.data_feed.amm_gateway_data_feed - ERROR - Error getting data from AmmDataFeed[quickswap_polygon_mainnet]Check network connection. Error: Error on POST https://localhost:15888/amm/price Error: {'message': 'Price query failed: Token not supported: LINK', 'httpErrorCode': 500, 'errorCode': 1013, 'stack': 'Error: Price query failed: Token not supported: LINK\n    at /home/nikita/gwstaging/dist/src/connectors/uniswap/uniswap.controllers.js:74:23\n    at Generator.throw (<anonymous>)\n    at rejected (/home/nikita/gwstaging/dist/src/connectors/uniswap/uniswap.controllers.js:6:65)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)'}
2023-07-19 15:01:18,770 - 439205 - hummingbot.connector.exchange.binance.binance_api_order_book_data_source.BinanceAPIOrderBookDataSource - INFO - Subscribed to public order book and trade channels...
2023-07-19 15:01:18,819 - 439205 - hummingbot.core.data_type.order_book_tracker - INFO - Initialized order book for BTC-USDT. 1/1 completed.
2023-07-19 15:01:19,001 - 439205 - hummingbot.strategy.script_strategy_base - WARNING - binance is not ready. Please wait...
2023-07-19 15:01:19,481 - 439205 - hummingbot.data_feed.amm_gateway_data_feed - ERROR - Error getting data from AmmDataFeed[quickswap_polygon_mainnet]Check network connection. Error: Error on POST https://localhost:15888/amm/price Error: {'message': 'Price query failed: Token not supported: WMATIC', 'httpErrorCode': 500, 'errorCode': 1013, 'stack': 'Error: Price query failed: Token not supported: WMATIC\n    at /home/nikita/gwstaging/dist/src/connectors/uniswap/uniswap.controllers.js:74:23\n    at Generator.throw (<anonymous>)\n    at rejected (/home/nikita/gwstaging/dist/src/connectors/uniswap/uniswap.controllers.js:6:65)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)'}
2023-07-19 15:01:19,527 - 439205 - hummingbot.data_feed.amm_gateway_data_feed - ERROR - Error getting data from AmmDataFeed[uniswap_polygon_mainnet]Check network connection. Error: Error on POST https://localhost:15888/amm/price Error: {'message': 'Price query failed: Token not supported: LINK', 'httpErrorCode': 500, 'errorCode': 1013, 'stack': 'Error: Price query failed: Token not supported: LINK\n    at /home/nikita/gwstaging/dist/src/connectors/uniswap/uniswap.controllers.js:74:23\n    at Generator.throw (<anonymous>)\n    at rejected (/home/nikita/gwstaging/dist/src/connectors/uniswap/uniswap.controllers.js:6:65)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)'}
2023-07-19 15:01:19,580 - 439205 - hummingbot.client.ui.hummingbot_cli - ERROR - Unhandled error in prompt_toolkit: Error on POST https://localhost:15888/amm/price Error: {'message': 'Price query failed: Token not supported: LINK', 'httpErrorCode': 500, 'errorCode': 1013, 'stack': 'Error: Price query failed: Token not supported: LINK\n    at /home/nikita/gwstaging/dist/src/connectors/uniswap/uniswap.controllers.js:74:23\n    at Generator.throw (<anonymous>)\n    at rejected (/home/nikita/gwstaging/dist/src/connectors/uniswap/uniswap.controllers.js:6:65)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)'}
NoneType: None
2023-07-19 15:01:19,580 - 439205 - hummingbot.client.ui.hummingbot_cli - ERROR - Unhandled error in prompt_toolkit: Error on POST https://localhost:15888/amm/price Error: {'message': 'Price query failed: Token not supported: WMATIC', 'httpErrorCode': 500, 'errorCode': 1013, 'stack': 'Error: Price query failed: Token not supported: WMATIC\n    at /home/nikita/gwstaging/dist/src/connectors/uniswap/uniswap.controllers.js:74:23\n    at Generator.throw (<anonymous>)\n    at rejected (/home/nikita/gwstaging/dist/src/connectors/uniswap/uniswap.controllers.js:6:65)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)'}
NoneType: None
2023-07-19 15:01:19,581 - 439205 - hummingbot.client.ui.hummingbot_cli - ERROR - Unhandled error in prompt_toolkit: Error on POST https://localhost:15888/amm/price Error: {'message': 'Price query failed: Token not supported: AAVE', 'httpErrorCode': 500, 'errorCode': 1013, 'stack': 'Error: Price query failed: Token not supported: AAVE\n    at /home/nikita/gwstaging/dist/src/connectors/uniswap/uniswap.controllers.js:74:23\n    at Generator.throw (<anonymous>)\n    at rejected (/home/nikita/gwstaging/dist/src/connectors/uniswap/uniswap.controllers.js:6:65)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)'}
NoneType: None
2023-07-19 15:01:19,581 - 439205 - hummingbot.client.ui.hummingbot_cli - ERROR - Unhandled error in prompt_toolkit: Error on POST https://localhost:15888/amm/price Error: {'message': 'Price query failed: Token not supported: LINK', 'httpErrorCode': 500, 'errorCode': 1013, 'stack': 'Error: Price query failed: Token not supported: LINK\n    at /home/nikita/gwstaging/dist/src/connectors/uniswap/uniswap.controllers.js:74:23\n    at Generator.throw (<anonymous>)\n    at rejected (/home/nikita/gwstaging/dist/src/connectors/uniswap/uniswap.controllers.js:6:65)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)'}
NoneType: None
2023-07-19 15:01:19,581 - 439205 - hummingbot.client.ui.hummingbot_cli - ERROR - Unhandled error in prompt_toolkit: Error on POST https://localhost:15888/amm/price Error: {'message': 'Price query failed: Token not supported: WMATIC', 'httpErrorCode': 500, 'errorCode': 1013, 'stack': 'Error: Price query failed: Token not supported: WMATIC\n    at /home/nikita/gwstaging/dist/src/connectors/uniswap/uniswap.controllers.js:74:23\n    at Generator.throw (<anonymous>)\n    at rejected (/home/nikita/gwstaging/dist/src/connectors/uniswap/uniswap.controllers.js:6:65)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)'}
NoneType: None

The text was updated successfully, but these errors were encountered:

nikspz · 2023-07-19T15:27:42Z

cc: @vic-en

Upd: Also happened on amm_arb (arbitrum_one)

logs_ammarbit.log
logs_gateway_app.log
ammarbit - Copy.txt

2023-07-20 06:36:30,004 - 444600 - hummingbot.core.utils.async_utils - ERROR - Unhandled error in background task: 'WETH'
Traceback (most recent call last):
  File "/home/nikita/staging1/hummingbot/core/utils/async_utils.py", line 9, in safe_wrapper
    return await c
  File "/home/nikita/staging1/hummingbot/strategy/amm_arb/amm_arb.py", line 234, in main
    await self.apply_slippage_buffers(profitable_arb_proposals)
  File "/home/nikita/staging1/hummingbot/strategy/amm_arb/amm_arb.py", line 260, in apply_slippage_buffers
    arb_side.amount = market.quantize_order_amount(arb_side.market_info.trading_pair, arb_side.amount)
  File "hummingbot/connector/connector_base.pyx", line 481, in hummingbot.connector.connector_base.ConnectorBase.quantize_order_amount
    return self.c_quantize_order_amount(trading_pair, amount)
  File "hummingbot/connector/connector_base.pyx", line 474, in hummingbot.connector.connector_base.ConnectorBase.c_quantize_order_amount
    order_size_quantum = self.c_get_order_size_quantum(trading_pair, amount)
  File "hummingbot/connector/connector_base.pyx", line 453, in hummingbot.connector.connector_base.ConnectorBase.c_get_order_size_quantum
    return self.get_order_size_quantum(trading_pair, order_size)
  File "/home/nikita/staging1/hummingbot/connector/gateway/amm/gateway_evm_amm.py", line 838, in get_order_size_quantum
    return max(self._amount_quantum_dict[base], self._amount_quantum_dict[quote])
KeyError: 'WETH'
2023-07-20 06:36:31,003 - 444600 - hummingbot.core.utils.async_utils - ERROR - Unhandled error in background task: 'WETH'
Traceback (most recent call last):
  File "/home/nikita/staging1/hummingbot/core/utils/async_utils.py", line 9, in safe_wrapper
    return await c
  File "/home/nikita/staging1/hummingbot/strategy/amm_arb/amm_arb.py", line 234, in main
    await self.apply_slippage_buffers(profitable_arb_proposals)
  File "/home/nikita/staging1/hummingbot/strategy/amm_arb/amm_arb.py", line 260, in apply_slippage_buffers
    arb_side.amount = market.quantize_order_amount(arb_side.market_info.trading_pair, arb_side.amount)
  File "hummingbot/connector/connector_base.pyx", line 481, in hummingbot.connector.connector_base.ConnectorBase.quantize_order_amount
    return self.c_quantize_order_amount(trading_pair, amount)
  File "hummingbot/connector/connector_base.pyx", line 474, in hummingbot.connector.connector_base.ConnectorBase.c_quantize_order_amount
    order_size_quantum = self.c_get_order_size_quantum(trading_pair, amount)
  File "hummingbot/connector/connector_base.pyx", line 453, in hummingbot.connector.connector_base.ConnectorBase.c_get_order_size_quantum
    return self.get_order_size_quantum(trading_pair, order_size)
  File "/home/nikita/staging1/hummingbot/connector/gateway/amm/gateway_evm_amm.py", line 838, in get_order_size_quantum
    return max(self._amount_quantum_dict[base], self._amount_quantum_dict[quote])
KeyError: 'WETH'
2023-07-20 06:36:32,003 - 444600 - hummingbot.core.utils.async_utils - ERROR - Unhandled error in background task: 'WETH'
Traceback (most recent call last):
  File "/home/nikita/staging1/hummingbot/core/utils/async_utils.py", line 9, in safe_wrapper
    return await c
  File "/home/nikita/staging1/hummingbot/strategy/amm_arb/amm_arb.py", line 234, in main
    await self.apply_slippage_buffers(profitable_arb_proposals)
  File "/home/nikita/staging1/hummingbot/strategy/amm_arb/amm_arb.py", line 260, in apply_slippage_buffers
    arb_side.amount = market.quantize_order_amount(arb_side.market_info.trading_pair, arb_side.amount)
  File "hummingbot/connector/connector_base.pyx", line 481, in hummingbot.connector.connector_base.ConnectorBase.quantize_order_amount
    return self.c_quantize_order_amount(trading_pair, amount)
  File "hummingbot/connector/connector_base.pyx", line 474, in hummingbot.connector.connector_base.ConnectorBase.c_quantize_order_amount
    order_size_quantum = self.c_get_order_size_quantum(trading_pair, amount)
  File "hummingbot/connector/connector_base.pyx", line 453, in hummingbot.connector.connector_base.ConnectorBase.c_get_order_size_quantum
    return self.get_order_size_quantum(trading_pair, order_size)
  File "/home/nikita/staging1/hummingbot/connector/gateway/amm/gateway_evm_amm.py", line 838, in get_order_size_quantum
    return max(self._amount_quantum_dict[base], self._amount_quantum_dict[quote])
KeyError: 'WETH'

Before

after restart:

rapcmia · 2023-07-20T11:18:26Z

Observed that this error would occur too when suddenly gateway connection has been stop and start again.

Setup simple AMM arb then start the strategy
Close the gateway connection by either stopping it or exit terminal
After successfully terminated the gateway connection, start it again until Client display gateway service is online
Start the AMM arb strategy again, observe behavior


2023-07-20 18:55:58,679 - 1207 - GatewayEVMAMM - INFO - Network status has changed to NetworkStatus.NOT_CONNECTED. Stopping networking...
2023-07-20 18:56:00,004 - 1207 - hummingbot.core.gateway.gateway_http_client - WARNING - Call to https://localhost:15888/amm/price failed. See logs for more details.
2023-07-20 18:56:00,004 - 1207 - GatewayEVMAMM - WARNING - Cannot connect to host localhost:15888 ssl:default [Connect call failed ('127.0.0.1', 15888)]
2023-07-20 18:56:00,004 - 1207 - hummingbot.core.gateway.gateway_http_client - WARNING - Call to https://localhost:15888/amm/price failed. See logs for more details.
2023-07-20 18:56:00,004 - 1207 - GatewayEVMAMM - WARNING - Cannot connect to host localhost:15888 ssl:default [Connect call failed ('127.0.0.1', 15888)]
2023-07-20 18:56:00,004 - 1207 - hummingbot.core.gateway.gateway_http_client - WARNING - Call to https://localhost:15888/amm/price failed. See logs for more details.
2023-07-20 18:56:00,004 - 1207 - GatewayEVMAMM - WARNING - Cannot connect to host localhost:15888 ssl:default [Connect call failed ('127.0.0.1', 15888)]
2023-07-20 18:56:00,005 - 1207 - hummingbot.core.gateway.gateway_http_client - WARNING - Call to https://localhost:15888/amm/price failed. See logs for more details.
2023-07-20 18:56:00,005 - 1207 - GatewayEVMAMM - WARNING - Cannot connect to host localhost:15888 ssl:default [Connect call failed ('127.0.0.1', 15888)]
2023-07-20 18:56:06,030 - 1207 - hummingbot.core.gateway.gateway_http_client - WARNING - Call to https://localhost:15888/amm/price failed. See logs for more details.
2023-07-20 18:56:06,031 - 1207 - GatewayEVMAMM - WARNING - Error on POST https://localhost:15888/amm/price Error: {'message': 'Price query failed: Token not supported: WMATIC', 'httpErrorCode': 500, 'errorCode': 1013, 'stack': 'Error: Price query failed: Token not supported: WMATIC\n    at /home/rapcmia/github/gateway/gateway-stg/dist/src/connectors/uniswap/uniswap.controllers.js:74:23\n    at Generator.throw (<anonymous>)\n    at rejected (/home/rapcmia/github/gateway/gateway-stg/dist/src/connectors/uniswap/uniswap.controllers.js:6:65)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)'}
2023-07-20 18:56:06,060 - 1207 - hummingbot.core.gateway.gateway_http_client - WARNING - Call to https://localhost:15888/amm/price failed. See logs for more details.

While this happened, other gateway commands seems to be working ok and able to connect other wallets. However when you try to start a script or strategy for gateway this issue occurs after the prompt yes Our workaround is to relaunch both gateway and client

logs_amm-uniswap-polygon.log

Arnique · 2023-07-26T16:32:35Z

Facing the same issue. Any updates?

nikspz · 2023-07-26T17:04:16Z

Currently for workaround:
If not working on Source (bot Client source + gateway source):
need to ./gateway-setup.sh step again:
https://docs.hummingbot.org/gateway/installation/#run-setup-script

fengtality · 2023-07-27T23:05:04Z

Currently for workaround: If not working on Source (bot Client source + gateway source): need to ./gateway-setup.sh step again: https://docs.hummingbot.org/gateway/installation/#run-setup-script

I did some testing by running a script and strategy, stopping Gateway, and then re-starting. Initially, I had some issues but after I ran ./gateway-setup.sh, I was able to stop and restart Gateway / client successfully for both the AMM Data Feed script and the AMM-ARB strategy.

Users using Gateway from source should run ./gateway-setup.sh after each release since configs for connectors may change.

rapcmia · 2023-07-28T07:01:18Z

Investigate this issue by running a set of user scenarios and this had me reproduce the issue on a 3/5 attempts

Screen.Recording.2023-07-28.at.2.37.27.PM.mov

Setup source client + gateway and make sure gateway-setup.sh else wont go Gateway: Online
Setup a Uniswap-Ethereum-Goerli for this test but should happened to other DEX connectors too ✅
Setup AMM-arb and start the strategy, after few minutes try to disconnect your gateway
After being disconnected, try to reconnect again. On other instances it reconnects ok but sometimes would return amm/price failed

logs_amm-uniswap-goerli.log
logs_gateway_app.log.2023-07-28.log

rapcmia · 2023-08-01T04:02:11Z

Ran tests on different nodeURLs ankr and infura and still issue occurs. On the screen recording below, observed that its running the amm_data_feed_examples and when gateway has disconnected and reconnected the error on price query occurred

Steps to reproduce:

Setup both latest development for source and gateway
Run amm_data_feed_example script
While running for couple of minutes, stop gateway then start again
Observe the issue on log panel

Screen.Recording.2023-08-01.at.11.43.28.AM.mov

Another observation, when the issue occurred we stop/start both gateway and client without restarting and seem to work ✅

Screen.Recording.2023-08-01.at.11.44.25.AM.mov

We have also ran test this by disconnecting from the network for both the script and amm-arbitrage and still works as expected. Still Looking for other possible scenarios

logs_amm_data_feed_example.log
logs_gateway_app.log.2023-08-01.log

nikspz · 2023-08-03T09:22:07Z

Description: Fix gateway connection issue described above
Acceptance criteria: gateway Instance work after restart, QA tests passed, PR including tests, etc
Bounty: P2
Bounty Amount: 50,000 HBOT
Status: OPEN

nikspz · 2023-08-15T00:52:52Z

Reproduced on the latest development
https://www.loom.com/share/68653d8735ab47d6be84d65914d5cd03
However noticed that there's a workaround that you could try to do the Source gateway restart with the delay / Restart the client and gateway with the delay (not start immediately) and it could help to fix the issue and start the script successfully

OjusWiZard · 2023-09-03T08:58:44Z

Heyy @nikspz, I've tried to fix this issue in the above PR. Could you take a look and check if it's eligible for the bounty?
Thanks :)

fix/ issue #164: Race condition in init() of connector classes

carlitogetaladajr · 2023-10-25T02:39:57Z

@OjusWiZard please fill up & submit the AML Policy Form via this link: https://forms.gle/p3zhnSpkP2eZnj9m9
This is a one-time process for each ETH Address used to receive HBOT bounty.

OjusWiZard · 2023-10-25T05:57:10Z

Hey @carlitogetaladajr, I've filled out the form, thanks!

nikspz · 2023-11-07T04:01:46Z

Thanks for your participation. The bounty has been sent to @OjusWiZard . The fix has been deployed on development branch and hummingbot version 1.21.0.

nikspz added the bug Something isn't working label Jul 19, 2023

nikspz changed the title ~~Gateway staging - after restart gateway failed to get access to the token list for amm_data_feed_example.py script~~ Gateway - after restart gateway failed to get access to the token list for amm_data_feed_example.py script Jul 20, 2023

nikspz added the staging label Jul 20, 2023

rapcmia changed the title ~~Gateway - after restart gateway failed to get access to the token list for amm_data_feed_example.py script~~ Gateway - getting price query failed error after restarting gateway connection Jul 20, 2023

nikspz mentioned this issue Jul 20, 2023

sync gateway / gateway staging -> main #166

Merged

2 tasks

nikspz added this to Bounties Board Aug 3, 2023

nikspz moved this to Open in Bounties Board Aug 3, 2023

OjusWiZard mentioned this issue Sep 3, 2023

fix/ issue #164: Race condition in init() of connector classes #197

Merged

2 tasks

nikspz moved this from Open to Assigned in Bounties Board Sep 4, 2023

nikspz assigned OjusWiZard Sep 4, 2023

nikspz added a commit to OjusWiZard/gateway that referenced this issue Sep 4, 2023

Merge branch 'development' into fix/hummingbot#164

1772a60

nikspz moved this from Assigned to Submitted in Bounties Board Sep 5, 2023

nikspz added a commit to OjusWiZard/gateway that referenced this issue Sep 15, 2023

Merge branch 'development' into fix/hummingbot#164

dfeb2cd

nikspz linked a pull request Sep 20, 2023 that will close this issue

fix/ issue #164: Race condition in init() of connector classes #197

Merged

2 tasks

nikspz added a commit to OjusWiZard/gateway that referenced this issue Sep 25, 2023

Merge branch 'development' into fix/hummingbot#164

c3ec250

rapcmia added a commit to OjusWiZard/gateway that referenced this issue Oct 6, 2023

Merge branch 'development' into fix/hummingbot#164

f4c777f

nikspz added a commit that referenced this issue Oct 16, 2023

Merge pull request #197 from OjusWiZard/fix/#164

0ff07f7

fix/ issue #164: Race condition in init() of connector classes

nikspz moved this from Submitted to Merged in Bounties Board Oct 16, 2023

nikspz closed this as completed Nov 7, 2023

github-project-automation bot moved this from Merged to Assigned in Bounties Board Nov 7, 2023

nikspz moved this from Assigned to Paid in Bounties Board Nov 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gateway - getting price query failed error after restarting gateway connection #164

Gateway - getting price query failed error after restarting gateway connection #164

nikspz commented Jul 19, 2023 •

edited

Loading

nikspz commented Jul 19, 2023 •

edited

Loading

rapcmia commented Jul 20, 2023

Arnique commented Jul 26, 2023

nikspz commented Jul 26, 2023 •

edited

Loading

fengtality commented Jul 27, 2023

rapcmia commented Jul 28, 2023

rapcmia commented Aug 1, 2023

nikspz commented Aug 3, 2023

nikspz commented Aug 15, 2023

OjusWiZard commented Sep 3, 2023

carlitogetaladajr commented Oct 25, 2023

OjusWiZard commented Oct 25, 2023

nikspz commented Nov 7, 2023

Gateway - getting price query failed error after restarting gateway connection #164

Gateway - getting price query failed error after restarting gateway connection #164

Comments

nikspz commented Jul 19, 2023 • edited Loading

nikspz commented Jul 19, 2023 • edited Loading

rapcmia commented Jul 20, 2023

Arnique commented Jul 26, 2023

nikspz commented Jul 26, 2023 • edited Loading

fengtality commented Jul 27, 2023

rapcmia commented Jul 28, 2023

rapcmia commented Aug 1, 2023

nikspz commented Aug 3, 2023

nikspz commented Aug 15, 2023

OjusWiZard commented Sep 3, 2023

carlitogetaladajr commented Oct 25, 2023

OjusWiZard commented Oct 25, 2023

nikspz commented Nov 7, 2023

nikspz commented Jul 19, 2023 •

edited

Loading

nikspz commented Jul 19, 2023 •

edited

Loading

nikspz commented Jul 26, 2023 •

edited

Loading