Degraded performance of the /payments Horizon endpoint

[orbitlens]

What version are you using?

horizon.stellar.org

horizon_version: "1.2.1-ddb2b8590b21b6a5fd5a127e7d95216b4d9819db"
ore_version: "stellar-core 13.0.0 (9ed3da29be1c2c932b946025ca2907646a9072f4)"

What did you do?

/payments endpoint has degraded performance and periodically returns 504 Timeout errors

Example request:
https://horizon.stellar.org/accounts/GAN7MLRQYPDU5VDWGGLPMQUDRCKK2B2YZDJLD3QVRDRBPEMLXHJY42SX/payments?limit=200&order=desc

The same request takes from 3 to 30 seconds to execute, sometimes returning timeout error. I didn't spot any pattern, so maybe it was caused by one of the servers behind the load balancer while others are operating normally.

[bartekn]

I checked Horizon logs from the last 7 days and could find only 49 requests to /accounts/{id}/payments that took more than 2 seconds (81 reqs above 1 second).

Can be a duplicate of #1808? Let's track it there.

[orbitlens]

Ok, I thought it has been fixed already.

Maybe something wrong with LB?

Here is what I got in the browser a few hours ago:

Request URL: https://horizon.stellar.org/accounts/GAN7MLRQYPDU5VDWGGLPMQUDRCKK2B2YZDJLD3QVRDRBPEMLXHJY42SX/payments?c=0.3051838452460005&cursor=125895185187344385&limit=200&order=asc
Status Code: 504 Gateway Timeout

Thanks for looking into this.

[bartekn]

Found your request in the logs. Returned 200 OK twice, in 22 and 45 sec. and ELB idle timeout is set to 60 sec. so it definitely should be a 200 OK for you. Now it gives a response in <1s for me.

Any chance you could inspect response body? Can it be your ISP proxy maybe?

[orbitlens]

No, definitely not ISP. I'm on the gigabit channel. All other resources were loading correctly.
Reproduced it several times.

Here is the dump from Chrome Dev Tools:

1. (loaded, but very slow) https://horizon.stellar.org/accounts/GAN7MLRQYPDU5VDWGGLPMQUDRCKK2B2YZDJLD3QVRDRBPEMLXHJY42SX/payments?order=asc&limit=200&c=0.28252693506078885

Status Code: 200 OK

Request/Response	DURATION
Request sent	​0.10 ms
Waiting (TTFB)	​32.09 s
Content Download	120.85 ms

2. (timeout) https://horizon.stellar.org/accounts/GAN7MLRQYPDU5VDWGGLPMQUDRCKK2B2YZDJLD3QVRDRBPEMLXHJY42SX/payments?c=0.3051838452460005&cursor=125895185187344385&limit=200&order=asc

Status Code: 504 Gateway Timeout
{
  "type": "https://stellar.org/horizon-errors/timeout",
  "title": "Timeout",
  "status": 504,
  "detail": "Your request timed out before completing.  Please try your request again. If you are submitting a transaction make sure you are sending exactly the same transaction (with the same sequence number)."
}

Request/Response	DURATION
Request sent	​0.17 ms
Waiting (TTFB)	​55.15 s
Content Download	​0.80 ms

3. (later on, the same request as 1) https://horizon.stellar.org/accounts/GAN7MLRQYPDU5VDWGGLPMQUDRCKK2B2YZDJLD3QVRDRBPEMLXHJY42SX/payments?order=asc&limit=200&c=0.0856698842787662

Status Code: 200 OK

Request/Response	DURATION
Request sent	0.11 ms
Waiting (TTFB)	710.11 ms
Content Download	120.54 ms

[bartekn]

Duh, I added status:200 filter when checking this 🤦. Yes, I can see 4 x 504. This is likely a duplicate of #1808.