nextest should bail after several SLOW runs

[jsantell]

Currently, SLOW runs on nextest are retried with a longer timeout, with seemingly no termination clause (e.g. after 10 retries), and will run until we manually terminate the job. We should see if we can limit this to a number of explicit attempts.

• Linux: https://github.com/subconsciousnetwork/noosphere/actions/runs/6724302582/job/18276250181?pr=698
• Macos: https://github.com/subconsciousnetwork/noosphere/actions/runs/6712976115/job/18243668361?pr=702

TRY 1 SLOW [> 60.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>120.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>180.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>240.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>300.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>360.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>420.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>480.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>540.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>600.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>660.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>720.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>780.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>840.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>900.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>960.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>1020.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>1080.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>1140.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>1200.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>1260.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>1320.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>1380.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>1440.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>1500.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>1560.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>1620.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>1680.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>1740.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>1800.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>1860.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>1920.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>1980.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>2040.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>2100.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>2160.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>2220.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>2280.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>2340.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>2400.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>2460.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content
  TRY 1 SLOW [>2520.000s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content

[jsantell]

Looks like we can set terminating after X tries for slow tests: https://nexte.st/book/slow-tests.html

[jsantell]

This is consuming a lot of CI resources: https://github.com/subconsciousnetwork/noosphere/actions/runs/7464684364/job/20312188121?pr=773