Stabilize test_streaming.py by scanning the data_dir, not the parent of data_dir

[moodyjon]

Unbundled from #3623

Parent of "data_dir" seems to be "/tmp" and contents are always changing which disturbs test.

Apparently it's not causing a problem due to OS version and behavior of that OS version? It's causing major problems on upgrade to 20.04 for testing.

[moodyjon]

Apparently it's not causing a problem due to OS version and behavior of that OS version? It's causing major problems on upgrade to 20.04 for testing.

Actually, I changed the build section of the workflow to "ubuntu-20.04" attempting to solve a build problem with the "py39_2" branch, but the testing sections are still "ubuntu-latest". So I'm not sure why test_streaming.py is not causing more trouble at head-of-line.

[eukreign]

@moodyjon can you explain the problem a little more? i don't quite understand what this is fixing. what is the value of self.daemon.conf.data_dir before and after calling os.path.dirname() on it?

[moodyjon]

When running tests in the GitHub action, I had some debugging changes in place to print the dir contents. What I saw appeared to the the contents of /tmp. We don't want to be looking in /tmp, we want to be looking in the test-specific /tmp, or perhaps one of the subdirs like <data_dir>/blobfiles. <data_dir>/uploads.

It's quite possible the test is looking in the wrong place even now. Should it be looking under /blobfiles?

On MacOS, its not /tmp, but /var/folders/NN/NNNNNNNNNNNNNNN/T seems to play that role:

...snip...
**** data_dir: /var/folders/46/44w2zhrx16b8gsvff9dxtr640000gq/T/tmpxub8flg6
**** dirname(data_dir): /var/folders/46/44w2zhrx16b8gsvff9dxtr640000gq/T
contents of data_dir: [<DirEntry 'blobfiles'>, <DirEntry 'lbc_regtest'>, <DirEntry 'lbrynet.sqlite'>, <DirEntry 'uploads'>, <DirEntry 'lbrynet.sqlite-shm'>, <DirEntry 'lbrynet.sqlite-wal'>, <DirEntry 'install_id'>, <DirEntry 'db_revision'>, <DirEntry 'wallets'>]
2022-08-01 10:34:19,500 - lbry.blob_exchange.client - WARNING - 1cb51442c8f335009bfef0f2b89ee103186fd3c9401eaa9985adaef34865012d92baa28b610982d1604bd5b00772ba2c not in availability response from 51.81.57.64:5567
2022-08-01 10:34:19,500 - lbry.blob_exchange.client - WARNING - {'blob_data_payment_rate': 'RATE_ACCEPTED', 'error': 'blob not found', 'lbrycrd_address': 'bJxKvpD96kaJLriqVajZ7SaQTsWWyrGQct'}
ok

----------------------------------------------------------------------
Ran 1 test in 29.240s

OK
(lbry-venv-3.9) swdev1@Jonathans-Mac-mini lbry-sdk % ls -l /var/folders/46/44w2zhrx16b8gsvff9dxtr640000gq/T/tmpxub8flg6                                                                 
ls: /var/folders/46/44w2zhrx16b8gsvff9dxtr640000gq/T/tmpxub8flg6: No such file or directory
(lbry-venv-3.9) swdev1@Jonathans-Mac-mini lbry-sdk % ls -l /var/folders/46/44w2zhrx16b8gsvff9dxtr640000gq/T
total 40
drwxr-xr-x   2 swdev1  staff     64 Jul 31 03:35 21E5423E-4194-4959-9E03-564C9CCCF85C
drwx------   2 swdev1  staff     64 Jul 27 04:43 AudioComponentRegistrar
drwxr-xr-x   2 swdev1  staff     64 Aug  1 03:35 B85D1854-0667-4892-962A-B344A7511D74
drwx------   2 swdev1  staff     64 Jul 27 04:43 SandboxHelper
...snip
drwx------@  2 swdev1  staff     64 Jul 27 04:43 com.apple.PressAndHold
drwx------@  2 swdev1  staff     64 Jul 27 04:43 com.apple.Preview
drwx------@  2 swdev1  staff     64 Jul 27 13:02 com.apple.QuickLookThumbnailing.extension.ThumbnailExtension-macOS
drwx------@  5 swdev1  staff    160 Jul 27 12:46 com.apple.Safari
drwx------@  3 swdev1  staff     96 Jul 27 22:22 com.apple.Safari.CacheDeleteExtension
drwx------   2 swdev1  staff     64 Jul 27 04:43 com.apple.SafariLaunchAgent
drwx------   2 swdev1  staff     64 Jul 27 04:43 com.apple.ScopedBookmarkAgent
drwx------@  3 swdev1  staff     96 Jul 27 04:43 com.apple.ScreenTimeAgent
drwx------@  2 swdev1  staff     64 Jul 27 04:43 com.apple.ScreenTimeWidgetApplication.ScreenTimeWidgetExtension
drwx------@  2 swdev1  staff     64 Jul 27 22:22 com.apple.TV.TVCacheExtension
drwx------   3 swdev1  staff     96 Jul 27 04:43 com.apple.TelephonyUtilities
drwx------@  3 swdev1  staff     96 Jul 27 04:44 com.apple.TestFlight.ServiceExtension
drwx------@  2 swdev1  staff     64 Jul 27 04:43 com.apple.TextEdit
drwx------@  2 swdev1  staff     64 Jul 27 04:44 com.apple.UsageTrackingAgent
drwx------   2 swdev1  staff     64 Jul 27 04:43 com.apple.accessibility.mediaaccessibilityd
drwx------   2 swdev1  staff     64 Jul 27 04:43 com.apple.akd
drwx------   2 swdev1  staff     64 Jul 27 04:43 com.apple.amp.mediasharingd
drwx------   2 swdev1  staff     64 Jul 27 04:43 com.apple.ap.adprivacyd
drwx------   2 swdev1  staff     64 Jul 27 04:44 com.apple.ap.promotedcontentd
drwx------   2 swdev1  staff     64 Jul 28 04:43 com.apple.appleaccountd
drwx------   2 swdev1  staff     64 Jul 27 04:44 com.apple.appstoreagent
drwx------   2 swdev1  staff     64 Jul 27 04:43 com.apple.avconferenced
...snip
drwx------   2 swdev1  staff     64 Jul 27 07:43 com.apple.mapspushd
drwx------@  3 swdev1  staff     96 Jul 27 05:43 com.apple.mediaanalysisd
drwx------@  2 swdev1  staff     64 Jul 27 04:47 com.apple.mobiletimer.WorldClockWidget
drwx------@ 20 swdev1  staff    640 Aug  1 10:11 com.apple.news.widget
drwx------@  3 swdev1  staff     96 Jul 27 04:43 com.apple.notificationcenterui
drwx------   3 swdev1  staff     96 Jul 27 04:43 com.apple.nsurlsessiond
drwx------   3 swdev1  staff     96 Jul 27 04:44 com.apple.parsec-fbf
drwx------@  3 swdev1  staff     96 Aug  1 10:11 com.apple.parsecd
drwx------   3 swdev1  staff     96 Jul 27 04:43 com.apple.passd
drwx------@  4 swdev1  staff    128 Jul 31 18:45 com.apple.photoanalysisd
drwx------@  4 swdev1  staff    128 Jul 27 05:43 com.apple.photolibraryd
drwx------   3 swdev1  staff     96 Jul 27 04:44 com.apple.pluginkit
drwx------   2 swdev1  staff     64 Jul 27 04:43 com.apple.printtool.agent
drwx------   2 swdev1  staff     64 Jul 27 05:13 com.apple.proactiveeventtrackerd
drwx------@  2 swdev1  staff     64 Jul 27 04:43 com.apple.quicklook.QuickLookUIService
drwx------   3 swdev1  staff     96 Jul 27 13:02 com.apple.quicklook.ThumbnailsAgent
drwx------   2 swdev1  staff     64 Jul 27 13:02 com.apple.quicklook.satellite.general
...snip
drwx------   3 swdev1  staff     96 Jul 29 08:54 go-build2075770330
drwx------   3 swdev1  staff     96 Jul 29 10:33 go-build2228263840
drwx------   3 swdev1  staff     96 Jul 29 10:32 go-build2256248549
drwx------   3 swdev1  staff     96 Jul 29 16:34 go-build2509153884
drwx------   3 swdev1  staff     96 Jul 29 10:32 go-build2738864756
drwx------   3 swdev1  staff     96 Jul 29 11:40 go-build3341503245
drwx------   2 swdev1  staff     64 Jul 29 15:50 go-build3923829984
drwx------   3 swdev1  staff     96 Jul 29 10:33 go-build4064678972
drwx------   3 swdev1  staff     96 Jul 29 10:32 go-build4080194559
drwx------   3 swdev1  staff     96 Aug  1 10:15 go-lbry-hub1240325475
drwx------   2 swdev1  staff     64 Aug  1 03:35 go-lbry-hub1447330241
drwx------   3 swdev1  staff     96 Jul 29 11:32 go-lbry-hub1640616760
drwx------   2 swdev1  staff     64 Aug  1 03:35 go-lbry-hub1643076011
drwx------   3 swdev1  staff     96 Jul 29 10:42 go-lbry-hub2121822017
drwx------   2 swdev1  staff     64 Aug  1 03:35 go-lbry-hub2185613297
drwx------   3 swdev1  staff     96 Jul 29 11:24 go-lbry-hub2419531561
...snip...
drwx------   2 swdev1  staff     64 Jul 27 06:33 metrickitd
drwx------   2 swdev1  staff     64 Jul 27 22:31 proactived
drwx------   2 swdev1  staff     64 Jul 29 16:00 pyright-14419-7WloCggLNmAV
drwx------   2 swdev1  staff     64 Jul 27 12:47 pyright-14712-0bTh8FTajx2C
drwx------   2 swdev1  staff     64 Jul 27 13:49 pyright-17423-Rr9Bx4EFwpf2
drwx------   2 swdev1  staff     64 Jul 27 13:50 pyright-17473-HGcHjJgjLPpQ
drwx------   2 swdev1  staff     64 Jul 27 13:58 pyright-18069-qyv1QtLLEJ9t
drwx------   2 swdev1  staff     64 Jul 27 04:43 pyright-2017-BGjdOgtBv98b
drwx------   2 swdev1  staff     64 Jul 27 16:33 pyright-24763-91WARyHdeoLM
drwx------   2 swdev1  staff     64 Jul 29 11:38 pyright-3572-1nvnCpyDn7Kv
drwx------   2 swdev1  staff     64 Jul 30 17:56 pyright-44147-Wt3gSD8Q5uww
drwx------   4 swdev1  staff    128 Aug  1 10:21 pyright-44147-vm0IYVinhaAM
drwx------   2 swdev1  staff     64 Jul 28 09:00 pyright-48021-7aUEDpaa44RN
drwx------   2 swdev1  staff     64 Jul 28 09:12 pyright-48367-STaTlFWk9UTR
drwx------   2 swdev1  staff     64 Jul 28 09:14 pyright-48563-BfDJIqIAU2VO
drwx------   2 swdev1  staff     64 Jul 28 09:16 pyright-48744-UFJBMlNrm7Y7
drwx------   2 swdev1  staff     64 Jul 28 09:27 pyright-49119-yGt04ajGIVM6
drwx------   2 swdev1  staff     64 Jul 28 09:37 pyright-51651-NnhBMuL3worV
...snip...
drwx------   2 swdev1  staff     64 Jul 29 14:53 tmps7hp0t_m
drwx------   4 swdev1  staff    128 Jul 29 16:17 tmpsigluuzs
drwx------   2 swdev1  staff     64 Jul 29 14:53 tmpsvphv0z9
drwx------   2 swdev1  staff     64 Jul 29 14:53 tmpte9f2ct4
drwx------   2 swdev1  staff     64 Jul 29 14:53 tmptn3xfe7r
drwx------   2 swdev1  staff     64 Jul 29 14:53 tmpv2rtbni1
drwx------   2 swdev1  staff     64 Jul 29 14:53 tmpxfv22nis
srwxr-xr-x   1 swdev1  staff      0 Jul 28 14:17 vscode-git-23dfe39b92.sock
srwxr-xr-x   1 swdev1  staff      0 Jul 27 04:43 vscode-git-261a0124a4.sock
srwxr-xr-x   1 swdev1  staff      0 Jul 29 16:34 vscode-git-94fec1e759.sock
srwxr-xr-x   1 swdev1  staff      0 Jul 29 17:26 vscode-git-a0b7dadb3c.sock
srwxr-xr-x   1 swdev1  staff      0 Jul 28 09:40 vscode-git-dc22470860.sock
srwxr-xr-x   1 swdev1  staff      0 Jul 30 17:56 vscode-git-fa7cb3d627.sock
drwxr-xr-x   3 swdev1  staff     96 Jul 30 17:56 vscode-typescript503
-rw-------   1 swdev1  staff    619 Jul 28 09:33 xcrun_db

[moodyjon]

Some of the other tests take a different approach. They start with the full_path of the stream, then scan the contents of dirname(full_path).

(lbry-venv-3.9) swdev1@Jonathans-Mac-mini lbry-sdk % env ENABLE_LEGACY_SEARCH=0 python -m unittest -vv -k test_stream_and_save_file_with_blobs tests/integration/datanetwork/test_streaming.py
test_stream_and_save_file_with_blobs (tests.integration.datanetwork.test_streaming.RangeRequests) ... 2022-08-01 10:47:23,665 - asyncio - WARNING - Executing <Task pending name='Task-1' coro=<CommandTestCase.asyncSetUp() running at /Users/swdev1/lbry-sdk/lbry/testcase.py:407> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/futures.py:384, <TaskWakeupMethWrapper object at 0x107168d90>()] created at /opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py:429> cb=[_run_until_complete_cb() at /opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py:184] created at /opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py:626> took 1.833 seconds
2022-08-01 10:47:28,024 - elasticsearch - WARNING - GET http://localhost:9200/_template/97030c9e644749a3a9c97d048b5a3bf8claims [status:404 request:0.030s]
/Users/swdev1/hub/hub/herald/session.py:595: DeprecationWarning: The explicit passing of coroutine objects to asyncio.wait() is deprecated since Python 3.8, and scheduled for removal in Python 3.11.
  await asyncio.wait([
/opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/events.py:80: DeprecationWarning: The explicit passing of coroutine objects to asyncio.wait() is deprecated since Python 3.8, and scheduled for removal in Python 3.11.
  self._context.run(self._callback, *self._args)
2022-08-01 10:47:34,362 - asyncio - WARNING - Executing <Task pending name='Task-710' coro=<RangeRequests.test_stream_and_save_file_with_blobs() running at /Users/swdev1/lbry-sdk/tests/integration/datanetwork/test_streaming.py:217> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/futures.py:384, <TaskWakeupMethWrapper object at 0x107d4bee0>()] created at /opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py:429> cb=[_run_until_complete_cb() at /opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py:184] created at /opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py:626> took 0.568 seconds
**** full_path: /var/folders/46/44w2zhrx16b8gsvff9dxtr640000gq/T/tmprer9xl9a/tmp5c_5ng__
**** dirname(full_path): /var/folders/46/44w2zhrx16b8gsvff9dxtr640000gq/T/tmprer9xl9a
contents of dirname(full_path): [<DirEntry 'blobfiles'>, <DirEntry 'tmp5c_5ng__'>, <DirEntry 'lbc_regtest'>, <DirEntry 'lbrynet.sqlite'>, <DirEntry 'uploads'>, <DirEntry 'lbrynet.sqlite-shm'>, <DirEntry 'lbrynet.sqlite-wal'>, <DirEntry 'install_id'>, <DirEntry 'db_revision'>, <DirEntry 'wallets'>]
ok

----------------------------------------------------------------------
Ran 1 test in 28.464s

OK

Looks to me as if self.daemon.conf.data_dir is the right one to scan for changing contents.

[eukreign]

@moodyjon okay, that makes sense now. please rebase and I'll merge.