add PYTHONWARNINGS=ignore::DeprecationWarning to integration tests

[cosmicexplorer]

Problem

Fixes #8539.

Solution

• Ignore deprecation warning messages on all integration tests.

Result

master is unbroken!

[stuhood]

This will unbreak master, but won't fix the issue that these warnings will render for users. Is it possible to fix them instead?

[Eric-Arellano]

This will unbreak master, but won't fix the issue that these warnings will render for users. Is it possible to fix them instead?

I recommend reverting the deprecation of datatype() and waiting to first replace collection.of() w/ a MyPy compatible implementation. This would remove the last use of datatype() in our codebase. Then, we can drop datatype() in one PR rather than needing to wait for a full deprecation cycle, per the policy we established with enum() and collection.of() that these are experimental APIs.

[cosmicexplorer]

This will unbreak master, but won't fix the issue that these warnings will render for users. Is it possible to fix them instead?

The larger issue would seem to me to be that seeing any deprecation warnings at all will cause our CI to break. Do we want to consider looking into fixing that before focusing on finishing up a migration process?

[blorente]

I agree that most of the deprecation warning will break CI and that's bad, but I wonder how they got past CI in the first place. The solutions seems to be to:
1 - Change our integration tests so that they don't rely on parsing stderr
2 - Fix this particular instance of deprecation warnings, either by fixing the uses of the deprecated datatype(), or reverting the deprecation. In this case, we'd be asserting that we will basically not allow deprecation cycles, because they will not pass CI in most cases.

I would really love us to go with option 1, but I don't know how feasible it is. In the interest of unblocking merging, I think either reverting or fixing datatype(), or the change in this PR is good.

Also, these deprecation warnings are for pants developers, right? Would it make sense to only output them in debug mode, or in when running pants from source?

[hrfuller]

@Eric-Arellano I am working on a fix for Collection.of today if that makes any difference here, not sure on timing exactly but hopefully by EOD.

[stuhood]

If this looks green, then fine with landing it.

I'll follow up with a thread to discuss deprecations and removing internal usages.

[blorente]

Restated the failing shards (after checking they were known flakes)

[stuhood]

It looks like this breaks a different integration test:

=================================== FAILURES ===================================

__________________ RunnerIntegrationTest.test_warning_filter ___________________

self = <pants_test.bin.test_runner_integration.RunnerIntegrationTest testMethod=test_warning_filter>

    def test_warning_filter(self):

      # We load the testprojects pants-plugins to get some testing tasks and subsystems.

      cmdline = [

        '--no-enable-pantsd',

        f"--pythonpath=+['{Path(get_buildroot(), 'testprojects/pants-plugins/src/python')}']",

        f"--backend-packages=+['test_pants_plugin']",

        # This task will always emit a DeprecationWarning.

        'deprecation-warning-task',

      ]

    

      warning_run = self.run_pants(cmdline)

      self.assert_success(warning_run)

      self.assertRegex(

        warning_run.stderr_data,

>       '\\[WARN\\].*DeprecationWarning: DEPRECATED: This is a test warning!')

E     AssertionError: Regex didn't match: '\\[WARN\\].*DeprecationWarning: DEPRECATED: This is a test warning!' not found in 'Scrubbed PYTHONPATH=/b/f/w:/b/f/w/pytest-with-requirements.pex:/pyenv-docker-build/versions/3.6.8/lib/python36.zip:/pyenv-docker-build/versions/3.6.8/lib/python3.6:/pyenv-docker-build/versions/3.6.8/lib/python3.6/lib-dynload:/b/f/w/pex_root/install/packaging-16.8-py2.py3-none-any.whl.e40536ef84ff020576b969f64434fa248c714237/packaging-16.8-py2.py3-none-any.whl:/b/f/w/pex_root/install/py-1.8.0-py2.py3-none-any.whl.3402fa58a6c20477b2e9efe86f4f43a119cd771a/py-1.8.0-py2.py3-none-any.whl:/b/f/w/pex_root/install/pycparser-2.19-py2.py3-none-any.whl.53078ed0a121a74ad0e5d5d8534cec1c8f49234e/pycparser-2.19-py2.py3-none-any.whl:/b/f/w/pex_root/install/cryptography-2.8-cp36-cp36m-linux_x86_64.whl.06c33cfecae0ab4203118f9cc408980b88ff1c2b/cryptography-2.8-cp36-cp36m-linux_x86_64.whl:/b/f/w/pex_root/install/traceback2-1.4.0-py2.py3-none-any.whl.5e1797d06c7ad5dc1603d4320126d8c992157962/traceback2-1.4.0-py2.py3-none-any.whl:/b/f/w/pex_root/install/pytest_cov-2.4.0-py2.py3-none-any.whl.18334e5c5b4fe92fac23389f65776f52e363bbd5/pytest_cov-2.4.0-py2.py3-none-any.whl:/b/f/w/pex_root/install/urllib3-1.25.6-py2.py3-none-any.whl.be3132b0776d8d906e57978f1591774617a1d4fb/urllib3-1.25.6-py2.py3-none-any.whl:/b/f/w/pex_root/install/twitter.common.confluence-0.3.11-py3-none-any.whl.db65f7c09e90802248102d3fe380f367d26b066e/twitter.common.confluence-0.3.11-py3-none-any.whl:/b/f/w/pex_root/install/chardet-3.0.4-py2.py3-none-any.whl.c4332e34e38b781695ce775973fe40663558a897/chardet-3.0.4-py2.py3-none-any.whl:/b/f/w/pex_root/install/typing_extensions-3.7.4-py3-none-any.whl.1edcbe8619e77859668b3f056ad84bcfdf2e0fcd/typing_extensions-3.7.4-py3-none-any.whl:/b/f/w/pex_root/install/PyYAML-5.1.2-cp36-cp36m-linux_x86_64.whl.9cf42c8cdcdf7383fe851f85465a864fa2f34348/PyYAML-5.1.2-cp36-cp36m-linux_x86_64.whl:/b/f/w/pex_root/install/asttokens-1.1.13-py2.py3-none-any.whl.00fa8ee883d8fd945420c0f59d79e2b8387d013b/asttokens-1.1.13-py2.py3-none-any.whl:/b/f/w/pex_root/install/pyOpenSSL-17.3.0-py2.py3-none-any.whl.f542e1c53dc88676d30ec1b1da93018ff42cb6a9/pyOpenSSL-17.3.0-py2.py3-none-any.whl:/b/f/w/pex_root/install/docutils-0.14-py3-none-any.whl.199b17a7fbabc943293d072079258c1dd5c68e0a/docutils-0.14-py3-none-any.whl:/b/f/w/pex_root/install/twitter.common.options-0.3.11-py3-none-any.whl.136eca7ec58c4a151b2d5b6a71d631ed1b3658b8/twitter.common.options-0.3.11-py3-none-any.whl:/b/f/w/pex_root/install/Pygments-2.3.1-py2.py3-none-any.whl.ee4bd72946de1ced4595abfa01a778b34f0cad86/Pygments-2.3.1-py2.py3-none-any.whl:/b/f/w/pex_root/install/contextlib2-0.5.5-py2.py3-none-any.whl.edad6efe1aebf29bea2d8a9e8dab93fbbc2555bc/contextlib2-0.5.5-py2.py3-none-any.whl:/b/f/w/pex_root/install/pyparsing-2.4.2-py2.py3-none-any.whl.50391d7edbea580d5d233c392696aef4d9056311/pyparsing-2.4.2-py2.py3-none-any.whl:/b/f/w/pex_root/install/atomicwrites-1.3.0-py2.py3-none-any.whl.2bce967cb4fb97fa3931f45e5b4decdf527bf276/atomicwrites-1.3.0-py2.py3-none-any.whl:/b/f/w/pex_root/install/pex-1.6.12-py2.py3-none-any.whl.d8a368099663ef2d23c79e3403d86115039740d7/pex-1.6.12-py2.py3-none-any.whl:/b/f/w/pex_root/install/cffi-1.12.3-cp36-cp36m-manylinux1_x86_64.whl.73be7de09a609474b1efc2d12432becbbd0f0de4/cffi-1.12.3-cp36-cp36m-manylinux1_x86_64.whl:/b/f/w/pex_root/install/thriftpy2-0.4.7-cp36-cp36m-linux_x86_64.whl.9c6f64a7936384b4b90cbbca1760defa3904a217/thriftpy2-0.4.7-cp36-cp36m-linux_x86_64.whl:/b/f/w/pex_root/install/certifi-2019.9.11-py2.py3-none-any.whl.9f399d4ea01e8af030fdb7731c76799123ef0839/certifi-2019.9.11-py2.py3-none-any.whl:/b/f/w/pex_root/install/pluggy-0.7.1-py2.py3-none-any.whl.742f368de18a3c89563e6d1a269f2b5b7f336d2d/pluggy-0.7.1-py2.py3-none-any.whl:/b/f/w/pex_root/install/www_authenticate-0.9.2-py3-none-any.whl.a7cb2145b3e03bc83c68b6a3bd8f7d26627568b4/www_authenticate-0.9.2-py3-none-any.whl:/b/f/w/pex_root/install/dataclasses-0.6-py3-none-any.whl.b0182a0043d72149b99128f2badc2b43ed38de40/dataclasses-0.6-py3-none-any.whl:/b/f/w/pex_root/install/fasteners-0.14.1-py2.py3-none-any.whl.99944bb9e5cc9cc4f6144bc3ee6173a78219b35a/fasteners-0.14.1-py2.py3-none-any.whl:/b/f/w/pex_root/install/wheel-0.31.1-py2.py3-none-any.whl.9162405b4c6d47ff554018389d29281b6e4854a6/wheel-0.31.1-py2.py3-none-any.whl:/b/f/w/pex_root/install/beautifulsoup4-4.6.3-py3-none-any.whl.f69b3796d967a649e5bef9622259dadb21e19d59/beautifulsoup4-4.6.3-py3-none-any.whl:/b/f/w/pex_root/install/Markdown-2.1.1-py3-none-any.whl.050752ed93b4e830b70076b3e1f28894753cb432/Markdown-2.1.1-py3-none-any.whl:/b/f/w/pex_root/install/twitter.common.lang-0.3.11-py3-none-any.whl.753335ecc81fb9377df5ad090881a9a3cfaf6aa9/twitter.common.lang-0.3.11-py3-none-any.whl:/b/f/w/pex_root/install/unittest2-1.1.0-py2.py3-none-any.whl.825429223a8ef4a056925c48c01a7f7a387cb1fd/unittest2-1.1.0-py2.py3-none-any.whl:/b/f/w/pex_root/install/pytest-3.6.4-py2.py3-none-any.whl.3cf4d2d91a6d84f9b62d5d5633d46f1dea9222bb/pytest-3.6.4-py2.py3-none-any.whl:/b/f/w/pex_root/install/pytest_timeout-1.2.1-py2.py3-none-any.whl.4581b560be35b67389220084292ed6b31c056bc3/pytest_timeout-1.2.1-py2.py3-none-any.whl:/b/f/w/pex_root/install/linecache2-1.0.0-py2.py3-none-any.whl.fd2cf72046aaaf68615409af851242d5c32b321f/linecache2-1.0.0-py2.py3-none-any.whl:/b/f/w/pex_root/install/twitter.common.collections-0.3.11-py3-none-any.whl.b3eeeb6a640fca93f1e271a2dd9a8a81fb47c030/twitter.common.collections-0.3.11-py3-none-any.whl:/b/f/w/pex_root/install/psutil-5.6.3-cp36-cp36m-linux_x86_64.whl.cced6b128e6ce4046f1d21b303db3e744bb27abf/psutil-5.6.3-cp36-cp36m-linux_x86_64.whl:/b/f/w/pex_root/install/pystache-0.5.3-py3-none-any.whl.0a04214ba1f2c3adf1b0141fd1b4fabe4c16109b/pystache-0.5.3-py3-none-any.whl:/b/f/w/pex_root/install/attrs-19.3.0-py2.py3-none-any.whl.01c47aecb2cee3cc26140dc70eb5d5066d174785/attrs-19.3.0-py2.py3-none-any.whl:/b/f/w/pex_root/install/py_zipkin-0.18.4-py2.py3-none-any.whl.cf7543ab85c6b8493274c465aa3e70d8bedd4118/py_zipkin-0.18.4-py2.py3-none-any.whl:/b/f/w/pex_root/install/twitter.common.log-0.3.11-py3-none-any.whl.c920dbc14b8e880aec2bb78850d101855d756952/twitter.common.log-0.3.11-py3-none-any.whl:/b/f/w/pex_root/install/pycodestyle-2.4.0-py2.py3-none-any.whl.0a958eee162effa42dbebab9ec410ad2e5c7fa7b/pycodestyle-2.4.0-py2.py3-none-any.whl:/b/f/w/pex_root/install/python_Levenshtein-0.12.0-cp36-cp36m-linux_x86_64.whl.ad09a3afa4125369de29eca4785858931354af78/python_Levenshtein-0.12.0-cp36-cp36m-linux_x86_64.whl:/b/f/w/pex_root/install/six-1.12.0-py2.py3-none-any.whl.56417fecea1347c6c047607ab9996cc97aec99de/six-1.12.0-py2.py3-none-any.whl:/b/f/w/pex_root/install/ply-3.11-py2.py3-none-any.whl.ced24d23e3ca145d26f6ac130df2de53c3e35188/ply-3.11-py2.py3-none-any.whl:/b/f/w/pex_root/install/setuptools-40.6.3-py2.py3-none-any.whl.4a47c8d55bbea45ef1029b493d822f7c1d55a196/setuptools-40.6.3-py2.py3-none-any.whl:/b/f/w/pex_root/install/twitter.common.dirutil-0.3.11-py3-none-any.whl.27719e6d70e5792a4eaa4b057e7ad31a39bceddd/twitter.common.dirutil-0.3.11-py3-none-any.whl:/b/f/w/pex_root/install/argparse-1.4.0-py2.py3-none-any.whl.dd8addeaf2a7bb1139fae1ac955d19f7c60bd9b2/argparse-1.4.0-py2.py3-none-any.whl:/b/f/w/pex_root/install/pathspec-0.5.9-py3-none-any.whl.8385d9631afea5f89b0b84977cd9d811d214d314/pathspec-0.5.9-py3-none-any.whl:/b/f/w/pex_root/install/more_itertools-7.2.0-py3-none-any.whl.f037f4802a5f30643d86c366cbf14ceb21d39f97/more_itertools-7.2.0-py3-none-any.whl:/b/f/w/pex_root/install/ansicolors-1.0.2-py3-none-any.whl.0088834e21b2c8f3df9b82794f4dca1c75af4f12/ansicolors-1.0.2-py3-none-any.whl:/b/f/w/pex_root/install/requests-2.22.0-py2.py3-none-any.whl.bb3699cc576016729d42f9bf4eccfdbcb292ec50/requests-2.22.0-py2.py3-none-any.whl:/b/f/w/pex_root/install/pywatchman-1.4.1-cp36-cp36m-linux_x86_64.whl.50cf862d6f5a3e154a8a9d916a4455646e459256/pywatchman-1.4.1-cp36-cp36m-linux_x86_64.whl:/b/f/w/pex_root/install/monotonic-1.5-py2.py3-none-any.whl.e6c7e442a796dc3284dafc29d5bee78b3d3b527a/monotonic-1.5-py2.py3-none-any.whl:/b/f/w/pex_root/install/setproctitle-1.1.10-cp36-cp36m-linux_x86_64.whl.5ac4fab07962f139db3d4d79e65f4246c729194c/setproctitle-1.1.10-cp36-cp36m-linux_x86_64.whl:/b/f/w/pex_root/install/coverage-4.5.4-cp36-cp36m-manylinux1_x86_64.whl.53442ad0427aece4db45460c3b9db6b525598f8f/coverage-4.5.4-cp36-cp36m-manylinux1_x86_64.whl:/b/f/w/pex_root/install/pyflakes-2.1.1-py2.py3-none-any.whl.728507c2086c87982c8c1c24ff46b1c2e6c46574/pyflakes-2.1.1-py2.py3-none-any.whl:/b/f/w/pex_root/install/idna-2.8-py2.py3-none-any.whl.645e82ab4157606d15a8766060dd690deebcc4f2/idna-2.8-py2.py3-none-any.whl:/b/f/w/pytest-with-requirements.pex/.bootstrap from the environment.\n\n'

pants_test/bin/test_runner_integration.py:28: AssertionError

[cosmicexplorer]

Fixed the above.

[stuhood]

@Eric-Arellano I am working on a fix for Collection.of today if that makes any difference here, not sure on timing exactly but hopefully by EOD.

#8540 landed to give us some breathing room on this.

@cosmicexplorer : My suggestion would be that we re-land #8509 and #8181 along with the remaining deprecation removals that @hrfuller has mentioned, because whether or not they are silenced in CI, they are triggering for our end users.

[cosmicexplorer]

Closing this as @Eric-Arellano has since removed datatype() entirely, allowing us to avoid the deprecation warnings for now. We can revisit this approach the next time we accidentally set off some warnings.