Tests: Python tests/test_lightningd.py

[jsarenik]

Issue and Steps to Reproduce

On Arch Linux when I run make check on f4447ff (where I fixed other check-related error) I get following:

...
test_payment_success_persistence (__main__.LightningDTests) ... Killing l1 in mid HTLC
lightning_channeld: Writing out status 65522: Broken pipe
ok
test_penalty_inhtlc (__main__.LightningDTests)
Test penalty transaction with an incoming HTLC ... Subdaemon lightning_onchaind hit error
Log dumped in crash.log
Fatal signal 6. Log dumped in crash.log
/home/jasan/lightning/tests/utils.py:388: ResourceWarning: unclosed <socket.socket fd=4, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
  pass
ok
test_penalty_outhtlc (__main__.LightningDTests)
Test penalty transaction with an outgoing HTLC ... skipped 'flaky test causing CI fails too often'
test_permfail (__main__.LightningDTests) ... ERROR

======================================================================
ERROR: test_permfail (__main__.LightningDTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_lightningd.py", line 1242, in test_permfail
    wait_for(lambda: l2.rpc.listpeers(l1.info['id'])['peers'][0]['channels'][0]['status'] == ['ONCHAIN:Tracking our own unilateral close', 'ONCHAIN:All outputs resolved: waiting 5 more blocks before forgetting channel'], timeout=1)
  File "tests/test_lightningd.py", line 68, in wait_for
    raise ValueError("Error waiting for {}", success)
ValueError: ('Error waiting for {}', <function LightningDTests.test_permfail.<locals>.<lambda> at 0x7fe6ed7ebae8>)

----------------------------------------------------------------------
Ran 56 tests in 3021.601s

FAILED (errors=1, skipped=2)
make: *** [Makefile:205: pytest] Error 1

[ZmnSCPxj]

All the "waiting for " tests are known to be somewhat flaky (i.e. they randomly fail due to timing problems; usually it happens if the event you are waiting for has already occurred by the time the waiting Python code executes); you can try rerunning the test again. If a few tries it still fails, we can take a look at it again.

[jsarenik]

Thanks for info. I will retry and update.

…

On Feb 28, 2018 6:57 AM, "ZmnSCPxj, ZmnSCPxj jxPCSmnZ" < ***@***.***> wrote: All the "waiting for " tests are known to be somewhat flaky (i.e. they randomly fail due to timing problems; usually it happens if the event you are waiting for has already occurred); you can try rerunning the test again. If a few tries it still fails, we can take a look at it again. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1126 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAO7VakP9gkbglS7yRgshg1LbqcVBkqQks5tZOq-gaJpZM4SVdiY> .

[jsarenik]

It is reproducibly failing here for me. Testing on glibc-based Arch Linux. Current HEAD, i.e. dbb7ea1.

$ make pytest
...
test_bitcoin_failure (__main__.LightningDTests) ... ok
test_channel_persistence (__main__.LightningDTests) ... Killing l2 in mid HTLC
lightning_channeld: Writing out status 65522: Broken pipe
valgrind -q --trace-children=yes --trace-children-skip=*bitcoin-cli* --error-exitcode=7 --log-file=/tmp/lightning-6jr0fmoh/test_channel_persistence/lightning-2//valgrind-errors.%p lightningd/lightningd --bitcoin-datadir=/tmp/lightning-6jr0fmoh/test_channel_persistence/bitcoind --lightning-dir=/tmp/lightning-6jr0fmoh/test_channel_persistence/lightning-2/ --port=16332 --allow-deprecated-apis=false --override-fee-rates=15000/7500/1000 --network=regtest --ignore-fee-limits=false --dev-broadcast-interval=1000 --dev-hsm-seed=6c696768746e696e672d32000000000000000000000000000000000000000000 --bitcoind-poll=1s --cltv-delta=6 --cltv-final=5 --locktime-blocks=5 --log-level=debug --dev-fail-on-subdaemon-fail --dev-debugger=channeld
FAIL

======================================================================
FAIL: test_channel_persistence (__main__.LightningDTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_lightningd.py", line 3060, in test_channel_persistence
    assert l1.rpc.listpeers()['peers'][0]['channels'][0]['msatoshi_to_us'] == 99980000
AssertionError

----------------------------------------------------------------------
Ran 6 tests in 183.546s

FAILED (failures=1, skipped=1)
make: *** [Makefile:205: pytest] Error 1

[jsarenik]

By the way I am running the tests with python-bitcoinlib in version 0.7.0 as I found mentioned in @cdecker 's Docker images. Is it safe to test with the current version, 0.9.0?

[cdecker]

That should not be an issue, the test is failing on an assert. Can you run with py.test -vvv tests/test_lightningd.py -k test_channel_persistence. It should also tell you how the assertion failed, i.e., the left and right side of the comparison. Wondering what might have happened there.

[jsarenik]

I managed to successfully run Python tests locally on master (be5d945) branch:

$ make pytest
...
test_update_fee_reconnect (__main__.LightningDTests) ... ok
test_waitanyinvoice (__main__.LightningDTests)
Test various variants of waiting for the next invoice to complete. ... ok
test_waitanyinvoice_reversed (__main__.LightningDTests)
Test waiting for invoices, where they are paid in reverse order ... ok
test_waitinvoice (__main__.LightningDTests)
Test waiting for one invoice will not return if another invoice ... ok
test_withdraw (__main__.LightningDTests) ... ok

----------------------------------------------------------------------
Ran 82 tests in 4547.542s

OK (skipped=3)
rm external/libwally-core/src/secp256k1/libsecp256k1.la

[jsarenik]

@cdecker First I installed all pip3 modules according to the current contrib/Dockerfile.builder. Now this is what I get:

$ py.test -vvv tests/test_lightningd.py -k test_channel_persistence
============================= test session starts ==============================
platform linux -- Python 3.6.4, pytest-3.0.5, py-1.5.2, pluggy-0.4.0 -- /usr/bin/python
cachedir: .cache
rootdir: /home/jasan/lightning, inifile:
plugins: test-groups-1.0.3, rerunfailures-3.1
collected 0 items / 1 errors

==================================== ERRORS ====================================
__________________ ERROR collecting tests/test_lightningd.py ___________________
ImportError while importing test module '/home/jasan/lightning/tests/test_lightningd.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.6/site-packages/_pytest/python.py:418: in _importtestmodule
    mod = self.fspath.pyimport(ensuresyspath=importmode)
/usr/lib/python3.6/site-packages/py/_path/local.py:668: in pyimport
    __import__(modname)
/usr/lib/python3.6/site-packages/_pytest/assertion/rewrite.py:216: in load_module
    py.builtin.exec_(co, mod.__dict__)
tests/test_lightningd.py:24: in <module>
    from lightning import LightningRpc
E   ModuleNotFoundError: No module named 'lightning'
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
=========================== 1 error in 0.36 seconds ============================

[jsarenik]

That seems to be just PYTHONPATH issue. I have installed global package (ran `pip3 install pylightning) and this is what I get now:

$ py.test -vvv tests/test_lightningd.py -k test_channel_persistence
============================= test session starts ==============================
platform linux -- Python 3.6.4, pytest-3.0.5, py-1.5.2, pluggy-0.4.0 -- /usr/bin/python
cachedir: .cache
rootdir: /home/jasan/lightning, inifile:
plugins: test-groups-1.0.3, rerunfailures-3.1
collected 82 items

tests/test_lightningd.py::LightningDTests::test_channel_persistence SKIPPED

============================= 81 tests deselected ==============================
=================== 1 skipped, 81 deselected in 0.07 seconds ===================

[jsarenik]

Just a note about running the separated test without global pylightning package installed:

export PYTHONPATH=$PWD/contrib/pylightning
py.test -vvv tests/test_lightningd.py -k test_channel_persistence

Important thing is to have the Python3 modules listed in contrib/Dockerfile.builder:

pip3 install python-bitcoinlib==0.7.0 pytest==3.0.5 \
  setuptools==36.6.0 pytest-test-groups==1.0.3 \
  flake8==3.5.0 pytest-rerunfailures==3.1

[jsarenik]

Closing this issue. Everything works now. Both on Arch Linux (glibc-based) and Alpine Linux (musl-based, only without valgrind).

[cdecker]

Thanks @jsarenik for persevering through the issues, glad you got it to work.

We could probably add pylightning==0.0.3 as a pip3 requirement, but the PYTHONPATH trick allows us to move forward in lock-step with any changes in the framework.

[jsarenik]

Just for reference, as this is not yet in documentation: #1538