Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Makefile: Change NO_VALGRIND to VALGRIND for pytest environment #4

Closed
wants to merge 40 commits into from
Closed

Makefile: Change NO_VALGRIND to VALGRIND for pytest environment #4

wants to merge 40 commits into from

Commits on May 31, 2018

  1. gossipd: don't publish private updates after channel_announce.

    We generate new ones anyway; removing this code changes fixes coming
    up which now only need to change one place.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed May 31, 2018
    Configuration menu
    Copy the full SHA
    a50a2cd View commit details
    Browse the repository at this point in the history
  2. tests: new helper to get pid of a particular subdaemon.

    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed May 31, 2018
    Configuration menu
    Copy the full SHA
    2e558f4 View commit details
    Browse the repository at this point in the history
  3. tests/test_lightningd.py: test new delayed gossip channel_announce be…

    …haviour.
    
    The gossip-query spec enhancements say not to forward an channel_announcement until
    you have receive a channel_update.  This test fails for now.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed May 31, 2018
    Configuration menu
    Copy the full SHA
    95efac4 View commit details
    Browse the repository at this point in the history
  4. gossipd: only broadcast channel_announcement once we have a channel_u…

    …pdate.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed May 31, 2018
    Configuration menu
    Copy the full SHA
    1bb6561 View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2018

  1. gossipd: announce own node only after channel announcement actually b…

    …roadcast.
    
    handle_pending_cannouncement might not actually add the announcment,
    as it could be waiting for a channel_update.  We need to wait for
    the actual announcement before considering announcing our node.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    39da5ea View commit details
    Browse the repository at this point in the history
  2. gossipd: announce nodes after channel announcement.

    In general, we need to only publish node announcements after
    publishing channel announcements, though we can accept node
    announcements as soon as we see channel announcements.  So we keep a
    flag for those node_announcement which haven't been broadcast yet.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    8890c0e View commit details
    Browse the repository at this point in the history
  3. features: define LOCAL_GOSSIP_QUERIES feature.

    From BOLT #9 proposed update.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    faa7145 View commit details
    Browse the repository at this point in the history
  4. wire: Update to lastest BOLT draft.

    This includes the gossip query messages.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    900961c View commit details
    Browse the repository at this point in the history
  5. gossipd: infrastructure to handle short_channel_id replies.

    We use the same system as for gossip: we trickle out replies when we're
    otherwise idle.
    
    This is minimal infrastructure: we don't actually process the
    query_short_channel_ids message yet, nor do we append node
    announcements.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    2302525 View commit details
    Browse the repository at this point in the history
  6. gossipd: send node announcements after short_channel_id replies.

    We use the same system as for gossip: we trickle out replies when we're
    otherwise idle.
    
    As we trickle out replies to query_short_channel_ids, we remember the
    pubkeys of nodes we mention.  At the end, we sort and uniquify, and
    then send any node_announcements we have for those.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    9ad02de View commit details
    Browse the repository at this point in the history
  7. gossipd: handle query_short_channel_ids message.

    This doesn't handle zlib yet.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    3683591 View commit details
    Browse the repository at this point in the history
  8. gossipd: wire up infrastructure to generate query_short_channel_ids msg.

    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    94581c6 View commit details
    Browse the repository at this point in the history
  9. tests: add dev-query-scids

    And write the test for it.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    65303a8 View commit details
    Browse the repository at this point in the history
  10. gossipd: add timestamp to each broadcast message.

    This lets us filter by timestamp.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    37b30a4 View commit details
    Browse the repository at this point in the history
  11. gossipd: ensure incoming timestamps are reasonable.

    This is kind of orthogonal to the other changes, but makes sense: if we
    would instantly or never prune the message, don't accept it.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    05b9e4f View commit details
    Browse the repository at this point in the history
  12. gossipd: handle gossip_timestamp_filter message.

    And initialize filter (to "never") when we negotiated LOCAL_GOSSIP_QUERIES,
    and send initial filter message.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    aefca01 View commit details
    Browse the repository at this point in the history
  13. gossipd: single-thread the gossip timer.

    We have a function called 'wake_pkt_out' which is really 'start
    gossiping', so rename it to 'wake_gossip_out'.
    
    In addition, it's fired both on a timer, and in response to our first
    gossip_timestamp_filter, which leads to very confusing (though,
    technically, not incorrect) behavior.
    
    Keep a single timer at all times, which now doubles as the flag to
    indicating we're syncing right now.  Set it once we're done syncing
    gossip.
    
    Technically this means we got from once-every-60-seconds to
    quiet-for-60-seconds-between-gossip, but that's OK.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    ebec857 View commit details
    Browse the repository at this point in the history
  14. gossipd: add dev-send-timestamp-filter command for testing timestamp …

    …filtering.
    
    Since we currently only (ab)use it to send everything, we need a way to
    generate boutique queries for testing.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    cae5351 View commit details
    Browse the repository at this point in the history
  15. gossipd: handle query_channel_range

    We send them all the short_channel_ids we have in a given range.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    6140421 View commit details
    Browse the repository at this point in the history
  16. gossip: dev-query-channel-range to test query_channel_range.

    We keep a crappy bitmap, and finish when their replies cover
    everything we asked.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    7e8dd23 View commit details
    Browse the repository at this point in the history
  17. gossipd: dev-set-max-scids-encode-size to artificially force "full" r…

    …eplies.
    
    We cap each reply at a single one, which forces the code into our
    recursion logic.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    f50d682 View commit details
    Browse the repository at this point in the history
  18. zlib: add as a requirement.

    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    3857e23 View commit details
    Browse the repository at this point in the history
  19. gossipd: support and use zlib encoding in short_channel_id encoding.

    We still use uncompressed if zlib turns out to be larger.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    de42538 View commit details
    Browse the repository at this point in the history
  20. test_lightningd.py: actually make sure we do zlib encoding.

    Previous replies weren't large enough; add another channel and then
    use IO tracing to make sure the reply is zlib encoded.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    d1b1fad View commit details
    Browse the repository at this point in the history
  21. gossipd: don't queue node_announce unless we've queued channel_announce.

    We *accept* a node_announce if we have a channel_announce, but we
    can't queue it until we queue the channel_announce, which we only do
    once we have recieved a channel_update.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    aa4f310 View commit details
    Browse the repository at this point in the history
  22. utils: make subd_pid return the *last* pid, in case we restarted daemon.

    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    002f2a2 View commit details
    Browse the repository at this point in the history
  23. Rename consider_own_node_announce to maybe_send_own_node_announce.

    Suggested-by: @cdecker
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    a4f1b50 View commit details
    Browse the repository at this point in the history
  24. pytest: test_gossip_jsonrpc can always test aliases.

    The whole test is under DEVELOPER anyway, but even if it weren't it
    would work fine.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    8ce1587 View commit details
    Browse the repository at this point in the history
  25. gossipd: minor cleanups.

    Suggested-by: @cdecker
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    05ec4b7 View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2018

  1. external: fix submodule handling for parallel builds, submodule URL c…

    …hanges.
    
    If we change an upstream URL, all submodules break.  Users would need
    to run 'git submodule sync'.  Note that the libbacktrace fix was merged
    upstream so this is no longer necessary, but it's good for future changes.
    
    Also, stress-testing reveals that git submodule fails locking
    '.git/config' when run in paralell.  It also segfaults and other
    problems.
    
    This is my final attempt to fix submodules; I've wasted far too many
    days on obscure problems it creates: I've already lost one copy of my
    repo to apparently unfixable submodule preoblems.  The next "fix" will
    be to simply import the source code so it works properly.
    
    Reported-by: @jsarenik
    Fixes: ElementsProject#1543
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 6, 2018
    1 Configuration menu
    Copy the full SHA
    5d77c6a View commit details
    Browse the repository at this point in the history
  2. ccan: update to have new configurator.

    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    3d7c352 View commit details
    Browse the repository at this point in the history
  3. configure: new script.

    You can use environment variables or the commandline to set defaults.
    
    It looks very autoconf, but you don't need to learn m4.
    
    Doesn't cover all the obscure flags, but it's easy to extend.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    e0dae59 View commit details
    Browse the repository at this point in the history
  4. configure: add --enable-valgrind/disable-valgrind.

    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    332b120 View commit details
    Browse the repository at this point in the history
  5. Makefile: import config.vars.

    We leave VALGRIND env var as an override for testing.
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    2966044 View commit details
    Browse the repository at this point in the history
  6. Makefile: not running configure first is so yesterday:

    Make should only run configure if config.var already exists:
    
        $ make
        ./configure --reconfigure
        ./configure: 65: .: config.vars: not found
        ./configure --reconfigure
        ./configure: 65: .: config.vars: not found
        Makefile:179: recipe for target 'ccan/config.h' failed
        make: *** [ccan/config.h] Error 2
    rustyrussell committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    0ca25a4 View commit details
    Browse the repository at this point in the history
  7. travis: use VALGRIND rather than NO_VALGRIND.

    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    ec68176 View commit details
    Browse the repository at this point in the history
  8. devtools/Makefile: add devtools/onion.c to DEVTOOLS_TOOL_SRC

    This way the object file correctly depends on external headers.  Currently
    a parallel build on a clean tree can give:
    
    ```
    In file included from ./common/sphinx.h:6:0,
                     from devtools/onion.c:5:
    ./bitcoin/pubkey.h:8:10: fatal error: secp256k1.h: No such file or directory
     #include <secp256k1.h>
              ^~~~~~~~~~~~~
    compilation terminated.
    <builtin>: recipe for target 'devtools/onion.o' failed
    ```
    
    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    fe2f384 View commit details
    Browse the repository at this point in the history
  9. libbacktrace: pull latest version, which has config.guess fix applied.

    Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
    rustyrussell committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    f660271 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    05beecf View commit details
    Browse the repository at this point in the history
  11. tests/test_lightningd.py: Negate VALGRIND

    Since the logic has changed from NO_VALGRIND to VALGRIND.
    Jan Sarenik committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    1fc517a View commit details
    Browse the repository at this point in the history