[RFC] Add site.conf option for fastd peer upgrade #1257

CodeFetch · 2017-11-10T11:20:33Z

The site.conf option delete_old_peers removes old peer entries with uci option "net" mesh_vpn
and their corresponding groups recursively.

Shouldn't this resolve #557?

BTW: Sorry for my inability to use Github. I don't like pull requests.

flobeier · 2017-11-10T15:02:38Z

Next time please just update your pull request instead of opening a new one.

christf · 2018-01-19T22:50:10Z

please adjust the commit message to include the module being changed.
gluon-mesh-vpn-fastd:

docs/user/site.rst

christf · 2018-01-19T22:59:55Z

package/gluon-mesh-vpn-fastd/luasrc/lib/gluon/upgrade/400-mesh-vpn-fastd

@@ -92,6 +92,64 @@ function add_groups(prefix, groups, parent)
 	end
 end

+local function match_custom(gp)


instead of defining match_custom using something like
if (string.find(gp, 'custom'))
then
print 'We have a match'
end
could be used for a little more brevity & clarity.

christf · 2018-01-19T23:02:16Z

docs/user/site.rst

@@ -230,6 +235,8 @@ mesh_vpn
          methods = {'salsa2012+umac'},
          -- configurable = true,
          -- syslog_level = 'warn',
+          -- delete_old_peers = false,
+          -- delete_custom_peers = false,


can you elaborate on the use case of delete_custom_peers?

This option is more about being able to retain custom peers. As the default for site values is always false I had to name it this way.
If you update from another community and your community does not support N2N or you want to abandon N2N or never planned on introducing it, you might want to get custom peers deleted also.

what is N2N? i don't know about custom peers and N2N either, so this should be documented with this Pull Request so everyone knows what it is

node to node VPN. I think it is great supporting that however I did not get this from the custom peers and rather would call them n2n_peers but that is just taking nit-picking to an extreme.
There is another PR open for supporting node to node VPN...

CodeFetch · 2018-01-24T05:11:35Z

I changed the title to make clear that I didn't want to get this merged as it is, but to drive forth the fallen asleep topic about handling peers on update.
The peer update mechanism is the only one I know of that has severe problems with cross-community flashing and old config retainment. Thus I thought that something like this could easily solve our problems until we have a configuration management system that fits Gluon better. There might be better ways to achieve this and that's why I ask you „Shouldn't this resolve the problem? Do you know a better way?“.

rotanid · 2018-02-15T14:30:10Z

@CodeFetch so what reactions would drive you to continue working on this to make it merge-ready? :D

CodeFetch · 2018-02-18T15:49:00Z

@rotanid Sorry, I'm very busy at the moment. I'm going to clean this up after initial xlat464 support.

rotanid · 2019-02-15T22:57:14Z

@CodeFetch it's been another year now, whats the progress/plan here?

i'm not sure why people open PRs and don't do anything afterwards - this way opening the PR is wasted time for the opener...

This commit introduces two additional fastd site.conf options: `sysupgrade_remove_old_peers` removes peer and peer group entries existing from an old configuration except those which were defined using the node2node VPN feature after a sysupgrade. `sysupgrade_remove_n2n_peers` removes existing peer and peer group entries which were defined using the node2node VPN feature (which have 'n2n_vpn' in their name) after a sysupgrade.

CodeFetch · 2019-02-24T04:33:59Z

@rotanid Done.

rotanid · 2019-02-25T00:43:34Z

@CodeFetch thanks!
i just noticed you're taking care of a feature here (n2n) which isn't supported by Gluon at the moment. (see #378 )
maybe you have it included in your local communities fork, but for upstream Gluon please remove the part about n2n for the time being.
after(!) doing it, it would be great if you could test your work.

CodeFetch · 2019-02-25T04:51:44Z

@rotanid That's why I didn't want to call it n2n_vpn, but custom. The user may want to keep some "custom" entries whether they are being set by the n2n feature or by hand.

mweinelt · 2019-02-25T11:46:46Z

IMO: If site.conf ships a peer group that should supersede the locally configured peer group of the same name, so just replace that, no knob please.
At the same time just ignore other peer groups to allow for local reconfiguration as was inteded by this feature.

Edit: Obvious gotcha is the missing possibility to delete an unused group, d'oh.

CodeFetch · 2019-02-26T12:07:11Z

@mweinelt This was discussed earlier and NeoRaider did not want to break the ability to incrementally add peers to peer groups.
@NeoRaider What was the reason to allow incremental addition of peers?

@rotanid I'll have a look at the node2node feature the next days. It shouldn't be that hard to get it merged, should it? christf said it is not considered before the babel release. I don't see a real reason for this. I just want to get these annoying 3 year old PRs merged. Having an established fastd node2node solution means I can be certain how the uci structure looks like and may simplify this PR. Thus you can set it blocked.

CodeFetch · 2019-03-02T07:30:17Z

@rotanid Okay I've realized that it was not clear what I want to achieve here and I was confused about it at some point, too.

At the moment there is only a single fastd mesh package called "gluon-mesh-vpn-fastd". This should be more modular. The package "gluon-mesh-vpn-fastd-core" should provide the configuration and upgrade scripts that are common to each flavor.
fastd has a "forward" option which enables forwarding of packets between peers. In a regular setup where a node is connected to one or more supernodes, only supernodes should forward packets between their peers. Thus the fastd configuration of a regular node always disables forwarding and it is not possible to add any custom peers there.
I propose to call this package "gluon-mesh-vpn-fastd-n2s".

The forwarding option of the node2node feature will always disable forwarding, too. The node2node feature should be able to define "custom" peers and ones defined using site.conf (If the node is a regular one it uses the node2node feature to use custom peerings defined by the user. However if it is a supernode it will use the node2node feature to connect with other supernodes and e.g. nameservers which will either be defined in the site.conf or manually). In that case the term "custom" and "n2n" peers cannot be used interchangeably anymore.
I propose to call this package "gluon-mesh-vpn-fastd-n2n".

In the future another package may come up to allow supernode functionality using gluon which will allow forwarding between peers using even another fastd instance providing connectivity to regular nodes. This package will have a configuration option to permit nodes to connect without authorization and will be able to handle site.conf and custom peer entries, too (thus the firmware does not always have to be flashed when a new node's public is being submitted).
I propose to call this package "gluon-mesh-vpn-fastd-s2n".

Thus "custom" peers should not only be a feature of n2n, but be provided by the "gluon-mesh-vpn-fastd-core" package. I'm going to change this pull-request to not rely on the naming of the options to decide whether a peer/peer group is custom, but introduce a boolean option called "custom" to peers/peer groups and fastd instances. If custom is set to true on one level of the configuration hierarchy all children will be considered custom. I'm also going to split the "gluon-mesh-vpn-fastd" package into "gluon-mesh-vpn-fastd-core" and "gluon-mesh-vpn-fastd-n2s".

christf · 2020-02-03T19:15:37Z

Today in review we decided the following:

There will be gluon-specific init-scripts that act on their default from site.conf without requiring a uci config.
in case a configuration exists for the package in question in /etc/config/gluon, this config will be allowed to enrich the content from site - in this case more peer groups can be added
custom peers and peer groups must be marked with a flag "preserve" that defines whether the config item is to be preserved on upgrade. The gluon-mesh-vpn-fastd-core package must be adjusted to reflect that.
there will be no data migration when switching to this scheme as we cannot determine whether a peer/peer group is a leftover or if it was placed intentionally by the user.

neocturne · 2020-02-04T01:09:06Z

@christf's comment sort of mixes up the short-term and long-term solutions I proposed: Short-term we will simply add a flag preserve, long-term we'd like to move additional configuration to /etc/config/gluon, making such flags unnecessary.

CodeFetch · 2020-02-04T07:01:53Z

So the only change is that the "custom" flag should be called "preserve" instead?

neocturne · 2020-02-05T23:21:52Z

There should be no site.conf option - old peers without preserve flag are removed unconditionally
Remove everything that mentions "n2n" (this is not an official Gluon feature)

Basically, the whole implementation can be simplified to:

First, remove all groups and peers with net "mesh_vpn" and without preserve
Then, run the usual setup for new groups and peers
I assume this can be done in 5 lines or fewer...

CodeFetch · 2020-02-06T02:05:41Z

@NeoRaider Maybe I misunderstand you...

* First, remove all groups and peers with net "mesh_vpn" and without `preserve`

I think one needs to check whether a group's subelements contain the preserve flag recursively first. That's what my code does.

CodeFetch · 2020-02-10T22:12:50Z

Meanwhile I've seen that there is a more elegant way to do this. Thus I'll look into it next week.

neocturne · 2020-05-10T11:49:10Z

Superseded by #2021.

CodeFetch mentioned this pull request Nov 10, 2017

Duplicate #1257 #1256

Closed

CodeFetch force-pushed the master branch 5 times, most recently from c2c52a7 to bc0002f Compare November 10, 2017 18:47

rotanid added the 0. type: enhancement The changeset is an enhancement label Nov 10, 2017

CodeFetch force-pushed the master branch 4 times, most recently from 500e1d8 to 524d965 Compare November 11, 2017 23:43

rotanid requested a review from christf January 10, 2018 08:38

christf reviewed Jan 19, 2018

View reviewed changes

rotanid added the needs work label Jan 22, 2018

CodeFetch changed the title ~~Add site.conf option for fastd peer upgrade~~ [RFC] Add site.conf option for fastd peer upgrade Jan 24, 2018

rotanid added the 2. status: merge conflict The merge has a conflict and needs rebasing label Sep 2, 2018

CodeFetch force-pushed the master branch from 524d965 to b0cb35e Compare February 24, 2019 04:24

CodeFetch force-pushed the master branch from b0cb35e to eaa110f Compare February 24, 2019 04:30

rotanid removed 2. status: merge conflict The merge has a conflict and needs rebasing needs work labels Feb 24, 2019

rotanid added the needs work label Mar 9, 2019

mweinelt added the 9. meta: reviewday Will be discussed on review day (1st Monday in even Months, 7:30pm CET/CEST) label Nov 4, 2019

neocturne self-assigned this Dec 2, 2019

christf removed the 9. meta: reviewday Will be discussed on review day (1st Monday in even Months, 7:30pm CET/CEST) label Feb 3, 2020

neocturne closed this May 10, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC] Add site.conf option for fastd peer upgrade #1257

[RFC] Add site.conf option for fastd peer upgrade #1257

CodeFetch commented Nov 10, 2017 •

edited

Loading

flobeier commented Nov 10, 2017

christf commented Jan 19, 2018

christf Jan 19, 2018

christf Jan 19, 2018

CodeFetch Jan 20, 2018

rotanid Jan 21, 2018

christf Jan 21, 2018 •

edited

Loading

CodeFetch commented Jan 24, 2018

rotanid commented Feb 15, 2018

CodeFetch commented Feb 18, 2018

rotanid commented Feb 15, 2019

CodeFetch commented Feb 24, 2019

rotanid commented Feb 25, 2019

CodeFetch commented Feb 25, 2019

mweinelt commented Feb 25, 2019 •

edited

Loading

CodeFetch commented Feb 26, 2019

CodeFetch commented Mar 2, 2019

christf commented Feb 3, 2020 •

edited

Loading

neocturne commented Feb 4, 2020

CodeFetch commented Feb 4, 2020

neocturne commented Feb 5, 2020

CodeFetch commented Feb 6, 2020

CodeFetch commented Feb 10, 2020

neocturne commented May 10, 2020

[RFC] Add site.conf option for fastd peer upgrade #1257

[RFC] Add site.conf option for fastd peer upgrade #1257

Conversation

CodeFetch commented Nov 10, 2017 • edited Loading

flobeier commented Nov 10, 2017

christf commented Jan 19, 2018

christf Jan 19, 2018

Choose a reason for hiding this comment

christf Jan 19, 2018

Choose a reason for hiding this comment

CodeFetch Jan 20, 2018

Choose a reason for hiding this comment

rotanid Jan 21, 2018

Choose a reason for hiding this comment

christf Jan 21, 2018 • edited Loading

Choose a reason for hiding this comment

CodeFetch commented Jan 24, 2018

rotanid commented Feb 15, 2018

CodeFetch commented Feb 18, 2018

rotanid commented Feb 15, 2019

CodeFetch commented Feb 24, 2019

rotanid commented Feb 25, 2019

CodeFetch commented Feb 25, 2019

mweinelt commented Feb 25, 2019 • edited Loading

CodeFetch commented Feb 26, 2019

CodeFetch commented Mar 2, 2019

christf commented Feb 3, 2020 • edited Loading

neocturne commented Feb 4, 2020

CodeFetch commented Feb 4, 2020

neocturne commented Feb 5, 2020

CodeFetch commented Feb 6, 2020

CodeFetch commented Feb 10, 2020

neocturne commented May 10, 2020

CodeFetch commented Nov 10, 2017 •

edited

Loading

christf Jan 21, 2018 •

edited

Loading

mweinelt commented Feb 25, 2019 •

edited

Loading

christf commented Feb 3, 2020 •

edited

Loading