Checklist for device integration testing

[mweinelt]

What needs to be verified to do integration testing for a new device. Let's create a checklist.

@Brother-Lal wanted to take care of that.

Make use of our notes from the meeting here: https://md.darmstadt.ccc.de/gluon-device-support-policy#

[rotanid]

additional ideas for this from my forum post: https://forum.freifunk.net/t/archer-c7-v-4-0/16384/7

[Brother-Lal]

Checklist for now:

- [ ] Flash from stock possible
- [ ] Flash back to stock possible
- [ ] sysupgrade is working (check with flashing + erasing settings)
- [ ] autoupdater working (implied by sysupgrade working?)
- [ ] Working WAN/LAN Ports + associated LED(s)
- [ ] Wifi LED working and blinking
- [ ] Wifi: 11s and AP Mode working in tandem / at least one of them
- [ ] Is there a possibility for user triggered reset (button,POE-Reset, etc)
- [ ] MAC: Is the official MAC(usually wlan0) the same as printed on the device/packaging?
- [ ] ... More?

Any other additions I might have missed?

[Adorfer]

MAC: clarifiy this point that this should be checked like "idem to meshid" (=default routername-suffix, i do not know how this will be on the map for babel)

[Adorfer]

• [ ] Config-mode is listeninging on a wired interface
• [ ] WAN/LAN ports are not inverted by accident. (perhaps change to "working WAN/LAN" to "working LAN/WAN and mapped correctly")

[rotanid]

@blocktrron @achterin maybe you can contribute to the suggested checklist

[rubo77]

- [ ] tested with x clients
- [ ] tested with x neighbors
- [ ] tested with x Mbit throughput

[csamsel]

- [ ] tested fastd: x Mbit VPN throughput
- [ ] tested tunneldigger: x Mbit VPN throughput

[T-X]

Throughput tests sounds like a nice idea! However I think we would need to specify and document how to do them then as we would otherwise get varying results.

[Brother-Lal]

Throughput tests will vary with several factor: MTU,bandwidth available, encryption, load on gateways, etc...
I wouldn't add that, as it's really depending on too many factors. A general area (10-13 MBits) would be ok, but not really dependable anyway....
The same with the clients, neighbours and different VPNs - without a real world scenario over a good timeframe (2/3 weeks) no conclusions can be drawn.

So, in my opinion: stick to basic yes/no function tests

[csamsel]

Throughput is a dealbreaker. If everything works in principle, but you can only achieve 2 mbit/s, the node is not usable. Sure you could change the question to

[ ] VPN throughput at least 10 mbit/s

but you'll have to measure it anyways. Might as well enter result then.

[mweinelt]

My opinion below:

Yay:

• must be flashable from vendor firmware
  • webinterface
  • tftp
  • other:
• must support upgrade mechanism
  • must have working sysupgrade
    • must keep/forget configuration (if applicable)
      think sysupgrade [-n] or firstboot
  • must have working autoupdate
    usually means: gluon profile name must match image name
• wired network
  • should support all network ports on the device
  • must have correct port assignment (WAN/LAN)
• wifi (if applicable)
  • association with AP must be possible on all radios
  • association with 802.11s mesh must be working on all radios
  • ap/mesh mode must work in parallel on all radios
• led mapping
  • power/sys led
    • lit while the device is on
    • should display config mode blink sequence
      (https://gluon.readthedocs.io/en/latest/features/configmode.html)
  • radio leds
    • should map to their respective radio
    • should show activity
  • switchport leds
    • should map to their respective port (or switch, if only one led present)
    • should show link state and activity
• reset/wps button must return device into config mode
• primary mac should match address on device label (or packaging) (https://gluon.readthedocs.io/en/latest/dev/hardware.html#notes)

Optional:

• flash back to stock, no strong opinion on that
  • shouldn't that be a request brought up with openwrt device integrators?

Nay:

• performance metrics (throughput, client count, neighbour count)
  • they depends on too many variables
  • performance limitations usually result from buying hardware that is too weak (in terms of cpu, memory, radios, antennas, etc.)
  • ...or generic load issues

[skorpy2009]

First a comment on @mweinelt:
wired network: should correctly map LEDs to ports: In my opinion all LEDs should be mapped correctly.
primary mac should match address on device (or packaging)

[rubo77]

Nay=No?

About throughput: I think it is important to test not the throughput, but test if the device is not crashing once there is much load (for example the Fonera Router runs fine with all tests, but it crashes, as soon as there are more than some clients and they start causing traffic):

• [] test device while running a lot of traffic

[mweinelt]

I'd like the developers to focus on getting devices to work. The userbase should start testing that integration asap and report back issues. The userbase are many people that can far better find bottlenecks. I'd like to remind you of #125 where a proper limit to client counts couldn't be found as there are too many variables to that.

[rotanid]

i vote for the current proposal by @mweinelt .

[rotanid]

current status: we decided to put the checklist on github as part of a pull request template, see #1433 for progress on this

[rotanid]

fixed by adding it to the gluon github wiki: https://github.com/freifunk-gluon/gluon/wiki/Device-Integration-checklist ( done by @mweinelt )