Use Travis CI for automated testing

[per1234]

I'd be happy to set up Travis CI on this repository when you're ready for it. Here is the list of tests I can add:

• Compile sketches for all board configurations
  • There is the option of publishing a CSV report of the compilation results (compile sizes and warning count for each sketch compiled) to a dedicated branch of the CI-Reports repository. These reports can be used to analyze the impact of a proposed change.
• Additional checks on bundled libraries
  • Check for common problems with library structure
  • Check for common problems with library.properties
  • Check for common problems with keywords.txt
• Code formatting check
  • Strictness levels:
    • 1: Code formatting configuration used by the Arduino IDE's Tools > Auto Format: https://github.com/arduino/Arduino/blob/master/build/shared/lib/formatter.conf
    • 2: Code formatting configuration used by Arduino on their example sketches: https://github.com/arduino/Arduino/blob/master/build/shared/examples_formatter.conf
    • 3: Custom code formatting configuration I wrote based on common practices in the official Arduino code base: Using Clang-Format for Automatic Source Code Formatting arduino/ArduinoCore-avr#71 (comment)
• File formatting checks:
  • UTF-8 BOM encoding
  • Files starting with a blank line
  • True tabs
  • Non-Unix line endings
  • Trailing whitespace
  • Extra blank lines at end of files
  • Files that don't end in a newline
• Spell check

My preference would be to all of the above, but you are welcome to pick and chose whichever ones you like.

Required infrastructure:

• Enable Travis CI for this repository:
  • Open this page https://travis-ci.org/account/repositories
  • Set the switch next to megaTinyCore to the on position
• If you want compilation report publishing:
  • Create a megaTinyCore branch in https://github.com/SpenceKonde/CI-Reports.
  • Open https://travis-ci.org/SpenceKonde/megaTinyCore/settings
  • Under the "Environment Variables" section, enter REPORT_GITHUB_TOKEN in the "Name" field.
  • Enter your personal access token in the "Value" field. This can be the same token used for ATTinyCore, or you can create a dedicated one if you prefer. If creating a new token, check the box next to public_repo to give it the necessary permissions.
  • Make sure the "Display value in build log" switch is in the off position.
  • Click the "Add" button.

[SpenceKonde]

Let's do everything except the code formatting checks (maybe will want to add them in later and mass-format the code, but I forsee this slowing development with failures due to code formatting that don't bear on functionality, and/or compromising the usefulness of the travis builds because they'd be erroring because of code formatting instead of revealing failures)

[per1234]

Question: In #33 (comment) you mentioned compiling BareMinimum for each board. Usually for the compilation tests I compile all the example sketches of the libraries bundled with the core. Do you want me to:

• compile the library examples only?
• compile BareMinimum only?
• compile both the library examples and BareMinimum in separate jobs, with the BareMinimum compilations first in the build queue?

because they'd be erroring because of code formatting instead of revealing failures

Since the time I set up Travis CI for ATTinyCore, I learned how to efficiently configure it so that the formatting checks are done in a separate job from the compilation tests. That means that a formatting problem will not cause the compilation test jobs to fail. It will cause the formatting check job to fail, and thus will still fail the CI build as a whole, but if you look at the Travis CI job list view, you can quickly see which jobs passed and which jobs failed to identify the general cause of the build failure before you even start looking at the job logs.

[SpenceKonde]

compile both the library examples and BareMinimum in separate jobs, with the BareMinimum compilations first in the build queue?
This one <3

That's wonderful re: separating the formatting checks into a separate job!

[per1234]

Compilation of some of the library examples fail for the 4xx chips with this error:
https://travis-ci.org/per1234/megaTinyCore/jobs/548004941#L1363

build_sketch /home/travis/Arduino/hardware/megaTinyCore/megaavr/libraries/SPI/examples/BarometricPressureSensor/BarometricPressureSensor.ino megaTinyCore:megaavr:atxy6:chip=406 false 1.8.9
Picked up JAVA_TOOL_OPTIONS: 
Loading configuration...
Initializing packages...
Preparing boards...
Verifying...
/home/travis/Arduino/hardware/megaTinyCore/megaavr/libraries/SPI/examples/BarometricPressureSensor/BarometricPressureSensor.ino: In function 'void loop()':
/home/travis/Arduino/hardware/megaTinyCore/megaavr/libraries/SPI/examples/BarometricPressureSensor/BarometricPressureSensor.ino:80:45: warning: left shift count >= width of type [-Wshift-count-overflow]
     long pressure = ((pressure_data_high << 16) | pressure_data_low) / 4;
                                             ^~
/home/travis/.arduino15/packages/arduino/tools/avr-gcc/7.3.0-atmel3.6.1-arduino5/bin/../lib/gcc/avr/7.3.0/../../../../avr/lib/avrxmega3/short-calls/crtattiny406.o:../../../../../crt1/gcrt1.S:311:(.init9+0x2): relocation truncated to fit: R_AVR_13_PCREL against symbol `exit' defined in .fini9 section in /home/travis/.arduino15/packages/arduino/tools/avr-gcc/7.3.0-atmel3.6.1-arduino5/bin/../lib/gcc/avr/7.3.0/avrxmega3/short-calls/libgcc.a(_exit.o)
/home/travis/.arduino15/packages/arduino/tools/avr-gcc/7.3.0-atmel3.6.1-arduino5/bin/../lib/gcc/avr/7.3.0/avrxmega3/short-calls/libgcc.a(_ctors.o): In function `__do_global_ctors':
/home/jenkins/workspace/avr-gcc-staging/label/debian7-x86_64/gcc-build/avr/avrxmega3/short-calls/libgcc/../../../../../gcc/libgcc/config/avr/lib1funcs.S:2481:(.init6+0xc): relocation truncated to fit: R_AVR_13_PCREL against symbol `__tablejump2__' defined in .text.libgcc section in /home/travis/.arduino15/packages/arduino/tools/avr-gcc/7.3.0-atmel3.6.1-arduino5/bin/../lib/gcc/avr/7.3.0/avrxmega3/short-calls/libgcc.a(_tablejump2.o)
collect2: error: ld returned 1 exit status
exit status 1

Is this expected (and therefore I should exclude those compilations from the CI build), or does it indicate a bug (and therefor I should leave the compilations in the CI build)?

[SpenceKonde]

I'm going to say I think that is a.... particularly gross manifestation of the code not fitting in flash.

[per1234]

OK, I'll exclude them. The Wire library examples with friendly error messages:
https://travis-ci.org/per1234/megaTinyCore/jobs/548004941#L1406

Sketch uses 3164 bytes (77%) of program storage space. Maximum is 4096 bytes.
Global variables use 330 bytes (128%) of dynamic memory, leaving -74 bytes for local variables. Maximum is 256 bytes.
Not enough memory; see http://www.arduino.cc/en/Guide/Troubleshooting#size for tips on reducing your footprint.

Are showing an SRAM shortage.

[SpenceKonde]

Leave the Wire ones without enough SRAM in for the 4k chips (which have 256 bytes of SRAM). The buffer is grotesquely oversized and that's a bug.

[SpenceKonde]

The examples that use Tone should fail on everything except 1616 and 3216 - these are due to pieces of the core not being done yet and should fail.

[SpenceKonde]

And that's another thing - are the Basics, Digital and Analog examples being compiled? They definitely should be in the list - they're probably the most important examples to be testing with.

[per1234]

I'm currently only compiling BareMinimum and the examples for the bundled libraries. I'll add compilations for those and any additional sketches you want.

The only limitation is that Travis CI allows a maximum of 200 jobs per build. I can recursively compile an entire folder of sketches and only add one job per board configuration, but if some of the sketches from a folder must be excluded then I need to break it down into one job per sketch for that folder. We're currently nowhere near 200 jobs, but as more custom menu options are added, the total number of board configurations multiplies. It has been a struggle keeping ATTinyCore under 200. I'm not sure if the same issue will happen in this project.

Of course, more compilations inevitably result in a longer CI build duration.

[SpenceKonde]

Do let me know which ones are being excluded.

I'm wondering if maybe these examples should be replaced with ones that demonstrate the same thing without needing to be excluded.

[SpenceKonde]

Will the travis configuration need to be adjusted when more boards and more menu options are added, or does that happen automagically?

[per1234]

maybe these examples should be replaced with ones that demonstrate the same thing without needing to be excluded.

Good idea! Current exclusions:

EEPROM library:

• eeprom_get for 416 and 406 due to exceeding available flash (103%)

SoftwareSerial library

• TwoPortReceive for 416 and 406 due to exceeding available flash (100%)

SPI library

• BarometricPressureSensor for 416 and 406 due to relocation truncated to fit: R_AVR_13_PCREL against symbol tablejump2'` issue

Wire library

• master_reader for 416 and 406 due to relocation truncated to fit: R_AVR_13_PCREL against symbol tablejump2'` issue
• SFRRanger_reader for 416 and 406 due to relocation truncated to fit: R_AVR_13_PCREL against symbol tablejump2'` issue
• slave_receiver for 416 and 406 due to relocation truncated to fit: R_AVR_13_PCREL against symbol tablejump2'` issue

[per1234]

Will the travis configuration need to be adjusted when more boards and more menu options are added, or does that happen automagically?

It needs to be manually adjusted every time.

There is a tool that could maybe eventually be used to automatically update the Travis CI build according to changes to boards.txt, but I haven't investigated the possibility at all:
esp8266/Arduino#5657
Travis CI doesn't allow the definition of jobs programmatically, so it would need to be done by an external script that automatically detects changes to boards.txt, rewrites .travis.yml, and then submits a PR for the updated .travis.yml. Tricky stuff.

[per1234]

are the Basics, Digital and Analog examples being compiled?

None of the "03.Analog" examples bundled with the Arduino IDE are going to compile for the boards that use the txy6 variant due to #27 (comment). The An pins are still defined for the txy4 variant, but I don't know whether that's only because you haven't gotten around to deleting them yet.

You can always add your own sketches to the core that are written specifically for Travis CI testing. Somewhere like /megaavr/travis-ci/tests would be a good place for them.

[per1234]

It turns out An pin names are used in some of the "01.Basics" and "02.Digital" sketches as well, so I'll need to do a single job for each of those sketches for each board configuration.

[SpenceKonde]

You know what? Leave the sketches that fail to compile due to An pin names in, and failing.

I will put An defines back in, in some form, so that code which uses them can be made to compile. I think it will cause more problems than it's worth to not support the An notation.

[SpenceKonde]

Uhm.... you're uh, probably going to want to pull current master and work over it with the formatting scripts again >.> I've been busy today.

All the planned variants are in now.

i would skip testing the 2k parts altogether, or just test like, bare minimum on them. They're all identical to larger variants, and the flash is so constrained that almost no examples will compile on them.

[SpenceKonde]

In terms of board menu options, presently the one coming that will need to be tested is the clock speed options.
The bod and EESAVE options are going to go in soon too, but they do not make any changes to the code, just the fuses, so Travis can't catch any problems with them.

[per1234]

you're uh, probably going to want to pull current master and work over it with the formatting scripts again

It will probably be quite a while before I have more volunteer time to work on this. I was really hoping it would get merged while there were no conflicts.

but they do not make any changes to the code, just the fuses, so Travis can't catch any problems with them.

Doing a compilation tests whether boards.txt is set up correctly. Usually I'm able to set Travis CI up so it tests these too without adding any additional jobs, since the clock menu has more options than any other custom menu.

[SpenceKonde]

I was hoping to get it merged back before there were merge conflicts too - but I couldn't keep my hands off it yesterday - had energy, nothing else to do, and am excited about the core. And besides, since the boards needed to be added to the travis configuration, I realized I needed to add those boards and variant files anyway for travis to run on those.

[SpenceKonde]

Travis was enabled and reporting set up.

[SpenceKonde]

So, I really need to get automated testing on this core....

There's a $25 cash or $50 in free stuff from my Tindie store bounty on this.

[per1234]

I'm working on it.