Pipeline Update

[LouisLu705]

Description

This pipeline change implements processing CSV files for meter and readings data in one place. This new process will check that the readings data does not violate expected constraints and outputs warnings as necessary if errors occur. Currently, this is tested on MAMAC data and works for CSV files which may come with several parameters to indicate how OED should process a certain CSV file such as:

onlyEndTime - CSV files with onlyEndTime values given
cumulative - CSV files with readings which are cumulative
Tgap - the allowed time gap in milliseconds that a gap may occur between two readings
Tlen - the allowed time variation in milliseconds that two readings may deviate from each other
cumulativeReset - true if the cumulative data can reset
resetStart - the start time of the allowed time range a cumulative CSV file may reset
resetEnd - the end time of the allowed time range a cumulative CSV file may reset

Fixes #513 and Fixes #546

See @huss comments below on changes that fixes #510, fixes #512, fixes #515, fixes #565 and fixes #641.

Type of change

(Check the ones that apply by placing an "x" instead of the space in the [ ] so it becomes [x])

• Note merging this changes the node modules
• Note merging this changes the database configuration.
• This change requires a documentation update

Checklist

• I have followed the OED pull request ideas
• I have removed text in ( ) from the issue request

Limitations

This merges in the csvPipeline. branch creataed by @lindavin.

Currently, we expect that dates are passed in as one of two formats:

(YYYY-MM-DD H:mm) or as (MM-DD-YYYY H:mm) == (2020/4/5 0:30) or as (4/5/2020 0:30)

If the dates are in neither of these forms there may be errors. There is also a slight performance decrease due to checking both formats so if we can default to one specific moment format we may improve performance slightly. The difference I found was 11 seconds longer when testing on OneMinuteTestingData.

Also, we need to connect the user interface to the pipeline so that users can decide if their data has certain parameters such as specific cumulativeResetStart and cumulativeResetEnd times as well as onlyEndTime. Note that currently onlyEndTime is set to be false and if no cumulatveResetStart or cumulativeResetEnd times are given the values respectively default to 0:00:00.000 and 23:59:99.999 indicating that a cumulativeReset may occur from the beginning of any day to the millisecond before the day ends and the next day begins.

We may also consider counting how many errors occur and the client may use a parameter to create a maximum cap before too many errors occur and we drop the CSV file and do not insert anything into the database.

Also, we may make the error messages more descriptive for example letting the user know which reading causes the error including that particular reading value, meter id, start timestamp and end timestamp.

There is now also redundant code in the pipeline that will need to be cleared and testing will need to be done to see if we can use this new pipeline process to handle different types of meters besides Mamac such as Obvius.

Here are some statistics from running the generatedTestData on my local machine with 8 GB of RAM.

fourDayFreqTestData:

real    0m0.395s
user    0m0.087s
sys     0m0.095s

generateFourHourTestingData:

real    0m0.922s
user    0m0.007s
sys     0m0.006s

generateTwentyThreeMinuteTestingData:

real    0m5.593s
user    0m0.000s
sys     0m0.017s

generateFifteenMinuteTestingData:

real    0m8.075s
user    0m0.010s
sys     0m0.011s

generateCosineTestingData:

real    0m5.260s
user    0m0.015s
sys     0m0.000s

generateSineSquaredTestingData:

real    0m0.181s
user    0m0.000s
sys     0m0.013s

generateCosineSquaredTestingData:

real    0m0.149s
user    0m0.000s
sys     0m0.011s

generateVariableAmplitudeTestingData:

• 15Freq1AmpSineTestData:

   real    0m16.320s
   user    0m0.000s
   sys     0m0.043s
  

• 15Freq2AmpSineTestData:

   real    0m16.065s
   user    0m0.001s
   sys     0m0.048s
  

• 15Freq3AmpSineTestData:

   real    0m16.329s
   user    0m0.010s
   sys     0m0.036s
  

• 15Freq4AmpSineTestData:

   real    0m15.876s
   user    0m0.012s
   sys     0m0.034s
  

• 15Freq5AmpSineTestData:

   real    0m15.612s
   user    0m0.001s
   sys     0m0.046s
  

• 15Freq6AmpSineTestData:

   real    0m16.351s
   user    0m0.011s
   sys     0m0.031s
  

• 15Freq7AmpSineTestData:

   real    0m15.882s
   user    0m0.010s
   sys     0m0.035s
  

generateOneMinuteTestingData:

real    2m1.651s
user    0m0.020s
sys     0m0.222s

[huss]

I thank @LouisLu705 for good work on a feature with changing needs. I know this is only in a draft state but I wanted to get in some comments now to keep the process moving. @LouisLu705 and I have talked about moving the files in src/server/test/web/csvPipeline/ to be similar to the test/db/data/ directory and excluding them from the header check to avoid that issue. Some CI test issues are due to including the upcoming CSV upload work and will be dealt with in that upcoming PR. I note that I have only done very limited testing of the pipeline and need to do more types of CSV. Finally, this PR will not be merged until the CSV upload PR has cleared due to dependency on that code.

[huss]

First, I want to thank @LouisLu705 for all the work on the pipeline. I don't have any additional comments on their code.

There are some CodeQL issues that will need to be addressed at some point. It might be after this PR is merged.

I'm pushed a lot of changes/commits to the PR. I kept making changes to finish tasks and it got somewhat out of hand where changes in related code outside the original PR is now included. The positive is that the pipeline and related code should now be fully integrated and working with all options. Given the scope of the work, my changes should be reviewed before they are merged with the PR. The changes are discussed in each commit but here are the highlights:

• The curl & web page now have the same abilities. This required many changes including unspecified values now default to what is stored on the meter (when appropriate) so Fixes CSV upload should start with DB data #641; the database was modified so it has values for every value that makes sense in the pipeline; some new abilities to handle all the options; update option now correctly works with DB so functional.
• Messages from the pipeline are now returned and displayed on a web page popup or as the return HTML for curl requests. The messages were enhanced to make them more useable. A maximum size for the returned message so won't report too many issues but all go to the log.
• MAMAC and Obvius now go through the pipeline. The duplicate capability from previous work is now gone. Previous code for the MAMAC drop zone was removed. Note that the reading length assumption of the old Obvius code is now gone as pipeline can figure out reading length. Fixes redundate pipeline file #510. Partly addresses Continued Obvius work #564.
• A new meter type of other was added so automatically created meters of unknown origin have a reasonable type. Fixes other meter type #565.
• The stream input option has been removed. The new pipeline seems to handle reasonable size CSV files so stream was not worth upgrading. Note testing shows that large files can fail if you don't give JS enough memory. This seems more a developer rather than production server issue.
• The old Metasys import code was removed. The new pipeline does everything it did and more. Fixes use of Metasys pipeline in OED code #512. Fixes missing meter on Metasys inport of data #515 as making it obsolete.
• The developer script for inputting test data now goes through the pipeline.
• Made DB have same default start/end date/time as moment(0) so they are consistent. Also changed some of the tests that happened to use this date/time and was causing failure. Many were left as not causing issue.
• A script with test CSV files is included that runs many tests against the pipeline. Note the hope is that many can be converted to Mocha/Chai tests so they become part of the continuous integration.
• Many other changes including removal of obsolete tests, few bug fixes, better error checking on some parameters, enhanced commenting, etc.

[lindavin]

This looks really good. I'm very glad that mamac and obvius utilize the same pipeline code now. I don't see any obvious JS errors. I also like the addition of the enums TimeSortTypesJS and BooleanTypesJS in validateCSVUploadParams. In some places we checked if a variable is equal to string 'true', for example, line 150 of /src/server/routes/csv.js. I think this could be changed to utilize BooleanTypeJS as a later TODO.

[huss]

After my review of the work by @LouisLu705 and the review of my changes by @lindavin, I think this is ready to merge. The CodeQL issues will be addressed at a later time.