Dart updates for ingest speed, correct time zone, tlmgrapher crash #779

ghost · 2018-04-27T21:28:57Z

and more

Still needs:

DART size optimization. DART grows way faster than the raw data right now. Need to eliminate some indexes and fields if possible.
New dart util needs to complete capability to remove files of data. To do this efficiently may need to add a packet_log_id to decom/reduced tables.
Investigate if string limit of 191 is wasting space
Decom needs to be faster. I increased the default num workers to 2, but it needs to run faster in general.

…dd status, and more

ghost · 2018-04-27T21:48:20Z

doesn't appear to be an issue. Varchars don't use the full space.

codecov-io · 2018-04-27T21:49:47Z

Codecov Report

Merging #779 into master will decrease coverage by 0.3%.
The diff coverage is 43.42%.

@@            Coverage Diff             @@
##           master     #779      +/-   ##
==========================================
- Coverage   85.65%   85.34%   -0.31%     
==========================================
  Files         159      160       +1     
  Lines       15041    15215     +174     
==========================================
+ Hits        12884    12986     +102     
- Misses       2157     2229      +72

Impacted Files	Coverage Δ
lib/cosmos/io/json_drb_object.rb	`96.66% <100%> (+0.15%)`	⬆️
lib/cosmos/top_level.rb	`84.91% <100%> (ø)`	⬆️
lib/cosmos/packets/packet_config.rb	`98.96% <100%> (+0.02%)`	⬆️
lib/cosmos/interfaces.rb	`100% <100%> (ø)`	⬆️
lib/cosmos/io/json_drb.rb	`96.79% <20%> (-1.9%)`	⬇️
lib/cosmos/interfaces/dart_status_interface.rb	`22% <22%> (ø)`
lib/cosmos/interfaces/serial_interface.rb	`73.33% <0%> (-5.84%)`	⬇️
lib/cosmos/io/serial_driver.rb	`90% <0%> (-2.86%)`	⬇️
...smos/tools/cmd_tlm_server/cmd_tlm_server_config.rb	`100% <0%> (ø)`	⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4c62f78...ba43cb8. Read the comment docs.

ghost · 2018-05-14T15:37:45Z

demo/config/system/system.txt

@@ -6,6 +6,7 @@ DECLARE_TARGET INST
 DECLARE_TARGET INST INST2
 DECLARE_TARGET EXAMPLE
 DECLARE_TARGET TEMPLATED
+DECLARE_TARGET DART


This basically makes DART a part of the Demo now. Don't we need to modify the Gemfile to make sure we're including all the DART dependencies. I think you have it commented out right now.

This adds a DART target, but it doesn't bring in all the gem requirements. I'll add DONT_CONNECT to the interface so that it doesn't add noise if DART is not running or used.

ghost · 2018-05-14T15:38:08Z

demo/config/targets/DART/cmd_tlm/dart_cmds.txt

@@ -0,0 +1,2 @@
+COMMAND DART CLEARERRORS BIG_ENDIAN "Clears error counters and messages"
+  APPEND_ID_PARAMETER PACKETID 8 INT 1 1 1 "Packet Id"


ghost · 2018-05-14T15:38:13Z

demo/config/targets/DART/cmd_tlm/dart_tlm.txt

@@ -0,0 +1,40 @@
+TELEMETRY DART STATUS BIG_ENDIAN "DART Status"
+  APPEND_ID_ITEM PACKETID 8 INT 1 "Packet Id"


ghost · 2018-05-14T15:39:19Z

demo/config/targets/DART/target.txt

@@ -0,0 +1,7 @@
+# Ignored Parameters
+# IGNORE_PARAMETER parameter_name
+IGNORE_PARAMETER PACKETID


ghost · 2018-05-14T15:39:26Z

demo/config/targets/DART/target.txt

+
+# Ignored Items
+# IGNORE_ITEM item_name
+IGNORE_ITEM PACKETID


ghost · 2018-05-14T15:58:30Z

lib/cosmos/dart/lib/dart_importer.rb

@@ -146,8 +165,22 @@ def import(filename, force)
          meta_id = ple.id
        end
      end
+
+      if ple_data_count >= 1000


Clever optimization

ghost · 2018-05-14T15:59:30Z

lib/cosmos/dart/lib/dart_packet_log_writer.rb

+        @ple_data << "('#{entry_time}','#{entry_time}',#{target_id},#{packet_id},'#{packet.received_time.dup.utc.to_s(:db)}',#{@packet_log_id},#{@file_size},#{@meta_id},#{@is_tlm},true)"
+      else
+        @ple_data << ",('#{entry_time}','#{entry_time}',#{target_id},#{packet_id},'#{packet.received_time.dup.utc.to_s(:db)}',#{@packet_log_id},#{@file_size},#{@meta_id},#{@is_tlm},true)"
+      end


Same comment as above.

ghost · 2018-05-14T16:01:34Z

lib/cosmos/interfaces/dart_status_interface.rb

+      super()
+      @first = true
+      @status_packet = System.telemetry.packet('DART', 'STATUS').clone
+      @status_packet.write('PACKETID', 1)


Can't you also just do a restore_defaults here?

Only commands have defaults

ghost · 2018-05-14T16:02:32Z

lib/cosmos/interfaces/dart_status_interface.rb

+      @first = true
+      @status_packet = System.telemetry.packet('DART', 'STATUS').clone
+      @status_packet.write('PACKETID', 1)
+      @clear_errors_command = System.commands.packet('DART', 'CLEARERRORS')


I didn't notice this before but I'd prefer CLEAR_ERRORS here with an underscore.

ghost · 2018-05-14T16:03:03Z

lib/cosmos/interfaces/dart_status_interface.rb

+      canceled = false
+      unless @first
+        if @query_time
+          sleep_time = 20.0 - (Time.now - @query_time)


I think the 20 should be a config parameter to the interface. Someone is going to want to speed up or slow this down. Perhaps you make a minimum limit on it. Not sure how long it takes but you're doing a bunch of DB stuff. Maybe 5s minimum?

ghost · 2018-05-14T19:50:41Z

Comments incorporated along with some other changes. ready for final review.

ghost · 2018-05-14T21:32:04Z

lib/cosmos/dart/lib/dart_database_cleaner.rb

+          start_time = first_packet.received_time
+          end_time = last_packet.received_time
+        rescue
+          if size == 128 or size == 0


What is this magic number?

Its the size of the normal COSMOS file header. I just checked in a change to remove the magic...

ghost · 2018-05-14T21:34:25Z

lib/cosmos/dart/lib/dart_database_cleaner.rb

+        ensure
+          reader.close
+        end
+        num_deleted = PacketLogEntry.where("time >= ? and time <= ? and packet_log_id = ?", start_time - 1.minute, end_time + 1.minute, packet_log.id).delete_all


What's with the +- 1 minute

This just provides some tolerance incase the timestamps in the log were a little out of order. The packet_log_id constraint makes sure nothing extra is deleted.

Dart updates for ingest speed, correct time zone, tlmgrapher crash, a…

7784446

…dd status, and more

ghost requested review from a user and donaldatball April 27, 2018 21:29

Merge branch 'master' into dart_updates

609d6af

ghost reviewed May 14, 2018

View reviewed changes

fix bugs. incorporate jasons comments. Make specs pass

7571c3d

ghost reviewed May 14, 2018

View reviewed changes

remove magic number

ba43cb8

ghost merged commit bb8fb1b into master May 15, 2018

ghost deleted the dart_updates branch May 15, 2018 22:06

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dart updates for ingest speed, correct time zone, tlmgrapher crash #779

Dart updates for ingest speed, correct time zone, tlmgrapher crash #779

ghost commented Apr 27, 2018

ghost commented Apr 27, 2018

codecov-io commented Apr 27, 2018 •

edited

Loading

ghost May 14, 2018

ghost May 14, 2018

ghost May 14, 2018

ghost May 14, 2018

ghost May 14, 2018

ghost May 14, 2018

ghost May 14, 2018

ghost May 14, 2018

ghost May 14, 2018

ghost May 14, 2018

ghost May 14, 2018

ghost May 14, 2018

ghost May 14, 2018

ghost commented May 14, 2018

ghost May 14, 2018

ghost May 15, 2018

ghost May 14, 2018

ghost May 15, 2018

		@@ -0,0 +1,2 @@
		COMMAND DART CLEARERRORS BIG_ENDIAN "Clears error counters and messages"
		APPEND_ID_PARAMETER PACKETID 8 INT 1 1 1 "Packet Id"

		@@ -0,0 +1,40 @@
		TELEMETRY DART STATUS BIG_ENDIAN "DART Status"
		APPEND_ID_ITEM PACKETID 8 INT 1 "Packet Id"

Dart updates for ingest speed, correct time zone, tlmgrapher crash #779

Dart updates for ingest speed, correct time zone, tlmgrapher crash #779

Conversation

ghost commented Apr 27, 2018

ghost commented Apr 27, 2018

codecov-io commented Apr 27, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented May 14, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Apr 27, 2018 •

edited

Loading