Moving towards multi-sensor-per-same-measurement fusion #652

kamilritz · 2019-10-08T14:57:13Z

The current EKF was designed to fuse velocity and position measurement in an efficient way. The current implementation does this by selecting a sensor source for each measurement and performing a measurement update containing measurement from different sensors.
Fusing the same measurement from different sources, e.g. GPS velocity and EV velocity, is also possible. The issue is that this happens through the same global _vel_pos_innov variable. Which can potentially be used for multiple sensors in the same loop. In general, the whole EKF interface is not designed to support logging of the same measurement from different sensors.

This PR tries to solve this issue by adapting the fuseVelPoseHeight() function. The function interface is changed, such that it takes multiple arguments and runs with a generic velPosHeight sensor. Therefore the computation of the height innovation is moved outside to the controlHeightFusion() function.
The controlGPSFusion(), controlExternalVisionFusion(), controlHeightFusion(), controlAuxFusion() and the new controlFakePosFusion() functions are all passing the innovation/-variances/-gates, observation-variances and a new fusion_mask as an argument in the function call to fuseVelPoseHeight().

The necessary changes on the Firmware side are laid out here.

These changes were test flown with different velPos and height sources. The test flight is based on flying a square followed by a yaw spin. The flight performance was in all cases as expected. The logs of the test flights can be found here:

@priseborough I would appreciate if you could have a look at this and give me rough feedback. So that I can move in the right direction.

priseborough · 2019-10-10T22:22:14Z

EKF/ekf_helper.cpp

 	// return the vertical position innovation test ratio
-	*hgt = sqrtf(_vel_pos_test_ratio[5]);
+	*hgt = sqrtf(_gps_vel_pos_test_ratio[5])+sqrtf(_ev_vel_pos_test_ratio[5]);


I think we should return the maximum.

I am still not sure what would be the appropriate solution. Can you give me some reasoning why we should use the maximum?

This is resolved in the newest version.

priseborough · 2019-10-10T22:28:52Z

EKF/vel_pos_fusion.cpp

+/**
+ * Update the EKF state with velocity and position measurements.
+ *
+ * @param innov  	Input/Output	Innovation which is fuse_mask is set to false or do not pass


Can you please reword the description for this. I have read it a few times and am not sure what it means.

You are totally right, this sentence does not make any sense. Thanks for noticing.

I think this line was part of an outdated version. Please have a look at the newest version.

kamilritz · 2019-10-11T08:32:50Z

I seem to break the python tests. But I do not manage to build ECL locally as a standalone. So I can not run and check the python tests. Any advise here? @priseborough

kamilritz · 2019-10-11T14:39:46Z

@priseborough I also make the get__innov interface to EKF cleaner.

bresch

Good work, I didn't review the logic in detail but added a few comments

bresch · 2019-10-14T13:45:54Z

EKF/control.cpp

-		if (_control_status.flags.ev_hgt) {
-			_fuse_height = true;
-		}
+		bool ev_fuse_mask[4]{false};


Suggested change

bool ev_fuse_mask[4]{false};

bool ev_fuse_mask[4]{};

It's the same in that specific case, but be aware that bool ev_fuse_mask[4]{true}; would initialize to {true, false, false, false}

bresch · 2019-10-14T13:50:50Z

EKF/control.cpp


 		// determine if we should use the horizontal position observations
 		if (_control_status.flags.ev_pos) {
-			_fuse_pos = true;
+			ev_fuse_mask[2] = true;


would be nice to have an enum class for those indexes. It's really hard to follow if you don't remember the indexes by heart

bresch · 2019-10-14T13:54:09Z

EKF/control.cpp

-			_fuse_vert_vel = true;
-			_fuse_hor_vel = true;
+
+			bool gps_fuse_mask[4]{false};


Suggested change

bool gps_fuse_mask[4]{false};

bool gps_fuse_mask[4]{};

bresch · 2019-10-14T13:54:54Z

EKF/control.cpp

@@ -1009,8 +998,14 @@ void Ekf::controlHeightSensorTimeouts()

 void Ekf::controlHeightFusion()
 {
-	// set control flags for the desired primary height source
+	bool height_fuse_mask[4]{false};


Suggested change

bool height_fuse_mask[4]{false};

bool height_fuse_mask[4]{};

bresch · 2019-10-14T14:01:15Z

EKF/control.cpp

 	}

+	if (height_fuse_mask[3]) {


Do you really want to add 63 lines to a function that is already way too long? :'(

@bresch No, I do not want to. I am not very happy how the height fusion is implemented at the moment and I will try to work on this next. If you want I can split it into two separate functions for now?

bresch · 2019-10-14T14:55:50Z

EKF/tests/pytest/test_utils.py

@@ -89,7 +89,7 @@ def initialized_ekf():
    dt_usec = 5000

    # Provide a few samples
-    for _ in range(1000):
+    for _ in range(2000):


why? Does it take more time to converge?

@bresch Apparently. I can not see a clear reason why it should take longer to get the tilt aligned. But with out increasing the number of samples, it does not pass the test.

The problem is that we need to know why. Maybe it worked before but shouldn't pass, maybe something broke now...

This points to a change in behavior, in what should be a zero-behavior-change refactor. This should be investigated, maybe it was fixed due to a bug before, but I think due to it taking longer to diverge there is more likely a bug now.

bresch · 2019-10-14T15:34:11Z

EKF/vel_pos_fusion.cpp

-				zeroCols(P, i, i);
+			// update individual measurement health status
+			if (obs_index == 0) {
+				_fault_status.flags.bad_vel_N = true;


you can write a function that does the change depending on the index plus two small functions set and clear to avoid having to pass the boolean:

void Ekf::setFaultStatusFlag(const int index) { changeFaultStatusFlag(index, true); } void Ekf::clearFaultStatusFlag(const int index) { changeFaultStatusFlag(index, false); } void Ekf::changeFaultStatusFlag(const int index, const bool status) { if (index == 0) { _fault_status.flags.bad_vel_N = status; } else if (index == 1) { _fault_status.flags.bad_vel_E = status; } else if (index == 2) { _fault_status.flags.bad_vel_D = status; } else if (index == 3) { _fault_status.flags.bad_pos_N = status; } else if (index == 4) { _fault_status.flags.bad_pos_E = status; } else if (index == 5) { _fault_status.flags.bad_pos_D = status; } }

Then you can simplify the whole for loop with :

for (int i = 0; i < _k_num_states; i++) { if (P[i][i] < KHP[i][i]) { // zero rows and columns zeroRows(P, i, i); zeroCols(P, i, i); healthy = false; setFaultStatusFlag(obs_index); } else { clearFaultStatusFlag(obs_index); } }

That is nice!

bresch · 2019-10-14T16:11:26Z

EKF/ekf_helper.cpp

@@ -1217,8 +1363,8 @@ void Ekf::get_ekf_soln_status(uint16_t *status)
 	soln_status.flags.const_pos_mode = !soln_status.flags.velocity_horiz;
 	soln_status.flags.pred_pos_horiz_rel = soln_status.flags.pos_horiz_rel;
 	soln_status.flags.pred_pos_horiz_abs = soln_status.flags.pos_horiz_abs;
-	bool gps_vel_innov_bad = (_vel_pos_test_ratio[0] > 1.0f) || (_vel_pos_test_ratio[1] > 1.0f);
-	bool gps_pos_innov_bad = (_vel_pos_test_ratio[3] > 1.0f) || (_vel_pos_test_ratio[4] > 1.0f);
+	bool gps_vel_innov_bad = (_gps_vel_pos_test_ratio[0] > 1.0f);


why only checking the X axis here?

bresch · 2019-10-14T16:11:53Z

EKF/ekf_helper.cpp

@@ -1182,22 +1324,26 @@ bool Ekf::reset_imu_bias()
 // Innovation Test Ratios - these are the ratio of the innovation to the acceptance threshold.
 // A value > 1 indicates that the sensor measurement has exceeded the maximum acceptable level and has been rejected by the EKF
 // Where a measurement type is a vector quantity, eg magnetometer, GPS position, etc, the maximum value is returned.
-void Ekf::get_innovation_test_status(uint16_t *status, float *mag, float *vel, float *pos, float *hgt, float *tas, float *hagl, float *beta)
+void Ekf::get_innovation_test_status(uint16_t *status, float *mag, float *gps_vel, float *gps_pos, float *ev_vel, float *ev_pos, float *hgt, float *tas, float *hagl, float *beta)


pass references?

bresch · 2019-10-14T16:14:34Z

EKF/ekf.h

-	// gets the innovations of velocity and position measurements
-	// 0-2 vel, 3-5 pos
-	void get_vel_pos_innov(float vel_pos_innov[6]);
+	// gets the GPS innovations of velocity and position measurements


All those comments are redundant as the names of the functions are clear. However, if the intent is to generate documentation, you would need to add more details

bresch

All good, except the python check that fails

bresch · 2019-10-16T11:41:29Z

EKF/ekf.h

@@ -724,4 +739,37 @@ class Ekf : public EstimatorInterface
 	// sensor measurement
 	float calculate_synthetic_mag_z_measurement(Vector3f mag_meas, Vector3f mag_earth_predicted);

+	// stop GPS fusion


Suggested change

// stop GPS fusion

bresch · 2019-10-16T11:41:41Z

EKF/ekf.h

+	// stop GPS fusion
+	void stopGpsFusion();
+
+	// stop GPS position fusion


Suggested change

// stop GPS position fusion

bresch · 2019-10-16T11:41:53Z

EKF/ekf.h

+	// stop GPS position fusion
+	void stopGpsPosFusion();
+
+	// stop GPS velocity fusion


Suggested change

// stop GPS velocity fusion

bresch · 2019-10-16T11:42:02Z

EKF/ekf.h

+	// stop GPS velocity fusion
+	void stopGpsVelFusion();
+
+	// stop GPS yaw fusion


Suggested change

// stop GPS yaw fusion

bresch · 2019-10-16T11:42:10Z

EKF/ekf.h

+	// stop GPS yaw fusion
+	void stopGpsYawFusion();
+
+	// stop external vision fusion


Suggested change

// stop external vision fusion

bresch · 2019-10-16T11:44:39Z

EKF/estimator_interface.h

-	// gets the innovations of velocity and position measurements
-	// 0-2 vel, 3-5 pos
-	virtual void get_vel_pos_innov(float vel_pos_innov[6]) = 0;
+	// gets the GPS innovations of velocity and position measurements


Suggested change

// gets the GPS innovations of velocity and position measurements

bresch · 2019-10-16T11:44:47Z

EKF/estimator_interface.h

-	virtual void get_vel_pos_innov(float vel_pos_innov[6]) = 0;
+	// gets the GPS innovations of velocity and position measurements
+	virtual void getGpsVelPosInnov(float hvel[2], float &vvel, float hpos[2], float &vpos) = 0;
+	// gets the GPS innovation variances of velocity and position measurements


Suggested change

// gets the GPS innovation variances of velocity and position measurements

bresch · 2019-10-16T11:47:08Z

EKF/tests/pytest/test_utils.py

@@ -89,7 +89,7 @@ def initialized_ekf():
    dt_usec = 5000

    # Provide a few samples
-    for _ in range(1000):
+    for _ in range(2000):


The problem is that we need to know why. Maybe it worked before but shouldn't pass, maybe something broke now...

bresch · 2019-10-16T11:52:32Z

EKF/vel_pos_fusion.cpp

+
+			healthy = false;
+
+			setVelPosFaultStatus(obs_index,true);


It's okay here because it is clear enough that the true/false is the value you want to apply to the obs_index, but good practice is to wrap the function into an other one that sets the boolean for you.

I know that there is no astyle check for ecl, but try to follow the rues as close as possible :D

Suggested change

setVelPosFaultStatus(obs_index,true);

setVelPosFaultStatus(obs_index, true);

bresch · 2019-10-16T11:52:46Z

EKF/vel_pos_fusion.cpp

+			setVelPosFaultStatus(obs_index,true);
+
+		} else {
+			setVelPosFaultStatus(obs_index,false);


Suggested change

setVelPosFaultStatus(obs_index,false);

setVelPosFaultStatus(obs_index, false);

bresch · 2019-10-16T11:57:00Z

Just as a side note, adding a comment to every variable and function isn't useful if it doesn't add any information: https://www.todaysoftmag.com/article/1120/clean-code-comments-and-formatting

dagar · 2019-10-16T11:59:49Z

EKF/ekf.h

+
+	void getAirspeedInnov(float &airspeed_innov);
+
+	void getAirspeedInnovVar(float &airspeed_innov_var);


If you're going to bother changing these how about we return by value whenever possible? A lot of them could then turn into inlined getters.

That is a nice idea. A lot of the float arrays that are returned are directly put into message fields, which are also a float array. So returning them by value as a Vector would be a lot of effort.
There are a lot of other functions that do not follow the requested standard and I suggest that we try to clean this up in a separate PR.

dagar · 2019-10-16T12:00:31Z

EKF/ekf.h

+
+	void getDragInnovRatio(float drag_innov_ratio[2]);
+
+	void getAirspeedInnov(float &airspeed_innov);


Suggested change

void getAirspeedInnov(float &airspeed_innov);

float getAirspeedInnov() const { return _airspeed_innov; }

jkflying · 2019-10-16T12:01:07Z

EKF/ekf_helper.cpp

+
+void Ekf::getGpsVelPosInnovRatio(float &hvel, float &vvel, float &hpos, float &vpos)
+{
+	memcpy(&hvel, _gps_vel_pos_test_ratio+HVEL, sizeof(float));


These ones with the references are much easer to read as:

Suggested change

memcpy(&hvel, _gps_vel_pos_test_ratio+HVEL, sizeof(float));

hvel = _gps_vel_pos_test_ratio[HVEL];

dagar · 2019-10-16T12:01:36Z

EKF/ekf.h

+
+	void getAirspeedInnov(float &airspeed_innov);
+
+	void getAirspeedInnovVar(float &airspeed_innov_var);


Suggested change

void getAirspeedInnovVar(float &airspeed_innov_var);

float getAirspeedInnovVar() const { return _airspeed_innov_var; }

jkflying · 2019-10-16T12:08:05Z

EKF/vel_pos_fusion.cpp

@@ -45,276 +45,175 @@
 #include <ecl.h>
 #include <mathlib/mathlib.h>

-void Ekf::fuseVelPosHeight()
+/**


This whole file is much cleaner, nice work!

dagar · 2019-10-16T12:11:53Z

EKF/ekf_helper.cpp

-void Ekf::get_vel_pos_innov(float vel_pos_innov[6])
+void Ekf::getGpsVelPosInnov(float hvel[2], float &vvel, float hpos[2],  float &vpos)
+{
+	memcpy(hvel, _gps_vel_pos_innov+0, sizeof(float) * 2);


Returning matrix::Vectors is also fine.

can you give an example how you mean that?

Actually this one isn't a great example, but in other cases we could easily/safely return a Vector by value rather than passing in an array or pointer to fill.

Examples

// Current PX4/ecl: // we often pass in a pointer to fill with 1, 2, or 3 floats void estimator_interface::get_velocity(float *vel); // wind velocity pass in vector to fill with 2 floats void Ekf::get_wind_velocity(float *wind); // SUGGESTION: // whenever possible we should prefer simple and safe getters that return by value matrix::Vector3f estimator_interface::get_velocity() const; matrix::Vector2f Ekf::get_wind_velocity() const;

I'm mainly trying to say don't be afraid to return these thing as values. It's easier to use, harder to misuse, and not going to be any slower.

kamilritz · 2019-10-18T09:49:09Z

All good, except the python check that fails

@bresch I found out why I did not pass the original python tiltAlignment test. In the current EKF version we fuse fakePos measurements in the beginning since we do not have GPS measurements. The rate at which this is happening is determined by this if statement. Since we fuse baro measurement early on, we set the variable _fuse_height to true at the rate of the baro measurements. In my previous version I did not use the _fuse_height variable anymore, but it was still in the if statement. There for the fake zero x and y position measurement were fused at a much lower rate causing the uncertainty in the orientation to converge slower. As an intermediate solution I set the _fuse_height variable to true every time height is fused. I checked with a debugger, that now the time of the successful tilt alignment during the test is unchanged.

PS: the python tests based on the initialized_ekf() function are probably flawed, since they push in IMU sample with delta_*_dt in microseconds instead of seconds.

@priseborough LGTM

priseborough

I've been through this twice and can't find anything that should impact on exisiting operation. The issue will be coordinating it with the changes in the ekf2 module because it is renaming accessor functions.

For PR's in the past this has created a deadlock that has delayed implementation as both PR's became conditional on each other.

I am OK with merging this provided the changes on the other side of the interface are also approved so we can do them at the same time.

mhkabir · 2019-11-05T14:55:47Z

EKF/ekf_helper.cpp

+
+void Ekf::stopGpsVelFusion()
+{
+	_gps_vel_innov.setZero();


There should be a control flag for GPS vel too.

I agree. The control_status is logged in the estimator_status.msg. Where would we put the the GPS velocity flag in the filter_control_status struct? At the end? Are people okay with changed meaning of the control_status.gps flag?

I think both of those should be ok, but I defer to others.

mhkabir · 2019-11-05T14:56:14Z

EKF/ekf_helper.cpp

+
+void Ekf::stopAuxVelFusion()
+{
+	_aux_vel_innov.setZero();


There should be a control flag for auxvel.

mhkabir · 2019-11-05T14:56:49Z

EKF/ekf_helper.cpp

+	_control_status.flags.opt_flow = false;
+	memset(_flow_innov,0.0f,sizeof(_flow_innov));
+	memset(_flow_innov_var,0.0f,sizeof(_flow_innov_var));
+	memset(&_optflow_test_ratio,0.0f,sizeof(_optflow_test_ratio));


Can we avoid the direct memsets?

mhkabir · 2019-11-05T14:59:27Z

EKF/ekf.h

+
+	void stopAuxVelFusion();
+
+	void stopFlowFusion();


Can we do similar for starting fusion? (setting the flags)

mhkabir · 2019-11-05T14:59:57Z

EKF/ekf_helper.cpp

@@ -54,13 +54,15 @@ bool Ekf::resetVelocity()

 	// reset EKF states
 	if (_control_status.flags.gps && _gps_check_fail_status.value==0) {
+		ECL_INFO_TIMESTAMPED("EKF reset velocity to GPS");


Do these need the EKF prefix? The module name should already be there.

That is true. I will remove them.

mhkabir · 2019-11-05T15:01:22Z

EKF/common.h

+		bool reject_sideslip: 1;	///< 9 - true if the synthetic sideslip observation has been rejected
+		bool reject_hagl: 1;		///< 10 - true if the height above ground observation has been rejected
+		bool reject_optflow_X: 1;	///< 11 - true if the X optical flow observation has been rejected
+		bool reject_optflow_Y: 1;	///< 12 - true if the Y optical flow observation has been rejected
 	} flags;


Why is this not per-sensor now?

That is a good question. That is a point that is open for discussion.

dagar · 2019-11-05T16:51:32Z

EKF/common.h

-		bool reject_hagl: 1;		///< 9 - true if the height above ground observation has been rejected
-		bool reject_optflow_X: 1;	///< 10 - true if the X optical flow observation has been rejected
-		bool reject_optflow_Y: 1;	///< 11 - true if the Y optical flow observation has been rejected
+		bool reject_hor_vel: 1;		///< 0 - true if horizontal velocity observations have been rejected


If you're going to modify these anyway I think we should consider keeping them as actual bools directly. The memory difference is negligible, and it's slightly more work to mask the bit constantly.

Then in the ekf2 frontend I'd like to publish them in a new simple estimator_status_flags that publishes at a very low rate (or change). It makes it much easier to review manually and it avoids having duplicate bits definitions in 3-4 locations.

Previous rant - PX4/PX4-Autopilot#13036 (comment)

I like this idea very much. The current bitmask approach also means that reordering the flags is not advisable.
I can help to implement that, but I think that this should go into a separate PR.

kamilritz · 2019-11-11T09:12:45Z

@mhkabir and @dagar thank you very much for your feedback. Your requests are in the process of being implemented and will be in new PR.

bresch · 2020-04-27T06:05:59Z

@kamilritz Do you think you can start again on this topic? This would be a great improvement I think

kamilritz · 2020-04-27T06:54:55Z

@kamilritz Do you think you can start again on this topic? This would be a great improvement I think

@bresch Good point. I will put this on my list. But it will take some time until I will get to it. If you need it soon, then feel free to do it yourself.

bresch · 2020-04-27T07:03:21Z

@kamilritz Okay, cool. It isn't something urgent but it would be great to have it for the next release (v1.12)

kamilritz mentioned this pull request Oct 8, 2019

[ECL] Moving towards multi-sensor-per-same-measurement fusion PX4/PX4-Autopilot#13127

Merged

priseborough reviewed Oct 10, 2019

View reviewed changes

kamilritz marked this pull request as ready for review October 14, 2019 13:19

bresch suggested changes Oct 14, 2019

View reviewed changes

kamilritz force-pushed the refactoring_velpos branch 2 times, most recently from f7b191c to 5d442c7 Compare October 15, 2019 12:56

bresch suggested changes Oct 16, 2019

View reviewed changes

dagar reviewed Oct 16, 2019

View reviewed changes

jkflying reviewed Oct 16, 2019

View reviewed changes

dagar reviewed Oct 16, 2019

View reviewed changes

jkflying reviewed Oct 16, 2019

View reviewed changes

dagar reviewed Oct 16, 2019

View reviewed changes

kamilritz force-pushed the refactoring_velpos branch 3 times, most recently from 8500302 to 95bc154 Compare October 18, 2019 09:43

kamilritz added 5 commits October 29, 2019 10:34

Refactoring velPos fusion

10c5547

Clean get*innov* interface

f2d321b

Add stop*Fusion function

9049ef7

Add print statement to reset vel or pos function

50780eb

Group velocity and position as a 3d vector

f1ee45a

kamilritz force-pushed the refactoring_velpos branch 2 times, most recently from c19441d to 000c3c7 Compare October 29, 2019 10:06

Rename IMU biases

905a296

kamilritz force-pushed the refactoring_velpos branch from 000c3c7 to 905a296 Compare October 29, 2019 10:19

priseborough reviewed Nov 5, 2019

View reviewed changes

mhkabir reviewed Nov 5, 2019

View reviewed changes

EKF/ekf_helper.cpp

void Ekf::stopAuxVelFusion()

{

_aux_vel_innov.setZero();

Copy link

Member

mhkabir Nov 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There should be a control flag for auxvel.

mhkabir reviewed Nov 5, 2019

View reviewed changes

EKF/ekf.h

void stopAuxVelFusion();

void stopFlowFusion();

Copy link

Member

mhkabir Nov 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we do similar for starting fusion? (setting the flags)

mhkabir reviewed Nov 5, 2019

View reviewed changes

dagar reviewed Nov 5, 2019

View reviewed changes

kamilritz mentioned this pull request Nov 11, 2019

[continue:] Moving towards multi-sensor-per-same-measurement fusion #668

Merged

kamilritz closed this Nov 11, 2019

	setVelPosFaultStatus(obs_index,true);
	setVelPosFaultStatus(obs_index, true);

	setVelPosFaultStatus(obs_index,false);
	setVelPosFaultStatus(obs_index, false);


		void getAirspeedInnov(float &airspeed_innov);

		void getAirspeedInnovVar(float &airspeed_innov_var);


		void getDragInnovRatio(float drag_innov_ratio[2]);

		void getAirspeedInnov(float &airspeed_innov);

	void getAirspeedInnov(float &airspeed_innov);
	float getAirspeedInnov() const { return _airspeed_innov; }

	memcpy(&hvel, _gps_vel_pos_test_ratio+HVEL, sizeof(float));
	hvel = _gps_vel_pos_test_ratio[HVEL];

	void getAirspeedInnovVar(float &airspeed_innov_var);
	float getAirspeedInnovVar() const { return _airspeed_innov_var; }

Moving towards multi-sensor-per-same-measurement fusion #652

Moving towards multi-sensor-per-same-measurement fusion #652

Conversation

kamilritz commented Oct 8, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kamilritz commented Oct 11, 2019

kamilritz commented Oct 11, 2019

bresch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bresch Oct 14, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bresch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bresch commented Oct 16, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dagar Oct 16, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kamilritz commented Oct 18, 2019

priseborough left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kamilritz commented Nov 11, 2019

bresch commented Apr 27, 2020

kamilritz commented Apr 27, 2020

bresch commented Apr 27, 2020

kamilritz commented Oct 8, 2019 •

edited

Loading

bresch Oct 14, 2019 •

edited

Loading

dagar Oct 16, 2019 •

edited

Loading