idle time for mouse-trajectories with sampling rate depending on movement

[sbrockhaus]

For technical reasons it is possible that the mouse-coordinates are recorded whenever the mouse changes the position (in contrast to data that is logged at a constant rate). To get an idea of such data, I include a minimal example that closely mimics my data at hand.
I think for the computation of idle time (and hovers, but I would stick to idle time for the moment), it is important how such data is treated.

library(mousetrap)

## type in timestamps and coordinates, similar to really observed data  
timestamps <- c(30, 50, 341, 349, 357, 365, 373, 381, 389, 397) - 30 # in ms (get rid of initiation time)
xpos <- c(80, 80, 75, 65, 53, 43, 32, 24, 17, 11)
ypos <- c(46, 45, 44, 41, 38, 35, 32, 30, 29, 27)

## it seems that the maximal rate of recording mouse-movements is every 8ms 
diff(timestamps)

## create array of mouse-trajectories 
temp <- matrix(c(timestamps, xpos, ypos), ncol=3, nrow=10)
colnames(temp) <- c("timestamps", "xpos", "ypos")
temp1 <- temp 
tr <- array(NA, dim = c(2, 10, 3), 
            dimnames = list(1:2, 1:10, c("timestamps", "xpos", "ypos")))
tr[1,,] <- temp
tr[2,,] <- temp1
tr

## create mousetrap object
mt_data <- list(data = data.frame(x1 = 1:2, x2 = 4:5), 
                trajectories = tr)
class(mt_data) <- "mousetrap"

Focusing on the timestamps, we see that when the mouse is moving, the logging rate is every 8 ms:

> diff(timestamps)
[1]  20 291   8   8   8   8   8   8   8

My intuition would be that the idle time is 20 + 291 = 311 ms for this trajectory (whether a lower threshold > 8 ms should be defined for times to count as idle time is a different question that should be discussed - when not making this threshold explicit it is determined by the sampling rate).

When using the mousetrap-package to compute the idle time on this data, we get per definition 0 idle time, as the coordinates are only logged in case of movement.

## compute measures 
mt_data <- mt_measures(mt_data, flip_threshold = 0, use = "trajectories", 
                       save_as = "measures", hover_threshold = 2000 )
## idle time is zero
mt_data$measures$idle_time

When doing a resampling of the observations to a regular grid using linear interpolation, again the idle time is 0 (or only non-zero in case of mistakes with the machine epsilon).

#### compute equidistant versions of the trajectories with timestamps every 10 milliseconds
mt_data <- mt_resample(mt_data, exact_last_timestamp = FALSE, 
                       save_as = "rs_10_trajectories",
                       step_size = 10L)  

## compute measures 
mt_data <- mt_measures(mt_data, flip_threshold = 0, use = "rs_10_trajectories", 
                       save_as = "measures_10", hover_threshold = 2000 )
## idle time is zero
mt_data$measures_10$idle_time

We already discussed that a possible solution would be to do linear interpolation (see #7). However, we agreed that liner interpolation is better than constant interpolation whenever the mouse is moving.
Maybe it would be a solution to make a mix of linear and constant interpolation, with constant interpolation for time-differences that are above a certain threshold (this threshold should be 0 per default for backwards-compatibility and in my current example it would be set to 8).
What do you think?

Otherwise the idle time could be computed as

difft <- diff(timestamps)
sum(difft[difft != min(difft)])

But in my opinion this second solution fits very poorly into the current package design.

[PascalKieslich]

That's a good point. I really like the solution you suggested - to mix constant and linear interpolation, and to use a threshold for determining when which type of interpolation should be performed. I'll see whether I can implement something like this in mt_resample.

[PascalKieslich]

I have implemented this option in mt_resample (see 612c65f).

If you use mt_resample with the following arguments:

#### compute equidistant versions of the trajectories with timestamps every millisecond
mt_data <- mt_resample(mt_data, exact_last_timestamp = FALSE, 
                       save_as = "rs_1_trajectories",
                       step_size = 1, constant_interpolation = 8)  

## compute measures 
mt_data <- mt_measures(mt_data, flip_threshold = 0, use = "rs_1_trajectories", 
                       save_as = "measures_1", hover_threshold = 2000 )
## idle time is zero
mt_data$measures_1$idle_time

You will get an idle time of 309 ms.

Note that this is 2 ms shorter than what you suggested. However, I think this makes sense for the following reason:

Your example data looks as follows:

timestamps <- c( 0, 20, 311, 319, ...)
xpos       <- c(80, 80,  75,  65, ...)
ypos       <- c(46, 45,  44,  41, ...)

Internally, mt_resample would now add another timestamp for the first and second timestamp, as they exceed the constant interpolation threshold, which would result in the following data:

timestamps <- c( 0, 19, 20, 310, 311, 319, ...)
xpos       <- c(80, 80, 80,  80,  75,  65, ...)
ypos       <- c(46, 46, 45,  45,  44,  41, ...)

Afterwards, mt_resample performs resampling with linear interpolation, but the newly added timestamps ensure that a constant period is assumed for timestamps 0 to 19 and 20 to 310 (if you use a resampling step_size of 1 ms - if you use a different threshold, the time periods without movement might be reduced depending on where the new timestamps are located exactly).

However, as the position of the mouse slightly differs, the movement has to occur somewhere which currently reduces the idle time by 1 ms for the period for timestamps 19 to 20 and 310 to 311.

We could even think about a different criterion for adding the constant timestamps, e.g., instead of subtracting 1 from the next timestamp, we could subtract the step_size used for resampling. What do you think?

[sbrockhaus]

Thank you very much, that' s great!

[sbrockhaus]

You are right that the movement has to occur slightly before it can be captured and thus the idle time has to be shorter than I originally suggested. I had a closer look at the code now and I really like the solution.

Concerning your question on how much earlier the movement occurred: Is there a way to find out how fast the logging works? I would rather use constant_interpolation than step_size for the criterion, as this is the number that is related to what happend in the logging. Just thinking about it, I suggest to subtract half of the constant_interpolation putting

current_timestamps <- c(current_timestamps[1:j],current_timestamps[j+1]-round(constant_interpolation/2),current_timestamps[(j+1):nlogs])

for line https://github.com/PascalKieslich/mousetrap/blob/master/R/preprocess.R#L731

What do you think?

I also looked at the following graph for different settings:

#### compute equidistant versions of the trajectories with timestamps every millisecond
mt_data <- mt_resample(mt_data, exact_last_timestamp = FALSE, 
                       save_as = "rs_1_trajectories",
                       step_size = 1, constant_interpolation = 8)  
plot(mt_data$rs_1_trajectories[1, , ]) # interpolated movement
points(mt_data$trajectories[1, , ], col=2, pch =2) # logged movement 

The second point I would like to bring up is the minimal timespan that is counted to be part of idle time:
At the moment this minimal threshold is defined by step_size and constant_interpolation.
From my point of view the mouse is only idle, i.e., without movement, if it does not move for at least a quarter of a second (half a second?). Of course this threshold is arbitrary, but counting time as idle time if the mouse does not move for 8ms seems very short to me.

[PascalKieslich]

Thanks for your suggestions!

I like the idea of subtracting half the constant_interpolation threshold and have implemented this instead of subtracting 1 (cf. 423036f). The only change I made is that I do not round the subtraction - this is another advantage of your idea: we are now independent of the timestamps unit (i.e., timestamps could now also be in seconds with decimals).

Regarding the discussion about a minimal timespan that needs to be exceeded for the mouse to be counted as idle or hovering: I think the easiest way to solve this is to report the hover_time in addition to the number of hovers in mt_measures. This means that the threshold for hovers (specified using hover_threshold) is also used for a period to be added to the hover_time (cf. b1e29ab).

[sbrockhaus]

Agreed, using constant_interpolation/2 sounds like a good idea. I also like your solution of implementing hover_time as an extra measure. Thus, the user can set the threshold for times counting as hovering.

[sbrockhaus]

Just as a comment. We have discussed that hovering can be defined with and without the initiation time. In mousetrap, mt_measures() computes hovers and hover_time such that the initiation time is part of those hovering measures.
The following code example computes the hovering variables without the initiation time:

library(mousetrap)
library(dplyr)

## compute the mouse-measures with a hover thershold of 500
mt_example <- mt_measures(mt_example, flip_threshold = 0, use = "trajectories", 
                          save_as = "measures", hover_threshold = 500 )
# mt_example$measures

## compute for hovers and for hover_time the corresponding variables without the initiation time 
mt_example$measures <- mutate(mt_example$measures, 
                           hovers_ini = ifelse(initiation_time > 500, 
                                               hovers - 1, hovers) )

mt_example$measures <- mutate(mt_example$measures, 
                           hover_time_ini = ifelse(initiation_time > 500, 
                                                   hover_time - initiation_time, hover_time) )

Do you agree or did I get something wrong?

[PascalKieslich]

I agree, looks goods good to me!