ch_data_exploration.qmd

---
editor: 
  markdown: 
    wrap: sentence
---

# Data exploration and cleaning {#data-exp}

```{r setup}
#| code-summary: "Load packages"
library(tidyverse)
library(readxl)
library(DT)
library(ggplot2)
library(kableExtra)
```

## Import and compile data

The data set is located on GitHub in an `xlsx` format.
The data are split into different tabs based on year.

```{r readData}
#| code-summary: "Import data set"

dat <- read_excel("data/growthData.xlsx", 
    sheet = "y2017") |>
  # remove some measurements from October (FALL2). 
  # These were measured again in November 
  # along with the rest of the quadrats.
  select(-DateFALL2,
         -HeightFALL2,
         -ObserverFALL2) |>
  # adding a variable that keeps track of which tab the data comes from
  add_column(tab_year = 2017) |>
  bind_rows(
    read_excel("data/growthData.xlsx", 
    sheet = "y2018") |>
  mutate(DateFALL = as.Date(DateFALL, "%d.%m.%Y"),
         tab_year = 2018)
         ) |>
  bind_rows(
    read_excel("data/growthData.xlsx", 
    sheet = "y2019") |>
    mutate(
      DateSUMMER = as.Date(DateSUMMER, "%d/%m/%Y"),
      DateFALL = as.Date(DateFALL, "%d/%m/%y"),
      tab_year = 2019)
  ) |>
  bind_rows(
    read_excel("data/growthData.xlsx", 
    sheet = "y2020") |>
    mutate(DateSPRING = as.Date(DateSPRING, "%d/%m/%Y"),
           tab_year = 2020)
  ) |>
  bind_rows(
    read_excel("data/growthData.xlsx", 
    sheet = "y2021") |>
      # data had explicit NAs in cells that were read as text:
    mutate(across(starts_with("Height"), as.numeric),
           tab_year = 2021)
  ) |>
  bind_rows(
    read_excel("data/growthData.xlsx", 
    sheet = "y2022") |>
    mutate(across(starts_with("Height"), as.numeric),
           Plot_no = as.numeric(Plot_no),
           Pin_no = as.numeric(Pin_no),
           tab_year = 2022) 
  )
```

There are some warnings when actual NAs (text) are converted to real NAs.

Here's what the data looks like after I just row bind them:

```{r}
DT::datatable(dat |> slice_sample(n = 10))
```

</br>

I need to make this into a long format.
There are multiple date and height columns that I want to combine.
I will split the spring and fall data (ignoring the summer data) into separate sets, and then combine them again later.

```{r}
#| code-summary: "Turn into long format"

# Spring data
dat_spring <- dat |>
  select(-contains(c("FALL", "SUMMER", "diff"))) |>
  pivot_longer(
    cols = contains("Height"),
    values_to = "Height_cm",
    values_drop_na = T
  ) |>
  separate_wider_delim(name,
    delim = "_",
    names = c("temp", "pinPosition"),
    too_few = "align_start"
  ) |>
  filter(str_detect(temp, "Rejected", negate = T)) |>
  mutate(pinPosition = case_when(
    is.na(pinPosition) ~ "single",
    .default = pinPosition
  )) |>
  select(-temp) |>
  add_column(season = "spring") |>
  rename(date = DateSPRING)

# names(dat) [!names(dat) %in% names(dat_spring) ]
# unique(dat_spring$pinPosition)


dat_fall <- dat |>
  select(-contains(c("SPRING", "SUMMER", "diff"))) |>
  # introduces NA, where NA was originally as text
  mutate(across(starts_with("Height"), as.numeric)) |>
  pivot_longer(
    cols = contains("Height"),
    values_to = "Height_cm",
    values_drop_na = T
  ) |>
  separate_wider_delim(name,
    delim = "_",
    names = c("temp", "pinPosition"),
    too_few = "align_start"
  ) |>
  filter(str_detect(temp, "Rejected", negate = T)) |>
  mutate(pinPosition = case_when(
    is.na(pinPosition) ~ "single",
    .default = pinPosition
  )) |>
  select(-temp) |>
  add_column(season = "fall") |>
  rename(date = DateFALL)

# A check looking into the warnings introduced when turning
# Hieght columns from characters to numeric. All fine.
# ch <- dat |>
#   select(-contains(c("SPRING", "SUMMER", "diff"))) |>
#   drop_na(HeightFALL) |>
#   unite("link", c(ID, DateFALL))
# num <- dat |>
#   select(-contains(c("SPRING", "SUMMER", "diff"))) |>
#   mutate(across(starts_with("Height"), as.numeric)) |>
#   drop_na(HeightFALL) |>
#   unite("link", c(ID, DateFALL))
# ch |>
#   filter(!link %in% num$link) |>
#   View()


# Combining the two
dat_long <- dat_spring |>
  bind_rows(dat_fall) |>
  # merge comments and note columns
  unite("Remarks",
    contains(c("Comment", "Notes")),
    sep = ". ",
    na.rm = TRUE
  ) |>
  # merge observer columns
  unite("Observer",
    contains("Observer"),
    sep = ". ",
    na.rm = TRUE
  ) |>
  mutate(year = year(date))

rm(dat_fall, dat_spring)
```

The long data is `r nrow(dat_long)` rows.
This is too much to display as an html table on this web site, but here is a random sample of 50 rows just to illustrate.

```{r}
DT::datatable(dat_long |> slice_sample(n = 50))
```

</br>

## Looking for data problems

### Treatment

A closer look at the Treatment variable.

```{r}
dat_long |>
  count(Treatment)
```

How can Treatment be NA?
Turns out these are all new pins, and all from the fall.
I will delete the rows for a different reason [below](#id).

### Dates

#### Seasons

Are the dates entered correctly to match the seasons?

```{r fig-datesCheck}
#| fig.cap: "Distribution of measurement dates (months)."
dat_long |>
  ggplot() +
  geom_bar(aes(x = month(date)))+
  facet_grid(year(date)~season)
```

There are some measurements in the fall of 2020 that are wrong.
It turns out the month and day have been switched for plots 8 and 9:

```{r fig-dateCheck2}
#| fig.cap: "Checking inconsistency in date entries."
dat_long |>
  mutate(year = year(date)) |>
  filter(year == 2020,
         season == "fall") |>
  ggplot() +
  geom_bar(aes(x = date),
           color = "yellow",
           fill = "orange")+
  theme(axis.text.y = element_blank()) +
  facet_grid(Plot_no~.)
```

```{r}
#| eval: false
#| code-summary: "Confirming tht day and month have been switched."
dat_long |>
  filter(Plot_no %in% c(8, 9),
         season == "fall",
         year(date) == 2020) |>
  View()
```

I will reverse these now.

```{r}
#| code-summary: "Fix date mistake"
dat_long <- dat_long |>
  mutate(date = case_when(
    Plot_no %in% c(8,9) & date == date("2020-06-10") ~ date("2020-10-06"),
    .default = date
  ))
```

```{r fig-dateCheck3}
#| fig.cap: "Checking measurement dates after fixing mistake."
dat_long |>
  mutate(year = year(date)) |>
  filter(year == 2020,
         season == "fall") |>
  ggplot() +
  geom_bar(aes(x = date),
           color = "yellow",
           fill = "orange")+
  theme(axis.text.y = element_blank()) +
  facet_grid(Plot_no~.)
```

Then there were some errors with the dates in the spring of 2019 @fig-datesCheck.

```{r }
dat_long |>
  filter(year == 2019,
         season == "spring") |>
  mutate(month = month(date)) |>
  group_by(date, month) |>
  count()
```

Here I will assume that it is only the month that is wrong.
Fixing this now:

```{r}
#| code-summary: "Fix date mistake"
dat_long |>
  mutate(date = case_when(
    year == 2019 & date == date("2019-10-05") ~ date("2019-05-10"),
    .default = date
  )) |>
  filter(year == 2019,
         season == "spring") |>
  mutate(month = month(date)) |>
  group_by(date, month) |>
  count()

# OK

dat_long <- dat_long |>
  mutate(date = case_when(
    year == 2019 & date == date("2019-10-05") ~ date("2019-05-10"),
    .default = date
  ))
```

#### Year

```{r fig-distYears}
#| fig.cap: "Distribution of data points over the years and seasons"
dat_long |>
  ggplot() +
  geom_bar(aes(
    x = factor(year(date)),
    fill = season)) +
  labs(x = "Year")
```

I wonder why there are so relatively few observation in spring 2020.

```{r}
options(knitr.kable.NA = '')

dat_long |>
  mutate(tab_year = paste0("tab_year", tab_year)) |>
  group_by(tab_year, year) |>
  count() |>
  spread(year, n) |>
  knitr::kable() |>
  kable_paper(full_width = F)
```

Turns out some of the 2020 data (those added to the 2020 tab) was give the wrong date.

```{r}
#| code-summary: "Fix year mistake"

dat_long |>
  mutate(date = case_when(
    tab_year == 2020 & year == 2022 ~ date - years(2),
    .default = date),
         year = year(date))|>
  mutate(tab_year = paste0("tab_year", tab_year)) |>
  group_by(tab_year, year) |>
  count() |>
  spread(year, n) |>
  knitr::kable() |>
  kable_paper(full_width = F)
  
# OK

dat_long <- dat_long |>
  mutate(date = case_when(
    tab_year == 2020 & year == 2022 ~ date - years(2),
    .default = date),
         year = year(date))
```

```{r fig-newDistYears}
#| fig.cap: "Distribution of data points over the years and seasons after moving some datapoints from 2022 to 2020."
dat_long |>
  ggplot() +
  geom_bar(aes(
    x = year,
    fill = season)) +
  labs(x = "Year")
```

### Pin position

```{r}
dat_long |>
  group_by(year, season) |>
  count(pinPosition) |>
  spread(pinPosition, n) |>
  kable() |>
  kable_paper()
```

I want to combine E1 with E2, H1 with H2, etc.

In addition, in 2018 I want to combine all pinPositions.
In 2018, V (venstre) can be made equivalent W (west) , and H is E.

```{r}
#| output: false
#| code-summary: 'Create pinPosition2 by combination'
dat_long |>
  mutate(pinPosition2 = case_match(
    pinPosition,
    c("H1", "H2", "V1", "V2") ~ "single",
    c("E1", "E2") ~ "E",
    c("W1", "W2") ~ "W",
    .default = pinPosition
  )) |>
  count(pinPosition2)
 # OK. This variable can be aggregated across

dat_long <- dat_long |>
  mutate(pinPosition2 = case_match(
    pinPosition,
    c("H1", "H2", "V1", "V2") ~ "single",
    c("E1", "E2") ~ "E",
    c("W1", "W2") ~ "W",
    .default = pinPosition
  )) 
```

```{r}
dat_long |>
  group_by(year, season) |>
  count(pinPosition2) |>
  spread(pinPosition2, n) |>
  kbl() |>
  kable_paper(full_width=F)
```

### ID variable {#id}

A closer look at the ID variable.

Here's the time series for a single pin, measured from the west.

```{r}
dat_long |>
  filter(grepl("^8.14", ID),
         pinPosition == "W2") |>
  arrange(year) |>
  select(ID, Remarks, year, season, Height_cm) |>
  datatable()
```

It appears the pin was replaced in the fall of 2019.
I must assume that what is recorded there is the height of the new pin, and is therefore not compareable to the spring value that same year.
The *new* annotation only last one time, i.e. it is not repeated the next season.
In the spring of 2021 the wire seems to have been replaced again, and then again in the fall.
In the spring of 2022 it was replaced a forth(?) time, according to the remarks.
But this time the ID is unchanged.

Conclusion.
I will calculate the growth per season, if and only if the spring and fall height are recorded on the same pin/wire.
That means I can remove seasons where the fall measurements are done on new wires.
There are also some IDs that have the suffix *old*.
In these cases there should always be one measurement for the same date with the prefix *new,* meaning I can delete all *old* measurements from the spring heights.

```{r}
dat_long |>
  separate_wider_regex(ID,
    c(ID_num = "\\d+.\\d++", text_in_ID = "\\w+"), 
    too_few = "align_start", 
    cols_remove = F) |>
  count(text_in_ID)

```

I will split the ID column into a numerical `ID_num`, and a `text_in_ID` field that contains the suffix, if any (e.g. *old* or *new*).
In the same go I remove those with *direction* (NØ or SV) in the ID.

```{r}
dat_long <- dat_long|>
  separate_wider_regex(ID,
    c(ID_num = "\\d+.\\d++", text_in_ID = "\\w+"), 
    too_few = "align_start", 
    cols_remove = F) |>
  filter(!text_in_ID %in% c("NØ", "SV"))
```

Then I remove those that are *old* in spring.
I thought I could also remove those that are new in the fall, but turn out these could be labeled *new* in spring, and then that label is kept though that season (and reset again in the next season).
First, here is a view of the occurrences of *old* and *new* IDs.

```{r}
dat_long <- dat_long |>
  mutate(text_in_ID = case_when(
    is.na(text_in_ID) ~ "-",
    .default = text_in_ID
  )) 

dat_long |>
  group_by(year, season) |>
  count(text_in_ID) |>
  spread(text_in_ID, n) |>
  select(-"-") |>
  kbl() |>
  kable_paper(full_width = F)

```

I need to remove those that are *new* in fall and that don't have any value labeled *new* in spring (i.e. that they are truly new that fall, and that the label is not simply carried from the spring record).
Similarly, I want to remove those that are *old* in fall, and don't have any values labeled *old* in the preceding spring.
First I create a link variable for the occurrences that I want to match against.

```{r}
newInSpring <- dat_long |>
  filter(season == "spring" & text_in_ID == "new") |>
  mutate(link = paste(ID_num, tab_year, pinPosition, sep= "_")) |>
  pull(link)

oldInFall <- dat_long |>
  filter(season == "fall" & text_in_ID == "old") |>
  mutate(link = paste(ID_num, tab_year+1, pinPosition, sep= "_")) |>
  pull(link)
```

Then I remove some records based on this link variable.

```{r}
dat_long <- dat_long |>
  mutate(link = paste(ID_num, tab_year, pinPosition, sep= "_")) |>
  filter(
    ifelse(text_in_ID == "new" & !link %in% newInSpring,
      season != "fall",
      TRUE),
    ifelse(text_in_ID == "old" & !link %in% oldInFall,
      season != "spring",
      TRUE)) |>
  select(-link)

dat_long |>
  group_by(year, season) |>
  count(text_in_ID) |>
  spread(text_in_ID, n) |>
  select(-"-") |>
  kbl() |>
  kable_paper(full_width = F)
```

There are some ID numbers that look a bit weird:

```{r}
dat_long |>
  filter(nchar(ID_num) > 5) |>
  distinct(ID_num)
```

`ID_num` is a character column, but we can have it as numeric and round to two decimal points.

```{r}
#| code-summary: 'fix weird ID_num'
dat_long <- dat_long |>
  mutate(ID_num = round(as.numeric(ID_num), 2))
```

### Species

The _Shapgnum_ species identities were recorded for each pin/wire from 2020 and onwards.

```{r}
#dat_long |>
#  count(Species_W) |>
#  datatable()
#  
dat_long |>
  count(Species_E) |>
  datatable()
```

```{r}
#| code-summary: 'Standardise factor levels'
dat_long <- dat_long |>
  mutate(
    Species_W = case_match(
      Species_W,
      "rub/pap" ~ "pap/rub",
      c("med (bal)", "med/bal") ~ "bal/med",
      .default = Species_W),
    Species_E = case_match(
      Species_E,
      "rub/pap" ~ "pap/rub",
      c("med (bal)", "med/bal") ~ "bal/med",
      "pap/med" ~ "med/pap",
      .default = Species_E)
    )
```

```{r}
dat_long |>
  group_by(year) |>
  count(Species_W, Species_E) |>
  ungroup() |>
  slice_head(n = 5) |>
  kbl()|>
  kable_paper(full_width=F)
```

> Species were recorded from year 2020.

### Seasonal growth

The main response variable will be seasonal growth, from one spring to the following fall.

```{r}
dat_season <- dat_long |>
  pivot_wider(
    id_cols = c(
      ID_num,
      Plot_no,
      Pin_no,
      pinPosition2,
      Treatment,
      year
    ),
    names_from = season,
    values_from = Height_cm,
    values_fn = mean,
    unused_fn = list(
      Species_W = list,
      Species_E = list
    )
  ) |>
  mutate(growth_cm = spring - fall)

dat_season |>
  pull(growth_cm) |>
  summary()
```

According to this, the peat grows 5 mm per season.
But there are some of NAs.
These arise when there is no measurement done for one of the seasons.
Let's look is there are any patterns in these NA's.

```{r}
dat_season |>
  filter(is.na(growth_cm)) |>
  group_by(year, Treatment) |>
  count() |>
  pivot_wider(names_from = Treatment, values_from = n) |>
  kbl() |>
  kable_paper()
```

-   EDGE has a large number of NA's in 2022.
    That is because the edge treatment was dropped.

-   There is no big increase over time (e.g. due to more wires being replaced).

-   The NA seem to be mostly random/evenly spread out


Let's also look at the distribution of the actual data across 
Treatments and years (i.e. after removing the NA's).

```{r}
dat_season |>
  filter(!is.na(growth_cm)) |>
  group_by(year, Treatment) |>
  count() |>
  pivot_wider(names_from = Treatment, values_from = n) |>
  kbl() |>
  kable_paper()
```

Edge, Hollow and hummocks are only included from 2021.

Removing the NA's from the dataset:
```{r}
#| code-summary: 'Removing NAs'
dat_season <- dat_season |>
  filter(!is.na(growth_cm))
```

### Unnest species list columns

The species identities are preserved in the dataset as list columns, where one to six species names (abbreviations) are combined.
It can be four records if there are two from spring (e.g. W1 and W2) and two from the fall.

I will first count the number of unique species names in the list columns, and then, if that returns a single species (which is not NA), I will extract the unique species name.
If the species differed between the records, I will add NAs.
Below is a proof of concept on a smaller sample of the data.

```{r proofConcept}
#| eval: false
#| code-summary: 'Proof of concept'

# draw a subset of data with four unique strata
temp <- dat_long |>
  filter(ID_num %in% c(2.80, 3.80),
         year == 2021)

temp <- temp |>
  mutate(Species_W = case_when(
# In one strata I will add a second species
    ID_num == 2.80 & season == "spring" & pinPosition == "W1" ~ "fake species",
# In the second strata I will add an NA
    ID_num == 2.80 & season == "spring" & pinPosition == "E1" ~ NA,
# In the third strata will add all NAs
    ID_num == 3.80 & pinPosition2 == "W" ~ NA,
# The fourth strata will have the same four species repeated 
    .default = Species_W
  ))

# Pivot wider, as I did above when I created dat_season
temp2 <- temp |> 
  pivot_wider(
    id_cols = c(
      ID_num,
      Plot_no,
      Pin_no,
      pinPosition2,
      Treatment,
      year
    ),
    names_from = season,
    values_from = Height_cm,
    values_fn = mean,
    unused_fn = list(
      Species_W = list,
      Species_E = list
    )
  ) |>
  mutate(growth_cm = spring - fall)

# Testing the method
temp2 |>
  rowwise() |>
  mutate(sameSpecies = length(unique(Species_W))<2,
         species = case_when(
           isTRUE(sameSpecies) ~ Species_W[[1]],
           .default = NA
         )) |>
  select(ID_num,
         pinPosition2,
         Species_W,
         sameSpecies,
         species) |>
  datatable()
# Looks OK
```

```{r}
#| code-summary: 'Extract species name if unique within the same growing season.'
dat_season <- dat_season |>
  rowwise() |>
  mutate(
    sameSpecies_W = length(unique(Species_W))<2,
    sameSpecies_E = length(unique(Species_E))<2,
    species_W = case_when(
      isTRUE(sameSpecies_W) ~ Species_W[[1]],
      .default = NA),
    species_E = case_when(
      isTRUE(sameSpecies_E) ~ Species_E[[1]],
      .default = NA)
         )
```

I'm pretty sure this has worked, since I tried in on the synthetic data, but there are just four cases of the species being different across the aggregated strata (mainly this would imply that there had been a species change during the growing season).

```{r}
dat_season |>
  count(sameSpecies_E, sameSpecies_W)
```

### Removing some plots

Plot 28 should be Hollow (all the time). This was a data punching mistake.
Also, plots 29 and 30 should be excluded all together.
Those are the two Edge plots. The _Edge_ treatment was discontinued.

```{r}
#| code-summary: 'Removing plots 28-30'
dat_season <- dat_season |>
  mutate(Treatment = case_when(
    Plot_no == 28 ~ "HOLLOW",
    .default = Treatment
  )) |>
  filter(!Plot_no %in% c(29,30)) |>
  select(-Species_W,
         -Species_E)
```



I came across the case by chance.
Let's see it there are more cases like this.
(This kind of problem can be avoided by having hierarchical datasets also far data field sheets and data punching).

Here is a little code to check for more than one treatment for the same plot ID.
```{r}
(dups <- dat_season |>
  group_by(Treatment) |>
  count(ID_num) |>
  ungroup() |>
  group_by(ID_num) |>
  count() |>
  filter(n > 1) |>
  pull(ID_num))
```
There were none of these cases.


### Final check

```{r fig-yearEnd}
#| fig-cap: 'Distribution of data points across years'
dat_season |>
  ggplot() +
  geom_bar(aes(
    x = year),
    fill = "darkkhaki") +
  labs(x = "Year")
```

The distribution across years looks much better now.

```{r fig-yearAndTrt}
#| fig-cap: 'Distribution of data points across years, conditioned on Treatment'

dat_season |>
  ggplot() +
  geom_bar(aes(
    x = year),
    fill = "darkkhaki") +
  labs(x = "Year") +
  facet_wrap(.~Treatment)
```

The Edge treatment is now all gone.

```{r fig-commomSPecies}
#| fig-cap: 'Distribution of data points across taxa'

dat_season |>
  pivot_longer(
    cols = starts_with("species"),
    values_to = "Species"
    ) |>
  filter(!is.na(Species),
         Species != "NA",
         Species != "dead") |>
  count(Species) |>
  arrange(n) |>
  mutate(Species = fct_inorder(Species)) |>
  ggplot() +
  geom_col(aes(x = Species, y = n),
           fill = "darkkhaki") +
  coord_flip()
```

*Shagnum papilosum, S. medium, S. tennuis* and *S. rubellum* are the four most common species in the dataset.

```{r}
dat_season |>
  group_by(year, Treatment) |>
  count() |>
  pivot_wider(names_from = Treatment, values_from = n) |>
  kbl() |>
  kable_paper()
```

The distribution of data points across treatments now looks much better as well.