-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy path01-Framework.Rmd
158 lines (85 loc) · 25.2 KB
/
01-Framework.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
---
output: html_document
editor_options:
chunk_output_type: console
---
# Theoretical background
Two main categories of topics are relevant to be discussed in this chapter, that taps into the theoretical background of this thesis: the background for the substantive example used and the features of the methodology to be applied. To this end, I will first introduce the characteristics of international large-scale assessments (ILSA) with the example of ICCS 2016 study and I will outline the operationalization of the main concepts analyzed in this research. Second, I will review literature on mixture models in order to identify and point out the most suitable technique and statistics to consider when analyzing this particular type of ILSA data. More specifically, mixed models will be introduced, particularly Latent Class Analysis which is a model-based approach to cluster individuals/cases into distinctive groups, called latent classes, based on their responses to a set of observed categorical variables [@goodman_exploratory_1974]. Different approaches to measurement invariance (MI) [@oecd_invariance_2019] in LCA will also be discussed.
## International Large-Scale Assessments
International Large-Scale Assessments (ILSAs) have been used to draw comparisons among countries on a variety of topics in education and, more broadly, for example, in adolescent development [@isac_indicators_2019]. These assessments can inform the public about influential factors on the micro and macro levels, foster interdisciplinary and international collaboration, and provide important data for studying the context and processes of education and development.
### IEA - ICCS 2016
The International Association for the Evaluation of Educational Achievement (IEA) International Civic and Citizenship Education Study (ICCS) produces internationally comparative data collected via student, school, and teacher questionnaires. Data from different waves of the ICCS survey is publicly available to researchers. The first time this study was applied was in 1999 to 28 countries and it was called CIVED, the second wave started using the name ICCS and was implemented in 2009 in 38 countries, the last study was performed in 2016 in 24 countries. The next cycle is scheduled for 2022 and 25 countries will participate. The main aim of the ICCS study is to investigate how young people are prepared to undertake their roles as citizens in a range of countries in the second decade of the 21st century [@wolfram_schulz_iccs_2018]. ICCS study evaluates students' knowledge and understanding of civics and citizenship, as well as their attitudes, perceptions, and activities related to civics and citizenship.
ICCS 2016 addressed the following research questions [@carsten_overview_2018]:
1. The way civic and citizenship education is implemented in participating countries, including the aim and principles for this learning area, the curricular approaches chosen to provide it, and changes and/or developments since 2009.
2. The extent of student's knowledge and understanding of civics and citizenship, and the factors associated with its variation across and within countries.
3. Student's current and expected future involvement in civic-related activities, their perceptions of their capacity to engage in these activities, and their perception of the value of civic engagement.
4. Student's belief about contemporary civic and civic issues in society, including those concerned with civic institutions, rules, and social principles (democracy, citizenship, and diversity), as well as their perceptions of their communities and threats to the world's future.
5. The ways in which schools organize civic and citizenship education, with a particular focus on general approaches, the processes used to facilitate civic engagement, interaction with their communities, and schools' and teacher's perceptions of the role of this learning area.
The 2016 study gathered data from more than 94000 students in 8th grade in about 3800 schools from 24 countries. Also, data from more than 37000 teachers in those schools and contextual data collected from school principals are included. An additional European questionnaire gathered data from almost 53000 students in 14 European countries and a Latin American student questionnaire from more than 25000 students from 5 Latin American countries. The student population is defined as students in 8th grade, in average 13.5 years of age in this study.
Of all 24 participant countries, 16 are from Europe, 5 are from Latin America, and 3 from Asia. In two of the participant countries, a sub-national entity participates. In Belgium, ICCS 2016 was implemented only in the Flemish education system and North Rhine-Westphalia state in Germany took part as a benchmarking participant.
The school's samples were designed as stratified two-stage cluster samples, first schools were randomly selected at the first stage with probability proportional to the size and intact classrooms were sampled at the second stage. Each country has a sample size of 150 schools approximately and a sample of students around 3000 and 4500. Additionally, around 15 teachers teaching the target grade from each school were sampled.
It is required at a minimum, that an analyst carrying out statistical analysis with ICCS have a good understanding of the conceptual foundations of ICCS [@wolfram_schulz_iccs_2018], the themes addressed, the populations targeted, the samples selected, the instruments used, and the production of the international database. All of this information is described in practical terms in the user guide [@kohler_iccs_2018]. Researchers using the database also need to make themselves familiar with the database structure and its variables.
For any secondary analysis of the study’s data such as the planned for this thesis, it must be taken into account the following [@kohler_iccs_2018] :
1. The unequal selection probabilities of the sampling units that necessitate the use of weights during computation of estimates;
2. The complex multistage cluster sample design that was implemented to ensure a balance between the research goals and cost-efficient operations; and
3. The rotated design of the civic knowledge test, wherein students completed only samples of the test items rather than the full set of test items.
The use of this type of data is tailored mainly to be used with specific statistical packages, that support complex survey design, computing population estimates and design-based standard errors.
### Students' endorsement of equal rights and opportunities
In the context of the ICCS study, young people’s attitudes toward equal rights for women and equal rights for ethnic/race groups were included in the first affective-behavioral domain of the assessment framework, attitudes towards the rights and responsibilities of groups in society. Given the profound challenges experienced by societies worldwide today and particularly the perceived backlash in tolerance toward minority groups and gender equality, information tapping into such attitudes in youth is highly relevant to both policy makers and researchers. At the policy level, for example, these topics are of priority for both the European educational policy agenda (reflected by the Strategic Framework for European Cooperation in Education and Training towards the European Education Area (2021 - 2030)) [@council_of_the_european_union_council_2021] as well as the UNESCO’s Education 2030 Agenda on sustainable goals [@unesco_education_2020]. For researchers, these are topics of high interest for gaining insights into the future of democratic societies and into information that can help inform educational practices targeting low levels of intolerance.
Indeed, several empirical studies were conducted in order to compare young people’s attitudes toward equal rights for women and equal rights for ethnic/race groups among several countries [@isac_indicators_2019; @isac_teaching_2018; @munck_measurement_2018; @dotti_sani_best_2017]. CFA, a variable-centered methodology, was performed to evaluate country invariance in [@isac_indicators_2019], which was achieved using additionally attitudes towards immigration indicators, this type of analysis is helpful to identify global behaviours of the whole population in the country and be able to compare them across countries. However, with this methodology, it is not possible to dig deeper into different subpopulations in each country and explain those average values.
The rise of engaged citizenship study [@hooghe_rise_2015] used latent class analysis to identify groups supporting duty-based and engaged citizenship norms, based on 12 indicators. The research included 21 countries, although no invariance analysis was evaluated to compare these groups across countries. The analysis was performed both in 1999 and in 2009 scales and the sizes of the groups in both periods are compared globally and for each country.
Variable-centered studies were able to show that, on average, young people in different countries, tend to display high levels of egalitarian attitudes (i.e., attitudes toward equal rights for women and equal rights for ethnic/race groups) and also point out countries that stand out (in terms of lowest and highest average scores). Such findings also indicated high variability across and within countries (especially when individual items were examined) pointing to the possibility that different sizeable groups of young people may show particular attitudinal configurations (e.g., egalitarian, non-egalitarian). However, empirical and appropriate person-centered studies (e.g., LCA) with evaluations of such expectations are absent in the literature. Not much research using these particular indicators has been performed in order to identify how many subpopulations can be identified in each country regarding young people's beliefs about equal rights and opportunities for different groups in society based on gender and ethnic/racial background. Neither how the patterns or behaviours of these subpopulations are composed and which of them are representatives and comparable across countries.
## Mixture models Latent Class Analysis
Parameters that describe a factor's effects in an ordinary generalized linear model are called fixed effects. Fixed effect applies to all categories of interest, gender, treatments, or any other manifest grouping variable. By contrast, random effects apply to a sample of all possible categories. GLM extends ordinary regression by allowing non normal responses and a link function of the mean. The generalized linear mixed model (GLMM) is a further extension that permits random and fixed effects in the linear predictor.
In this type of analysis, a contingency table is treated as a finite mixture of unobserved tables generated under a conditional independence structure at categories of a latent variable [@agresti_categorical_2013].
A GLMM with discrete data creates a mixture of linear predictor values using a latent variable, in this case, the unobserved random effect vector instead of being continuous and assumed to have a normal distribution, it is a qualitative mixture distribution.
### Person-centered approach
ANOVA, multiple regression, mixed models are variable-centered approaches that focus on relations among variables and assume that the sample studied come from a homogeneous population. Mixture models (finite mixture models) have taken the place of the framework for a person-center analytic approach. The difference between these two approaches is that the person-centered approach focuses on identifying unobserved subpopulations composed of similar individuals or cases and involves modeling a mixture outcome distribution.
The most common technique to find homogeneous groups based on observed variables is cluster analysis [@srivastava_methods_2002; @mukhopadhyay_complex_2016]. There are different methods that can be used to identify these groups of cases, but they are lacking in giving statistical indices and tests for the optimal number of clusters. The most common techniques to determine the best number of groups are based on tabular or graphical interpretation of the researcher.
The mixture of different distributions indicates population heterogeneity, this means that observations arise from a finite number of unobserved subpopulations in the target populations. This is key when we want to identify different patterns in the sample.
Latent class modeling defines a model for the probability of having a particular response pattern. This probability is a weighted average (or mixture) of the class-specific probabilities for these patterns. The item responses of an individual are mutually independent given the individual's class membership. Like cluster analysis, it is possible to assign individuals to the latent classes. The probability of belonging to a particular class given the responses (posterior class membership probability) can be obtained by the Bayes' rule [@michalos_latent_2014].
In summary, mixture modeling provides an important complement to traditional variable-centered analytical approaches. It offers the opportunity for researchers to identify unknown a priori homogeneous classes of individuals based on the measures of interest, examine the features of heterogeneity across the classes, evaluate the effects of covariates on the class membership, assess the relationship between the class membership and other outcomes, and study transitions between the latent class memberships over time. As a matter of fact, person-centered approaches and variable-centered approaches can be integrated into a general mixture modeling framework so that one can better understand the relationships among variables and the pattern of such relationships [@muthen_integrating_2000].
### Latent Class Analysis
As indicated previously a mixture model assumes that some of its parameters differ across unobserved subgroups, latent classes, or mixture components and particularly a latent class model is a mixture model for a set of categorical items. The first LCA approach was improved by [@lazarsfeld_latent_1968] and [@goodman_exploratory_1974].
A latent class model assumes the existence of a latent categorical variable such that the observed response variables are conditionally independent, given that variable. In other words, LCA can directly assess the theory that distinctive groups of people share specific attitudes [@hagenaars_applied_2002]. Depending on the response variable in the model, the analysis is called Latent Profile Analysis if it is continuous (Normal) and Latent Class Analysis if the response variable is categorical (Multinomial) [@muthen_integrating_2000]. In this study categorical outcomes will be used.
The goal of LCA is to identify unobserved subgroups based on similar response patterns. In contrast with cluster analysis, LCA is a model-based approach to classify. It identifies subgroups based on posterior membership probabilities rather than dissimilarity measures such as Euclidean or Mahalanobis distance [@michalos_latent_2014]. The general probability model underlying LCA allows for formal statistical procedures for determining the number of clusters, and more interpretable results stated in terms of probabilities.
LCA assumes conditional independence, that the observed categorical indicators are mutually independent once the categorical latent variable is conditioned out. Assuming the conditional independence, the joint probability of all observed indicator variables is described as [@wang_structural_2020]:
\begin{align}
P(u_1,u_1,...,u_Q)= \sum_{k=1}^K{P(C=k)P(u_1|C=k)P(u_2|C=k)...P(u_Q|C=k)} \label{eq01}
\end{align}
From Bayes' formula, the posterior probabilities for each individual to be in different classes are estimated as:
\begin{align}
P(C=k|u_1,u_2,..u_Q)=\frac{P(C=k)P(u_1|C=k)P(u_2|C=k)...P(u_Q|C=k)}{P(u_1,u_2,...,u_Q)} \label{eq02}
\end{align}
where $P(C=k)$ are the unconditional probabilities ($\sum_{k=1}^KP(C=k)=1$) and $P(u_Q|C=k)$ are the conditional probabilities.
The unconditional probabilities are latent class probabilities, and the average of the probabi-lities can be interpreted as the prevalence of latent class (relative frequency of class membership) or the proportion of the population expected to belong to a latent class [@muthen_finite_1999]. The conditional probabilities are conditional item-response probabilities, measurement parameters, representing the likelihood of endorsing specific characteristics of the observed items, given a specific class membership.
Conditional probabilities close to 1.0 indicate that members in the corresponding latent class endorse a category of the item; on the contrary, a very small probability indicates that they do not endorse the characteristic of the item. When a conditional item-response probability is close to $1/J$, where J is the number of categories in the item, the conditional probability is considered as random probability, thus the latent class membership is not predictive of the patterns of item response.
The conditional item-response probability is defined as in (\ref{eq03}) and (\ref{eq04}).
\begin{align}
P(u_q = u_{qj}|C=k) = \frac{1}{1+exp(-L_{jk})} \label{eq03}
\end{align}
\begin{align}
L_{jk}=ln(\frac{P_{jk}}{1-P_{jk}}) \label{eq04}
\end{align}
which is the logit for $u_{qj}$ given in latent class $k$. A logit of 0 means that the conditional item probability $P_{jk}=0.5$, when the logit takes an extreme value as -15 then $P_{jk}=0$. On the contrary, a logit with a positive extreme value 15, $P_{jk}=1$. These conditional item response probabilities provide information about how the latent classes differ from each other, for this reason, are used to define the estimated classes.
## Measurement invariance
Measurement invariance can be defined as a conditional independence property of the measurement model with respect to a set of sub-populations within the parent population (e.g., gender, countries, or time). Measurement invariance is an important prerequisite for using multi-indicator assessment instruments to examine group differences [@miranda_measurement_2018]. If the measurement properties of an instrument differ between observed groups (non-invariance), it is not possible to compare the differences between the groups [@bialowolski_influence_2016; @kankaras_measurement_2011; @michalos_latent_2014].
The importance of cross-countries comparisons is at the heart of large-scale international surveys. Instruments that assess subjective attitudes (e.g., attitudes towards migrants) and psychological traits such as perseverance, aims for the validity and comparability of survey results. Reflective latent constructs measured through self-reports, for example, are particularly affected by subtle linguistic differences in the translated questionnaires and by broader cultural differences. These may introduce variation in participants’ understanding of survey questions, and therefore in the relationship between their responses and the target latent construct. Similarly, when confronted with Likert items (*Strongly Agree*, *Agree*, *Disagree*, *Strongly disagree*), or with subjective rating scales (*on a scale from 1 to 10*), cultural norms may mediate the response process of participants. As a result, international surveys may fall short of their objective to facilitate comparisons across countries [@oecd_invariance_2019].
Multigroup Latent Class Analysis tests whether the number of classes is stable across the known groups and if the measurement part of the model is equivalent across these groups.
#### 1. Heterogeneous model
The first model to measure invariance is an *unconstrained model* in which the compared groups exhibit the same number of classes but the parameters defining those classes are freely estimated across groups. This means that assumes that the only similarity between groups is the number of classes identified and allows that response patterns (conditional probabilities) and class sizes vary among groups. Although the number of classes in all groups may be the same, direct between-country comparisons are not possible in this step because the meaning of latent classes may be substantially different. A completely unrestricted multi-group latent class model is equivalent to the estimation of a separate 3-class LC model for each group [@davidov_cross-cultural_2011].
#### 2. Partial homogeneity
The second model to test is the *semi-constrained model* in which equality constraints are imposed across the observed groups. The measurement part of the model (conditional probabilities) is restricted to be equal in all observed groups. For each group, the meaning of latent classes is invariant of the group and cross-group comparisons are meaningful. Yet, the size of the classes (i.e., the relative importance of each class) may still vary. Most applicable and desirable in cross-cultural studies [@davidov_cross-cultural_2011].
To test for invariance, the unconstrained model and the semi-constrained models are compared using the likelihood ratio test (LRT) and information criteria such as AIC, BIC, aBIC [@olivera-aguilar_assessing_2018]. A statistically significant LRT indicates a substantial decrease in model fit such that the semi-constrained model should be rejected. The model with the smallest AIC, BIC, aBIC value is selected as the best-fitting model.
If the semi-constrained model is rejected, this means, lower information criteria for the unconstrained model and LRT statistically significant, there is no evidence to assume measurement invariance. In this case, latent classes are characterized differently across the observed groups and differences in the prevalence of the profiles across the groups cannot be meaningfully determined.
For invariance to exist, the semi-constrained model should show a better fit to the data than the unconstrained model. Only after establishing the stability of the classes' definition across the different groups, it is possible to compare groups and evaluate the differences in class prevalence.
#### 3. Complete homogeneity
The stricter level of invariance is where all parameters are constrained across countries, and the prevalence of latent classes are restricted to be equal across groups (i.e., the percentage of individuals assigned to different classes will be equal in all groups). This last assumption will imply that the identified groups of individuals with similar scoring patterns are identical in all the groups with identical numbers of individuals assigned to each group [@davidov_cross-cultural_2011].
If the fully constrained model fits best, it can be concluded that there are no differences in how the known groups are represented in each profile. In contrast, if the fully constrained model is rejected but the semi-constrained model holds means that although the profiles have the same meaning in each group, there are differences in how the individuals are distributed across classes.
Meeting this last assumption ensures the highest level of cross-country comparability but may be difficult to achieve in cross-cultural studies.
When the number of observations per group is small, likelihood ratio tests have limited power; while with large groups, violations of invariance detected in such tests may be inconsequential for the substantive inferences [@oecd_invariance_2019]. The problem is compounded by the fact that in realistic settings (when violations of measurement invariance may be due to cultural or language specificities), the hypotheses are not independent, neither across items nor across groups.
#### Special cases
In case that the fully constrained and semi-constrained models are rejected, it can be studied if some latent classes are measurement invariant or not and/or if some items are invariant or not. That means that the assumption of measurement invariance can be relaxed for some classes and/or items. This can be done successively until one finds such a less restrictive model that does not fit the data worse than the totally unrestricted model.
If the number of classes differs between groups, then it can be tested whether the classes that are present in all groups are measurement invariant or not. This means if one group has 2 classes and another group has 3 classes a 3-class multigroup model with full measurement invariance can be tested, where the size of the third class in the first group would be zero. This strategy is recommended for a small number of groups. When a large number of groups is tested another strategy is recommended as it will take so much time in computing and compare all parameters to identify the ones that are invariant or that should be free.
The appropriate strategy for many groups is to conduct a multigroup LCA where full measurement invariance is assumed across the groups and that the number of classes does not differ across those groups. For this, the appropriate number of classes should be identified for each group and test if just one class is different between them, if this is rejected an extra class should be added to identify if there are two different classes among them. If the double of classes is found as the best fit means that none of the classes is measurement invariant, because different classes by country are needed. This strategy has the advantage of having a higher power to detect small classes that exist in several groups but that would not be detected in country-specific analysis because their size within a group might be too small.
\clearpage