[paper] writing #332

DominiqueMakowski · 2025-01-13T16:17:47Z

Let's get this bad boi out.

@mattansb @bwiernik what would you write about the two backends, emmeans and marginaleffects, their main differences etc.

Do they both rely on the delta method?

DominiqueMakowski · 2025-01-13T16:28:57Z

@strengejacke I think I'll need your help for the marginal section, the different types of marginalization etc. it's quite blurry in my head to be honest ^^

DominiqueMakowski · 2025-01-13T17:31:27Z

@easystats/core-team remaining bits:

add details about types of marginalization
Drawbacks/benefits of both backends
Polishing

mattansb · 2025-01-14T06:22:44Z

emmeans relies on the delta method for non-linear transformations when those are done as part of the estimation.

@DominiqueMakowski where do you want my input, and what should it include? I must admit I have 0 familiarity with modelbased - I use marginaleffects, emmeans and ggeffects directly for plotting and estimation, so I have no idea what would be relevant here.

DominiqueMakowski · 2025-01-14T08:25:44Z

I must admit I have 0 familiarity

Your torment over you can make the switch now 😛

Mostly if you could take a look at the technical details section that talks about the backends

strengejacke · 2025-01-14T10:14:14Z

I must admit I have 0 familiarity with modelbased

modelbased currently covers classical predict() (non-focal held constant at representative values) or estimates marginal means (non-focal averaged/weighted), the latter using emmeans resp. marginaleffects via estimate_means(). estimate_means() should return identical results for both emmeans and marginaleffects backends, i.e. currently we support marginaleffects by mimicking the emmeans-approach of marginalzation.

DominiqueMakowski · 2025-01-28T10:01:38Z

paper/paper.md

+
+The modelbased package simplifies the extraction of these effects, providing a clear interface to understand how different predictors interact with outcomes in various scenarios.
+
+[details about types of marginalization]


@strengejacke I think here we could mention our 2 main types of marginalization (essentially copypasta our nice docs on that)

I can start working/helping on this paper by mid of February, or earlier if it's just minor stuff.

DominiqueMakowski · 2025-01-28T10:02:38Z

paper/paper.md

+  The term "Least-Squares Means" was somewhat misleading as it suggested a method specific to least-squares estimation, hence its renaming to `emmeans` in 2016 to clarify its scope for a wider range of models including generalized linear models, mixed models, and Bayesian models.
+- `marginaleffects` (REF) was more recently introduced and also employs the delta method to approximate variance estimation. It is compatible with a wider range of models and allows for more flexibility in the specification of the marginal effects to be estimated.
+
+[What's the difference / benefits / drawbacks of using one or ther other?]


@mattansb any other interesting facts to mention here?

Reference for marginaleffects would be https://www.jstatsoft.org/article/view/v111i09

Arel-Bundock, V., Greifer, N., & Heiss, A. (2024). How to Interpret Statistical Models Using marginaleffects for R and Python. Journal of Statistical Software, 111(9), 1–32. https://doi.org/10.18637/jss.v111.i09

Oh there's a lot that I can add.

I think the biggest difference would be:

emmeans used a reference grid (that has by default a "complete" combination of all predictors) to generate expected values, by default conditioning on the mean of numeric values and marginalizing over categorical/binary variables, using a linear function of the model's coefficients (and v-cov matrix to get SEs) to give "predictions at the mean" (predictions for an average observation).

marginaleffects uses unit level predictions to generate two counterfactual values - the difference of which is then taken (with SEs computed using the delta method), and averaged across all units. By default, the original model frame is used.

Of course, emmeans can also the delta method and can build complex reference grids (that aren't actually "grid" like), and marginaleffects can also generate linear perditions at the mean.

Using the delta method is more computationally costly than using a linear combination (though marginaleffects is very efficient). Using linear combinations with orthogonal "grids" also often means that results from emmeans directly correspond to a models coefficients (which is a benefit for those who are used to looking at regressions tables to understand their models - this can be shown with an example).

...

Would you like me to add all of this in? Just some of this?

Would you like me to add all of this in? Just some of this?

Anything you think is relevant, but I think it'll be good to be quite thorough and detailed here as the inner workings of marginal stuff are not often clearly explained so having details is good!

Alright - I'll give it some time tomorrow.

and some edits

mattansb · 2025-01-29T14:14:27Z

I've added the expanded "technical details" section.

Some general comments:

Statement of need

I think the "statement of need" needs some restructuring, because the actual need it addresses is that emmeans and marginaleffects are sometimes hard to use - "the modelbased package is designed to be user-friendly, with a focus on simplicity and flexibility". That is, it doesn't provide any functionality in and of itself, but it provides a (very!) convenient wrapper with good defaults.

Predictions

"For generalized linear models (GLMs), while the distinction between prediction and confidence intervals do not apply" - why not? I teach prediction intervals for non-gaussian models.
See also:
https://fromthebottomoftheheap.net/2017/05/01/glm-prediction-intervals-i/
https://fromthebottomoftheheap.net/2017/05/01/glm-prediction-intervals-ii/

Marginal effects

The different "marginals" need to be stated better here - the "marginal" in "marginal means" is not the same "marginal" that is in "marginal effects".
In fact, I'm not sure, from reading the paper, if estimate_contrasts() gives marginal differences or average differences? Etc..

strengejacke · 2025-01-29T14:21:43Z

The different "marginals" need to be stated better here - the "marginal" in "marginal means" is not the same "marginal" that is in "marginal effects".
In fact, I'm not sure, from reading the paper, if estimate_contrasts() gives marginal differences or average differences? Etc..

Since I updated my mixed models slides for teaching after working on modelbased, maybe we can use some of the "definitions" I use on my slides (hopefully, these are accurate):

After fitting a model, it is useful generate model-based estimates. Usually, there are three quantities of interest:

marginal means: to better understand the relationship of predictors with the outcome. These are the average expected values, or adjusted predictions, of the response variable for different combinations of predictor values.
marginal effects: to evaluate the average strength of an effect (slope, average coefficient).
marginal contrasts: to look for (statistically significant) differences between groups, which helps us, for instance, analyzing “social inequalities”.

The key difference:

marginal means (or: adjusted predictions) return averages of the outcome variable, which allows you to say for instance “the average health score for a person at the age of sixty is 80 points”.
marginal effects return averages of coefficients, which allows you to say, for instance, “the average effect of age (or ageing) on the health score is an decrease 5 points”.
marginal contrasts return the average difference between two groups, typically indicated by different factor levels, which allows you to say “the average difference in health scores for lower and higher status groups at the same age is 15 points”.

Briefly:

marginal means show the association of the dependent variable and one or more independent variables;
marginal effects show you the average strength of a coefficient;
marginal contrasts show you average differences in the outcome between groups

strengejacke · 2025-01-29T14:21:56Z

marginal differences or average differences?

What would be the difference between these differences?

mattansb · 2025-01-29T15:28:27Z

library(marginaleffects)

mtcars$vs <- factor(mtcars$vs)
mtcars$gear <- factor(mtcars$gear)

mod <- glm(am ~ vs * hp * gear, 
           family = binomial(),
           data = mtcars)

# unit level differences
comparisons(mod, variables = "vs",
            type = "response")

# average differences (mean unit level differences)
avg_comparisons(mod, variables = "vs",
                type = "response")
#> 
#>  Estimate Std. Error      z Pr(>|z|)   S  2.5 % 97.5 %
#>    -0.166      0.195 -0.851    0.395 1.3 -0.547  0.216
#> 
#> Term: vs
#> Type:  response 
#> Comparison: 1 - 0
#> 




# marginal means over a regular grid
avg_predictions(mod, variables = "vs",
                # a regular grid
                newdata = datagrid(hp = mean, gear = levels),
                type = "response")

# marginal differences (differences between marginal means)
avg_predictions(mod, variables = "vs",
                # a regular grid
                newdata = datagrid(hp = mean, gear = levels),
                type = "response",
                hypothesis = ~ reference)
#> 
#>  Hypothesis Estimate Std. Error     z Pr(>|z|)   S  2.5 % 97.5 %
#>       1 - 0   -0.302      0.239 -1.26    0.207 2.3 -0.772  0.167
#> 
#> Type:  response
#>

mattansb · 2025-01-29T15:31:01Z

I think we should avoid using the ambiguous term "marginal" as much as possible with regard to effects/slopes/differences (probably still okay with regard to "marginal means" since that can only be the "average" kind, not the "conditional" kind).

strengejacke · 2025-01-29T15:45:15Z

(probably still okay with regard to "marginal means" since that can only be the "average" kind, not the "conditional" kind).

Unless you deal with mixed models, where you can have "marginal" and "conditional" effects/predictions/means (or whatever you'd like to call them).

strengejacke · 2025-01-29T15:48:43Z

Ok, I see, I think your distinction refers to the marginalize argument. I personally think, and maybe we should state that clearly, that we use "marginal" in terms of "marginalizing out" / "integrating out" / "averaging over/across". Then, marginal means/effects/contrasts makes sense. Whether the means/effects/contrasts refer to a specific person, an average person or the "broader population" (counterfactuals) depends on how we "marginalizing out" / "integrating out" / "averaging over/across".

strengejacke · 2025-01-29T18:33:36Z

Here are the modelbased equivalents:

library(modelbased)
library(marginaleffects)

mtcars$vs <- factor(mtcars$vs)
mtcars$gear <- factor(mtcars$gear)

mod <- glm(am ~ vs * hp * gear, family = binomial(), data = mtcars)

# average differences (mean unit level differences)
avg_comparisons(mod, variables = "vs")
estimate_contrasts(mod, "vs", marginalize = "population")

# marginal means over a regular grid
avg_predictions(mod, variables = "vs", newdata = datagrid(vs = levels, hp = mean, gear = levels))
estimate_means(mod, "vs", predict = "response")

# note the corrected prediction type
avg_predictions(mod, variables = "vs",
                newdata = datagrid(vs = levels, hp = mean, gear = levels),
                type = "invlink(link)")
estimate_means(mod, "vs")

# marginal differences (differences between marginal means)
avg_predictions(mod, variables = "vs",
                newdata = datagrid(hp = mean, gear = levels),
                type = "response",
                hypothesis = ~ reference)
estimate_contrasts(mod, "vs", comparison = ~ reference)

DominiqueMakowski · 2025-02-05T10:12:50Z

We should also probably add in the paper & in the docstrings some examples about when people might want to use the non-default option for estimate (e.g., "population"). It's not obvious to me, and if we don't clarify that we might end up with an option that nobody ever uses just because they don't know they should

strengejacke · 2025-02-05T11:21:01Z

We may add a vignette, I'd say, which is immediately online and independent from the package version on CRAN?

DominiqueMakowski · 2025-02-05T11:31:43Z

Yes, though that is probably independent from throwing a few words in about potential usecases. Did you encounter scenarios where you preferred to use estimate="population" instead of "average"?

strengejacke · 2025-02-05T11:38:10Z

It's definitely needed for G-computation (ATE/ATT/ATU - average treatment effects, for the treated and untreated), because you provide a subset of the data with only a specific level of your focal term, and then contrasts usually fail (you only have 1 level - nothing to compare). Then you need estimate = "population", which will create the counterfactuals (i.e. new observations with different levels for the focal terms) and then you have "enough" levels for comparison.

library(modelbased)

dat <- marginaleffects::get_dataset("lottery")
dat <- subset(dat, win_big == 1 | win == 0)
dat$win_big <- as.factor(dat$win_big)

mod <- lm(
  earnings_post_avg ~ win_big * (
    tickets + man + work + age + education + college + year +
      earnings_pre_1 + earnings_pre_2 + earnings_pre_3),
  data = dat
)

# ATE
estimate_contrasts(mod, "win_big")
#> Marginal Contrasts Analysis
#> 
#> Level1 | Level2 | Difference |   SE |          95% CI | t(280) |     p
#> ----------------------------------------------------------------------
#> 1      | 0      |      -5.56 | 2.35 | [-10.19, -0.94] |  -2.37 | 0.019
#> 
#> Variable predicted: earnings_post_avg
#> Predictors contrasted: win_big
#> Predictors averaged: tickets (2.6), man (0.69), work (0.78), age (53), education (14), college (0.47), year (2e+03), earnings_pre_1 (15), earnings_pre_2 (16), earnings_pre_3 (16)
#> p-values are uncorrected.

# ATT
estimate_contrasts(mod, "win_big", newdata = subset(dat, win_big == 1), estimate = "population")
#> Marginal Contrasts Analysis
#> 
#> Level1 | Level2 | Difference |   SE |          95% CI | t(280) |      p
#> -----------------------------------------------------------------------
#> 1      | 0      |      -9.49 | 1.82 | [-13.08, -5.90] |  -5.20 | < .001
#> 
#> Variable predicted: earnings_post_avg
#> Predictors contrasted: win_big
#> Predictors averaged: tickets (2.6), man (0.69), work (0.78), age (53), education (14), college (0.47), year (2e+03), earnings_pre_1 (15), earnings_pre_2 (16), earnings_pre_3 (16)
#> p-values are uncorrected.

# ATU
estimate_contrasts(mod, "win_big", newdata = subset(dat, win_big == 0), estimate = "population")
#> Marginal Contrasts Analysis
#> 
#> Level1 | Level2 | Difference |   SE |         95% CI | t(280) |     p
#> ---------------------------------------------------------------------
#> 1      | 0      |      -4.91 | 2.59 | [-10.01, 0.19] |  -1.90 | 0.059
#> 
#> Variable predicted: earnings_post_avg
#> Predictors contrasted: win_big
#> Predictors averaged: tickets (2.6), man (0.69), work (0.78), age (53), education (14), college (0.47), year (2e+03), earnings_pre_1 (15), earnings_pre_2 (16), earnings_pre_3 (16)
#> p-values are uncorrected.

# error when estimate not "population"
estimate_contrasts(mod, "win_big", newdata = subset(dat, win_big == 0))
#> Error: Sorry, calculating marginal contrasts failed with following error:
#> Items
#>   of 'old' not found in column names: [estimate]. Consider
#>   skip_absent=TRUE.
#> 
#> It seems that not all required levels of the focal
#>   terms are available in the provided data. If you want predictions
#>   extrapolated to a hypothetical target population, try setting
#>   `estimate="population".

(see also https://marginaleffects.com/chapters/gcomputation.html)

DominiqueMakowski · 2025-02-05T11:42:25Z

interesting

strengejacke · 2025-02-05T14:24:58Z

interesting

It's useful for casual inference and (randomized) trials. Imagine you have 2 or more time points, and some drop outs over time. You could impute missing data, but that ideally requires MCAR. Often, drop out is rather systematic, like lower educated or less healthy persons drop out. Then you can compensate for this drop out either by calculating inverse probability weights or use g-computation to calculate accurate ATE/ATT/ATU.

rempsyc · 2025-02-08T03:55:30Z

Please edit the rmd file from now on and not the md file

@strengejacke

@strengejacke feel free to put yourself wherever though

paper/paper.Rmd

strengejacke · 2025-02-09T13:56:55Z

paper/paper.Rmd

+
+## Marginal means and effects
+
+The concept of "marginal" in statistical modeling typically refers to the analysis of the effect of one or more "focus" variables while all other variables are held constant at specific values


I'm not quite happy with this first paragraph, because it focuses on "effects" (i.e. slopes), while we want to talk about relationship, comparisons/contrasts and effects/slopes. We should maybe focus on the "averaging" aspect (and maybe even move "contrasts" into an own section? - but that's not very necessary)

lol you know best here

strengejacke · 2025-02-09T14:01:03Z

paper/paper.Rmd

+**[TODO: details about types of marginalization]**
+
+
+## Technical details


Maybe we can shorten this section a bit? The paper becomes quite long, and I'm not sure if the technical details of the backend packages are helpful to understand what modelbased is and how it works, and what's the idea behind our package?

yes we can streamline a bit, although in general I like the idea of having slightly more substance in that paper compared to some of our other ones (that whole domain deserves it and it will be useful as a reference to readers)

strengejacke · 2025-02-09T14:01:36Z

paper/paper.Rmd

+
+Of the two, `emmeans` [@russell2024emmeans] is the more senior package and was originally known as `lsmeans` (for "Least-Squares Means").
+This term has been historically used to describe what are now more commonly referred to as "Estimated Marginal Means" or EMMs:
+predictions made over a regular grid—a counter-factual dataset containing all combinations of the categorical predictors in the model and typically the mean of numerical predictors.


I'm not convinced it's always a "counterfactual" data set ;-)

strengejacke · 2025-02-09T14:03:01Z

paper/paper.Rmd

+# Examples
+


examples look good to me!

@strengejacke

@strengejacke get's his wish!

DominiqueMakowski added 2 commits January 13, 2025 16:16

Update paper.md

e8fb785

Update paper.md

c242f84

DominiqueMakowski added 3 commits January 13, 2025 16:35

marginal

a52afbb

backends

0acb491

getting somewhere

0985a4f

DominiqueMakowski linked an issue Jan 13, 2025 that may be closed by this pull request

Contrasting slopes: differences between backends #333

Open

Merge branch 'main' into paper

16add3a

Merge branch 'main' into paper

3a919b1

DominiqueMakowski mentioned this pull request Jan 14, 2025

JOSS paper #127

Closed

strengejacke and others added 4 commits January 15, 2025 12:20

Merge branch 'main' into paper

51178f6

Merge branch 'main' into paper

c5f547b

Merge branch 'main' into paper

e1b7406

Update paper.md

ad7aa9b

DominiqueMakowski commented Jan 28, 2025

View reviewed changes

strengejacke and others added 2 commits January 28, 2025 11:22

Merge branch 'main' into paper

b4856fb

Technical details

5ef34a0

and some edits

Merge branch 'main' into paper

1c5018a

strengejacke and others added 2 commits February 5, 2025 14:19

Merge branch 'main' into paper

0f27e1e

Update paper.md

4919bbe

This comment was marked as outdated.

Sign in to view

Update paper.md

243e2d2

rempsyc added 2 commits February 7, 2025 17:43

Merge branch 'main' into paper

4acf169

rmd, references, pdf, etc.

2b61c3c

DominiqueMakowski added 3 commits February 9, 2025 08:30

Merge branch 'main' into paper

056d875

Affs: Daniel as senior

a62af74

@strengejacke feel free to put yourself wherever though

fix pdf

067592c

This comment was marked as resolved.

Sign in to view

fix yaml

e0c853e

strengejacke reviewed Feb 9, 2025

View reviewed changes

paper/paper.Rmd Show resolved Hide resolved

strengejacke added 4 commits February 9, 2025 14:24

statement of need

a93ea5a

key concepts

145ac6a

minor

2c9afbc

marginal effects

c4f70d3

strengejacke reviewed Feb 9, 2025

View reviewed changes

rempsyc and others added 2 commits February 9, 2025 11:31

turn link-citations to false to fix references

1a0df5c

Update paper.Rmd

81ceaec

@strengejacke get's his wish!


		The modelbased package simplifies the extraction of these effects, providing a clear interface to understand how different predictors interact with outcomes in various scenarios.

		[details about types of marginalization]


		## Marginal means and effects

		The concept of "marginal" in statistical modeling typically refers to the analysis of the effect of one or more "focus" variables while all other variables are held constant at specific values

		[TODO: details about types of marginalization]


		## Technical details

[paper] writing #332

Are you sure you want to change the base?

[paper] writing #332

Conversation

DominiqueMakowski commented Jan 13, 2025

DominiqueMakowski commented Jan 13, 2025

DominiqueMakowski commented Jan 13, 2025

mattansb commented Jan 14, 2025

DominiqueMakowski commented Jan 14, 2025

strengejacke commented Jan 14, 2025 • edited Loading

Choose a reason for hiding this comment

strengejacke Jan 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

strengejacke Jan 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattansb commented Jan 29, 2025

strengejacke commented Jan 29, 2025

strengejacke commented Jan 29, 2025

mattansb commented Jan 29, 2025 • edited Loading

mattansb commented Jan 29, 2025

strengejacke commented Jan 29, 2025

strengejacke commented Jan 29, 2025

strengejacke commented Jan 29, 2025

DominiqueMakowski commented Feb 5, 2025

strengejacke commented Feb 5, 2025

DominiqueMakowski commented Feb 5, 2025

strengejacke commented Feb 5, 2025

DominiqueMakowski commented Feb 5, 2025

This comment was marked as outdated.

strengejacke commented Feb 5, 2025

rempsyc commented Feb 8, 2025

This comment was marked as resolved.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

strengejacke commented Jan 14, 2025 •

edited

Loading

strengejacke Jan 28, 2025 •

edited

Loading

strengejacke Jan 28, 2025 •

edited

Loading

mattansb commented Jan 29, 2025 •

edited

Loading