Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nginx: if root is in Nix store, use path's hash as ETag #48337

Merged
merged 6 commits into from
Apr 18, 2019
Merged

nginx: if root is in Nix store, use path's hash as ETag #48337

merged 6 commits into from
Apr 18, 2019

Conversation

lukateras
Copy link
Member

Motivation for this change

Resolves #25485. Usage example:

$ realpath /var/www
/nix/store/wnrhnnpdj3x50j5xz38zp1qxs1ygwccw-site
$ curl --head localhost
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 28 Sep 2018 06:09:25 GMT
Content-Type: text/html
Content-Length: 50
Last-Modified: Thu, 01 Jan 1970 00:00:01 GMT
Connection: keep-alive
ETag: "wnrhnnpdj3x50j5xz38zp1qxs1ygwccw"
Accept-Ranges: bytes

cc @Akii @chris-martin @domenkozar

P.S. I didn't commit patch into Nixpkgs repo because that seems to be the preference, but it would be nice to have the patch reviewed. Tell me if I should commit it anyway.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: nginx

Partial log (click to expand)

test -d '/nix/store/r4bzjgc4y1k1f27p2g4a3660pydg6m11-nginx-1.14.0/logs' \
        || mkdir -p '/nix/store/r4bzjgc4y1k1f27p2g4a3660pydg6m11-nginx-1.14.0/logs'
make[1]: Leaving directory '/build/nginx-1.14.0'
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/r4bzjgc4y1k1f27p2g4a3660pydg6m11-nginx-1.14.0
shrinking /nix/store/r4bzjgc4y1k1f27p2g4a3660pydg6m11-nginx-1.14.0/bin/nginx
strip is /nix/store/p9akxn2sfy4wkhqdqa3li97pc6jaz3r1-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/r4bzjgc4y1k1f27p2g4a3660pydg6m11-nginx-1.14.0/bin
patching script interpreter paths in /nix/store/r4bzjgc4y1k1f27p2g4a3660pydg6m11-nginx-1.14.0
checking for references to /build in /nix/store/r4bzjgc4y1k1f27p2g4a3660pydg6m11-nginx-1.14.0...

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: nginx

Partial log (click to expand)

test -d '/nix/store/vkdh0nnyfg6j9fdmky7x2img0vgx0jzz-nginx-1.14.0/logs' \
        || mkdir -p '/nix/store/vkdh0nnyfg6j9fdmky7x2img0vgx0jzz-nginx-1.14.0/logs'
make[1]: Leaving directory '/build/nginx-1.14.0'
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/vkdh0nnyfg6j9fdmky7x2img0vgx0jzz-nginx-1.14.0
shrinking /nix/store/vkdh0nnyfg6j9fdmky7x2img0vgx0jzz-nginx-1.14.0/bin/nginx
strip is /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/vkdh0nnyfg6j9fdmky7x2img0vgx0jzz-nginx-1.14.0/bin
patching script interpreter paths in /nix/store/vkdh0nnyfg6j9fdmky7x2img0vgx0jzz-nginx-1.14.0
checking for references to /build in /nix/store/vkdh0nnyfg6j9fdmky7x2img0vgx0jzz-nginx-1.14.0...

@GrahamcOfBorg
Copy link

Success on x86_64-darwin (full log)

Attempted: nginx

Partial log (click to expand)

        || mkdir -p '/nix/store/nc4hcpmi05q8wb0f4hp56q5cn72lip4d-nginx-1.14.0/logs'
test -d '/nix/store/nc4hcpmi05q8wb0f4hp56q5cn72lip4d-nginx-1.14.0/html' \
        || cp -R html '/nix/store/nc4hcpmi05q8wb0f4hp56q5cn72lip4d-nginx-1.14.0'
test -d '/nix/store/nc4hcpmi05q8wb0f4hp56q5cn72lip4d-nginx-1.14.0/logs' \
        || mkdir -p '/nix/store/nc4hcpmi05q8wb0f4hp56q5cn72lip4d-nginx-1.14.0/logs'
make[1]: Leaving directory '/private/tmp/nix-build-nginx-1.14.0.drv-0/nginx-1.14.0'
post-installation fixup
strip is /nix/store/53nysl8bqwxihwzs1pgwka20nf8mbvlp-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/nc4hcpmi05q8wb0f4hp56q5cn72lip4d-nginx-1.14.0/bin
patching script interpreter paths in /nix/store/nc4hcpmi05q8wb0f4hp56q5cn72lip4d-nginx-1.14.0

@grahamc
Copy link
Member

grahamc commented Oct 14, 2018 via email

@grahamc
Copy link
Member

grahamc commented Oct 14, 2018 via email

@lukateras
Copy link
Member Author

Yes, I believe so. Browsers will use ETag as a validation token to ensure content didn't change, and that maps onto Nix store semantics very well. There is also some syntax involved and that's conformant too (double quotes, optionally weak prefix but it doesn't apply here).

This patch seems to generate RFC 7232 compliant tags.

Additionally, Nginx will also properly handle this ETag and respond with 304 Not Modified if client sends HEAD request with If-None-Match: "<ETag>" header.

@lukateras
Copy link
Member Author

Included patch into the PR.

@lukateras
Copy link
Member Author

lukateras commented Oct 14, 2018

Come to think of it, there is a concern: if Nix store path contains impure symlinks outside of Nix store, and Nginx has disable_symlinks off; set, this patch will cause such symlinked path to be cached indefinitely.

Meaning, my assumption was that if root is in Nix store, it will only contain symlinks (if any) that point inside of Nix store. Is that a valid assumption to make?

Situation is complicated by the fact that ngx_http_set_etag function doesn't really have access to path of the file that it's going to serve, only to its last modified time and content length.

If this is undesirable, I think I have an idea how to make it work while keeping patch relatively small and contained: make sure that last modified time is 1970-01-01 00:00:00 before checking if root is in Nix store.

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: nginx

Partial log (click to expand)

test -d '/nix/store/cc32dm6gg32a5wcy9lhbv6r9fvr9wmn7-nginx-1.14.0/logs' \
        || mkdir -p '/nix/store/cc32dm6gg32a5wcy9lhbv6r9fvr9wmn7-nginx-1.14.0/logs'
make[1]: Leaving directory '/build/nginx-1.14.0'
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/cc32dm6gg32a5wcy9lhbv6r9fvr9wmn7-nginx-1.14.0
shrinking /nix/store/cc32dm6gg32a5wcy9lhbv6r9fvr9wmn7-nginx-1.14.0/bin/nginx
strip is /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/cc32dm6gg32a5wcy9lhbv6r9fvr9wmn7-nginx-1.14.0/bin
patching script interpreter paths in /nix/store/cc32dm6gg32a5wcy9lhbv6r9fvr9wmn7-nginx-1.14.0
checking for references to /build in /nix/store/cc32dm6gg32a5wcy9lhbv6r9fvr9wmn7-nginx-1.14.0...

@GrahamcOfBorg
Copy link

Success on x86_64-darwin (full log)

Attempted: nginx

Partial log (click to expand)

        || mkdir -p '/nix/store/cj2njbxkxqmk4j10iz0pd18maav79n8y-nginx-1.14.0/logs'
test -d '/nix/store/cj2njbxkxqmk4j10iz0pd18maav79n8y-nginx-1.14.0/html' \
        || cp -R html '/nix/store/cj2njbxkxqmk4j10iz0pd18maav79n8y-nginx-1.14.0'
test -d '/nix/store/cj2njbxkxqmk4j10iz0pd18maav79n8y-nginx-1.14.0/logs' \
        || mkdir -p '/nix/store/cj2njbxkxqmk4j10iz0pd18maav79n8y-nginx-1.14.0/logs'
make[1]: Leaving directory '/private/tmp/nix-build-nginx-1.14.0.drv-0/nginx-1.14.0'
post-installation fixup
strip is /nix/store/53nysl8bqwxihwzs1pgwka20nf8mbvlp-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/cj2njbxkxqmk4j10iz0pd18maav79n8y-nginx-1.14.0/bin
patching script interpreter paths in /nix/store/cj2njbxkxqmk4j10iz0pd18maav79n8y-nginx-1.14.0

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: nginx

Partial log (click to expand)

        || mkdir -p '/nix/store/f6zgnxw75r98yr723kix1a3x8v7nmdr9-nginx-1.14.0/logs'
make[1]: Leaving directory '/build/nginx-1.14.0'
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/f6zgnxw75r98yr723kix1a3x8v7nmdr9-nginx-1.14.0
shrinking /nix/store/f6zgnxw75r98yr723kix1a3x8v7nmdr9-nginx-1.14.0/bin/nginx
strip is /nix/store/p9akxn2sfy4wkhqdqa3li97pc6jaz3r1-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/f6zgnxw75r98yr723kix1a3x8v7nmdr9-nginx-1.14.0/bin
patching script interpreter paths in /nix/store/f6zgnxw75r98yr723kix1a3x8v7nmdr9-nginx-1.14.0
checking for references to /build in /nix/store/f6zgnxw75r98yr723kix1a3x8v7nmdr9-nginx-1.14.0...
/nix/store/f6zgnxw75r98yr723kix1a3x8v7nmdr9-nginx-1.14.0

@domenkozar
Copy link
Member

I think whatever we do here, it shouldn't be the default. We can make it convenient to turn on Nix+Nginx behaviour for those that understand what it does.

@lukateras
Copy link
Member Author

lukateras commented Oct 16, 2018

Why? If Nix store symlinks are handled properly, this shouldn't cause any regressions whatsoever. Since this Nginx is Nixpkgs-distributed, it seems that it would make sense for it to correctly work with Nix.

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: nginx

Partial log (click to expand)

test -d '/nix/store/3hlqw53imrs5l7v34gj4i6f0xi2sjq09-nginx-1.14.0/logs' \
        || mkdir -p '/nix/store/3hlqw53imrs5l7v34gj4i6f0xi2sjq09-nginx-1.14.0/logs'
make[1]: Leaving directory '/build/nginx-1.14.0'
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/3hlqw53imrs5l7v34gj4i6f0xi2sjq09-nginx-1.14.0
shrinking /nix/store/3hlqw53imrs5l7v34gj4i6f0xi2sjq09-nginx-1.14.0/bin/nginx
strip is /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/3hlqw53imrs5l7v34gj4i6f0xi2sjq09-nginx-1.14.0/bin
patching script interpreter paths in /nix/store/3hlqw53imrs5l7v34gj4i6f0xi2sjq09-nginx-1.14.0
checking for references to /build in /nix/store/3hlqw53imrs5l7v34gj4i6f0xi2sjq09-nginx-1.14.0...

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: nginx

Partial log (click to expand)

test -d '/nix/store/gcgzsqbr0shcp55p7lcp02wm24hwwg07-nginx-1.14.0/logs' \
        || mkdir -p '/nix/store/gcgzsqbr0shcp55p7lcp02wm24hwwg07-nginx-1.14.0/logs'
make[1]: Leaving directory '/build/nginx-1.14.0'
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/gcgzsqbr0shcp55p7lcp02wm24hwwg07-nginx-1.14.0
shrinking /nix/store/gcgzsqbr0shcp55p7lcp02wm24hwwg07-nginx-1.14.0/bin/nginx
strip is /nix/store/p9akxn2sfy4wkhqdqa3li97pc6jaz3r1-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/gcgzsqbr0shcp55p7lcp02wm24hwwg07-nginx-1.14.0/bin
patching script interpreter paths in /nix/store/gcgzsqbr0shcp55p7lcp02wm24hwwg07-nginx-1.14.0
checking for references to /build in /nix/store/gcgzsqbr0shcp55p7lcp02wm24hwwg07-nginx-1.14.0...

@GrahamcOfBorg
Copy link

Success on x86_64-darwin (full log)

Attempted: nginx

Partial log (click to expand)

        || mkdir -p '/nix/store/pys7fcv41sbyj3qqr27lmab3k9p2cyvk-nginx-1.14.0/logs'
test -d '/nix/store/pys7fcv41sbyj3qqr27lmab3k9p2cyvk-nginx-1.14.0/html' \
        || cp -R html '/nix/store/pys7fcv41sbyj3qqr27lmab3k9p2cyvk-nginx-1.14.0'
test -d '/nix/store/pys7fcv41sbyj3qqr27lmab3k9p2cyvk-nginx-1.14.0/logs' \
        || mkdir -p '/nix/store/pys7fcv41sbyj3qqr27lmab3k9p2cyvk-nginx-1.14.0/logs'
make[1]: Leaving directory '/private/tmp/nix-build-nginx-1.14.0.drv-0/nginx-1.14.0'
post-installation fixup
strip is /nix/store/53nysl8bqwxihwzs1pgwka20nf8mbvlp-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/pys7fcv41sbyj3qqr27lmab3k9p2cyvk-nginx-1.14.0/bin
patching script interpreter paths in /nix/store/pys7fcv41sbyj3qqr27lmab3k9p2cyvk-nginx-1.14.0

@lukateras
Copy link
Member Author

lukateras commented Oct 16, 2018

I've pushed two commits: one adds missing realpath error check (@gebner, thanks!) and another one fixes impure symlink handling (see #48337 (comment)).

This patch shouldn't cause any regressions anymore.

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: nginx

Partial log (click to expand)

test -d '/nix/store/d1mpq32i9qyvpiw1x4qq3dj2k3w6r0w6-nginx-1.14.0/logs' \
        || mkdir -p '/nix/store/d1mpq32i9qyvpiw1x4qq3dj2k3w6r0w6-nginx-1.14.0/logs'
make[1]: Leaving directory '/build/nginx-1.14.0'
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/d1mpq32i9qyvpiw1x4qq3dj2k3w6r0w6-nginx-1.14.0
shrinking /nix/store/d1mpq32i9qyvpiw1x4qq3dj2k3w6r0w6-nginx-1.14.0/bin/nginx
strip is /nix/store/p9akxn2sfy4wkhqdqa3li97pc6jaz3r1-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/d1mpq32i9qyvpiw1x4qq3dj2k3w6r0w6-nginx-1.14.0/bin
patching script interpreter paths in /nix/store/d1mpq32i9qyvpiw1x4qq3dj2k3w6r0w6-nginx-1.14.0
checking for references to /build in /nix/store/d1mpq32i9qyvpiw1x4qq3dj2k3w6r0w6-nginx-1.14.0...

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: nginx

Partial log (click to expand)

test -d '/nix/store/7nsm2sz5q5xilp0kqhzqa33a99n2nnr7-nginx-1.14.0/logs' \
        || mkdir -p '/nix/store/7nsm2sz5q5xilp0kqhzqa33a99n2nnr7-nginx-1.14.0/logs'
make[1]: Leaving directory '/build/nginx-1.14.0'
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/7nsm2sz5q5xilp0kqhzqa33a99n2nnr7-nginx-1.14.0
shrinking /nix/store/7nsm2sz5q5xilp0kqhzqa33a99n2nnr7-nginx-1.14.0/bin/nginx
strip is /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/7nsm2sz5q5xilp0kqhzqa33a99n2nnr7-nginx-1.14.0/bin
patching script interpreter paths in /nix/store/7nsm2sz5q5xilp0kqhzqa33a99n2nnr7-nginx-1.14.0
checking for references to /build in /nix/store/7nsm2sz5q5xilp0kqhzqa33a99n2nnr7-nginx-1.14.0...

@GrahamcOfBorg
Copy link

Success on x86_64-darwin (full log)

Attempted: nginx

Partial log (click to expand)

        || mkdir -p '/nix/store/cpq2ca2652p6icr2xxprlk49qqgnqj8d-nginx-1.14.0/logs'
test -d '/nix/store/cpq2ca2652p6icr2xxprlk49qqgnqj8d-nginx-1.14.0/html' \
        || cp -R html '/nix/store/cpq2ca2652p6icr2xxprlk49qqgnqj8d-nginx-1.14.0'
test -d '/nix/store/cpq2ca2652p6icr2xxprlk49qqgnqj8d-nginx-1.14.0/logs' \
        || mkdir -p '/nix/store/cpq2ca2652p6icr2xxprlk49qqgnqj8d-nginx-1.14.0/logs'
make[1]: Leaving directory '/private/tmp/nix-build-nginx-1.14.0.drv-0/nginx-1.14.0'
post-installation fixup
strip is /nix/store/53nysl8bqwxihwzs1pgwka20nf8mbvlp-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/cpq2ca2652p6icr2xxprlk49qqgnqj8d-nginx-1.14.0/bin
patching script interpreter paths in /nix/store/cpq2ca2652p6icr2xxprlk49qqgnqj8d-nginx-1.14.0

@domenkozar
Copy link
Member

@yegortimoshenko because we're changing expected behavior in nginx, it could have implications we can't predict. Although to be fair, current "cache forever" is pretty bad so maybe it's not that bad :)

@grahamc
Copy link
Member

grahamc commented Nov 30, 2018

We should probably merge this.

@aszlig aszlig requested a review from gebner April 18, 2019 08:33
@lukateras
Copy link
Member Author

@aszlig Ack! Thank you for working on this.

@domenkozar domenkozar merged commit 9bc23f3 into NixOS:master Apr 18, 2019
yorickvP pushed a commit to serokell/nixpkgs that referenced this pull request Apr 23, 2019
nginx: if root is in Nix store, use path's hash as ETag
yorickvP pushed a commit to serokell/nixpkgs that referenced this pull request Apr 23, 2019
nginx: if root is in Nix store, use path's hash as ETag
aszlig added a commit to headcounter/shabitica that referenced this pull request May 15, 2019
Since NixOS/nixpkgs#48337 we no longer have the
issue that we get the same ETags for different store paths but the hash
of the store path is now the ETag.

This means, that we no longer need to disable caching if the patch is
applied to nginx.

Unfortunately, the patch is only in NixOS Unstable at the moment, hence
the conditional on whether the patch exists. Even if the patch is
backported to 19.03, we'd still need it in 18.09, which we currently
still support as well.

In order to apply the patch for eg. NixOS 19.03, something like this
needs to be put in the system configuration:

  { pkgs, lib, ... }:

  {
    # ... other configuration definitions

    services.nginx.package = pkgs.nginx.overrideAttrs (drv: {
      patches = (drv.patches or []) ++ lib.singleton (fetchurl {
        url = "https://raw.githubusercontent.com/NixOS/nixpkgs/master/"
            + "pkgs/servers/http/nginx/nix-etag-1.15.4.patch";
        sha256 = "0w7sbvfrf0s20lyfr99r5d13rd97nd3c4n569n9ldy7a1r7nx019";
      });
    });

    # ... other configuration definitions
  }

Signed-off-by: aszlig <aszlig@nix.build>
@aszlig
Copy link
Member

aszlig commented May 23, 2019

Heads-up everyone: After testing this in a few production instances, it seems that some browsers still get cache hits for new store paths (and changed contents) for some reason. I highly suspect that it might be due to the last-modified header (as mentioned in #48337 (comment)).

Going to test this with last-modified disabled for a little while and if this is the case I think we should improve that patch by disabling last-modified if serving from a store path.

@lukateras
Copy link
Member Author

@aszlig
Copy link
Member

aszlig commented May 23, 2019

@yegortimoshenko: Yep, in theory it should, see also #48337 (comment).

@juskrey
Copy link

juskrey commented Sep 16, 2019

Some browsers indeed show very strange caching behavior when both ETag and last-modified are present. At least Chrome seems to keep etag-ged file for a while when last-modified is old

@aszlig
Copy link
Member

aszlig commented Dec 30, 2019

@juskrey: Mhm, I just could confirm exactly this. I think we also need to disable last-modified if we provide an ETag from the store.

@aszlig
Copy link
Member

aszlig commented Dec 30, 2019

Here is a small NixOS VM test reproducing the issue even on Firefox, pull request with the test and fix following soon...

aszlig added a commit to aszlig/nixpkgs that referenced this pull request Dec 30, 2019
This is what I've suspected a while ago[1]:

> Heads-up everyone: After testing this in a few production instances,
> it seems that some browsers still get cache hits for new store paths
> (and changed contents) for some reason. I highly suspect that it might
> be due to the last-modified header (as mentioned in [2]).
>
> Going to test this with last-modified disabled for a little while and
> if this is the case I think we should improve that patch by disabling
> last-modified if serving from a store path.

Much earlier[2] when I reviewed the patch, I wrote this:

> Other than that, it looks good to me.
>
> However, I'm not sure what we should do with Last-Modified header.
> From RFC 2616, section 13.3.4:
>
> - If both an entity tag and a Last-Modified value have been
>   provided by the origin server, SHOULD use both validators in
>   cache-conditional requests. This allows both HTTP/1.0 and
>   HTTP/1.1 caches to respond appropriately.
>
> I'm a bit nervous about the SHOULD here, as user agents in the wild
> could possibly just use Last-Modified and use the cached content
> instead.

Unfortunately, I didn't pursue this any further back then because
@pbogdan noted[3] the following:

> Hmm, could they (assuming they are conforming):
>
>  * If an entity tag has been provided by the origin server, MUST
>    use that entity tag in any cache-conditional request (using If-
>    Match or If-None-Match).

Since running with this patch in some deployments, I found that both
Firefox and Chrome/Chromium do NOT re-validate against the ETag if the
Last-Modified header is still the same.

So I wrote a small NixOS VM test with Geckodriver to have a test case
which is closer to the real world and I indeed was able to reproduce
this.

Whether this is actually a bug in Chrome or Firefox is an entirely
different issue and even IF it is the fault of the browsers and it is
fixed at some point, we'd still need to handle this for older browser
versions.

Apart from clearing the header, I also recreated the patch by using a
plain "git diff" with a small description on top. This should make it
easier for future authors to work on that patch.

[1]: NixOS#48337 (comment)
[2]: NixOS#48337 (comment)
[3]: NixOS#48337 (comment)

Signed-off-by: aszlig <aszlig@nix.build>
@aszlig
Copy link
Member

aszlig commented Dec 30, 2019

Done: #76697

dtzWill pushed a commit to dtzWill/nixpkgs that referenced this pull request Jan 2, 2020
This is what I've suspected a while ago[1]:

> Heads-up everyone: After testing this in a few production instances,
> it seems that some browsers still get cache hits for new store paths
> (and changed contents) for some reason. I highly suspect that it might
> be due to the last-modified header (as mentioned in [2]).
>
> Going to test this with last-modified disabled for a little while and
> if this is the case I think we should improve that patch by disabling
> last-modified if serving from a store path.

Much earlier[2] when I reviewed the patch, I wrote this:

> Other than that, it looks good to me.
>
> However, I'm not sure what we should do with Last-Modified header.
> From RFC 2616, section 13.3.4:
>
> - If both an entity tag and a Last-Modified value have been
>   provided by the origin server, SHOULD use both validators in
>   cache-conditional requests. This allows both HTTP/1.0 and
>   HTTP/1.1 caches to respond appropriately.
>
> I'm a bit nervous about the SHOULD here, as user agents in the wild
> could possibly just use Last-Modified and use the cached content
> instead.

Unfortunately, I didn't pursue this any further back then because
@pbogdan noted[3] the following:

> Hmm, could they (assuming they are conforming):
>
>  * If an entity tag has been provided by the origin server, MUST
>    use that entity tag in any cache-conditional request (using If-
>    Match or If-None-Match).

Since running with this patch in some deployments, I found that both
Firefox and Chrome/Chromium do NOT re-validate against the ETag if the
Last-Modified header is still the same.

So I wrote a small NixOS VM test with Geckodriver to have a test case
which is closer to the real world and I indeed was able to reproduce
this.

Whether this is actually a bug in Chrome or Firefox is an entirely
different issue and even IF it is the fault of the browsers and it is
fixed at some point, we'd still need to handle this for older browser
versions.

Apart from clearing the header, I also recreated the patch by using a
plain "git diff" with a small description on top. This should make it
easier for future authors to work on that patch.

[1]: NixOS#48337 (comment)
[2]: NixOS#48337 (comment)
[3]: NixOS#48337 (comment)

Signed-off-by: aszlig <aszlig@nix.build>
(cherry picked from commit ccf55be)
aszlig added a commit that referenced this pull request Jan 2, 2020
This is what I've suspected a while ago[1]:

> Heads-up everyone: After testing this in a few production instances,
> it seems that some browsers still get cache hits for new store paths
> (and changed contents) for some reason. I highly suspect that it might
> be due to the last-modified header (as mentioned in [2]).
>
> Going to test this with last-modified disabled for a little while and
> if this is the case I think we should improve that patch by disabling
> last-modified if serving from a store path.

Much earlier[2] when I reviewed the patch, I wrote this:

> Other than that, it looks good to me.
>
> However, I'm not sure what we should do with Last-Modified header.
> From RFC 2616, section 13.3.4:
>
> - If both an entity tag and a Last-Modified value have been
>   provided by the origin server, SHOULD use both validators in
>   cache-conditional requests. This allows both HTTP/1.0 and
>   HTTP/1.1 caches to respond appropriately.
>
> I'm a bit nervous about the SHOULD here, as user agents in the wild
> could possibly just use Last-Modified and use the cached content
> instead.

Unfortunately, I didn't pursue this any further back then because
@pbogdan noted[3] the following:

> Hmm, could they (assuming they are conforming):
>
>  * If an entity tag has been provided by the origin server, MUST
>    use that entity tag in any cache-conditional request (using If-
>    Match or If-None-Match).

Since running with this patch in some deployments, I found that both
Firefox and Chrome/Chromium do NOT re-validate against the ETag if the
Last-Modified header is still the same.

So I wrote a small NixOS VM test with Geckodriver to have a test case
which is closer to the real world and I indeed was able to reproduce
this.

Whether this is actually a bug in Chrome or Firefox is an entirely
different issue and even IF it is the fault of the browsers and it is
fixed at some point, we'd still need to handle this for older browser
versions.

Apart from clearing the header, I also recreated the patch by using a
plain "git diff" with a small description on top. This should make it
easier for future authors to work on that patch.

[1]: #48337 (comment)
[2]: #48337 (comment)
[3]: #48337 (comment)

Signed-off-by: aszlig <aszlig@nix.build>
(cherry picked from commit ccf55be)
Reason: The issue breaks setups that serve static content via Nix store
        paths. I've also backported the NixOS VM test from Python to
        Perl.
dtzWill pushed a commit to dtzWill/nixpkgs that referenced this pull request Feb 25, 2020
While our ETag patch works pretty fine if it comes to serving data off
store paths, it unfortunately broke something that might be a bit more
common, namely when using regexes to extract path components of
location directives for example.

Recently, @devhell has reported a bug with a nginx location directive
like this:

  location ~^/\~([a-z0-9_]+)(/.*)?$" {
    alias /home/$1/public_html$2;
  }

While this might look harmless at first glance, it does however cause
issues with our ETag patch. The alias directive gets broken up by nginx
like this:

  *2 http script copy: "/home/"
  *2 http script capture: "foo"
  *2 http script copy: "/public_html/"
  *2 http script capture: "bar.txt"

In our patch however, we use realpath(3) to get the canonicalised path
from ngx_http_core_loc_conf_s.root, which returns the *configured* value
from the root or alias directive. So in the example above, realpath(3)
boils down to the following syscalls:

  lstat("/home", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
  lstat("/home/$1", 0x7ffd08da6f60) = -1 ENOENT (No such file or directory)

During my review[1] of the initial patch, I didn't actually notice that
what we're doing here is returning NGX_ERROR if the realpath(3) call
fails, which in turn causes an HTTP 500 error.

Since our patch actually made the canonicalisation (and thus additional
syscalls) necessary, we really shouldn't introduce an additional error
so let's - at least for now - silently skip return value if realpath(3)
has failed.

However since we're using the unaltered root from the config we have
another issue, consider this root:

  /nix/store/...-abcde/$1

Calling realpath(3) on this path will fail (except if there's a file
called "$1" of course), so even this fix is not enough because it
results in the ETag not being set to the store path hash.

While this is very ugly and we should fix this very soon, it's not as
serious as getting HTTP 500 errors for serving static files.

I added a small NixOS VM test, which uses the example above as a
regression test.

It seems that my memory is failing these days, since apparently I *knew*
about this issue since digging for existing issues in nixpkgs, I found
this similar pull request which I even reviewed:

NixOS#66532

However, since the comments weren't addressed and the author hasn't
responded to the pull request, I decided to keep this very commit and do
a follow-up pull request.

[1]: NixOS#48337

Signed-off-by: aszlig <aszlig@nix.build>
Reported-by: @devhell
(cherry picked from commit f6c7635)
aszlig added a commit that referenced this pull request Mar 28, 2020
While our ETag patch works pretty fine if it comes to serving data off
store paths, it unfortunately broke something that might be a bit more
common, namely when using regexes to extract path components of
location directives for example.

Recently, @devhell has reported a bug with a nginx location directive
like this:

  location ~^/\~([a-z0-9_]+)(/.*)?$" {
    alias /home/$1/public_html$2;
  }

While this might look harmless at first glance, it does however cause
issues with our ETag patch. The alias directive gets broken up by nginx
like this:

  *2 http script copy: "/home/"
  *2 http script capture: "foo"
  *2 http script copy: "/public_html/"
  *2 http script capture: "bar.txt"

In our patch however, we use realpath(3) to get the canonicalised path
from ngx_http_core_loc_conf_s.root, which returns the *configured* value
from the root or alias directive. So in the example above, realpath(3)
boils down to the following syscalls:

  lstat("/home", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
  lstat("/home/$1", 0x7ffd08da6f60) = -1 ENOENT (No such file or directory)

During my review[1] of the initial patch, I didn't actually notice that
what we're doing here is returning NGX_ERROR if the realpath(3) call
fails, which in turn causes an HTTP 500 error.

Since our patch actually made the canonicalisation (and thus additional
syscalls) necessary, we really shouldn't introduce an additional error
so let's - at least for now - silently skip return value if realpath(3)
has failed.

However since we're using the unaltered root from the config we have
another issue, consider this root:

  /nix/store/...-abcde/$1

Calling realpath(3) on this path will fail (except if there's a file
called "$1" of course), so even this fix is not enough because it
results in the ETag not being set to the store path hash.

While this is very ugly and we should fix this very soon, it's not as
serious as getting HTTP 500 errors for serving static files.

I added a small NixOS VM test, which uses the example above as a
regression test.

It seems that my memory is failing these days, since apparently I *knew*
about this issue since digging for existing issues in nixpkgs, I found
this similar pull request which I even reviewed:

#66532

However, since the comments weren't addressed and the author hasn't
responded to the pull request, I decided to keep this very commit and do
a follow-up pull request.

[1]: #48337

Signed-off-by: aszlig <aszlig@nix.build>
Reported-by: @devhell
Acked-by: @7c6f434c
Acked-by: @yorickvP
Merges: #80671
Fixes: #66532
aszlig added a commit that referenced this pull request Mar 28, 2020
While our ETag patch works pretty fine if it comes to serving data off
store paths, it unfortunately broke something that might be a bit more
common, namely when using regexes to extract path components of
location directives for example.

Recently, @devhell has reported a bug with a nginx location directive
like this:

  location ~^/\~([a-z0-9_]+)(/.*)?$" {
    alias /home/$1/public_html$2;
  }

While this might look harmless at first glance, it does however cause
issues with our ETag patch. The alias directive gets broken up by nginx
like this:

  *2 http script copy: "/home/"
  *2 http script capture: "foo"
  *2 http script copy: "/public_html/"
  *2 http script capture: "bar.txt"

In our patch however, we use realpath(3) to get the canonicalised path
from ngx_http_core_loc_conf_s.root, which returns the *configured* value
from the root or alias directive. So in the example above, realpath(3)
boils down to the following syscalls:

  lstat("/home", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
  lstat("/home/$1", 0x7ffd08da6f60) = -1 ENOENT (No such file or directory)

During my review[1] of the initial patch, I didn't actually notice that
what we're doing here is returning NGX_ERROR if the realpath(3) call
fails, which in turn causes an HTTP 500 error.

Since our patch actually made the canonicalisation (and thus additional
syscalls) necessary, we really shouldn't introduce an additional error
so let's - at least for now - silently skip return value if realpath(3)
has failed.

However since we're using the unaltered root from the config we have
another issue, consider this root:

  /nix/store/...-abcde/$1

Calling realpath(3) on this path will fail (except if there's a file
called "$1" of course), so even this fix is not enough because it
results in the ETag not being set to the store path hash.

While this is very ugly and we should fix this very soon, it's not as
serious as getting HTTP 500 errors for serving static files.

I added a small NixOS VM test, which uses the example above as a
regression test.

It seems that my memory is failing these days, since apparently I *knew*
about this issue since digging for existing issues in nixpkgs, I found
this similar pull request which I even reviewed:

#66532

However, since the comments weren't addressed and the author hasn't
responded to the pull request, I decided to keep this very commit and do
a follow-up pull request.

[1]: #48337

Signed-off-by: aszlig <aszlig@nix.build>
Reported-by: @devhell
Acked-by: @7c6f434c
Acked-by: @yorickvP
Merges: #80671
Fixes: #66532
(cherry picked from commit e1d63ad)
aszlig added a commit that referenced this pull request Mar 28, 2020
While our ETag patch works pretty fine if it comes to serving data off
store paths, it unfortunately broke something that might be a bit more
common, namely when using regexes to extract path components of
location directives for example.

Recently, @devhell has reported a bug with a nginx location directive
like this:

  location ~^/\~([a-z0-9_]+)(/.*)?$" {
    alias /home/$1/public_html$2;
  }

While this might look harmless at first glance, it does however cause
issues with our ETag patch. The alias directive gets broken up by nginx
like this:

  *2 http script copy: "/home/"
  *2 http script capture: "foo"
  *2 http script copy: "/public_html/"
  *2 http script capture: "bar.txt"

In our patch however, we use realpath(3) to get the canonicalised path
from ngx_http_core_loc_conf_s.root, which returns the *configured* value
from the root or alias directive. So in the example above, realpath(3)
boils down to the following syscalls:

  lstat("/home", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
  lstat("/home/$1", 0x7ffd08da6f60) = -1 ENOENT (No such file or directory)

During my review[1] of the initial patch, I didn't actually notice that
what we're doing here is returning NGX_ERROR if the realpath(3) call
fails, which in turn causes an HTTP 500 error.

Since our patch actually made the canonicalisation (and thus additional
syscalls) necessary, we really shouldn't introduce an additional error
so let's - at least for now - silently skip return value if realpath(3)
has failed.

However since we're using the unaltered root from the config we have
another issue, consider this root:

  /nix/store/...-abcde/$1

Calling realpath(3) on this path will fail (except if there's a file
called "$1" of course), so even this fix is not enough because it
results in the ETag not being set to the store path hash.

While this is very ugly and we should fix this very soon, it's not as
serious as getting HTTP 500 errors for serving static files.

I added a small NixOS VM test, which uses the example above as a
regression test.

It seems that my memory is failing these days, since apparently I *knew*
about this issue since digging for existing issues in nixpkgs, I found
this similar pull request which I even reviewed:

#66532

However, since the comments weren't addressed and the author hasn't
responded to the pull request, I decided to keep this very commit and do
a follow-up pull request.

[1]: #48337

Signed-off-by: aszlig <aszlig@nix.build>
Reported-by: @devhell
Acked-by: @7c6f434c
Acked-by: @yorickvP
Merges: #80671
Fixes: #66532
(cherry picked from commit e1d63ad)
stigok pushed a commit to stigok/nixpkgs that referenced this pull request Jun 12, 2020
While our ETag patch works pretty fine if it comes to serving data off
store paths, it unfortunately broke something that might be a bit more
common, namely when using regexes to extract path components of
location directives for example.

Recently, @devhell has reported a bug with a nginx location directive
like this:

  location ~^/\~([a-z0-9_]+)(/.*)?$" {
    alias /home/$1/public_html$2;
  }

While this might look harmless at first glance, it does however cause
issues with our ETag patch. The alias directive gets broken up by nginx
like this:

  *2 http script copy: "/home/"
  *2 http script capture: "foo"
  *2 http script copy: "/public_html/"
  *2 http script capture: "bar.txt"

In our patch however, we use realpath(3) to get the canonicalised path
from ngx_http_core_loc_conf_s.root, which returns the *configured* value
from the root or alias directive. So in the example above, realpath(3)
boils down to the following syscalls:

  lstat("/home", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
  lstat("/home/$1", 0x7ffd08da6f60) = -1 ENOENT (No such file or directory)

During my review[1] of the initial patch, I didn't actually notice that
what we're doing here is returning NGX_ERROR if the realpath(3) call
fails, which in turn causes an HTTP 500 error.

Since our patch actually made the canonicalisation (and thus additional
syscalls) necessary, we really shouldn't introduce an additional error
so let's - at least for now - silently skip return value if realpath(3)
has failed.

However since we're using the unaltered root from the config we have
another issue, consider this root:

  /nix/store/...-abcde/$1

Calling realpath(3) on this path will fail (except if there's a file
called "$1" of course), so even this fix is not enough because it
results in the ETag not being set to the store path hash.

While this is very ugly and we should fix this very soon, it's not as
serious as getting HTTP 500 errors for serving static files.

I added a small NixOS VM test, which uses the example above as a
regression test.

It seems that my memory is failing these days, since apparently I *knew*
about this issue since digging for existing issues in nixpkgs, I found
this similar pull request which I even reviewed:

NixOS#66532

However, since the comments weren't addressed and the author hasn't
responded to the pull request, I decided to keep this very commit and do
a follow-up pull request.

[1]: NixOS#48337

Signed-off-by: aszlig <aszlig@nix.build>
Reported-by: @devhell
Acked-by: @7c6f434c
Acked-by: @yorickvP
Merges: NixOS#80671
Fixes: NixOS#66532
(cherry picked from commit e1d63ad)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

nginx serving nix store paths sets last-modified header to unix+1 timestamp
9 participants