Varnish Cache not being hit at all?

[rossmc]

I've installed Turpentine on our staging site and it seems as if Varnish is either not caching or being hit and it is just passing the request to NGINX. The TTFB is still at approx 850ms.

The site is on Ubuntu 14.04, NGINX, varnish-3.0.7, Turpentine v0.6.5, Magento 1.9.2.

I've turned on Enabled Debug Info and Logging but I'm getting no error messages.

I only see the following in the turpentine.log when I clear the Varnish cache :
2015-09-12T15:42:15+00:00 DEBUG (7): TURPENTINE: Cleared ESI cache for client (9dcd02d74b13e0b2610a765af415212d) on event: core_session_abstract_add_message

This is what I have in my Daemon Opts in /etc/default/varnish

DAEMON_OPTS="-a :80 \
             -T localhost:6082 \
             -f /var/www/staging/var/default.vcl \
             -S /etc/varnish/secret \
             -p esi_syntax=0x2 \
             -p cli_buffer=16384 \
             -s malloc,256m"

Here one of the things I changed which wasn't in the instructions was -f /etc/varnish/default.vcl \ to -f /var/www/staging/var/default.vcl \. While I think this is correct it has made absolutely no difference.

When I run either of the following commands I see a bunch or requests in the console.

varnishtop -i RxURL
varnishtop -i TxURL

The requests include the homepage or page I'm visiting i.e. / or /cloths and all the images, css & js on our server for Magento, 36 in total. I believe the RxURL is for incoming requests to varnish and TxURL is for requests passed on to NGINX. Is this correct?

Also when I check the varnish CLI with the varnishadm command I see nothing of note:

varnish> backend.list 
200        
Backend name                   Refs   Admin      Probe
default(127.0.0.1,,8080)       1      probe      Healthy (no probe)
admin[0](127.0.0.1,,8080)      1      probe      Healthy (no probe)
varnish> vcl.list
200        
available       6 boot
available       0 1313bccf1348fc87c3d41220efd85b3382a5cfde64f60d9a0a3aa26b9de12a5d
active          2 ade416cc09cd1378c6d3c322774ec1c77d3c29e2655d3391a1760e59a0c3bed6

varnish> ban.list 
200        
Present bans:
varnish> 

When I run the varnishstat command and refresh our homepage a few time the hitrate remains at 0, but some other stats change.

Hitrate ratio:        0        0        0
Hitrate avg:     0.0000   0.0000   0.0000

    1544         9,99         0.01 client_conn - Client connections accepted
    1520         9,99         0.01 client_req - Client requests received
    1519         9,99         0.01 backend_conn - Backend conn. success
      1         0.00         0.00 backend_reuse - Backend conn. reuses

I would really appreciate any advice on how I can debug this issue further or sort out whats not working with it. Below is my Downloaded Varnish Config from System > Cache Management. Perhaps this is not being loaded by varnish?

C{
#include <stdlib.h>
#include <stdio.h>
#include <time.h>
#include <pthread.h>
static pthread_mutex_t lrand_mutex = PTHREAD_MUTEX_INITIALIZER;
void generate_uuid(char* buf) {
pthread_mutex_lock(&lrand_mutex);
long a = lrand48();
long b = lrand48();
long c = lrand48();
long d = lrand48();
pthread_mutex_unlock(&lrand_mutex);
sprintf(buf, "frontend=%08lx%04lx%04lx%04lx%04lx%08lx",
a,
b & 0xffff,
(b & ((long)0x0fff0000) >> 16) | 0x4000,
(c & 0x0fff) | 0x8000,
(c & (long)0xffff0000) >> 16,
d
);
return;
}
}C
import std;
director default round-robin {
{
.backend = {
.host = "127.0.0.1";
.port = "8080";
.first_byte_timeout = 300s;
.between_bytes_timeout = 300s;
}
}
}
director admin round-robin {
{
.backend = {
.host = "127.0.0.1";
.port = "8080";
.first_byte_timeout = 21600s;
.between_bytes_timeout = 21600s;
}
}
}
acl crawler_acl {
"127.0.0.1";
}
acl debug_acl {
}
/* -- REMOVED
sub generate_session {
if (req.url ~ ".*[&?]SID=([^&]+).*") {
set req.http.X-Varnish-Faked-Session = regsub(
req.url, ".*[&?]SID=([^&]+).*", "frontend=\1");
} else {
C{
char uuid_buf [50];
generate_uuid(uuid_buf);
VRT_SetHdr(sp, HDR_REQ,
"\030X-Varnish-Faked-Session:",
uuid_buf,
vrt_magic_string_end
);
}C
}
if (req.http.Cookie) {
std.collect(req.http.Cookie);
set req.http.Cookie = req.http.X-Varnish-Faked-Session +
"; " + req.http.Cookie;
} else {
set req.http.Cookie = req.http.X-Varnish-Faked-Session;
}
}
sub generate_session_expires {
C{
time_t now = time(NULL);
struct tm now_tm = *gmtime(&now);
now_tm.tm_sec += 3600;
mktime(&now_tm);
char date_buf [50];
strftime(date_buf, sizeof(date_buf)-1, "%a, %d-%b-%Y %H:%M:%S %Z", &now_tm);
VRT_SetHdr(sp, HDR_RESP,
"\031X-Varnish-Cookie-Expires:",
date_buf,
vrt_magic_string_end
);
}C
}
-- */
sub vcl_recv {
if (req.restarts == 0) {
if (req.http.X-Forwarded-For) {
set req.http.X-Forwarded-For =
req.http.X-Forwarded-For + ", " + client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
}
if (req.http.Accept-Encoding) {
if (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} else if (req.http.Accept-Encoding ~ "deflate") {
set req.http.Accept-Encoding = "deflate";
} else {
unset req.http.Accept-Encoding;
}
}
if (req.http.User-Agent ~ "iP(?:hone|ad|od)|BlackBerry|Palm|Googlebot-Mobile|Mobile|mobile|mobi|Windows Mobile|Safari Mobile|Android|Opera (?:Mini|Mobi)") {
set req.http.X-Normalized-User-Agent = "mobile";
} else if (req.http.User-Agent ~ "MSIE") {
set req.http.X-Normalized-User-Agent = "msie";
} else if (req.http.User-Agent ~ "Firefox") {
set req.http.X-Normalized-User-Agent = "firefox";
} else if (req.http.User-Agent ~ "Chrome") {
set req.http.X-Normalized-User-Agent = "chrome";
} else if (req.http.User-Agent ~ "Safari") {
set req.http.X-Normalized-User-Agent = "safari";
} else if (req.http.User-Agent ~ "Opera") {
set req.http.X-Normalized-User-Agent = "opera";
} else {
set req.http.X-Normalized-User-Agent = "other";
}
if (!true || req.http.Authorization ||
req.request !~ "^(GET|HEAD|OPTIONS)$" ||
req.http.Cookie ~ "varnish_bypass=1") {
if (req.url ~ "^(/media/|/skin/|/js/|/)(?:(?:index|litespeed)\.php/)?admin") {
set req.backend = admin;
}
return (pipe);
}
set req.url = regsuball(req.url, "([^:])//+", "\1/");
if (req.url ~ "^(/media/|/skin/|/js/|/)(?:(?:index|litespeed)\.php/)?") {
set req.http.X-Turpentine-Secret-Handshake = "1";
if (req.url ~ "^(/media/|/skin/|/js/|/)(?:(?:index|litespeed)\.php/)?admin") {
set req.backend = admin;
return (pipe);
}
if (req.http.Cookie ~ "\bcurrency=") {
set req.http.X-Varnish-Currency = regsub(
req.http.Cookie, ".*\bcurrency=([^;]*).*", "\1");
}
if (req.http.Cookie ~ "\bstore=") {
set req.http.X-Varnish-Store = regsub(
req.http.Cookie, ".*\bstore=([^;]*).*", "\1");
}
if (req.url ~ "/turpentine/esi/get(?:Block|FormKey)/") {
set req.http.X-Varnish-Esi-Method = regsub(
req.url, ".*/method/(\w+)/.*", "\1");
set req.http.X-Varnish-Esi-Access = regsub(
req.url, ".*/access/(\w+)/.*", "\1");
if (req.http.X-Varnish-Esi-Method == "esi" && req.esi_level == 0 &&
!(true || client.ip ~ debug_acl)) {
error 403 "External ESI requests are not allowed";
}
}
if (req.http.host !~ "(staging\-uat\.our-domain\.ie)") {
return (pass);
}
if (req.http.Cookie !~ "frontend=" && !req.http.X-Varnish-Esi-Method) {
if (client.ip ~ crawler_acl ||
req.http.User-Agent ~ "^(?:ApacheBench/.*|.*Googlebot.*|JoeDog/.*Siege.*|magespeedtest\.com|Nexcessnet_Turpentine/.*)$") {
set req.http.Cookie = "frontend=crawler-session";
} else {
return (pipe);
}
}
if (true &&
req.url ~ ".*\.(?:css|js|jpe?g|png|gif|ico|swf)(?=\?|&|$)") {
unset req.http.Cookie;
unset req.http.X-Varnish-Faked-Session;
return (lookup);
}
if (req.url ~ "^(/media/|/skin/|/js/|/)(?:(?:index|litespeed)\.php/)?(?:admin|api|cron\.php|checkout|checkout/cart|customer|advancednewsletter|onestepcheckout|onepage|exporter|contacts|iphone|join|sitemap.xml)" ||
req.url ~ "\?.*__from_store=") {
return (pipe);
}
if (true &&
req.url ~ "(?:[?&](?:__SID|XDEBUG_PROFILE)(?=[&=]|$))") {
return (pass);
}
if (true && req.url ~ "[?&](utm_source|utm_medium|utm_campaign|utm_content|utm_term|gclid|cx|ie|cof|siteurl)=") {
set req.url = regsuball(req.url, "(?:(\?)?|&)(?:utm_source|utm_medium|utm_campaign|utm_content|utm_term|gclid|cx|ie|cof|siteurl)=[^&]+", "\1");
set req.url = regsuball(req.url, "(?:(\?)&|\?$)", "\1");
}
return (lookup);
}
}
sub vcl_pipe {
unset bereq.http.X-Turpentine-Secret-Handshake;
set bereq.http.Connection = "close";
}
sub vcl_hash {
hash_data(req.url);
if (req.http.Host) {
hash_data(req.http.Host);
} else {
hash_data(server.ip);
}
hash_data(req.http.Ssl-Offloaded);
if (req.http.X-Normalized-User-Agent) {
hash_data(req.http.X-Normalized-User-Agent);
}
if (req.http.Accept-Encoding) {
hash_data(req.http.Accept-Encoding);
}
if (req.http.X-Varnish-Store || req.http.X-Varnish-Currency) {
hash_data("s=" + req.http.X-Varnish-Store + "&c=" + req.http.X-Varnish-Currency);
}
if (req.http.X-Varnish-Esi-Access == "private" &&
req.http.Cookie ~ "frontend=") {
hash_data(regsub(req.http.Cookie, "^.*?frontend=([^;]*);*.*$", "\1"));
}
if (req.http.X-Varnish-Esi-Access == "customer_group" &&
req.http.Cookie ~ "customer_group=") {
hash_data(regsub(req.http.Cookie, "^.*?customer_group=([^;]*);*.*$", "\1"));
}
return (hash);
}
sub vcl_hit {
}
sub vcl_fetch {
set req.grace = 15s;
set beresp.http.X-Varnish-Host = req.http.host;
set beresp.http.X-Varnish-URL = req.url;
if (req.url ~ "^(/media/|/skin/|/js/|/)(?:(?:index|litespeed)\.php/)?") {
unset beresp.http.Vary;
set beresp.do_gzip = true;
if (beresp.status != 200 && beresp.status != 404) {
set beresp.ttl = 15s;
return (hit_for_pass);
} else {
if (beresp.http.Set-Cookie) {
set beresp.http.X-Varnish-Set-Cookie = beresp.http.Set-Cookie;
unset beresp.http.Set-Cookie;
}
unset beresp.http.Cache-Control;
unset beresp.http.Expires;
unset beresp.http.Pragma;
unset beresp.http.Cache;
unset beresp.http.Age;
if (beresp.http.X-Turpentine-Esi == "1") {
set beresp.do_esi = true;
}
if (beresp.http.X-Turpentine-Cache == "0") {
set beresp.ttl = 15s;
return (hit_for_pass);
} else {
if (true &&
bereq.url ~ ".*\.(?:css|js|jpe?g|png|gif|ico|swf)(?=\?|&|$)") {
set beresp.ttl = 28800s;
set beresp.http.Cache-Control = "max-age=28800";
} elseif (req.http.X-Varnish-Esi-Method) {
if (req.http.X-Varnish-Esi-Access == "private" &&
req.http.Cookie ~ "frontend=") {
set beresp.http.X-Varnish-Session = regsub(req.http.Cookie,
"^.*?frontend=([^;]*);*.*$", "\1");
}
if (req.http.X-Varnish-Esi-Method == "ajax" &&
req.http.X-Varnish-Esi-Access == "public") {
set beresp.http.Cache-Control = "max-age=" + regsub(
req.url, ".*/ttl/(\d+)/.*", "\1");
}
set beresp.ttl = std.duration(
regsub(
req.url, ".*/ttl/(\d+)/.*", "\1s"),
300s);
if (beresp.ttl == 0s) {
set beresp.ttl = 15s;
return (hit_for_pass);
}
} else {
set beresp.ttl = 3600s;
}
}
}
return (deliver);
}
}
sub vcl_deliver {
if (req.http.X-Varnish-Faked-Session) {
set resp.http.Set-Cookie = req.http.X-Varnish-Faked-Session +
"; expires=" + resp.http.X-Varnish-Cookie-Expires + "; path=/";
if (req.http.Host) {
if (req.http.User-Agent ~ "^(?:ApacheBench/.*|.*Googlebot.*|JoeDog/.*Siege.*|magespeedtest\.com|Nexcessnet_Turpentine/.*)$") {
set resp.http.Set-Cookie = resp.http.Set-Cookie +
"; domain=" + regsub(req.http.Host, ":\d+$", "");
} else {
if(req.http.Host ~ "") {
set resp.http.Set-Cookie = resp.http.Set-Cookie +
"; domain=";
} else {
set resp.http.Set-Cookie = resp.http.Set-Cookie +
"; domain=" + regsub(req.http.Host, ":\d+$", "");
}
}
}
set resp.http.Set-Cookie = resp.http.Set-Cookie + "; httponly";
unset resp.http.X-Varnish-Cookie-Expires;
}
if (req.http.X-Varnish-Esi-Method == "ajax" && req.http.X-Varnish-Esi-Access == "private") {
set resp.http.Cache-Control = "no-cache";
}
if (true || client.ip ~ debug_acl) {
set resp.http.X-Varnish-Hits = obj.hits;
set resp.http.X-Varnish-Esi-Method = req.http.X-Varnish-Esi-Method;
set resp.http.X-Varnish-Esi-Access = req.http.X-Varnish-Esi-Access;
set resp.http.X-Varnish-Currency = req.http.X-Varnish-Currency;
set resp.http.X-Varnish-Store = req.http.X-Varnish-Store;
} else {
unset resp.http.X-Varnish;
unset resp.http.Via;
unset resp.http.X-Powered-By;
unset resp.http.Server;
unset resp.http.X-Turpentine-Cache;
unset resp.http.X-Turpentine-Esi;
unset resp.http.X-Turpentine-Flush-Events;
unset resp.http.X-Turpentine-Block;
unset resp.http.X-Varnish-Session;
unset resp.http.X-Varnish-Host;
unset resp.http.X-Varnish-URL;
unset resp.http.X-Varnish-Set-Cookie;
}
}

[rossmc]

Could this have something to do with the cookies in the request?

From the chrome inspector I see that cookies are in the header.

Status Code:200 OK
Request Headersview source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip, deflate, sdch
Accept-Language:en-US,en;q=0.8
Authorization:Basic ZGV2aG9pOk5hc0RhdzEx
Cache-Control:max-age=0
Connection:keep-alive
Cookie:store=ie; adminhtml=6a879bc4bdd224727cbca06a5be87f4d; frontend=b6e7bcaa471baca15fba63417a44dad6; currency=EUR; adminhtml=6a879bc4bdd224727cbca06a5be87f4d; frontend=b6e7bcaa471baca15fba63417a44dad6; cookielaw=1
Host:staging-uat.our-domain.ie
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.95 Safari/537.36
Response Headersview source
Cache-Control:no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Connection:close
Content-Encoding:gzip
Content-Type:text/html; charset=UTF-8
Date:Mon, 14 Sep 2015 09:39:14 GMT
Expires:Thu, 19 Nov 1981 08:52:00 GMT
Pragma:no-cache
Server:nginx/1.4.6 (Ubuntu)
Set-Cookie:frontend=b6e7bcaa471baca15fba63417a44dad6; expires=Mon, 14-Sep-2015 10:39:13 GMT; Max-Age=3600; path=/; domain=staging.our-domain.ie
Transfer-Encoding:chunked
Vary:Accept-Encoding
X-Frame-Options:SAMEORIGIN
X-Powered-By:HHVM/3.9.0

I disabled the modules causing the store, currency & cookielaw cookies to be set on when a user visits a page, so that ownly the typical Magento frontend cookie is set but this did not make any difference.

[rossmc]

Below is a an excerpt from the beginning of our varnishlog

0 CLI          - Rd ping
0 CLI          - Wr 200 19 PONG 1442236902 1.0
11 SessionOpen  c 62.my.ip.?? 13425 :80
11 SessionClose c pipe
13 BackendOpen  b default 127.0.0.1 52309 127.0.0.1 8080
13 BackendXID   b 477110995
13 TxRequest    b GET
13 TxURL        b /
13 TxProtocol   b HTTP/1.1
13 TxHeader     b Host: staging-uat.our-domain.ie
13 TxHeader     b Pragma: no-cache
13 TxHeader     b Cache-Control: no-cache
13 TxHeader     b Authorization: Basic ZGV2aG9pOk5hc0RhdzEx
13 TxHeader     b Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
13 TxHeader     b Upgrade-Insecure-Requests: 1
13 TxHeader     b User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36
13 TxHeader     b Accept-Language: en-US,en;q=0.8,sv;q=0.6
13 TxHeader     b Cookie: store=ie; frontend=adf456dc6c13037ed010b9c11b89a4a7; currency=EUR; frontend=adf456dc6c13037ed010b9c11b89a4a7; cookielaw=1
13 TxHeader     b X-Forwarded-For: 62.my.ip.??
13 TxHeader     b Accept-Encoding: gzip
13 TxHeader     b X-Normalized-User-Agent: chrome
13 TxHeader     b X-Varnish: 477110995
13 TxHeader     b Connection: close
13 BackendClose b default
11 ReqStart     c 62.my.ip.?? 13425 477110995
11 RxRequest    c GET
11 RxURL        c /
11 RxProtocol   c HTTP/1.1
11 RxHeader     c Host: staging-uat.our-domain.ie
11 RxHeader     c Connection: keep-alive
11 RxHeader     c Pragma: no-cache
11 RxHeader     c Cache-Control: no-cache
11 RxHeader     c Authorization: Basic ZGV2aG9pOk5hc0RhdzEx
11 RxHeader     c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
11 RxHeader     c Upgrade-Insecure-Requests: 1
11 RxHeader     c User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36
11 RxHeader     c Accept-Encoding: gzip, deflate, sdch
11 RxHeader     c Accept-Language: en-US,en;q=0.8,sv;q=0.6
11 RxHeader     c Cookie: store=ie; frontend=adf456dc6c13037ed010b9c11b89a4a7; currency=EUR; frontend=adf456dc6c13037ed010b9c11b89a4a7; cookielaw=1
11 VCL_call     c recv pipe
11 VCL_call     c hash
11 Hash         c /
11 Hash         c staging-uat.our-domain.ie
11 Hash         c 
11 Hash         c chrome
11 Hash         c gzip
11 VCL_return   c hash
11 VCL_call     c pipe pipe
11 Backend      c 13 default default
11 ReqEnd       c 477110995 1442236902.654249191 1442236903.732433319 0.000072241 0.006358385 1.071825743
11 StatSess     c 62.my.ip.?? 13425 1 1 1 1 0 0 0 0
18 SessionOpen  c 62.my.ip.?? 10304 :80
18 SessionClose c pipe
19 BackendOpen  b default 127.0.0.1 52314 127.0.0.1 8080
19 BackendXID   b 477110998
19 TxRequest    b GET
19 TxURL        b /media/css/27b97561a63eda4a63cace3ddd5697de.css
19 TxProtocol   b HTTP/1.1
19 TxHeader     b Host: staging-uat.our-domain.ie
19 TxHeader     b Pragma: no-cache
19 TxHeader     b Cache-Control: no-cache
19 TxHeader     b Authorization: Basic ZGV2aG9pOk5hc0RhdzEx
19 TxHeader     b Accept: text/css,*/*;q=0.1
19 TxHeader     b User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36
19 TxHeader     b Referer: http://staging-uat.our-domain.ie/
19 TxHeader     b Accept-Language: en-US,en;q=0.8,sv;q=0.6
19 TxHeader     b Cookie: store=ie; frontend=adf456dc6c13037ed010b9c11b89a4a7; currency=EUR; cookielaw=1; frontend=adf456dc6c13037ed010b9c11b89a4a7
19 TxHeader     b X-Forwarded-For: 62.my.ip.??
19 TxHeader     b Accept-Encoding: gzip
19 TxHeader     b X-Normalized-User-Agent: chrome
19 TxHeader     b X-Varnish: 477110998
19 TxHeader     b Connection: close
19 BackendClose b default
18 ReqStart     c 62.my.ip.?? 10304 477110998
18 RxRequest    c GET
18 RxURL        c /media/css/27b97561a63eda4a63cace3ddd5697de.css
18 RxProtocol   c HTTP/1.1
18 RxHeader     c Host: staging-uat.our-domain.ie
18 RxHeader     c Connection: keep-alive
18 RxHeader     c Pragma: no-cache
18 RxHeader     c Cache-Control: no-cache
18 RxHeader     c Authorization: Basic ZGV2aG9pOk5hc0RhdzEx
18 RxHeader     c Accept: text/css,*/*;q=0.1
18 RxHeader     c User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36
18 RxHeader     c Referer: http://staging-uat.our-domain.ie/
18 RxHeader     c Accept-Encoding: gzip, deflate, sdch
18 RxHeader     c Accept-Language: en-US,en;q=0.8,sv;q=0.6
18 RxHeader     c Cookie: store=ie; frontend=adf456dc6c13037ed010b9c11b89a4a7; currency=EUR; cookielaw=1; frontend=adf456dc6c13037ed010b9c11b89a4a7
18 VCL_call     c recv pipe
18 VCL_call     c hash
18 Hash         c /media/css/27b97561a63eda4a63cace3ddd5697de.css
18 Hash         c staging-uat.our-domain.ie
18 Hash         c 
18 Hash         c chrome
18 Hash         c gzip
18 VCL_return   c hash
18 VCL_call     c pipe pipe
18 Backend      c 19 default default
18 ReqEnd       c 477110998 1442236903.731236696 1442236903.860495329 0.000040770 0.000084400 0.129174232
18 StatSess     c 62.my.ip.?? 10304 0 1 1 1 0 0 0 0
11 SessionOpen  c 62.my.ip.?? 41882 :80
11 SessionClose c pipe
13 BackendOpen  b default 127.0.0.1 52315 127.0.0.1 8080
13 BackendXID   b 477110999
13 TxRequest    b GET
13 TxURL        b /media/wysiwyg/logo.png
13 TxProtocol   b HTTP/1.1
13 TxHeader     b Host: staging-uat.our-domain.ie
13 TxHeader     b Pragma: no-cache
13 TxHeader     b Cache-Control: no-cache
13 TxHeader     b Authorization: Basic ZGV2aG9pOk5hc0RhdzEx
13 TxHeader     b Accept: image/webp,image/*,*/*;q=0.8
13 TxHeader     b User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36
13 TxHeader     b Referer: http://staging-uat.our-domain.ie/
13 TxHeader     b Accept-Language: en-US,en;q=0.8,sv;q=0.6
13 TxHeader     b Cookie: store=ie; frontend=adf456dc6c13037ed010b9c11b89a4a7; currency=EUR; cookielaw=1; frontend=adf456dc6c13037ed010b9c11b89a4a7
13 TxHeader     b X-Forwarded-For: 62.my.ip.??
13 TxHeader     b Accept-Encoding: gzip
13 TxHeader     b X-Normalized-User-Agent: chrome
13 TxHeader     b X-Varnish: 477110999
13 TxHeader     b Connection: close
13 BackendClose b default
11 ReqStart     c 62.my.ip.?? 41882 477110999
11 RxRequest    c GET
11 RxURL        c /media/wysiwyg/logo.png
11 RxProtocol   c HTTP/1.1
11 RxHeader     c Host: staging-uat.our-domain.ie
11 RxHeader     c Connection: keep-alive
11 RxHeader     c Pragma: no-cache
11 RxHeader     c Cache-Control: no-cache
11 RxHeader     c Authorization: Basic ZGV2aG9pOk5hc0RhdzEx
11 RxHeader     c Accept: image/webp,image/*,*/*;q=0.8
11 RxHeader     c User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36
11 RxHeader     c Referer: http://staging-uat.our-domain.ie/
11 RxHeader     c Accept-Encoding: gzip, deflate, sdch
11 RxHeader     c Accept-Language: en-US,en;q=0.8,sv;q=0.6
11 RxHeader     c Cookie: store=ie; frontend=adf456dc6c13037ed010b9c11b89a4a7; currency=EUR; cookielaw=1; frontend=adf456dc6c13037ed010b9c11b89a4a7
11 VCL_call     c recv pipe
11 VCL_call     c hash
11 Hash         c /media/wysiwyg/logo.png
11 Hash         c staging-uat.our-domain.ie
11 Hash         c 
11 Hash         c chrome
11 Hash         c gzip
11 VCL_return   c hash
11 VCL_call     c pipe pipe
11 Backend      c 13 default default
11 ReqEnd       c 477110999 1442236903.870586395 1442236903.996061563 0.011831999 0.000076056 0.125399113
11 StatSess     c 62.my.ip.?? 41882 0 1 1 1 0 0 0 0
11 SessionOpen  c 62.my.ip.?? 45282 :80
11 SessionClose c pipe

[aricwatson]

My first suggestion (since I always forget this step) is to be sure you clicked 'apply varnish config' to make sure that Turpentine has applied its VCL to the Varnish instance? It sounds to me like Varnish is just passing everything through.

[rossmc]

@aricwatson I have clicked apply varnish config but it makes no difference :(

[aricwatson]

OK - couple other things to rule out - have you disabled all the cache types on the cache management page - other than the two varnish cache types? Are you using any other caching extensions that might conflict? Also be sure that you have the compiler turned off...

Also - when testing, be sure to reload your test page two or three times - since the first request will just be piped through Apache/Nginx to get a good session, if you only load the page once you won't see any Varnish "stuff" happening.

Sorry if these are things you already know of have already tried, just making sure we aren't missing anything :)

[rossmc]

@aricwatson Yes compiler is turned off. We don't have any other types of HTTP caches.

I've been reloading the page several times when testing.

I've cleared the Magento cache also rm -rf var/cache

[aricwatson]

OK, I think your problem may be related to a bug we just fixed in 0.6.6 - please try updating to current master version and test. Specifically - I think it might be related to #904

[rossmc]

@aricwatson Still no difference after upgrade.

I cleared Magento cache, logged in & out of admin, re-applied Varnish config & restarted varnish on server. After reloading test page TTFB still at approx 850ms and I see no difference on varnishstat - Hitrate avg is still at 0, that means Varnish cache is being by-passed and everything is being passed to NGINX, right?

[addison74]

I always make Varnish works on ports with Apache/Nginx before installing Turpentine extension.

1. check if varnish and nginx is listening on the right ports. 800ms load time is like using Nginx directly
2. how did you installed Turpentine? as root? go and check permissions of your Magento files
3. do a right configuration in backend paying attention to secret string. check the path to varnish configuration.
4. in cache management click on save Varnish configuration to disk they Apply.
5. restart nginx and varnish services.

Do you have some rules in iptables to limit the number of connections?

[rossmc]

@addison74

1. I checked the ports, nginx is listening correctly listen 8080; as is varnish -a :80 \. I can also access the website on both ports, i.e. staging.our-domain.com:8080.
2. I installed it with sudo apt-get by following the instructions here. However originally I installed version 4.03, when this did not work for me I uninstalled version 4 and installed version 3 as I saw version 4 was only recently supported by Turpentine. Could this be causing some conflict?

I followed the rest of your instructions and double checked the secret string was correct and with the \n appended. Still no change after applying varnish config & restarting nginx & varnish.

I'm not familiar with iptables and have done nothing with them.

$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination  

Could this have something to do with the cookies on the page request?

[addison74]

1. how did you check for ports? Netstat or ps commands?

2. Varnish 4 is causing trouble on Debian 8 (systemd famous issue). You should add Varnish repository to install Varnish 3 correctly.

3. If you check in Chrome the headers can you see Varnish tags (with debug enabled in Turpentine settings of course)?

Please see your post 2. There are no Varnish tags there (if debug is enabled). When Turpentine is working and the configuration file is loaded you don't see information about server or hhvm. Turpentine is hiding this information by default. Conclusion is only one, there is an issue related to applying Turpentine VCL file.

[rossmc]

@addison74

This is the result of the netstat & ps command, it seems to show the ports are set up correctly

$ sudo netstat -nlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      944/sshd        
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1175/sendmail: MTA:
tcp        0      0 127.0.0.1:6082          0.0.0.0:*               LISTEN      6368/varnishd   
tcp        0      0 127.0.0.1:587           0.0.0.0:*               LISTEN      1175/sendmail: MTA:
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      6370/varnishd   
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      6252/nginx      
tcp6       0      0 :::22                   :::*                    LISTEN      944/sshd        
tcp6       0      0 :::9000                 :::*                    LISTEN      6421/hhvm       
tcp6       0      0 :::80                   :::*                    LISTEN      6370/varnishd   
udp        0      0 0.0.0.0:53808           0.0.0.0:*                           613/dhclient    
udp        0      0 0.0.0.0:68              0.0.0.0:*                           613/dhclient    
udp6       0      0 :::38865                :::*                                613/dhclient    
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     8995     1175/sendmail: MTA: /var/run/sendmail/mta/smcontrol
unix  2      [ ACC ]     SEQPACKET  LISTENING     7460     450/systemd-udevd   /run/udev/control
unix  2      [ ACC ]     STREAM     LISTENING     6665     1/init              @/com/ubuntu/upstart
unix  2      [ ACC ]     STREAM     LISTENING     8368     761/dbus-daemon     /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     8667     946/acpid           /var/run/acpid.socket

$ ps aux |grep varnishd
root      6368  0.0  0.1 117300  1412 ?        Ss   16:47   0:00 /usr/sbin/varnishd -P /var/run/varnishd.pid -a :80 -T localhost:6082 -f /var/www/staging/var/default.vcl -S /etc/varnish/secret -p esi_syntax=0x2 -p cli_buffer=16384 -s malloc,256m
nobody    6370  0.0  0.3 271040  3780 ?        Sl   16:47   0:00 /usr/sbin/varnishd -P /var/run/varnishd.pid -a :80 -T localhost:6082 -f /var/www/staging/var/default.vcl -S /etc/varnish/secret -p esi_syntax=0x2 -p cli_buffer=16384 -s malloc,256m
ubuntu    6497  0.0  0.0  10460   940 pts/0    S+   16:53   0:00 grep --color=auto varnishd

I'm using varnish-3.0.7 so I don't think I should have an issue with version 4, I removed it and replaces it with 3.

I agree with your conclusion but how to debug it further and fix?

[rossmc]

@addison74

I see from issue 109, that I can check to see which Varnish VCL by running this command: varnishadm vcl.show "$(varnishadm vcl.list | grep '^active' | awk '{print $3}')" it seems to show the correct one?

# varnishadm vcl.show "$(varnishadm vcl.list | grep '^active' | awk '{print $3}')"
C{
#include <stdlib.h>
#include <stdio.h>
#include <time.h>
#include <pthread.h>
static pthread_mutex_t lrand_mutex = PTHREAD_MUTEX_INITIALIZER;
void generate_uuid(char* buf) {
pthread_mutex_lock(&lrand_mutex);
long a = lrand48();
long b = lrand48();
long c = lrand48();
long d = lrand48();
pthread_mutex_unlock(&lrand_mutex);
sprintf(buf, "frontend=%08lx%04lx%04lx%04lx%04lx%08lx",
a,
b & 0xffff,
(b & ((long)0x0fff0000) >> 16) | 0x4000,
(c & 0x0fff) | 0x8000,
(c & (long)0xffff0000) >> 16,
d
);
return;
}
}C
import std;
backend default {
.host = "127.0.0.1";
.port = "8080";
.first_byte_timeout = 300s;
.between_bytes_timeout = 300s;
}
backend admin {
.host = "127.0.0.1";
.port = "8080";
.first_byte_timeout = 21600s;
.between_bytes_timeout = 21600s;
}
acl crawler_acl {
"127.0.0.1";
}
acl debug_acl {
}
/* -- REMOVED
sub generate_session {
if (req.url ~ ".*[&?]SID=([^&]+).*") {
set req.http.X-Varnish-Faked-Session = regsub(
req.url, ".*[&?]SID=([^&]+).*", "frontend=\1");
} else {
C{
char uuid_buf [50];
generate_uuid(uuid_buf);
VRT_SetHdr(sp, HDR_REQ,
"\030X-Varnish-Faked-Session:",
uuid_buf,
vrt_magic_string_end
);
}C
}
if (req.http.Cookie) {
std.collect(req.http.Cookie);
set req.http.Cookie = req.http.X-Varnish-Faked-Session +
"; " + req.http.Cookie;
} else {
set req.http.Cookie = req.http.X-Varnish-Faked-Session;
}
}
sub generate_session_expires {
C{
time_t now = time(NULL);
struct tm now_tm = *gmtime(&now);
now_tm.tm_sec += 3600;
mktime(&now_tm);
char date_buf [50];
strftime(date_buf, sizeof(date_buf)-1, "%a, %d-%b-%Y %H:%M:%S %Z", &now_tm);
VRT_SetHdr(sp, HDR_RESP,
"\031X-Varnish-Cookie-Expires:",
date_buf,
vrt_magic_string_end
);
}C
}
-- */
sub vcl_recv {
if (req.restarts == 0) {
if (req.http.X-Forwarded-For) {
set req.http.X-Forwarded-For =
req.http.X-Forwarded-For + ", " + client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
}
if (req.http.Accept-Encoding) {
if (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} else if (req.http.Accept-Encoding ~ "deflate") {
set req.http.Accept-Encoding = "deflate";
} else {
unset req.http.Accept-Encoding;
}
}
if (req.http.User-Agent ~ "iP(?:hone|ad|od)|BlackBerry|Palm|Googlebot-Mobile|Mobile|mobile|mobi|Windows Mobile|Safari Mobile|Android|Opera (?:Mini|Mobi)") {
set req.http.X-Normalized-User-Agent = "mobile";
} else if (req.http.User-Agent ~ "MSIE") {
set req.http.X-Normalized-User-Agent = "msie";
} else if (req.http.User-Agent ~ "Firefox") {
set req.http.X-Normalized-User-Agent = "firefox";
} else if (req.http.User-Agent ~ "Chrome") {
set req.http.X-Normalized-User-Agent = "chrome";
} else if (req.http.User-Agent ~ "Safari") {
set req.http.X-Normalized-User-Agent = "safari";
} else if (req.http.User-Agent ~ "Opera") {
set req.http.X-Normalized-User-Agent = "opera";
} else {
set req.http.X-Normalized-User-Agent = "other";
}
if (!true || req.http.Authorization ||
req.request !~ "^(GET|HEAD|OPTIONS)$" ||
req.http.Cookie ~ "varnish_bypass=1") {
if (req.url ~ "^(/media/|/skin/|/js/|/)(?:(?:index|litespeed)\.php/)?admin") {
set req.backend = admin;
}
return (pipe);
}
set req.url = regsuball(req.url, "([^:])//+", "\1/");
if (req.url ~ "^(/media/|/skin/|/js/|/)(?:(?:index|litespeed)\.php/)?") {
set req.http.X-Turpentine-Secret-Handshake = "1";
if (req.url ~ "^(/media/|/skin/|/js/|/)(?:(?:index|litespeed)\.php/)?admin") {
set req.backend = admin;
return (pipe);
}
if (req.http.Cookie ~ "\bcurrency=") {
set req.http.X-Varnish-Currency = regsub(
req.http.Cookie, ".*\bcurrency=([^;]*).*", "\1");
}
if (req.http.Cookie ~ "\bstore=") {
set req.http.X-Varnish-Store = regsub(
req.http.Cookie, ".*\bstore=([^;]*).*", "\1");
}
if (req.url ~ "/turpentine/esi/get(?:Block|FormKey)/") {
set req.http.X-Varnish-Esi-Method = regsub(
req.url, ".*/method/(\w+)/.*", "\1");
set req.http.X-Varnish-Esi-Access = regsub(
req.url, ".*/access/(\w+)/.*", "\1");
if (req.http.X-Varnish-Esi-Method == "esi" && req.esi_level == 0 &&
!(false || client.ip ~ debug_acl)) {
error 403 "External ESI requests are not allowed";
}
}
if (req.http.Cookie !~ "frontend=" && !req.http.X-Varnish-Esi-Method) {
if (client.ip ~ crawler_acl ||
req.http.User-Agent ~ "^(?:ApacheBench/.*|.*Googlebot.*|JoeDog/.*Siege.*|magespeedtest\.com|Nexcessnet_Turpentine/.*)$") {
set req.http.Cookie = "frontend=crawler-session";
} else {
return (pipe);
}
}
if (true &&
req.url ~ ".*\.(?:css|js|jpe?g|png|gif|ico|swf)(?=\?|&|$)") {
unset req.http.Cookie;
unset req.http.X-Varnish-Faked-Session;
return (lookup);
}
if (req.url ~ "^(/media/|/skin/|/js/|/)(?:(?:index|litespeed)\.php/)?(?:admin|api|cron\.php|checkout|checkout/cart|customer|advancednewsletter|onestepcheckout|onepage|turpentine|exporter|contacts|iphone|join|sitemap.xml)" ||
req.url ~ "\?.*__from_store=") {
return (pipe);
}
if (true &&
req.url ~ "(?:[?&](?:__SID|XDEBUG_PROFILE)(?=[&=]|$))") {
return (pass);
}
if (true && req.url ~ "[?&](utm_source|utm_medium|utm_campaign|utm_content|utm_term|gclid|cx|ie|cof|siteurl)=") {
set req.url = regsuball(req.url, "(?:(\?)?|&)(?:utm_source|utm_medium|utm_campaign|utm_content|utm_term|gclid|cx|ie|cof|siteurl)=[^&]+", "\1");
set req.url = regsuball(req.url, "(?:(\?)&|\?$)", "\1");
}
return (lookup);
}
}
sub vcl_pipe {
unset bereq.http.X-Turpentine-Secret-Handshake;
set bereq.http.Connection = "close";
}
sub vcl_hash {
hash_data(req.url);
if (req.http.Host) {
hash_data(req.http.Host);
} else {
hash_data(server.ip);
}
hash_data(req.http.Ssl-Offloaded);
if (req.http.X-Normalized-User-Agent) {
hash_data(req.http.X-Normalized-User-Agent);
}
if (req.http.Accept-Encoding) {
hash_data(req.http.Accept-Encoding);
}
if (req.http.X-Varnish-Store || req.http.X-Varnish-Currency) {
hash_data("s=" + req.http.X-Varnish-Store + "&c=" + req.http.X-Varnish-Currency);
}
if (req.http.X-Varnish-Esi-Access == "private" &&
req.http.Cookie ~ "frontend=") {
hash_data(regsub(req.http.Cookie, "^.*?frontend=([^;]*);*.*$", "\1"));
}
if (req.http.X-Varnish-Esi-Access == "customer_group" &&
req.http.Cookie ~ "customer_group=") {
hash_data(regsub(req.http.Cookie, "^.*?customer_group=([^;]*);*.*$", "\1"));
}
return (hash);
}
sub vcl_hit {
}
sub vcl_fetch {
set req.grace = 15s;
set beresp.http.X-Varnish-Host = req.http.host;
set beresp.http.X-Varnish-URL = req.url;
if (req.url ~ "^(/media/|/skin/|/js/|/)(?:(?:index|litespeed)\.php/)?") {
unset beresp.http.Vary;
set beresp.do_gzip = true;
if (beresp.status != 200 && beresp.status != 404) {
set beresp.ttl = 15s;
return (hit_for_pass);
} else {
if (beresp.http.Set-Cookie) {
set beresp.http.X-Varnish-Set-Cookie = beresp.http.Set-Cookie;
unset beresp.http.Set-Cookie;
}
unset beresp.http.Cache-Control;
unset beresp.http.Expires;
unset beresp.http.Pragma;
unset beresp.http.Cache;
unset beresp.http.Age;
if (beresp.http.X-Turpentine-Esi == "1") {
set beresp.do_esi = true;
}
if (beresp.http.X-Turpentine-Cache == "0") {
set beresp.ttl = 15s;
return (hit_for_pass);
} else {
if (true &&
bereq.url ~ ".*\.(?:css|js|jpe?g|png|gif|ico|swf)(?=\?|&|$)") {
set beresp.ttl = 28800s;
set beresp.http.Cache-Control = "max-age=28800";
} elseif (req.http.X-Varnish-Esi-Method) {
if (req.http.X-Varnish-Esi-Access == "private" &&
req.http.Cookie ~ "frontend=") {
set beresp.http.X-Varnish-Session = regsub(req.http.Cookie,
"^.*?frontend=([^;]*);*.*$", "\1");
}
if (req.http.X-Varnish-Esi-Method == "ajax" &&
req.http.X-Varnish-Esi-Access == "public") {
set beresp.http.Cache-Control = "max-age=" + regsub(
req.url, ".*/ttl/(\d+)/.*", "\1");
}
set beresp.ttl = std.duration(
regsub(
req.url, ".*/ttl/(\d+)/.*", "\1s"),
300s);
if (beresp.ttl == 0s) {
set beresp.ttl = 15s;
return (hit_for_pass);
}
} else {
set beresp.ttl = 7200s;
}
}
}
return (deliver);
}
}
sub vcl_deliver {
if (req.http.X-Varnish-Faked-Session) {
set resp.http.Set-Cookie = req.http.X-Varnish-Faked-Session +
"; expires=" + resp.http.X-Varnish-Cookie-Expires + "; path=/";
if (req.http.Host) {
if (req.http.User-Agent ~ "^(?:ApacheBench/.*|.*Googlebot.*|JoeDog/.*Siege.*|magespeedtest\.com|Nexcessnet_Turpentine/.*)$") {
set resp.http.Set-Cookie = resp.http.Set-Cookie +
"; domain=" + regsub(req.http.Host, ":\d+$", "");
} else {
if(req.http.Host ~ "") {
set resp.http.Set-Cookie = resp.http.Set-Cookie +
"; domain=";
} else {
set resp.http.Set-Cookie = resp.http.Set-Cookie +
"; domain=" + regsub(req.http.Host, ":\d+$", "");
}
}
}
set resp.http.Set-Cookie = resp.http.Set-Cookie + "; httponly";
unset resp.http.X-Varnish-Cookie-Expires;
}
if (req.http.X-Varnish-Esi-Method == "ajax" && req.http.X-Varnish-Esi-Access == "private") {
set resp.http.Cache-Control = "no-cache";
}
if (false || client.ip ~ debug_acl) {
set resp.http.X-Varnish-Hits = obj.hits;
set resp.http.X-Varnish-Esi-Method = req.http.X-Varnish-Esi-Method;
set resp.http.X-Varnish-Esi-Access = req.http.X-Varnish-Esi-Access;
set resp.http.X-Varnish-Currency = req.http.X-Varnish-Currency;
set resp.http.X-Varnish-Store = req.http.X-Varnish-Store;
} else {
unset resp.http.X-Varnish;
unset resp.http.Via;
unset resp.http.X-Powered-By;
unset resp.http.Server;
unset resp.http.X-Turpentine-Cache;
unset resp.http.X-Turpentine-Esi;
unset resp.http.X-Turpentine-Flush-Events;
unset resp.http.X-Turpentine-Block;
unset resp.http.X-Varnish-Session;
unset resp.http.X-Varnish-Host;
unset resp.http.X-Varnish-URL;
unset resp.http.X-Varnish-Set-Cookie;
}
}

[rossmc]

@addison74 also with debugging enabled I see no varnish tags in the headers and the X-Powered-By:HHVM/3.9.0 tag is still in the header.

[addison74]

I am not using Nginx as webserver, but you should check your configuration file because you need to add more statements then usual when Nginx is behind a proxy server like Varnish. Do a Google search for this scenario, there is a plenty of tutorials.

If you want to test Magento + Varnish + Turpentine I suggest installing Webmin/Virtualmin pack, it comes with everything you need to run Magento into virtual hosts. Apache needs only ports changing to run. Webmin allows you to switch from Apache to Nginx very easy.

[addison74]

Warm the cache.

Stop varnish service. Can you navigate inside your website?
Stop nginx service. Can you navigate inside your website?

My first conclusion Nginx is not well configured.

[addison74]

just an example: http://mikkel.hoegh.org/2012/07/24/varnish-as-reverse-proxy-with-nginx-as-web-server-and-ssl-terminator/

Please note there statements like: proxy_pass, proxy_set_header, ...

Are you using them into your configuration file?

[rossmc]

@addison74

Thanks for the good advice, I look into that and get back to you.

BTW I can't access the website with varnish or nginx stopped.

this is my current nginx configuration file

server {
         # Listen on port 80 as well as post 443 for SSL connections.
        listen 8080;
        #listen 443 default ssl;

    #server_name localhost;
    server_name staging-uat.our-domain.ie www.staging-uat.our-domain.ie;


    # Specify path to your SSL certificates.
   #ssl_certificate /etc/nginx/certificates/yourcertificate.crt;
    #ssl_certificate_key /etc/nginx/certificates/yourcertificate.key;

    # Path to the files in which you wish to
    # store your access and error logs.
    #access_log /path/to/your/logs/access_log;
    #error_log /path/to/your/logs/error_log;

    # If the site is accessed via mydomain.com
    # automatically redirect to www.magento.localhost.com.
    #if ($host = 'staging' ) {
        #rewrite ^/(.*)$ http://www.staging/$1permanent;
   #}

    root /var/www/staging/;
    auth_basic "Restricted website - authorised access only";
    auth_basic_user_file /etc/nginx/.htpasswd;

    location / {
        index index.html index.htm index.php;
        try_files $uri $uri/ @handler;
    }

    #include hhvm.conf;  # INCLUDE HHVM HERE

    # Deny access to specific directories no one
    # in particular needs access to anyways.
    location /app/ { deny all; }
    location /includes/ { deny all; }
    location /lib/ { deny all; }
    location /media/downloadable/ { deny all; }
    location /pkginfo/ { deny all; }
    location /report/config.xml { deny all; }
    location /var/ { deny all; }

    # Allow only those who have a login name and password
    # to view the export folder. Refer to /etc/nginx/htpassword.
    #location /var/export/ {
    #    auth_basic "Restricted";
    #    auth_basic_user_file htpasswd;
    #    autoindex on;
    #}

    # Deny all attempts to access hidden files
    # such as .htaccess, .htpasswd, etc...
    location ~ /\. {
         deny all;
         access_log off;
         log_not_found off;
   }

    # This redirect is added so to use Magentos
    # common front handler when handling incoming URLs.
    location @handler {
        rewrite / /index.php;
    }

    # Forward paths such as /js/index.php/x.js
    # to their relevant handler.
    location ~ .php/ {
        rewrite ^(.*.php)/ $1 last;
    }

    ##
    # Rewrite for versioned CSS+JS via filemtime
    ##
    location ~* ^.+\.(css|js)$ {
        rewrite ^(.+)\.(\d+)\.(css|js)$ $1.$3 last;
        expires 31536000s;
        access_log off;
        log_not_found off;
        add_header Pragma public;
        add_header Cache-Control "max-age=31536000, public";
    }
    ##
    # Aggressive caching for static files
    # If you alter static files often, please use 
    # add_header Cache-Control "max-age=31536000, public, must-revalidate, proxy-revalidate";
    ##
    location ~* \.(asf|asx|wax|wmv|wmx|avi|bmp|class|divx|doc|docx|eot|exe|gif|gz|gzip|ico|jpg|jpeg|jpe|mdb|mid|midi|mov|qt|mp3|m4a|mp4|m4v|mpeg|mpg|mpe|mpp|odb|odc|odf|odg|odp|ods|odt|ogg|ogv|otf|pdf|png|pot|pps|ppt|pptx|ra|ram|svg|svgz|swf|tar|t?gz|tif|tiff|ttf|wav|webm|wma|woff|wri|xla|xls|xlsx|xlt|xlw|zip)$ {
        expires 31536000s;
        access_log off;
        log_not_found off;
        add_header Pragma public;
        add_header Cache-Control "max-age=31536000, public";
    }

       # Handle the exectution of .php files.
    location ~ .php$ {
        if (!-e $request_filename) {
            rewrite / /index.php last;
        }
        expires off;

        # --HHVM CONFIG START--
        fastcgi_pass   127.0.0.1:9000;
        fastcgi_index  index.php;
        fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include        fastcgi_params;
        try_files $uri $uri/ @handler; 
        # --HHVM CONFIG END--

        fastcgi_param MAGE_RUN_CODE default;
        fastcgi_param MAGE_RUN_TYPE store;
        include fastcgi_params;
    }
}

[addison74]

If your website is cached by Varnish when you stop nginx service you can continue visiting the website.

You don't need a lot of statements in your nginx configuration file. For example those with expire tag. Varnish will do the job once it's working.

Check also if there are any other configuration files for Nginx to avoid conflict.

Create a simple configuration file for the beginning. There are many tutorials you can find using Google searching for nginx behind varnish.

[rossmc]

I get a 504 gateway error when I use proxy_pass, proxy_set_header and proxy_redirect settings.

These settings are for setting up NGINX as an SSL terminator which process ssl and passes the decrypted request to Varnish. Fabrizio Branca has a post on the same thing which explains it better.

I've checked out some other posts on NGINX + VARNISH and there NGINX configuration is the same as mine, for example this post by Ash Smith

[rossmc]

@aricwatson @addison74

Got it working! I started from scratch by scrapping our staging server and creating a new one from an AWS image of our origional staging server before I installed varnish, and it works. I think it must have been a conflict in varnish caused by installing varnish v4 first.

Still not working perfectly but the other issues are not related to this so I will close the issue but if you can offer any insight into the below issues I'm facing I would be very grateful.

1) Also, I'm loosing my cart items when I refresh the page homepage but when I go to an inside page I see the item in the cart but the html & css is all wrong. The topcart is being generated by a template in our theme app/design/frontend/blacknwhite/default/template/checkout/cart/topcart.phtml.

2) When I have a query string such as ?___from_store=0&___store=ie appended to the URL varnish does not work, or at least is much slower.

3) Varnish cache is not working when I access the page from chrome on ubuntu. I cleared cache, cookies etc. It's chome for Linux Version 40.0.2214.95 (64-bit). Seems to work in all other browsers so not a huge issue.

I'll try to sort these out before opening another issue but if you have any tips they'd be more than welcome.

Thanks :)

[campan]

Same problem with the cart and login as it is described by @rossmc in question#2
We tried many configurations but every time when we add product to cart it disappears on home page in top cart section but appears when we go to categories page or product page. How can we correctly exclude it from caching?

[aricwatson]

every time when we add product to cart it disappears on home page in top cart section but appears when we go to categories page or product page.

I'm not sure what you mean by it disappears? Can you elaborate or share screenshots?

[campan]

not sure that screenshot will be helpful but anyway it is attached:
Usually when we add products the zero turns to qunatity of products which were already added, but with the turpentine cache the top cart section always shows 0 while if we go to any category or product page we see just actual quantity of products which are displayed in that section. Same problem with the login section.

[aricwatson]

with the turpentine cache the top cart section always shows 0 while if we go to any category or product page we see just actual quantity of products which are displayed in that section.

So, are you saying that if you go to a category page you're seeing a number of items in the cart that's the same as the number of items in the category? That's strange.

Or are you saying that on category and product pages the number of items in the cart is displaying properly whereas it always shows '0' on the homepage? If that's the case, I'd look into what ESI policies are set up for that homepage block (the one with the cart) - it sounds like it's not being hole-punched.

[campan]

Thx, we already solved that by proper configuration of cache module.
Have a nice day.

[aricwatson]

Good to hear!