peterjaap/varnish6.vcl

@peterjaap I'll continue look at this and suggest small fixes on Monday

@ThijsFeryn thanks! I updated the gist accordingly.

@ThijsFeryn we also have some rules in some of our VCLs to bypass routes that are never cached in Magento, does that make sense?

     # Bypass customer, shopping cart, checkout, admin
     if (req.url ~ "/customer" || req.url ~ "/checkout" || req.url ~ "/admin") {
        return (pass);
    }

@ThijsFeryn we also have some rules in some of our VCLs to bypass routes that are never cached in Magento, does that make sense?

     # Bypass customer, shopping cart, checkout, admin
     if (req.url ~ "/customer" || req.url ~ "/checkout" || req.url ~ "/admin") {
        return (pass);
    }

If these pages return Cache-Control: private response headers or variations like Cache-Control: no-cache or Cache-Control: no-store, there is no need to explicitly have those rules for /customer, /checkout and /admin in VCL. Varnish will respect the Cache-Control header and bypass these requests from hitting the cache.

However, if the Cache-Control header doesn't return the right value, you need to write VCL.

Long story short: only write VCL in situations where the built-in VCL doesn't provide a native solution.

@peterjaap The Varnish6.vcl file below you diff has the following code:

sub process_graphql_headers {
    if (req.http.X-Magento-Cache-Id) {
        hash_data(req.http.X-Magento-Cache-Id);

        # When the frontend stops sending the auth token, make sure users stop getting results cached for logged-in users
        if (req.http.Authorization ~ "^Bearer") {
            hash_data("Authorized");
        }
    }

    hash_data(req.http.Store);

    hash_data(req.http.Content-Currency);
}

This code implies that authorized /graphql calls can be cached in Magento. Is that the desired behavior? I was under the impression that authorized /graphql calls should bypass the cache.

Please clarify.

@ThijsFeryn from the Magento 2 docs;

Adobe Commerce and Magento Open Source can cache pages rendered from the results of certain GraphQL queries with full-page caching. Full-page caching improves response time and reduces the load on the server. Without caching, each page might need to run blocks of code and retrieve large amounts of information from the database. Only queries submitted with an HTTP GET operation can be cached. POST queries cannot be cached.

Indeed, I feel Authorized graphql calls should bypass the cache.

Further optimization we do is skipping Varnish fully for certain requests..
Varnish doesn't support SSL so because we hit nginx for static files and immediately serve them by Nginx...

For instance: /static are only files which can be served with nginx

location /media/customer/ { deny all; }
location /media/import/   { deny all; }
location /magento_version { deny all; }

location /static {
  expires max;

  # Remove signature of the static files that is used to overcome the browser cache
  location ~ ^/static/version {
    rewrite ^/static/(version\d*/)?(.*)$ /static/$2 last;
  }

  add_header Cache-Control "public";
  add_header X-Frame-Options "SAMEORIGIN";

  allow all;
}

I also published this one: https://gist.github.com/JeroenBoersma/6ece510192ce4292c79d010284544c56

Further optimization we do is skipping Varnish fully for certain requests.. Varnish doesn't support SSL so because we hit nginx for static files and immediately serve them by Nginx...

For instance: /static are only files which can be served with nginx

location /media/customer/ { deny all; }
location /media/import/   { deny all; }
location /magento_version { deny all; }

location /static {
  expires max;

  # Remove signature of the static files that is used to overcome the browser cache
  location ~ ^/static/version {
    rewrite ^/static/(version\d*/)?(.*)$ /static/$2 last;
  }

  add_header Cache-Control "public";
  add_header X-Frame-Options "SAMEORIGIN";

  allow all;
}

@JeroenBoersma
But that is if you use Nginx as a TLS proxy. I actually advise using Hitch as a TLS proxy:

• It's developed by Varnish Software
• It's open source
• It's a pure TLS proxy that has no HTTP awareness
• It supports both PROXY protocol version 1 & 2

See https://www.varnish-software.com/developers/tutorials/terminate-tls-varnish-hitch/ for more information about Hitch and Varnish.

@peterjaap Here's the notes I added for my team, with pwa studio references for usage

Seems magento 2.4.4 and above, and latest PWA studio have improved cacheability for GQL requests
https://devdocs.magento.com/guides/v2.4/graphql/caching.html#customers

Magento generates a x-cache-id variation - https://github.com/magento/magento2/blob/192b28038dac2e993c65b49fb8b5af10cb520945/app/code/Magento/PageCache/etc/varnish6.vcl#L116-L148

PWA studio constantly grabs it from the response, and updates it to attach to the next requests so that when you log in etc you change customer groups / get cacheability
magento/pwa-studio@626ea1f#diff-b638977b4ea6fd3f29b668736551ed752c5b3915805df1af2f1a7579a006aacbR27

https://github.com/magento/pwa-studio/blob/626ea1f1c384654703501783b219bc7c4efa672d/packages/peregrine/lib/Apollo/links/gqlCacheLink.js#L41

@peterjaap The Varnish6.vcl file below you diff has the following code:

sub process_graphql_headers {
    if (req.http.X-Magento-Cache-Id) {
        hash_data(req.http.X-Magento-Cache-Id);

        # When the frontend stops sending the auth token, make sure users stop getting results cached for logged-in users
        if (req.http.Authorization ~ "^Bearer") {
            hash_data("Authorized");
        }
    }

    hash_data(req.http.Store);

    hash_data(req.http.Content-Currency);
}

This code implies that authorized /graphql calls can be cached in Magento. Is that the desired behavior? I was under the impression that authorized /graphql calls should bypass the cache.

Please clarify.

This specific example implies that caching happens only when X-Magento-Cache-Id is set then, it additionally may add Authorization to it. Although, by default, GraphQL API allows doing queries without Authorization for stuff like catalog, so definitely looks strange.

Also, Authorization makes the cache user-specific, which removes any benefit of caching it in the Varnish server side, as if you have a spike of 1000 users, each of which will generate 1000 variations of the same cache.
I would probably remove Authorization header from the request in vcl_recv if X-Magento-Cache-Id is specified to prevent caching of customer-sensitive data in the first place. As a side effect, it should increase the cache hit ratio, as I imagine Authorization is added automatically if the request is sent when the user gets logged in. But that is implementation specific of the frontend that uses GraphQL, which should always separate customer-specific queries form cacheable ones.

@JeroenBoersma But that is if you use Nginx as a TLS proxy. I actually advise using Hitch as a TLS proxy:

• It's developed by Varnish Software
• It's open source
• It's a pure TLS proxy that has no HTTP awareness
• It supports both PROXY protocol version 1 & 2

See https://www.varnish-software.com/developers/tutorials/terminate-tls-varnish-hitch/ for more information about Hitch and Varnish.

@ThijsFeryn If it would be just SSL termination than that's a true point, in this case it's a combined effort for file serving and (generated) content you get a performance gain by serving the files directly from the server and using Varnish for the generated content. There is no need to cache these static files on the same server in Varnish.

A customer notified us with a list of more marketing pixels....

I think we can include these in the list of parameters to...
https://maxchadwick.xyz/tracking-query-params-registry/ props to @mpchadwick

Source which links to the above list: https://www.rumvision.com/blog/make-sure-to-check-your-caching-rules/

A customer notified us with a list of more marketing pixels....

I think we can include these in the list of parameters to... https://maxchadwick.xyz/tracking-query-params-registry/ props to @mpchadwick

Source which links to the above list: https://www.rumvision.com/blog/make-sure-to-check-your-caching-rules/

@JeroenBoersma: here's the updated VCL:

    if (req.url ~ "(\?|&)(_branch_match_id|_bta_[a-z]+|campid|customid|_ga|gclid|gclsrc|gdf[a-z]+|cx|dm_i|ef_id|epik|ie|igshid|cof|hsa_[a-z]+|_ke|mk[a-z]{3}|msclkid|(mtm|matomo)_[a-z]+|pcrid|p(iwi)?k_[a-z]+|redirect(_log)?_mongo_id|siteurl|s_kwcid|sb_referer_host|si|trk_[a-z]+|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=") {
        set req.url = regsuball(req.url, "(_branch_match_id|_bta_[a-z]+|campid|customid|_ga|gclid|gclsrc|cx|dm_i|ef_id|epik|ie|igshid|cof|hsa_[a-z]+|_ke|mk[a-z]{3}|msclkid|(mtm|matomo)_[a-z]+|pcrid|p(iwi)?k_[a-z]+|redirect(_log)?_mongo_id|siteurl|s_kwcid|sb_referer_host|si|trk_[a-z]+|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=[-_A-z0-9+()%.]+&?", "");
        set req.url = regsub(req.url, "[?|&]+$", "");
    }

@peterjaap and @ThijsFeryn magento/magento2#37524

These marketing pixels should be ignored when hitting graphql otherwise you'll endup regsub way more than you expect. Sometimes, likely unexpectedly, there's a relative URL (with query params) hidden inside the URL.

You have an error on line 44, the close bracket before the || needs removing.

@chrisastley thanks!

Can we add the "gad_source" parameter?
Source: https://support.google.com/google-ads/answer/13327296?hl=en & https://www.rumvision.com/use-cases/making-sure-paid-traffic-has-the-best-page-speed-possible/

@Melleschepers added!

sub process_graphql_headers {
if (req.http.X-Magento-Cache-Id) {
hash_data(req.http.X-Magento-Cache-Id);

    # When the frontend stops sending the auth token, make sure users stop getting results cached for logged-in users
    if (req.http.Authorization ~ "^Bearer") {
        hash_data("Authorized");
    }
}

hash_data(req.http.Store);

hash_data(req.http.Content-Currency);

}
Can we remove Authorization token from frontend as ‘X-Magento-Cache-Id’ already there which provide Authorization so no need of bearer token?

Hello @peterjaap

Could you add nb_klid in marketing parameters ?
klaviyo is adding like below;
?utm_source=Klaviyo&utm_medium=email&utm_campaign=Flash%20Sale%20%7C%20BOGO50%20%7C%20Panther%20Bracelets&nb_klid=01GYYXF9WQJE9K5KMWHXCCJ2EJ&_kx=149hMbYs3n64hbonnZsQJb9Bbn2cWv9qxmrHmgLSqLA.YwSEB3

And I wonder can GA4 follow links even after removing UTM and marketing tags by varnish.vcl?

@eneskirca I've added nb_klid to the list.

The VCL only strips off these parameters for internal usage. This is done so Varnish can calculate the hash correctly, thereby increasing the hit rate. The actual parameter stays in the GET URL, so Javascript based tools like GA4 can still read out the parameter from the GET querystring.

@peterjaap

• Can I use this file for Varnish 7 as well?
• These placeholder, f. ex. {{ host }}, do I need to replace them manually or can I somehow insert the proper values?

@codedge

• I'm not sure, but if it compiles it should work
• You need to generate the VCL from the Magento backend

@codedge

• I'm not sure, but if it compiles it should work
• You need to generate the VCL from the Magento backend

@codedge @peterjaap Compatibility shouldn't be a problem at all.

The Varnish version doesn't really matter that much difference, it's the VCL version you're using that is important. And vcl 4.1; is the right version marker to stay compatible.

Can we add the "gbraid" parameter? :)
Source: https://support.google.com/analytics/answer/11367152?hl=en

@Melleschepers done!

Updated the tracking parameters list to match https://github.com/mpchadwick/tracking-query-params-registry/blob/master/_data/params.csv

@Melleschepers

Notable differences

1. VCL Version and Compatibility

• Magento VCL: Uses vcl 4.0 for compatibility, even though Varnish version 6 or 7 is used.
• Optimized VCL: Uses vcl 4.1—a more current version—while still maintaining compatibility with Varnish 6 and 7.

2. Request Handling (sub vcl_recv)

• Magento VCL: Focuses on request normalization (removing marketing parameters, handling cache control for specific URL paths) but lacks improvements like query parameter sorting or more sophisticated request cleaning.
• Optimized VCL:
  • Query String Optimization: Sorts query parameters using std.querysort() to improve cache efficiency.
  • Empty Query Parameter Removal: Eliminates empty query strings for normalization.
  • Host Header Cleanup: Removes port numbers from the Host header for better cache key normalization.
  • Httpoxy Vulnerability Mitigation: Unsets the proxy header to mitigate potential vulnerabilities.
  • Grace Period for Healthy Backend: Dynamically adjusts the grace period based on backend health.
  • Advanced Query Param Removal: Removes more marketing-related query parameters compared to the original Magento VCL, optimizing cache objects even further.

3. Cookie and Header Management

• Magento VCL: Uses std.collect() on cookies but doesn't perform additional cookie header collapsing or aggressive cleanup of unnecessary headers.
• Optimized VCL: Collapses multiple Cookie headers and removes headers like proxy for more efficient processing and enhanced security.

4. Cache Key Generation (sub vcl_hash)

• Magento VCL: Hashes certain request headers and applies logic for GraphQL and Magento's X-Magento-Vary cookies, but has room for better structure and logic consolidation.
• Optimized VCL:
  • Consolidation: Handles GraphQL requests and Magento cache ID more elegantly by ensuring either the X-Magento-Cache-Id or fallback headers (Store, Content-Currency) are used for hashing.
  • General Security: Adds more structured logic for SSL offloading, improving HTTPS handling and hash key integrity.

5. Backend Response Handling (sub vcl_backend_response)

• Magento VCL:
  • Basic TTL Control: Caches 200 and 404 responses but unsets cookies in a less optimized way.
  • Hit-For-Pass: Implements a basic Hit-For-Pass strategy when responses are uncachable.
• Optimized VCL:
  • Grace and Stale Content: Keeps stale content for up to 3 days, using asynchronous revalidation to refresh the cache while still serving expired content.
  • Set-Cookie Header: Proactively unsets Set-Cookie for cacheable content (GET and HEAD requests) to improve cache hit rates.
  • Strict TTL Control: Sets a short (120s) TTL for uncacheable content and mismatched GraphQL cache IDs to avoid excessive pass-throughs.

6. Header Management on Delivery (sub vcl_deliver)

• Magento VCL: Unsets a range of headers such as X-Magento-Tags, X-Powered-By, and Server, but includes debugging information in a basic form.
• Optimized VCL:
  • Improved Debugging: Adds more specific debugging headers (X-Magento-Cache-Debug), including detailed cache hit/miss reporting.
  • More Aggressive Header Cleanup: Unsets additional headers like X-Varnish, Via, and Link to streamline responses and reduce unnecessary data sent to clients.

7. Security Enhancements

• Magento VCL: Implements basic security by removing sensitive headers but lacks more advanced threat mitigations.
• Optimized VCL: Includes explicit X-Forwarded-Proto logic for HTTPS, better security header management (e.g., removing proxy headers), and optimizations like the removal of the HTTPoxy vulnerability.

8. GraphQL Request Handling

• Magento VCL: Caches only authorized GraphQL requests with appropriate X-Magento-Cache-Id headers, but with slightly less efficient logic for hashing and caching.
• Optimized VCL: More streamlined and optimized logic for handling GraphQL cache keys, including fallback mechanisms to ensure cache key integrity even if X-Magento-Cache-Id is not set.

9. Simplified Logic and Performance Improvements

• The Optimized VCL is much cleaner, with more consolidated logic, fewer repetitive conditions, and more robust performance enhancements (e.g., cookie collapsing, better query handling, etc.). It also introduces enhanced debugging features and more granular cache control based on real-time conditions such as backend health and dynamic grace periods.