ETag and HTTP Caching

Improve performance of your apps by taking full advantage of HTTP with ETag and caching. Use ETag, If-Match and If-None-Match header to detect data editing conflicts.

RESTful API Engine fully embraced the HTTP features that were invented to make the Internet faster.

ETag

A web resource may have an optional ETag header also known as an entity tag. Its value must be a unique identifier for the resource content. The engine sets the ETag header to the hash value of the current version of the resource. Web browsers associate the entity tag with the URL of the fetched resource automatically.

If-None-Match

Browsers specify the entity tag in the If-None-Match header when a web application makes the next attempt to get the previously fetched resource. The engine produces the response data, calculates its hash value, and compares the new ETag with the If-None-Match value in the request header. If both values match, then there is no need to transfer the resource content to the client. The engine will set the response status to 304 Not Modified and keep the response body empty. Browsers understand this status code as the instruction to use the previously fetched content.

The ETag and If-None-Match headers provide a simple mechanism to reduce the volume of the client/server traffic. This is especially important when BLOB resources are fetched by the client app repeatedly. Large photos, video, and sound clips are downloaded to the client only once if there are no changes. The subsequent requests to the same resource will result in “304” responses from the server as long as its content has not changed.

The network trace shows two GET requests fetching the list of products. The first request with the "200" status has fetched the resource data. The second request with the "304" status has fetched an empty body and had to re-use the cached data of the previous response instead.
The network trace shows two GET requests fetching the list of products. The first request with the "200" status has fetched the resource data. The second request with the "304" status has fetched an empty body and had to re-use the cached data of the previous response instead.

The performance gains are achieved through the reduced response size. The engine will still produce the resource content on the server and calculate its hash values in response to each network request from the client.

If-Match

CRUD operations may find another use for the entity tags. The entity tag specified in the If-Match header of the request will let the RESTful API Engine to detect the midair collisions. The engine will figure the current ETag of the resource when PUT, POST, PATCH, or DELETE are specified as the HTTP method of the request. If the values of ETag and If-Match are not equal, then the error with the 412 Precondition Failed status is returned. Simply put, the error means that another user has changed the resource after it was fetched by the client app.

The value of the ETag header is automatically copied to the etag property of the _links.self hypermedia control in the resource fetched with the $app.restful method.

Client apps can include the etag property in the method argument when changing data or invoking custom actions. If the original resource content with the modified fields is specified in the body property of the $app.restful method argument, then set the etag to true for conflict detection. Otherwise set the etag value to the _links.self.etag property of the previously fetched resource.

Here is an example of the code that will demonstrate the midair collision.

JavaScript
1234567891011121314151617181920212223242526272829303132333435window.addEventListener('load', e => {
    document.querySelector("h1").addEventListener("click", e => {
        // fetch the arbitrary product resource
        $app.restful({
            url: '/v2/products/17'
        }).then(product => {
            // change the unitPrice
            product.unitPrice = product.unitPrice + 0.1;
            // **** "Edit" Attempt #1 *****
            // Send the modified body to the URL specified in
            // the "edit" hypermedia control to PATCH the resource.
            $app.restful({
                url: product._links.edit,
                body: product,
                etag: true
            }).then(changedProduct => {
                // alert the user to the new UnitPrice
                alert(
                    'The new price of "' + changedProduct.productName +
                    '" is ' + changedProduct.unitPrice);
                // The second attempt to patch the product resource will fail 
                // since the etag in "product._links.self.etag" does not match
                // the ETag of the changed product resource on the server.
                product.unitPrice = product.unitPrice + 0.1;
                // **** "Edit" Attempt #2 *****
                $app.restful({
                    url: product._links.edit,
                    body: product,
                    etag: true
                })
                    .catch(restfulException);
            });
        });
    });
});

Add the code to the bottom of the closure in the spa4.js file of the SPA4 client app. The code will run when users click the h1 element showing the name of the app at the top of the page.

The following sequence of events will take place:

  1. User clicks the header of the app.
  2. The product with the primary key specified in the url parameter of the $app.restful method argument is fetched from the /v2/products/17 resource (see the sample response data).
  3. The unitPrice property of the product variable is increased by $0.01. The variable is pointing to the object fetched in the previous step.
  4. The first attempt to edit the resource is performed. The result of the edit is the new state of the resource in the changedProduct argument. The alert will show the name of the changed product and its unitPrice.
  5. The unitPrice property of the same product variable is increased again by the same amount.
  6. The second attempt to edit the resource is performed with the body property in argument still pointing to the product variable. The $app.restful method will use the “old” entity tag extracted from the variable as the value of the If-Match header. The request fails since the ETag of the resource on the server does not match.

The "GET" request fetches the resource from the server. The first "PATCH" request changes the resource field successfully. The second "PATCH" request fails with the "412 Precondition Failed" status due to the midair conflict.
The "GET" request fetches the resource from the server. The first "PATCH" request changes the resource field successfully. The second "PATCH" request fails with the "412 Precondition Failed" status due to the midair conflict.

The screenshot shows the alert displayed after the first attempt to edit the product.

image4.png

The error message will be reported when the alert is closed since the second attempt to edit the product has failed. The engine will recommend specifying the different entity tag in the If-Match header.

image2.png
The ETag of the resource specified in the If-Match header of the PATCH request allows detecting the data editing conflicts.

Both attempts to edit the product will succeed if the etag property is removed from the $app.restful argument of the second attempt.

ETag is the version number of the resource. Entity tag makes it possible to detect conflicts when specified in the If-Match header.

A better approach is to use the resource that was fetched during the first attempt to edit the product. Replace references to the product variable with the changedProduct and keep etag property for robust conflict detection.

JavaScript
1234567891011121314151617181920212223242526// fetch the arbitrary product resource
$app.restful({
    url: '/v2/products/17'
}).then(product => {
    // change the unitPrice
    product.unitPrice = product.unitPrice + 0.1;
    // **** "Edit" Attempt #1 *****
    $app.restful({
        url: product._links.edit,
        body: product,
        etag: true
    }).then(changedProduct => {
        // alert the user to the new UnitPrice
        alert(
            'The new price of "' + changedProduct.productName +
            '" is ' + changedProduct.unitPrice);
        changedProduct.unitPrice = changedProduct.unitPrice + 0.1;
        // **** "Edit" Attempt #2 *****
        $app.restful({
            "url": changedProduct._links.edit,
            "body": changedProduct,
            "etag": true
        })
            .catch(restfulException);
    });
});

If the body property is not referencing the original resource fetched with the $app.restful method, then set the etag property explicitly. This is the example of the ad-hoc body parameter with the reference to the etag value.

JavaScript
123456789101112131415161718192021222324252627// fetch the arbitrary product resource
$app.restful({
    url: '/v2/products/17'
}).then(product => {
    // change the unitPrice
    product.unitPrice = product.unitPrice + 0.1;
    // **** "Edit" Attempt #1 *****
    $app.restful({
        url: product._links.edit,
        body: product,
        etag: true
    }).then(changedProduct => {
        // alert the user to the new UnitPrice
        alert(
            'The new price of "' + changedProduct.productName +
            '" is ' + changedProduct.unitPrice);
        // **** "Edit" Attempt #2 *****
        $app.restful({
            "url": changedProduct._links.edit,
            "body": {
                unitPrice: changedProduct.unitPrice + 0.1
            },
            "etag": changedProduct._links.self.etag
        })
            .catch(restfulException);
    });
});

Conflict detection requires a certain amount of work on the server. Do not specify the etag parameter when invoking the $app.restful method if the simultaneous editing of the resources in the client apps is not expected.

Cache-Control

The combination of the ETag and If-None-Match headers allows avoiding the redundant transfer of the resource data to the client. The elimination of the GET requests to the previously fetched resources will provide an enormous performance boost both to the client app and the backend application.

Universal Data Caching

The framework in the foundation of the applications created with Code On Time implements the universal data caching. The simple rules provided by developers can instruct the app to keep some of the fetched data in the application cache for the specified duration. The cached data is shared by the entire user base. This eliminates the database interactions and significantly reduces the response time of the backend application.

This is the example of the universal caching rules directing the Data Aquarium framework to keep the responses from the Suppliers and Categories controllers in the cache for 1 minute. The rules are defined in ~/app/touch-settings.json configuration file.

JSON
12345678910{
  "server": {
    "cache": {
      "rules": {
        "suppliers|categories": {
          "duration": 1
        }
      }
    }
}

By default, only the private caching of the RESTful resources is allowed on the client. Entity tags coupled with the 304 Not Modified status code instruct the clients to reuse the previously fetched data.

The built-in RESTful API Engine piggybacks on the same rules to cache the responses associated with the specific URLs in the application cache on the server. Furthermore it instructs the clients to keep the responses in their own cache for the same duration by including the max-age parameter in the Cache-Control header.

The engine navigates the segments of the requested URL and figures the data controller that will be producing the response. If there are caching rules, then the engine will try to locate the response in the cache. If the previous response is not available, then the physical execution of the request will take place with the subsequent placement to the server cache. In either case the max-age parameter is specified in the Cache-Control header.

The caching of the RESTful resource greatly reduces the response time of the fist fetch by the client. The max-age parameter eliminates the further requests completely for the specified duration.

The single page application SPA4 with CRUD fetches the lookup resources of categories and suppliers when the Edit Product form is displayed.

image7.png
The items of the Supplier Company Name and CategoryName lookups are fetched from the RESTful Application with GET requests. The physical data transfer happens one time only for each lookup for each subsequent opening of the Edit Product form.

The network log shows the requests executed by the client after the following sequence of user actions:

Firefox Developer Tools provide a complete list of HTTP requests executed by the web application. Other browsers may not provide the full picture of the network traffic between the client and the server.
The RESTful Application responds to the lookup requests with the HTTP status code 304. It tells the client that the ETag of the corresponding resource is unchanged and the previously fetched copy of the response can be used instead.
  • STEP 1: Click the Aniseed Syrup product in the product list.
    • The OPTION request is performed by the browser to validate the CORS policy of the backend application.
    • The /v2/products/3 resource is fetched with the response code 200.
    • The Edit Product form is constructed, which causes one PATCH request preceded by the OPTIONS check. The application generates the form using the metadata reported by the RESTful API Engine. The form layout is reused in the subsequent request.
    • A request to GET the suppliers fetches the SupplierID lookup data. It is preceded by the OPTIONS validation request.
    • A request to GET the categories fetches the CategoryID lookup data. It is also preceded by the OPTIONS validation request.
  • Click the Cancel button to close the form.
  • STEP 2: Click the same product row the second time.
    • The /v2/products/3 resource is fetched with the response code 200. The previous PATCH request to this resource has invalidated the entity tag in the view of the browser. The resource data is fetched again even though the data has not changed on the server.
    • Suppliers are fetched with the 200 status code directly from the client cache.
      max-age | local cache
    • Categories are fetched with the 200 status code directly from the client cache.
      max-age | local cache
  • Click the Cancel button to close the form.
  • STEP 3: Click the Aniseed Syrup product row the third time.
    • The /v2/products/3 resource is fetched from cache after the 304 status code is reported by the server indicating that the content has not changed. The request was processed on the server but no data was transmitted to the client.
      GET | 304 | local cache
    • Suppliers are fetched with the 200 status code directly from the client cache.
      max-age | local cache
    • Categories are fetched with the 200 status code directly from the client cache.
      max-age | local cache
  • Click the Cancel button to close the form.
  • STEP 4: Click the Alice Mutton product in the product list.
    • The /v2/products/17 resource is fetched with the response code 200. It is preceded by the OPTIONS validation request.
    • Suppliers are fetched with the 200 status code directly from the client cache.
      max-age | local cache
    • Categories are fetched with the 200 status code directly from the client cache.
      max-age | local cache
  • Click the Cancel button to close the form.
  • STEP 5: Wait for 30 minutes and click the Alice Mutton product the second time.
    • The /v2/products/17 resource is requested but not fetched. The status code of the response is 401 Unauthorized. The access token has expired.
    • Method $app.restful performs automatic refresh of the access token by executing the POST request to the /oauth2/v2/token resource.
    • Method $app.restful tries again to request the /v2/products/17 resource with a new access token. The resource is fetched with the response code 200.
    • Suppliers are fetched from cache after the 304 status code is reported by the server indicating that the resource content has not changed.
      GET | 304 | local cache
    • Categories are fetched from cache after the 304 status code is reported by the server indicating that the resource content has not changed.
      GET | 304 | local cache
  • Click the Cancel button to close the form.
  • STEP 6: Click the Alice Mutton product the third time.
    • The /v2/products/17 resource is fetched from cache after the 304 status code is reported by the server indicating that the resource content has not changed.
      GET | 304 | local cache
    • Suppliers are fetched with the 200 status code directly from the client cache.
      max-age | local cache
    • Categories are fetched with the 200 status code directly from the client cache.
      max-age | local cache
  • Web applications are taking full advantage of the HTTP Caching when working with the RESTful API Engine of applications created with Code On Time. Native client apps may also mirror the behavior of the browsers by inspecting the ETag and Cache-Control headers of responses and specifying the previously fetched entity tags in the If-None-Match header of requests.

    Public API Key

    If an API Key is specified in the URL and caching is enabled, then the Cache-Control header is set to the public value. It allows responses to be stored in the shared caches on the way to the client.

    Here is the sample item Baked Goods from the categories collection resource. It includes the API Key aptly named “public” in the self hypermedia control and in the value of the picture field.

    JSON
    1234567891011121314{
        "_links": {
            "self": {
                "href": "/v2/public/categories/217"
            }
        },
        "categoryId": 217,
        "categoryName": "Baked Goods",
        "description": "Breads, cookies, pastries",
        "picture": "/v2/public/categories/MjE3L3BpY3R1cmUvag.jpeg",
        "fileName": "bread.jpeg",
        "contentType": "image/jpeg",
        "length": 13155
    }

    The API Key allows loading the resource directly in the browser window.

    image5.png

    Perform the following sequence of actions while monitoring the requests:

    • STEP 1: Open a new browser window, open Developer Tools, and enter the full URL of the picture directly in the address bar to see the /v2/public/categories/MjE3L3BpY3R1cmUvag.jpeg resource. Note that you file name may be different than the one in this example.
      • The image is fetched with the GET request.
        GET | 200
    • STEP 2: Place cursor in the browser address bar and press Enter key
      • The image is fetched from the local private cache. The network request is not performed.
        max-age | local cache
    • STEP 3: Place cursor in the browser address bar and press Enter key one more time.
      • Once more the image is fetched from the private cache without any help from the server.
        max-age | local cache
    • STEP 4: Click the Refresh button on the left-hand side of the address bar in the browser.
      • The browser makes a GET request to the server with the If-None-Match header set to the ETag of the image in the local cache.
      • The server responds with the 304 Not Modified status and nothing in the body.
      • The image is fetched from the local cache again.
        GET | 304 | local cache

    Each successful request to GET the image is followed by a 404 error. Most browsers will attempt to load the "favico.ico" when an URL is entered in the address bar. The 404 response simply indicate that there is no icon in the application root on the server.
    Each successful request to GET the image is followed by a 404 error. Most browsers will attempt to load the "favico.ico" when an URL is entered in the address bar. The 404 response simply indicate that there is no icon in the application root on the server.

    RESTful API Engine will include the max-age and s-maxage parameters in the Cache-Control header if the path or query parameter specifies an API Key. The first parameter will tell the browsers to keep the response in the private cache. The second parameter will tell the shared caches and proxies to keep the public response for the specified duration. This will completely eliminate the handling of the incoming requests by the application server, since the shared cache server will have the response ready for the clients.

    JSON and BLOB responses to the requests with the API keys  specified in the URLs are configured for public caching by the browsers, shared caches, and proxies.
    JSON and BLOB responses to the requests with the API keys specified in the URLs are configured for public caching by the browsers, shared caches, and proxies.

    Embedding With Caching

    RESTful API Engine does not suffer from the under-fetching problem typical to the Level 2 REST APIs and below. Developers can fetch multiple resources in a single request thanks to the hypermedia.

    Consider the products collection with the limit of 1 item presented below. The links supplierId and categoryId in the one and only item in the collection are marked with the embeddable property.

    JSON
    12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061{
        "_links": {
            "self": {
                "href": "/v2/products?limit=1"
            },
            "first": {
                "href": "/v2/products?page=0&limit=5"
            },
            "lookup-supplierId": {
                "href": "/v2/suppliers?count=true&fields=supplierId,companyName",
                "embeddable": true
            },
            "lookup-categoryId": {
                "href": "/v2/categories?count=true&fields=categoryId,categoryName",
                "embeddable": true
            },
            "create": {
                "href": "/v2/products",
                "method": "POST"
            },
            "do-something": {
                "href": "/v2/products/do-something",
                "method": "POST"
            },
            "report1": {
                "href": "/v2/products/report1"
            },
            "schema": {
                "href": "/v2/products?_schema=true&limit=1"
            }
        },
        "collection": [
            {
                "_links": {
                    "self": {
                        "href": "/v2/products/17"
                    },
                    "supplierId": {
                        "href": "/v2/products/17/supplier-id",
                        "embeddable": true
                    },
                    "categoryId": {
                        "href": "/v2/products/17/category-id",
                        "embeddable": true
                    }
                },
                "productId": 17,
                "productName": "Alice Mutton",
                "supplierId": 7,
                "supplierCompanyName": "Pavlova, Ltd.",
                "categoryId": 6,
                "categoryName": "Meat/Poultry",
                "quantityPerUnit": "20 - 1 kg tins",
                "unitPrice": 16.9000,
                "unitsInStock": 97,
                "unitsOnOrder": 23,
                "reorderLevel": 5,
                "discontinued": false
            }
        ]
    }

    Modify the URL of the request to have the _embed parameter with the value that enumerates the embeddable resources. Also add the _links=false parameter to remove the hypermedia controls from the output

    /v2/products?limit=1&_embed=supplierId,categoryId&_links=false

    The supplierId and categoryId field values in the collection item of the response are objects with the properties of the embedded resources. The engine automatically replaces the original fields with the corresponding embeddable resources.

    JSON
    1234567891011121314151617181920212223242526272829303132333435363738394041{
        "collection": [
            {
                "productId": 17,
                "productName": "Alice Mutton",
                "supplierId": {
                    "supplierId": 7,
                    "companyName": "Pavlova, Ltd.",
                    "contactName": "Ian Devling",
                    "contactTitle": "Marketing Manager",
                    "Products2": "/v2/products/17/supplier-id/products2",
                    "address": "74 Rose St. Moonie Ponds",
                    "city": "Melbourne",
                    "region": "Victoria",
                    "postalCode": "3058",
                    "country": "Australia",
                    "phone": "(03) 444-2343",
                    "fax": "(03) 444-6588",
                    "products": "/v2/products/17/supplier-id/products"
                },
                "supplierCompanyName": "Pavlova, Ltd.",
                "categoryId": {
                    "categoryId": 6,
                    "categoryName": "Meat/Poultry",
                    "description": "Prepared meats",
                    "picture": "/v2/products/17/category-id/picture",
                    "fileName": null,
                    "contentType": null,
                    "length": null,
                    "products": "/v2/products/17/category-id/products"
                },
                "categoryName": "Meat/Poultry",
                "quantityPerUnit": "20 - 1 kg tins",
                "unitPrice": 16.9000,
                "unitsInStock": 97,
                "unitsOnOrder": 23,
                "reorderLevel": 5,
                "discontinued": false
            }
        ]
    }

    Take a note of the time it took to produce the response in a tool like Postman. Try execution of the same request a few times in the row. The response time will decrease thanks to the caching of Suppliers and Categories.

    Remove the limit=1 parameter from the URL of the request and take a note of the time it takes to complete. About 80 product items will be included in the collection with the supplierId and categoryId fields resolved. Subsequent requests to the same URL are executing up to 40 times faster.

    Add the Products data controller to the caching rules and you will discover further improvement in the response time, which was up to 200 times faster compared to the app configuration without caching in our own test.

    Learn about fetching the data in the exact shape that is needed for the client app in the No More Under-Fetching Or Over-Fetching segment of the RESTful Workshop.

    Latest Version

    HTTP Caching does improve performance but makes it impossible to manage the data in the client apps. Universal Data Caching supports the exemptions. Developers can specify the roles that will not be affected by the caching rules. If the identity of the user associated with the access token or API Key has the role or scope of an exemption, then the engine will return the current version of the resource.

    RESTful API Engine provides the means of reading the latest version of resources by users that are not exempt from caching.

    Consider the categories collection resource in the next snippet.

    The hypermedia control self has the max-age property letting the developers know that there is the caching rule set for this resource. The caching duration is 60 seconds. There is also the latest-version hypermedia control that provides access to the latest version of the resource content.

    JSON
    1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465{
        "_links": {
            "self": {
                "href": "/v2/categories",
                "max-age": 60
            },
            "latest-version": {
                "href": "/v2/categories/latest-version"
            },
            "first": {
                "href": "/v2/categories?page=0&limit=10"
            },
            "create": {
                "href": "/v2/categories",
                "method": "POST"
            },
            "schema": {
                "href": "/v2/categories?_schema=true"
            }
        },
        "collection": [
            {
                "_links": {
                    "self": {
                        "href": "/v2/categories/217"
                    }
                },
                "categoryId": 217,
                "categoryName": "Baked Goods",
                "description": "Breads, cookies, pastries",
                "picture": "/v2/categories/MjE3L3BpY3R1cmUvag.jpeg",
                "fileName": "bread.jpeg",
                "contentType": "image/jpeg",
                "length": 13155
            },
            {
                "_links": {
                    "self": {
                        "href": "/v2/categories/1"
                    }
                },
                "categoryId": 1,
                "categoryName": "Beverages",
                "description": "Soft drinks, coffees, teas, beers, and ales",
                "picture": "/v2/categories/1/picture",
                "fileName": null,
                "contentType": null,
                "length": null
            },
            {
                "_links": {
                    "self": {
                        "href": "/v2/categories/2"
                    }
                },
                "categoryId": 2,
                "categoryName": "Condiments",
                "description": "Sweet and savory sauces, relishes, spreads, and seasonings",
                "picture": "/v2/categories/2/picture",
                "fileName": null,
                "contentType": null,
                "length": null
            }
        ]
    }

    Here is the latest version of the categories collection resource. Note the latest-version segment is in the path of the hypermedia links. This magic suffix in the URL path will create a caching exemption that overrides the server caching rules.

    JSON
    12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061{
        "_links": {
            "self": {
                "href": "/v2/categories/latest-version"
            },
            "first": {
                "href": "/v2/categories/latest-version?page=0&limit=10"
            },
            "create": {
                "href": "/v2/categories/latest-version",
                "method": "POST"
            },
            "schema": {
                "href": "/v2/categories/latest-version?_schema=true"
            }
        },
        "collection": [
            {
                "_links": {
                    "self": {
                        "href": "/v2/categories/217/latest-version"
                    }
                },
                "categoryId": 217,
                "categoryName": "Baked Goods",
                "description": "Breads, cookies, pastries",
                "picture": "/v2/categories/latest-version/MjE3L3BpY3R1cmUvag.jpeg",
                "fileName": "bread.jpeg",
                "contentType": "image/jpeg",
                "length": 13155
            },
            {
                "_links": {
                    "self": {
                        "href": "/v2/categories/1/latest-version"
                    }
                },
                "categoryId": 1,
                "categoryName": "Beverages",
                "description": "Soft drinks, coffees, teas, beers, and ales",
                "picture": "/v2/categories/1/picture/latest-version",
                "fileName": null,
                "contentType": null,
                "length": null
            },
            {
                "_links": {
                    "self": {
                        "href": "/v2/categories/2/latest-version"
                    }
                },
                "categoryId": 2,
                "categoryName": "Condiments",
                "description": "Sweet and savory sauces, relishes, spreads, and seasonings",
                "picture": "/v2/categories/2/picture/latest-version",
                "fileName": null,
                "contentType": null,
                "length": null
            }
        ]
    }

    If the hypermedia controls are used in the client application, then it is trivial to switch between the cached and the latest version of a resource. It is easy to know when the cached data is being served by the backend application.

    If the max-age parameter is specified in the self hypermedia control, then the data is cached. Use the latest-version control to access the latest resource content.

    This tutorial is part of the RESTful Workshop designed to empower individual developers and teams to create amazing enterprise-quality backends for mobile and web applications. Zero experience and no code is required to provide your data with the REST API conforming to the Level 3 of Richardson Maturity Model with the built-in OAuth 2.0 and Open ID Connect.