Improve API performance with CDN

Cover image taken from Unsplash

This post of part of API performance improvement mini series 🚀

• Improve API performance with CDN(this post)
• Improve API performance with application caching

---

Background

Content Delivery Network (CDN) can be very powerful for performance improvement. I always thought that CDN is only useful for static contents. Recently I read a blog on CDN from a friend of mine, and learned that CDN can be used to improve API performance as well.

The idea is to offload some of the incoming requests (potentially 99%) to CDN so that servers and databases would not be so intense in the event of spiking traffics.

If the cache is hit, the request won't be directed to the API server.

I studied and tested, and here is my initial observation.

In this post, I will share what I have learned so far, with example code, setup and performance test results

I will be using Cloud CDN, but the concept should apply to all CDN providers as well.

How to cache

Generally the contents that are cacheable by CDN will have the following characteristic:

• GET request, with status code 200, 203, 204, 206, 300, 301, 302, 307, 308, 404, 405, 410, 421, 451, or 501
• Cache-Control: public directive, an Expires header, or a static content type.

How long to cache

For static content, you can set as long as you want. But for time sensitive content like API response, we can decide the expiration based on the following factors borrowed from the docs:

Near real-time updates: live feeds for sporting events, traffic, registration platform, hot sales

Setting 1 to 10 seconds expiration could be enough, depends on the requirement

Frequent updates (weekly, daily, or hourly): weather information, front-page news images

Setting to 1 hour, 1 day or 1 week, depends on the requirement

Infrequent updates: website logo, CSS, JavaScript files

As webpack generates files with different hash on each build, generally setting to 6 months or even a year should be okay

How to select which API to cache

As we know, CDN should respect Cache-Control header, unless we enable FORCE_CACHE_ALL mode. We can add the headers on our code to control which API to cache, and how long it should be cached.

Code snippet below is written in Go with fiber, details on the code are omitted, only show the related code to control the cache

Source code is available at cncf-demo/cache-server

We write a middleware, to accept cache duration

func allowCache(dur int) fiber.Handler {
    return func(c *fiber.Ctx) error {
        c.Set(fiber.HeaderCacheControl, fmt.Sprintf("public, max-age=%d", dur))
        return c.Next()
    }
}

Then setting the default to no-store

// default to no store
app.Use(func(c *fiber.Ctx) error {
    c.Set(fiber.HeaderCacheControl, "no-store")
    return c.Next()
})

And apply to the API that needs to be cached

// without cache
app.Get("/", func(c *fiber.Ctx) error {
    // simulate database call
    // for 200ms
    time.Sleep(200 * time.Millisecond)
    return c.JSON(fiber.Map{"result": "ok"})
})

// cache for 5 seconds
app.Get("/cache", allowCache(5), func(c *fiber.Ctx) error {
    // simulate database call
    // for 200ms
    time.Sleep(200 * time.Millisecond)
    return c.JSON(fiber.Map{"result": "ok"})
})

Deploy the code to test

Create a Kubernetes Cluster and deploy the example cacheable API server by running

cd examples/cache-server
kubectl apply -f cache-server-with-cdn.yaml

Update backend service to return Cache ID and cache status on response header

Get the public IP address

kubectl get ingress

Verify the response header with curl, with cache status miss

curl PUBLIC_IP_ADDRESS -v

# only show relevant values are here
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 15
< Cache-Control: no-store
< Cache-ID: KUL-5a71ba6a # here is Cache ID
< Cache-Status: miss # here is Cache-Status
<
{"result":"ok"}

Test the API performance

Visual testing

Open up the browser with the public IP address, and submit request to / and /cache, twice on each API.

/ is without cache, /cache set to cached for 5 seconds

Verify on the Cache-Status header for cache status

The result shows that non-cacheable API (/) results in different response time, mainly caused by the network latency, but not performance improvement. Meanwhile cacheable API (/cache) improved from 281ms to 5ms, about 56X faster.

Benchmark test

Note that I am running the test on local terminal, so the factor of internet speed is ignored. Benchmark is tested with bombardier with the following parameters

-c 200 -n 100000 

-c Maximum number of concurrent connections
-n Number of requests

The API server only run on one replica, with the following resources

resources:
   requests:
      memory: "64Mi"
      cpu: "250m"

Here is the benchmark result for non-cacheable API /, average with 899.61 requests per second, took 1m52s to finish, and average latency 225.16ms

$ bombardier -c 200 -n 100000 IP_ADDRESS

Bombarding IP_ADDRESS with 100000 request(s) using 200 connection(s)
 100000 / 100000 [=====] 100.00% 885/s 1m52s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec       899.61     760.42   10838.69
  Latency      225.16ms    17.68ms   533.07ms
  HTTP codes:
    1xx - 0, 2xx - 100000, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:   197.52KB/s

Here is the benchmark result for cacheable API /cache, average with 12274.13 requests per second, took 8s to finish, and average latency 16.27ms

$ bombardier -c 200 -n 100000 PUBLIC_IP_ADDRESS/cache
Bombarding PUBLIC_IP_ADDRESS/cache with 100000 request(s) using 200 connection(s)
 100000 / 100000 [=====] 100.00% 12162/s 8s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec     12274.13    1594.21   17255.73
  Latency       16.27ms     3.97ms   115.79ms
  HTTP codes:
    1xx - 0, 2xx - 100000, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:     3.42MB/s

The result of cacheable API is impressive 🚀. We get about 14X improvement in latency and RPS 🤯.

Metrics are available from Google Cloud console after running the tests, region in red indicate requests were served from cache

Region pointing down indicate requests served from cache

Each cache entry in CDN cache is identified by a cache key. When a request comes into the cache, the cache converts the URI of the request into a cache key, and then compares it with keys of cached entries. If it finds a match, the cache returns the object associated with that key.

For example, a request for example.com/cat.jpg won't use the cache entry for example.com/cat.jpg?1234. To share cache entries, can be configure with custom cache key.

Users that are physically located in the same region would be served from same cache server and should hit the cache, if the cache server has the record already, regardless the User Agent. Learn more on available locations of Cloud CDN

Sometimes I want to bypass cache

For example, if you have certain groups of users that always wanted to bypass cache even if the contents were set to cache, and the contents were previously cached, Cloud CDN allows us to define request headers to bypass the cache.

Based on the configuration below, if the Bypass-Cache request header is present, Cloud CDN will bypass the cache, more info on how to configure cache bypass

Cache-Status become unreachable with Bypass-Cache header

curl -H 'Bypass-Cache: 1' PUBLIC_IP_ADDRESS/cache -v

# only show relevant values are here
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 15
< Cache-Control: public, max-age=5
< Cache-ID: KUL-33361a1f # here is Cache ID
< Cache-Status: uncacheable # here is Cache-Status
<

Upon remove the header, should get cache status hit, stale or miss

Conclusion

CDN is a very powerful tool and also much cheaper than servers and databases costs. However, do make sure that know how to invalidate cache in the case of emergency, and choose wisely which contents/APIs to cache and the expiry time.

For GraphQL, to utilise CDN may need to serve over http with GET request, or use CDN that made for GraphQL, for example GraphCDN.

For gRPC, no at the moment, may consider application caching.

The concept shows in the blog post should apply to other CDN providers, not only limited to Cloud CDN

Source code is available at cncf-demo/cache-server

---

This post of part of API performance improvement mini series 🚀

• Improve API performance with CDN(this post)
• Improve API performance with application caching