In the last post, Samuell (from the og-aws Slack group) figured out his negative TTL problems in Cloudfront. During that session, we actually solved two different problems. This post is to finish off that conversation and address the other problem, which was how to use the default error page in Cloudfront.
Since the details I share below about Samuell are duplicated from the last post, let me add that he’s a fighter. I was only on a screen-share with him for about an hour to fix both problems, but let me tell you that we were both struggling. We had a timezone difference, that he accommodated for, not me (his IP, website, and company, were from India, Brazil, and Czech Republic if I remember correctly). We also had a language barrier, which wasn’t helped by my terrible microphone (I’ve since replaced it). Also, during the screenshare, the software I chose to use wouldn’t work with his push-to-talk microphone, so everytime he wanted to talk to me, he had to switch focus back to the screenshare window and then talk. I’m sure it was exhausting for him, but he dealt with all the issues with a happy attitude. It was honestly a pleasure to speak with him!
Tabs vs Spaces: Spaces
Favorite IDE: Visual Studio Code
Current OS: Windows 10
iPhone vs Android: Android
Favorite Superhero: I don’t know
Twitter Handle: @sam_uell1
Samuell uses a CMS on Amazon Web Services to manage his website and related content. Users can upload files through the CMS; those files get placed into a single S3 bucket. A Cloudfront web distribution is setup with that S3 bucket as origin. The Cloudfront distribution has configuration for customized error responses, but don’t have a customized error page (the default one is in use).
A picture is worth at least a paragraph in this instance:
Samuell’s setup included a CMS, an S3 bucket, and a Cloudfront for CDN
Default error responses for HTTP codes 400, 403, and 404
In the previous post, I gave some more details about the configuration of the Cloudfront distribution. Feel free to reference them, but there’s nothing special about them.
I never verified this in the screen-share, but Samuell shared that the S3 bucket had a public policy for reads, and the Cloudfront distribution’s origin was the S3 bucket endpoint. static website hosting was not enabled, and the static website endpoint was not the origin. Because the static website was neither enabled nor in use, the S3 error page configuration is irrelevant.
The S3 bucket primarily, if not exclusively, holds web assets (images, stylesheets, scripts). It doesn’t hold any static pages or downloadable files. There would never be a reason to link to those assets, except as assets on a page.
The symptoms were originally caused by the S3/Cloudfront timing and negative TTL issue I posted last time. As Samuell went down that rabbit hole, he found another issue he resolved to fix independently — the default, nice-looking 403 (S3’s 404 equivalent) page from Cloudfront wasn’t showing up, only a very unhelpful XML document with a terrible user experience. Here is the XML document for a known-missing image (test.png).
Error document for a known-missing file in the Cloudfront distribution
Chrome console for the error document
Chrome console for the error document
His expectation was the default error page from Cloudfront, a nice-looking page that told users what the problem was. Instead, he got an XML document with IDs that only a programmer can make sense of. His expectation, I assume, was more along the lines of this fun 404 page from Lego:
Fun 404 page from Lego website, not from Cloudfront
The problem here is the expectation. The XML document is the default error page. There is no special, default 404 or 403 page that Cloudfront will show. As with most AWS services, Cloudfront is incredibly customizable, but lacking some default actions that would make it more valuable.
When specifying the error pages in the Cloudfront configuration, you have the option to customize the error response. If you select yes, you must provide the resource (likely an HTML page) to respond with. If you select no, Cloudfront will show whatever the origin shows, and S3 shows the XML document.
Custom response options in Cloudfront
If you choose to provide a customized error response, you can also change the HTTP response code to 404 Not Found from 403 Forbidden. That can be helpful for programmers, but honestly won’t make much difference for the general public.
To reiterate, an option for a default, nice-looking 404 Not Found page in Cloudfront does not exist. But that doesn’t mean we can’t make one.
The solution to the generic problem is to create your own custom error pages. Because Cloudfront does not provide any nice default, you’ll have to do it yourself. To make it easier, I made a Youtube video for setting up custom error pages on a Cloudfront distribution with an S3 origin. Here it is:
However, this is not the solution that Samuell decided to go with, so the video will be the only reference to that particular solution.
Remember that Samuell uses this bucket and distribution as a CDN. There should be no static pages, no navigation, and no reason to be browsing through these particular assets. We looked through the Cloudfront statistics trying to see if anyone had been using them, or getting 403/404s. Here are some of the graphs we looked at, and the conclusions they brought us to.
A majority of traffic avoids errors on the CDN
Errors are neither consistent nor sustained
After looking at these graphs and the popular objects report, Samuell decided it wasn’t worth adding custom error pages — he would add them later, if ever needed, based on a Cloudwatch alarm that he created. The alarm watches the number of errors and notifies him if they ever exceed his threshold.
Another debugging item that we could have added was access logging from Cloudfront. When enabled, all access logs from Cloudfront edge nodes are uploaded to a configured S3 bucket in your account within 15 minutes of processing. Because they’re delayed, they’re not helpful in diagnosing live bugs or critical production issues, but they could have better verified that Samuell was indeed the only person getting errors on his distribution. Combined with Amazon Athena, you can have full text searching of your logs for extremely cheap.
Amazon caters to customizability, not usability. A default, nice-looking 404 page would be really nice and take AWS all of 1 day to produce. Instead, they opt for features like changing the status code, a different error page for each origin status code, and negative TTLs. Don’t get me wrong, these are great features, but they do have a tendency to miss the 90% case in favor of the 10% case.
Cloudfront does not have alerts out of the box. In setting up the alerts, Samuell was surprised to find that Cloudfront didn’t have any setup automatically. This is another case of the customizability vs usability, but an important one to note. Out of the box, Amazon will not notify you of errors in Cloudfront configuration, error rates in deliverability, or anything else. No notifications for you.
Amazon Cloudfront doesn’t give alerts out of the box
You can create your own 404 page in minutes. Watch the video I posted. You can be done in less time than it takes Cloudfront to propagate your changes. Get yourself a royalty-free image from Google search, put in “404 Not Found” in some big letters, and you’re off to the races. Or, use one of these pre-built 404 pages I found.
Guardian DevOps is a free service that puts you in contact with DevOps and SRE experts to solve your infrastructure, automation, and monitoring problems. Tag us in a post on Twitter @GuardianDevOps, and together we’ll solve your problems in real time. Sponsored by Blue Matador.
Tweet to @GuardianDevOps Follow @guardiandevops
Blue Matador is an automated monitoring and alerting platform. Out-of-the-box, Blue Matador identifies your AWS and computing resources, understands your baselines, manages your thresholds and sends you only actionable alerts. No more anxiety wondering, “Do I have an alert for that?” Blue Matador has you covered.
