Status Page
As we start to build up the server and our services, we felt it was a good time to start building up the status page and make it public, while also explaining how it works.
We use Checkly, which is a free status page / uptime monitoring site.
This site checks if a given service is alive/responding. However, this does not check if a service is running properly.
With Browser Checks;
The check for dotNOC (this blog) will check to see if a service's homepage loads. If it responds with a valid HTML page, that does not otherwise contain an error, it will show as online. This is normally good enough for what we need but isn't a golden bullet for checking.
With Discord API Checks;
We do a check against guilds endpoint which allows us to check if the API is online and responding without issues if we look for a specific user, message, or channel. Normally, if this endpoint is down, there is likely a larger API outage. Conversely, if the endpoint for checking roles is down, (which is required for the whitelist) this would not be reflected by this check.
This check runs on a GET request on https://discord.com/api/v8/guilds/<guildid> which then responds with the following JSON if it is functioning properly.
What happens when it goes down?
Due to costs, rate limiting & needs from our environment we only do status checks every X minutes. Thus there might be a delta of that time before we know something is down.
However, when that happens we get notified in two ways:
Discord
All failure notifications are posted into a discord channel that all devCRU staff can see, these notifications look a little like this...
This shows the name of the service thats failed, how long it takes for the check to complete, the location it is checked from (to account for outages related to the checking server), an internal tag name and the time.
There is also a link embedded to the outage report as well as a notification that displays when the service comes back online.
Emails
Select devCRU also will receive email, which look a bit like this.
The emails show the same information and contains a direct link to the outage report.
We will add more services to this page as we bring them online. We will also write further posts on these tools as we expand.
