20110607 - RESOLVED - partial outage - Kansas City

Earlier today we suffered an outage at our mkc1 (Kansas City) data center.  Here are some details about today's event:

  • At 2:50 AM (CDT) we detected elevated temperatures on our equipment.  We immediately contacted the facilities people to look into the problem.
  • At 3:52 AM the facilities people responded that they were working on the issue and temperatures should start to come down.  Shortly after that we noticed that the temperatures were still rising.
  • At 5:05 AM one set of UPSs shut down due to the elevate temperatures.  This reduced our capacity by approximately 50%.
  • At 5:32 AM the facilities people responded that the problem was worse than expected.  A specialist was dispatched to investigate the problem.
  • At 6:00 AM I was on-site at the data center monitoring the situation.
  • At approximately 7:00 AM The A/C specialist was on-site.
  • At 8:24 AM The appropriate replacement parts were in the process of being put in to place.
  • At approximately 08:30 AM we began restoring our system to full capacity.
  • At 9:24 AM The A/C system was back to full capacity.

It was explained to me that this problem was due to multiple simultaneous failures in the A/C system.  The first temperature rise was due to fan failures which resulted in a compressor shutdown in the primary system.  This problem appears to have started about an hour our two prior the warning we received from our equipment.  Some time around 4:00 AM a similar failure occurred in the backup system.  The resulting temperature rise caused the UPSs to shutdown a short time later.

To help prevent this from occurring again, I've been told that additional backup A/C capacity will soon be added.  The added system will also use a separate cooling source for more protection.

Please let me know if you have any questions about this.

-Bob

We Answer Your Questions: FAQ

Q: What is the maximum e-mail attachment size?

A: The ETRN.com e-mail servers do not limit the size of individual e-mail attachments. The ETRN.com e-mail servers do impose a 400 MB maximum total message size limit. Individual customers can choose a smaller message size limit. We can also customize the handling of "over-sized" e-mails. Please contact us to discuss your specific needs. A couple of important facts:

1. Attachments are typically encoded in what is called Base64[1]. As a result, the actual length of MIME-compliant Base64-encoded binary data is usually about 137% of the original file size.

2. E-mails often contain both plain text and HTML components. This also increases the overall size of the e-mail.