Log of problems with XO


Background: how we ended up using XO

MonkeyBrains (as of Jan 2007) hosts most of its network in the carrier neutral data-center in the 200 Paul Ave, San Francisco, CA building. The 5th floor of the 200 Paul Ave colocation center (run by DRT and GNi) only allows minimal power usage, so we investigated our other options in the building. We started to rent space from XO because they will set up racks with 40 amps of service. XO is on the 2nd floor of the building. United Layer rents a fraction of the 3rd floor, Level3 is on the 4th floor, etc.

For several years, we only had PRIs from XO, and they have worked flawlessly in that realm. Other services have been lacking, specifically, the colocation. This page is a resource for other ISPs so that they can think of things to write into their contracts regardless of the provider. Also, this is documentation of our specific problems (and successes) with XO. Perhaps even XO can use this page as a guide to better understanding small businesses in need of telecom services.

Poor cooling

During October and November of 2006, the temperature in the XO facility was much hotter than any other data center I had ever been in. Ambient temperatures were in excess of 80 degrees Fahrenheit. It was like a sauna in there! Due to XO's over subscription of the cooling system and rapid expansion in their colocation customer base (Technoratti brought in 40 racks worth of equipment and jacked up the temperature), Monkey Brains requested a credit for services. Our customers asked us for a credit, and we replied that we would ask for one and pass on any savings directly to them. XO declined to credit us anything. If you get a contract with XO, write in the ambient air temperature into your SLA.

The ambient air temperature in San Francisco is -- on average -- COLD. Why do data centers not use more fresh air in their cooling systems?

Rudy
As you know our internal air exchange systems were thrown off balance by the addition of a large number of customer cabinets in mid to late September.
At that time XO solicited the services of Morrison Herschfield, a design engineering group, to review and propose solutions for the appropriate distribution of both cooling air and the exhaust from the various configurations of cabinets and cages within the facility.
The engineers from MH visited the site twice and their findings are in our possession. They have recommended an augment of the overall available cooling air and a re-direction of the output from the fixed ducts.
To meet these needs XO has sought and acquired the necessary three vendor quotes for the HVAC capacity upgrade. We are bringing another 30 ton unit on line, making the total available within the facility 100 tons. The electrical work to connect the unit will start next week, the fabrication of the duct work will take some 7 to 10 days after which that unit will be plumbed and brought on line. Time frame for this effort to be complete is roughly 11-10-2006.
The addition of additional output ducts to re-direct the existing air (and allow us to remove the fans) is being reviewed and we should have word on how soon we can implement that phase shortly. It will be in the immediate time frame but right now we do not have the lead time on the new "barrel" ducts which have been proposed. We will have further word on these items Friday and can then acquire the quotes to install and deploy them.
This is our immediate status, you can contact me with any further questions.
Regards
Mike
I found this picture on the Internet... current (May 14, 2007) cooling situation in XO:

Power failures

January of 2007 saw 3 power failures. The first one lasted 4 seconds and was due to the UPS failing. The power then automatically bypassed the UPS and was 'live from the street' without any UPS protection. 3 nights later, that circuit with the power tripped the 600 Amp breaker. Apparently, XO had oversubscribed the power as well as the cooling. Another customer found the flipped breaker and returned it to the on position after power had been out for one and a half hours. No XO techs showed up to the site that night. The outage was around 11pm. The next day, the circuit breaker cover was off and a fan was pointing right at it to prevent it from tripping again. 2 weeks later, XO hired some electricians to fix the problems. The power was out for an hour, and they:
  • replaced the possibly damaged breaker
  • rerouted some of the racks of equipment onto another 600 amp breaker
  • set up the wiring to allow the new UPS to be installed later that day
    • The new UPS came up and all power was switched back to UPS protected.
    Repeatedly, we inquired about getting some kind of credit for the outages. The electricity was out for 2.5 hours in one month. The number of man hours to get our 8 racks of equipment back up and stable was about 50. Verbally, we were told, yes, yes, let me get back to you about the credit, you are due a credit, etc. Finally, I got an email from my Sales rep one month after the fact saying we would get NO CREDITs. Repeatedly, we asked for an official report on the outage so we could pass that along to our customers. Rudy requested that document repeatedly from several XO representatives. A month after the fact, another customer emailed me that they had gotten XO's report. I called them AGAIN, and Jeffery finally emailed me the document. The verbiage in it was total nonsense. They failed to note that they had 620 amps of customers on a 600 amp circuit! Here is their not so accurate or informative report:
    Reason for Outage
    All Times Reported as Pacific Standard Time 
     RFO Prepared By:    Katherine Strasz
 Date RFO Prepared:   January 31, 2007
 City:   San Francisco
 Trouble Description:     Power failure at 200 Paul
 Trouble Ticket Opened:   January 18, 2007
 Trouble Resolved:   January 31, 2007
    Summary: On January 18th, 2007 at approximately 4:40pm the XO Collocation Facility located at 300 Paul in San Francisco, CA experienced service interruption on the 300 KVA Leibert UPS equipment under a low voltage / reverse power condition. The duration of the interruption was approximately 3 seconds. XO initiated immediate troubleshooting and on review determined that the unit should be replaced rather than repaired to avoid future impact to our customers. The new unit was ordered for expedited delivery.

    On January 26, 2007 at approximately 11:00pm a major distribution breaker failed at the facility. After resetting the breaker [Who reset the breaker? A customer fresh off a flight from Las Vegas, not XO personnel], service was restored at approximately 1:00am. A review of the equipment determined that the breaker had been damaged during the previous outage event and now needed to be replaced as well. At this time MOP 31866 was created to coordinate maintenance efforts to replace both the faulty UPS, the distribution breaker, and work to remediate the underlying cause of the failure.

    On Tuesday, January 30, 2006 XO received the replacement UPS and breaker and began preparations to bring it online. On Wednesday, at 4:30AM XO personnel and the electrical contractors removed the failed UPS from service, the breaker was replaced, and a new load distribution plan was implemented for the facility. This was completed at approximately 5:35AM following which the crews began exchanging the failed UPS with the new unit in anticipation of the scheduled maintenance window during which the new unit would be brought online.

    At approximately 11:00pm XO again implemented Emergency Maintenance Event 31866 and brought the new UPS online with a momentary switch not detected by the bulk of the systems within the collo

    Cause: A previously undetected circuit defect in the AC distribution within the collo lead to a pre-mature failure of the UPS. The failure of the UPS also affected the distribution breaker.

    Resolution: The faulty unit and breaker affected by the outage were replaced during the standard maintenance window. The load within the collo was re-distributed by XO taking this opportunity to bring another UPS on line to service the facility. This UPS, a 225KVA unit, was equipped with two new strings of batteries just prior to the cutover. This new UPS will now protect approximately 30% of the load that was on the original unit while providing circuit redundancy to a significant portion of the collo space.


    Rudy requested a refund for the entire month to cover the hours MonkeyBrains and their customers spent due to XO negligence. I was offered 2 or 3 days refund for two of my cabinets over there. The XO billing system doesn't show the people on the other end of the phone where the services are, apparently. I tried to tell them that other account id#'s I have are also in 200 Paul Ave, but their records show that the cabinets are in my office! Surprise, my office is suddenly a data center - NOT! Anyhow, I was requesting a refund for 8 cabinets for one month and was offered 2 cabinets for 2 days. In fact, the XO trouble ticket system did not even show my accounts were affected by the power outage. The support person -- following the bureaucratic cahnnels -- requested documentation that we did indeed experience a power outage. The only thing I had was the RFO (above) sent to me from Jeffery Jackson. I sent it in and was informed that the RFO was prepared for United Layer and not me, so it was no good. At the end of the day, XO denies that the power ever went out! I emailed Jeffery several times asking him to resend me the RFO and to attach a note stating that Monkey Brains was indeed affected -- he never responded.

    Bad advice

    MoneyBrains decided to purchase additional XO cabinet space in another facility across town. To link the cabinets, we needed to purchase a connection from XO (frankly, it should be free to connect between XO spaces in my mind). The XO sales engineer told us to purchase a DS3 to fiber media converter so we could clink our networks using our Cisco switches. Nope. It was a media converter and not a protocol converter. We were getting DS3 signal over fiber, no good. So, they got us our original request in the end: an Ethernet handoff. We went for the 100Mbps connection... It took them about 6 months to get this working. And we were stuck with $1000+ in hardware we purchased per XO's advice. Bummer. We asked for a credit for the hardware they advised us to purchase. We were told, NO CREDIT. (What about a free month of connectivity? NO!)

    I talked with another provider today (TW Telecom) and they informed me that they provide "DS3 modems" (a device small like a DSL modem) that converts the DS3 signal to Ethernet for free to all their DS3 customers who need them.

    Rudy this is what you need. I had my engineer take a look at it. You can make an offer that is lower than $899 especially if buying two.. Email seller to see if they have two..

    Andy R.
    an ebay auction for a "Transition Networks DS3-T3/E3 Media Conv SCSCF3013-100" media converter was attached

    Feb 2007 Update: we sold the media converts on eBay for $168.31.

    Preferential treatment to new customers

    We requested an additional cabinet on February 22, 2007. It took until May 7th to get in there. No big deal, I guess, BUT, it seems other new customers were given preference over us. If XO straight up told us: "Hey, we got a new customer we are trying to impress with a speedy turn around and bumped your install back and moved it to a different cabinet. To compensate you we are going to waive the install fee and give you a free month of service." But they DID not offer a setup fee waive or a free month of service for our new cabinet. We currently have 10 cabinets with them. We indicated which cabinets we preferred when we asked for the new cabinet. When install time came around, we had to move to other cabinets as those had been 'reserved for another customer'. Other people can reserve cabinets, but we can't? We have 10 cabinets there and are given less preference than other people. Also, there is a power shortage, so to turn up our cabinet, we had to agree to use less power elsewhere that we are already. Yep, to use a new 20amp circuit, we had to disable an existing (luckily not yet used) circuit.

    Mystery Taxes

    When you get a quote from XO, they will not estimate the taxes for you (our first bill was a shock in that we were never informed that we would have tax on colocation -- I don't pay tax on the office space I rent, why do I pay tax on the 4 square feet per cabinet I lease from XO?).
    If you get a $1000 quote from them, expect your monthly bill to be $1180 +/- $100. :)

    Good PRIs

    I feel I should say some good things as well. The PRIs have been flawless for several years now. The techs at 651 Brannan and 200 Paul Ave (both named Mike) are real sharp and helpful. Also, Andy and Eric took Alex and me out to an excellent lunch two months ago. I would say, we are mostly happy with XO, but feel they don't give a crap when things start to break down -- things that are specifically their responsibility.

    A badge fee was waived

    Sep 13, 2007 update: the badge fee is fairly small (> 1% of our monthly bill with XO), but we needed another one and XO waived the fee. I didn't ask why; I'm just assuming that at 10+ cabinets we have reached a revenue level for XO that they can let the tiny things slide. Of course they should: the quicker and more efficient small things like badges are delivered to us, the more time we can spend on growing our business (spending more money with XO) and less time on maintaining the business.

    Conclusion

    In summary, the XO sales team (which all the other departments refer you back to) is too busy to help out their customers with anything other than getting new paperwork for new revenue propositions. Customer retention, people! Customer retention! Currently, we are looking into the future a couple of years and XO does not look like the most viable colocation option -- this could change. Specifically, we need to have some XO people (with enough power in the company to actually change things) meet with us, resolve our issues, and help us move forward.