Unless you work for a company in an IT capacity you probably have no idea of the kind of infrastructure that goes into keeping a business with hundreds of employees up and running. Multiply that number by millions and you may wonder just how much power, man hours, insurance, cost, logistics and everything else that goes into keeping that business operational. Welcome to Verizon Wireless (VZW), the nations largest cellular provider.
Reviews Editor James Pikover and I were invited by VZW to visit one of their top-secret Mobile Switching Centers. We were sworn to secrecy and asked to provide everything short of a blood sample before entering, or even receiving the address. An interesting note: you might live a block down from one such center and be none the wiser. Go ahead and look out your window. Do you see a rather benign and bland structure with a few Verizon trucks parked in its lot? It could be one of them.
The Verizon Mobile Switching Center that we visited is just one of many located in Southern California. I can tell you that we traveled far and wide to reach its location, and that we almost passed out from heat exhaustion.
We did a double take to be sure the address provided was correct. The building and property are the definition of nondescript. Inside we were greeted by a single security guard who prompted us to sign in and wait for someone to come fetch us. A few minutes and two security doors later we entered the brains of the Mobile Switching Center. It’s a sharp, open room with a dozen or so work stations, broken only by a handful of TV sets hung on the wall displaying everything from financial news to the weather channel. All of it is practically sterile, and somewhat reminiscent of Homer’s workplace in the nuclear power plant…on the “Simpsons”. The command center seats up to about 15 employees, but requires only 2-3 people on duty to keep watch on the mostly automized system.
Adjacent to the command center, just steps away and hidden behind another security door, are stacks of servers that process calls and data for millions of customers across Southern California, and sometimes beyond. Verizon doesn’t share specifics when it comes to how much data or how many users are active on any individual site (though they did share that up to 5 million users are attached to this particular center during peak hours), but keen emphasis is placed on redundancy for all communications. All VZW centers have server redundancy between individual servers, then server racks, all the way to entire server rooms. But it doesn’t stop there. Half of the building we visited are actually redundant; there are two nearly identical rooms so if any one entire section of the building stops functioning, the back up takes over instantly. And if both fail, then other centers locally– and eventually nationally — take over.
The data room filled with servers house thousands of computer chips, which individually emit a lot of heat. Combined it could bring a home air conditioning system to the melting point within seconds, yet the room is cool like a hospital, more akin to a grocery aisle freezer section’s temperature thanks to a water cooled AC system, which channels cool air through a 2-foot raised floor. That is to say that the servers aren’t on the ground; in fact, they’re attached only to the ceiling, which anyone who has lived through an earthquake knows is a more stable place to be than the shaky floor (assuming, of course, that you have an infrastructure for it).
The ceiling of the room is outfitted with a dizzying amount of colored cords and pipes, most funneling fiber optic cables that transfer vast amounts of data to and fro. The setup itself is amazing to behold, but at the same time it’s nearly impossible to grasp the massive undertaking of open communications to millions of people in any other way.
This server room is divided into a few sections: 3G , 4G LTE, and older 1x and 2x networks, as well as call-only servers. The 3G, the older generation network, which supports up to EVDO A+, takes up the bulk (about 50%) of the room because, as VZW told us, it’s much less efficient than LTE. The same can be said about 1x and 2x, which are available networks (if you live in areas with poor cell reception, you may seem them on your phone occasionally), which are slowly phasing out but take up another chunk of space on the floor.
As for LTE? The 4G servers fill one rack alone, including redundancy. In fact, all of LTE covered by the region this site covers is controlled by that single server. The level of efficiency is magnitudes better than any previous generation of cellular data. When LTE is more widespread not only will it actively include talk and data (today most LTE devices only use LTE for data while talk and texts are sent through the 3G networks), it’ll also do a better job on every end. Better efficiency means less power requirements, faster speeds, and in effect less problems for both users and carriers. VZW asked us to not show any photos of their LTE servers up close, as it could give away proprietary company secrets.
The amount of power required to commit to such a Mobile Switching Center is enormous, but it’s also extremely different from what you might expect. While power comes from the city as an AC current (like we all use on a regular basis), maintaining constant data communications with no loss whatsoever requires a steady flow of electricity. That requires DC power. AC, or alternating current, is great for household appliances and everyday use of nearly everything most people use. Shavers, computers, lights, etc. We don’t even notice that everything flickers because it does so so quickly. But we notice the fluctuations.
That doesn’t happen on DC, or direct current. The reason we don’t use direct current is because it’s far less efficient for most people (no one needs that much power), and it’s extremely dangerous. If you get zapped by something plugged in you might get burnt, but it’s more often than not a minor injury. DC can kill you because of just how much power goes through, and more importantly, how much power goes through consecutively. If you want data communications to continue sans interruption though, DC is the way to go.
So at this station power is received from the city as AC power. It’s then converted to DC power inside the building, and flows through three sections: the backup generators, the sulfuric-based batteries, and then onto the building itself. Power goes through the building in this way to make sure if the city fails delivering power the generators can kick in instantly, and if the generators fail the batteries can kick in instantly, with no loss of communication. In the event that power is interrupted, the two gigantic generators can provide up to 96 hours of juice with the 80% filled gas tanks. With additional fuel pumped regularly they can continue indefinitely (and VZW is building another generator and another massive air conditioning unit to accommodate it in the coming months).
If those generators failed, a room filled with over a dozen racks of a dozen 50 pound sulfur-ammonia based batteries take over support the center for up to 12 hours – it’s a last ditch resort, one further evidenced by all the caution signs and extra care they’ve taken in the event one (or many) of the batteries spills or leaks. The ground surrounding the batteries is lowered by two feet so any leak is instantly contained, and the ground tiles have a special coating that the sulfuric acid won’t burn through.
Furthermore, in the event of some apocalyptic disaster, VZW will be able to continue providing a field of communication for 108 hours on its own with no outside help. That’s four and a half days with no additional fuel pumped in or the batteries recharged, running as it would on any normal day. And on top of that, the center has been outfitted with showers and sleeping quarters to support the employees families should such an emergency occur. The reasoning behind this is simple: keeping communications open during any emergency is critical for everyone, but individuals need to make sure their families are safe. This center will at least provide a haven for the employees families so they can keep working and maintain the infrastructure when it’s most needed.
Verizon Wireless doesn’t limit it’s coverage to unmovable centers. The company also has a fleet of GoaTs and CoLTs during large events or emergencies. The Generator on a Truck (GoaT) and Cell on a Light Truck (CoLT) are two vehicles that VZW dispatches when a major concert takes place and the company needs to make sure everyone can stay connected, transmitted signals to and from their stationary centers with high-powered microwave transceivers. You’ve probably seen a GoaT or CoLT during a major storm or event. If you’re on the east coast, perhaps recently during hurricane Sandy.
Getting an inside glimpse into the workings of a top tier service provider helped us appreciate the scope of their business. And keep in mind this is just one of many. The company is constantly expanding it’s wireless network, as we’ve seen across all of Southern California with a growing LTE network that spans most of Los Angeles county, as well as the majority of the continental US. Construction was just starting on a fourth air conditioning unit in this particular center, and after that another power generator, all to make room for more internal servers. How the company is building it’s infrastructure out for wireless communication is almost mind-boggling in scale, and yet Verizon Wireless is the largest carrier in the US for a reason. After this trip, I can see why.
James Pikover contributed (heavily) to this piece.
3 responses to “Post Title”