December was a tense month for the Vancouver Olympic Committee (VANOC). It had to conduct a full technical rehearsal for the Olympic Games, which take place from February 12–28 (with the Paralympics running directly afterward). Everything seemed strangely quiet, as hundreds of workers silently toiled on rows of PCs, acting as if the Games were already in motion.
Personnel pretended to log times and scores on the carefully prepared applications. Information was sent to servers that pretended to disseminate megabytes of data to the world’s media outlets. And at the Games’ various venues, officially appointed saboteurs did their best to cause trouble.
They attempted to break into computer networks, make systems fail, and, acting as the hand of God, would approach key personnel and strike them down with fictional illnesses, forcing their teams to resort to contingency plans. It was all designed to test just how much punishment the precisely orchestrated ecosystem could take while achieving its goal – to get information from the Olympics out to everyone who needed it.
One Team that Will not Receive Medals
This group of saboteurs, nicknamed the ‘shadow team’, comprises experts from the VANOC, Atos Origin, and Bell Canada, one of the telecommunications companies supporting the Games. “They have been told to do what they need to do”, explains Magnus Alvarsson, chief integrator at Atos Origin, the French company that has handled the entire IT infrastructure for the Games as a primary contractor since 2000.
| "We had to interconnect the venues to the broadcast center and the data center. To do that, Bell laid hundreds of kilometers of redundant fiber" |
| Ward Chaplin, CIO for the Games at the VANOC |
Atos Origin has developed, in preparation for the Olympic Games, a set of IT scenarios that could arise. “Then the telecommunications company starts looking at scenarios that we have, and how to change them for the specific conditions in Vancouver”, notes Alvarsson.
The shadow team also draws on the expertise of personnel from Olympic partners, future organizing committees, sports federations and the International Olympic Committee (IOC), which oversees the entire Olympic operation.
During the technical rehearsal, key members from the shadow team sit in a meeting room, hatching plots to try and catch out the staff. Two or three officials at each venue will also be working to put various spanners in as many wheels as possible.
“We will test for social engineering, for people trying to control the system – the full spectrum”, says Alvarsson. “Typical scenarios include people not getting to work because of inclement weather, food poisoning, and infrastructure collapse. We might game a power outage in a critical component, or the communications network failing.”
All in all, there are roughly 600 different scenarios listed in a playbook that the team uses to rehearse the Olympic Games. Should any event occur, they can refer to the playbook for a detailed set of procedures to follow, and to assess the responsibilities of key personnel.
"Typical scenarios include people not getting to work because of inclement weather, food poisoning, and infrastructure collapse. We might game a power outage in a critical component, or the communications network failing." |
| Magnus Alvarsson, Atos Origin |
These scenarios are stored in an Access database. When the week-long technical rehearsal is done, the shadow team reports back on the technical performance of the operational staff. If there is anything that can be improved upon, it is noted for the future. There are two technical rehearsals during the project to create the Olympic Games, designed to help the team learn from its mistakes and bolster security as much as possible before the event actually happens – because once the torch is lit, there is no going back.
Avoiding a Repeat of History
Security, both physical and logical, is a prime concern for Olympics organizers, who have plenty of incidents to draw on from past Olympic Games. In Beijing in 2008, Chinese police killed and arrested various members of an alleged terrorist gang, said to be plotting an attack on the Beijing Games.
Cyber threats also played a part in the 2008 Olympic Games in Beijing. Predictably, phishing scammers set up a fake ticket sales site and began harvesting credit card details.
The Olympic Games in Vancouver faces its own challenges, and in many ways could be the most targeted to date. A broad group of activists called the Olympic Resistance Network has been mounting a campaign against the 2010 Games, citing issues such as the theft of indigenous land, adverse affects on poverty-stricken communities, environmental damage and rapidly increasing public debt. Ironically, the measures taken to reduce the risk of physical threats at the Games have also served to anger activist groups, which highlight increased surveillance operations and more funding for law enforcement groups as areas of concern to them.
The IT system supporting the 2010 Olympics breaks down into two broad areas: the core Games management system (GMS), and the information diffusion system (IDS).
The GMS handles a variety of applications designed to keep the logistics of the Games up and running. Applications cover medical encounters to govern the health and medical management of athletes and report to the IOC’s Medical Commission. The sports injuries and qualifications system collects data on all of the 6000 athletes and processes those eligible to compete, judged on around 1000 criteria.
An entire IT system manages transportation of everyone – from media and volunteers to officials and athletes – as part of the Games. Other systems govern workforce management and human resources, accreditation and access privileges, and even the arrival and departure of VIPs. A new part of the IT landscape for the 2010 Games is a volunteer portal, which enables volunteers to apply to help the Olympic Games online.
The IDS enables information from the Olympic Games to get out to the media and the public. It breaks down into several key areas. An on-venue results system created by timing company, Omega, times the various events and delivers the information to the IDS. These results are channeled to various recipients via different applications. A commentator information system (CIS) displays the results to broadcasters, while an intranet called INFO2010 provides the results, along with background information, biographies, and schedules to the ‘Olympic family’ of media, athletes and officials.
An internet data feed sends information from INFO2010 and the CIS to internet-based customers, including broadcasters, while a separate feed provides the information in an XML format specific to the World News Press agencies. There is even a print distribution mechanism used specifically for on-site media and athletes.
Who’s the Boss?
Overall security for the Olympic Games is co-ordinated by the Vancouver 2010 Integrated Security Unit (ISU), which is a force combining the Royal Canadian Mounted Police, Department of National Defense, different Vancouver police departments, and others, including the Canadian Security Intelligence Service (CSIS). In terms of physical security, the Vancouver ISU has worked with organizations including the Canadian arm of the North American Aerospace Defense Command (CANR), preparing for activities in the air, as well as those on the ground.
This security effort has several subcomponents. The VANOC has its own security team responsible for day-to-day operational issues, and when it comes to cyber security, Atos Origin is responsible for keeping IT systems locked down.
Significantly, there is little contact between the IT security team and the ISU until the Olympic Games begin, indicating that the ISU tends to treat logical security as a black-box solution and give Atos Origin a large degree of autonomy. When the Games start, there will be a liaison between the ISU and the Technology Operations Centre (TOC) that forms the nerve center of the Games’ IT operation.
Atos Origin began designing the IT systems for the 2010 Olympic Games four years ago. It developed a master plan for the Games in 2005, and began in earnest in 2006 by designing the system architecture for the event. It included 800 servers, supported by 2000 staff across 35 Olympic venues.
Although the IT systems supporting the Olympic Games are undoubtedly complex, their goal is remarkably simple: keep the information flowing at all costs. When designing and testing the systems, the organizers always have this in mind.
One-way System
This isn’t the only simplified means of advancing security that the IT team uses. The internal IT systems run on a closed network, sealed off from the public internet apart from specific touch points necessary for it to communicate information to the external world. The rule is that it only allows information to be transmitted out, not in.
“We have to provide information in real time to the spectators and that is our absolute priority”, says Patrick Adiba, executive vice president for the Olympic Games and major events at Atos Origin. “That means that even if things sometimes don’t work as planned, it doesn’t really matter as long as we can deliver the results at the right time.”
The design and testing phases incorporate classic elements of business continuity management (BCM), including designing redundancy into the systems at an early stage so that it does not require retrofitting later on. In traditional BCM, systems and processes are analyzed to see how critical they are to the continued operation of an organization. It is no different here.
To reinforce the focus on business continuity, the structure of the IT operation is very hierarchical. Everything is co-ordinated from the headquarters at the VANOC, which is located in Richmond, a suburb just south of Vancouver. There, a collection of logistical staff manage central operations from the TOC. The TOC is backed up by a hot site that takes about two hours to recover. If the VANOC headquarters was hit by an attack and incapacitated, it would take two hours to bring the Olympic Games back online.
However, the TOC staff will generally only cope with around 2% of any issues that are raised. The rest of them will be managed at the venues. Technology experts on the ground will be monitoring systems and conditions, ready to deal with anything that they can at the site, to avoid the TOC from becoming overloaded.
In some instances, it will be necessary to report issues and developments back to the central operations center, if for no other reason than to let staff there correlate events. It may be that a seemingly innocuous development at one venue could be simultaneously occurring at a number of sites, belying a more significant emerging problem. Generally, single-user issues would be dealt with locally, but if a more major issue arose, like a power outage in an area, then it would be reported back centrally.
The Games Must Go On
That business continuity focus extends to the data network connecting the various venues at the Games, including Whistler, the alpine skiing venue, which lies 75 miles north of the Richmond TOC. “We had to interconnect the venues to the broadcast center and the data center”, explains Ward Chaplin, CIO for the Games at the VANOC. To do that, “Bell laid hundreds of kilometers of redundant fiber.”
Security and availability go hand-in-hand, and thus design and deployment teams had to assess which systems were critical. According to the level of criticality, they designed between two and four levels of redundancy into the systems. In the timing systems, for example, there are four different levels of redundancy to ensure that availability continues. Depending on criticality, the organization would use alternate systems from different manufacturers as a means of providing additional failover protection.
"If you can think of a security company, then we are probably using their products" |
| Patrick Adiba, Atos Origin |
Most of this design is already complete, as Atos Origin – which has been involved with the Olympic Games since 1992 before becoming the primary contractor in 2000 – is able to reuse much of its equipment and design. After all, what isn’t broken shouldn’t be fixed.
“We emphasize knowledge capture and knowledge transfer. We capture what we do right and wrong, and document it simply”, says Adiba. “From Games to Games, we see what we can reapply. In the last four Games, we managed to use 50% of our technology from one Games to another. That’s the software, but also the policies and procedures.”
One thing that changed to a certain extent in between Beijing and Vancouver was the level of virtualization that the design teams were able to implement. “In Beijing, we used serial virtualization”, says Alvarsson. “Here, we do virtualization on all of the information diffusion systems.” Technology decisions are made four years before the Games. “We were not completely comfortable with virtualization at that point”, he adds. Consequently, the design team chose not to virtualize Windows at all. In fact, the Windows-based systems are running XP service pack 3. The company was not comfortable upgrading to Vista so soon before the Games, and all technology platforms are, in any case, frozen one year before the event, meaning that Windows 7 was entirely out of the question.
Nevertheless, there are still change management issues to consider, even as the Olympic Games are in progress. During the Beijing Games, 400 change requests were received for the IT systems while the Games were going on. The IT team makes a judgment call on each request to see how critical it is, and whether it should be implemented. It ended up accepting just 30 of the change requests.
Is This a War?
When conducting testing and simulation, IT teams use four levels of severity, in a manner similar to the way that NORAD uses DEFCON five through to DEFCON one. Severity level 1 is the harshest, and if something gets to that point, it means that an individual competition, or the Games as a whole, is in danger of being disrupted.
All of the software designed to support the Olympic Games was developed as part of a deployment process that stretched between 2007 and 2008, culminating in the launch of the volunteer coordination systems exactly two years before the event. The time window from 2008 to 2010 has been spent testing the systems and their ability to cope with different events. The period from late 2009 until the Games this year has been spent building up the capacity of production systems and getting them ready for deployment at different venues.
Deployment of production systems starts in the system laboratory. It is here that production systems are built and tested. “We have to make sure that the environment is representative of the Games. You have to ensure that it will behave the same”, says Adiba.
More than 60 applications have to be integrated during the deployment phase, and they are divided into cells corresponding to individual sports. On the busiest day during July 2009, there were up to 70 people working on this integration.
It was only during January 2010 that systems were deployed at venues. The site not only had to be built, but it also had to be physically secure before the equipment could be deployed. Aside from the security, venue availability is also a restraining factor. Deployment on site can be a tricky process because, in some cases, the company doesn’t have much access to the venues at all. In Athens, for example, delayed construction schedules meant that IT teams had only days to get equipment in, and make sure that it was operating properly.
By the time you read this, the Olympic Games will be in full swing. With a $2.5bn budget overall, and with a technology budget accounting for hundreds of millions, it is unsurprising that Adiba says: “If you can think of a security company, then we are probably using their products.” It seems that there is a piece of the challenge – and a piece of that juicy Olympic pie – for everyone.