TECHNICAL SECURITY AND COUNTERMEASURES
WHITE PAPER FOR WATER UTILITIES
Ron Booth, Senior Security Consultant, CH2M Hill
Dan Ryan, Director, Environmental Health & Safety, USFilter Operating Services
Chuck Hewell, VP, Houston Regional Operations, ECO Resources

December 12, 2001

Table of Contents
Section I: Introduction
Section II: The Test of Time
Section III: Updating Emergency Action/Contingency Plans
Section IV: Acknowledging the Threats
A) Infrastructure: Physical Damage
B) Infrastructure: Damage to Chemical Storage Areas
C) Contamination with a Biological or Chemical Agent
D) Cyber Terrorism
Section V: Analytical Capabilities
Section VI: Evaluation
Section VII: Conclusions
Appendix A: Facility Security Survey
Appendix B: Frequently Asked Questions from NIPC
Appendix C: InfraGard from NIPC
Appendix D: Additional Information

I. Introduction
The horrific events of September 11th have obliged us to critically consider what new and redoubled efforts should be made to further enhance security at water treatment facilities. Our emergency preparedness and action plans must be reevaluated. Written plans can no longer be limited to natural disasters such as hurricanes, floods, tornadoes and earthquakes. Nor can they be limited to catastrophic equipment failure, power outages, fire, chemical spills or organized labor issues. Today's plans should be updated to include terrorist and sabotage threats as well.

Municipalities must now mandate that the security and vulnerability of its water facilities be observed, analyzed and assessed to determine if there are security issues that need corrective action or further evaluation. To this end, several agencies and organizations are prepared to assist in compiling a summary of security and emergency management best practices relevant to many of these types of facilities. This paper is designed to familiarize you with what best practices are available to your facilities as well as other resources you can draw upon. In addition, it can serve as a means of which you can compare what your current practices are with other like facilities.

II. The Test of Time
Dating as far back as World War II, water utilities have always been recognized as being vulnerable to acts of terrorism or sabotage. As a result, security features have since been "designed in" construction and operation plans as standard equipment. This often includes components such as perimeter fencing, outside lighting, single entrance points, fire suppression equipment, chemical leak detectors, smoke detectors, intrusion alarms, real-time monitoring of system pressures, turbidity, chemical dosages and other process variables.

Over the years, the aforementioned safeguards have provided a certain level of comfort for city leaders and plant operators. In fact, the technology available to date has allowed many utilities to reduce staff and/or shift coverage by way of automation. Automated systems such as Supervisory Control and Data Acquisition (SCADA) systems successfully collect data, monitor operations, make process adjustments and alert on-call operators of critical alarm conditions for timely response.

Existing security systems and procedures that may have served us well in the past may not be sufficient to meet the new level of threat that faces our water utilities today or in the future. Some improvements can be as simple and inexpensive as installing video surveillance equipment and/or ensuring facilities have adequate outside lighting. Other system enhancements/considerations are provided in Appendix A.

In 1998, the Clinton administration referenced a growing potential vulnerability with respect to critical infrastructures (e.g. telecommunications, energy, banking, transportation, water systems, and emergency services) in its white paper "The Clinton Administration's Policy on Critical Infrastructure Protection: Presidential Directive 63", 1998. The goal of this directive was to achieve an initial operating capability across the United States with the ability to protect our nation's critical infrastructures from intentional acts that would significantly diminish their abilities.

In addition to the presidential directive, industry best practices for security encouraged governments and the private sector to map out joint strategies that did not rely on increased government regulation or non-funded government mandates. These strategies were designed to complement the efforts of market forces responsible for developing and introducing more inclusive and secure information system technologies. They were also designed to enable private sector owners and operators, in their own right, to achieve and maintain adequate security.

III. Updating Emergency Action/Contingency Plans
In light of the terrorist attacks of September 11th, utility owners and plant operators must reevaluate their standard operating procedures, contingency plans, emergency action plans, crisis communication plans and the like. These written programs and procedures as they exist today, tend to focus more on natural disasters such as floods, tornadoes, earthquakes, blizzards and droughts. Utility owners and operators now recognize the need to expand their library of written programs and procedures to include more man-made scenarios. Man-made events were previously thought of as operator errors, industrial slug loads and chemical spills, to name a few. For the majority of utilities, written programs paid very little attention to sabotage or terrorism prevention and response.

IV. Acknowledging the Threats
Today's plans should be updated to include terrorist and sabotage activities, such as physical destruction, biological contamination, chemical contamination and cyber attacks. Although some of these activities may be in the form of a threat, all occurrences must be considered legitimate until they can be disqualified. Threats are made simply to create chaos, confusion and fear. Terrorist attacks may target various essential functions of our water systems:

A) Infrastructure: Physical Damage
Physical destruction is one of the more likely scenarios regarding terrorist and sabotage attacks on a water utility. Destruction of water and wastewater infrastructures would have longer lasting effects than those created by a chemical or bacterial assault. Destruction of wastewater treatment plants or their collection system components (gravity sewers, force mains, lift stations, etc.) would pollute bodies of surface water including ponds, creeks, streams, rivers, lakes and oceans, as potentially millions of gallons of raw or partially treated sewage could be released.

Over extended periods of time, this could ultimately affect ground water supplies. An attack such as this would cause public health and sanitation concerns, adverse effects on aquatic life and could risk source water supplies for drinking water plants. Physical attacks on water treatment plants and related source or distribution systems could take various forms. The bombing of dams, critical treatment plant processes, or a water tower, for instance, would play havoc on a community's ability to produce safe, pressurized water for consumption or fire-fighting. Destruction of electrical power grids or gas lines servicing a water or wastewater treatment plant would significantly hinder or halt water utility operations for an indefinite period of time as well.

The best defense for preventing physical attacks and infrastructure damage is awareness and treating all threats as real.

B) Infrastructure: Damage to Chemical Storage Areas
Sabotage or physical damage to a utility's chemical inventory, for example, a storage area or chemical railroad car, would have severe consequences for plant staff, emergency response personnel, neighbors within the zone of influence and the environment. Once the initial consequences of such an attack are addressed, the secondary concern would be the facility's inability to feed that chemical until temporary measures were put in place or the system was repaired. Chlorine, common to many water and wastewater treatment plants has a low boiling point and rapidly becomes a gas when released under normal conditions. Chlorine reacts with moisture on the skin, throat and lungs to create hydrochloric acid causing burning of the skin and inflammation of body tissues. Chlorine was successfully used during World War I as a form of chemical warfare, resulting in mass casualties.

Utilities should consider "designing-out" the use of extremely hazardous chemicals, replacing them with less dangerous chemicals or physical treatment processes where possible.

Additional measures/considerations to reduce the risk of physical destruction are provided in Appendix A.

C) Contamination with a Biological or Chemical Agent
While the effects of a biological or chemical attack can potentially cause water contamination with resultant illness or death, it is theorized that the episode itself would be short-lived and not widespread. If a terrorist were able to breach the security of a water treatment plant (and the benefits of dilution), the injected contaminants would have to be strong enough to survive treatment plant processes typically consisting of coagulation, flocculation, sedimentation, disinfection, and in many facilities, filtration.

An ever-growing concern of water officials, government agencies and professional trade organizations is the potential for a biological or chemical event within the distribution system. Injection within the distribution system circumvents the security measures put in place at the treatment plant, bypasses the protective barriers previously mentioned (filtration, chlorination, etc.), and negates the benefit of dilution when comparing the capacity of the raw water supply (aquifer, lake, reservoir, etc.) with the capacity of the distribution system piping in the affected zone. This scenario assumes the injection point is down stream of any large clear well or water tower.

Communities and local law enforcement are nearly defenseless for an act of terrorism or sabotage of this nature, as it could take place at a remote fire hydrant or even a hose bib at the home of any resident. The best line of defense is public awareness and increased neighborhood watch campaigns.

Another defense mechanism would be the installations of backflow devices in every home. In Europe, backflow prevention devices are present in all distribution system connections as a preventive measure. In the United States, several incidents of accidental distribution system contamination by backflow are reported every year. This installation would be costly, could take years to install and a would-be terrorist would not be above removing a backflow device to complete his mission. Another defensive mechanism would be lock-down caps and/or protective seals that may eliminate fire hydrants as a point of entry. These improvement measures would need to be assessed on a case-by-case basis to evaluate cost versus risk.

In most cases, the incident would be isolated to a specific pressure zone, subdivision, or street. It would be quickly identified and promptly communicated to affected areas. Distribution system maps should be kept current and readily available so isolation plans can be put into motion. Once isolated, the contaminated water would be contained for further treatment or disposal.

Some concern has been raised regarding the quality control at chemical manufacturing/packaging plants and within the chemical transportation industry. It is conceivable that chemicals ordered and delivered to water utilities could be tainted during the packaging process or somewhere along the transportation route. In this case, utility operators are subject to exposure and would actually be injecting the chemical or biological agent unknowingly into the water system.

Table 1 summarizes some primary and secondary indicators that a biological or chemical event may have occurred.

Table 1
Biological/Chemical Attack Indicators

Primary	Secondary	Symptoms (General)
Symptoms of Victims	Dead Animals	Excessive twitching, runny nose, sweating, drooling, pinpoint pupils, urinating, defecating, vomiting, convulsions, eye & lung burning, coughing, choking, headache, nausea, diarrhea
Mass Casualties	Things out of Place
Warning Given or Credit Taken	Unexplained liquids
	Strange Smells
	Surge of 911 calls

With the recent rash of Anthrax cases involving the U.S. mail, water utility personnel are not exempt from potential exposure. Instructions for proper handling of mail and packages can be obtained from the U.S. Postal Service and should be reviewed with utility staff.

The Centers for Disease Control (CDC) is now working on identification and characterization of biological agents of concern for water utilities. A panel of experts has been assembled to prepare a reference document. The document will include a list of agents that could be used in a hoax or an actual event. The identification of detection procedures and methods, the oral dose of concern, the effectiveness of standard water treatment equipment and process chemicals against the agent, and information on how to inactivate the agent with equipment and/or chemical treatment will be included in this research.

D) Cyber Terrorism
Cyber terrorism also posses a threat to the nation's water utilities. As previously mentioned, many utilities have employed SCADA systems to improve monitoring, process control, compliance, security and overall reliability. Experienced hackers can access SCADA systems that operate over the Internet. Once logged in, hackers can deactivate process alarms, change chemical federates, start/stop equipment and so on. The results of this intrusion can pose a wide variety of environmental, health and safety concerns to plant employees and the general public.

Many system managers installed firewall protection as part of their "Y2K" contingency plans. The likelihood of a cyber attack by terrorists is less likely than a cyber attack from a disgruntled employee. The insider threat, which could result from, acts of disgruntled current or former employees or from accidental introductions of foreign software or viruses by loading of non-authorized software on employee SCADA computers.

The National Infrastructure Protection Center (NIPC), located in the FBI headquarters in Washington, D.C., conducts information sharing with the public and private sector owners and operators of critical infrastructures. The NIPC monitors, warns, and investigates unlawful acts involving computer and information technologies. The agency manages computer intrusion investigations and supports law enforcement related to cyber crimes. NIPC can be accessed at www.nipc.gov. Appendix B provides additional information.

The National Infrastructure Assurance Plan operating as a public-private partnership further assesses cyber attack vulnerabilities, recommends plans to eliminate significant vulnerabilities, and proposes systems for identifying and preventing major attacks. The plan will also include a protocol for alerting and containing the attack.

V. Analytical Capabilities
Since most water utilities do not have laboratories equipped to identify exotic biological or chemical agents, utility personnel must work proactively with contract laboratories to develop an emergency procedure for sample analysis. Having appropriate contacts in the laboratory community is imperative should a biological or chemical event be suspected. Most contract laboratories provide full-service, ranging from heavy metals to pesticides to volatile organics. This will cover most low-tech poisons that are easy to acquire and deliver. Gas Chromatography/Mass Spectrometry (GC/MS) libraries can be utilized to identify 30,000+ known organic compounds. For biological concerns, a separate contract laboratory may need to be identified.

Early on-line warning systems that can detect anomalies in source water quality are currently not available for the threats posed by advanced biological and chemical terrorism. While certain contamination could be detected by more conventional, on-line monitors such as pH, Dissolved Oxygen (D.O.), turbidity, particle counters, conductivity, fluorometers, etc., these continuous monitors would not necessarily flag the onset of a terrorist event.

Further research and development will be expedited to develop additional monitoring and early warning systems capable of providing real-time information for treatment plant operators to react accordingly. However, even with more sophisticated monitoring equipment at remote booster stations, water towers, and elsewhere throughout the distribution system, the only way to detect the act of sabotage or terrorism that occurs by back feeding chemical toxins or microbiological agents through a garden hose at a residence, would require home test kits. Again, this is a question of cost versus perceived risk.

VI. Evaluation
An assessment of the facility's security posture addresses most industry standard concerns for physical, biological, chemical and cyber security.

The work accomplished in preparing a public utility vulnerability assessment report should include the following activities:

• A kick-off meeting with the utility senior staff to review the overall objectives and receive the utility's observations on security threats vulnerabilities
  
• Physical site observation and walkthrough of facilities
  
• Review of previous security documentation
  
• Compilation of a summary of security best practices from external sources This should include:
  
• Reviewing the results of comprehensive questionnaires on security practices - these questionnaires were sent to "like" utilities that verbally agreed to complete the questionnaire
  

Exhaustive literature research on best security practices

Appendix A is a system evaluation form that can be used on a preliminary basis, as EPA and their partners develop a more comprehensive vulnerability assessment tool.

VII. Conclusions
The EPA continues its efforts to develop methodology and training materials for water systems to assess vulnerability. It is anticipated that these materials will be available by the end of 2001 and an Emergency Operations Manual released in 2002. Numerous organizations and other groups (public and private) are actively addressing terrorism, with the following agencies aggressively taking the lead on issues specific to water utilities:

• Federal Bureau of Investigation (FBI)
  
• Federal Emergency Management Association (FEMA)
  
• Department of Defense and Energy (DOD/DOE)
  
• National Institutes of Health (NIH)
  
• Office of Emergency Preparedness
  
• Agency for Toxic Substances and Disease Registry (ATSDR)
  
• Centers for Disease Control (CDC).
  
  

Although water and wastewater treatment plants remain under a heightened level of security, there have been no nationwide warnings from federal authorities that water utilities are under immediate threat by terrorist activity. Our nation's public utility systems have always been secure. However, the time has come to identify improvement opportunities which include good operating practices, sound, cost-effective security measures and the development and installation of early warning and monitoring systems at source water and remote distribution points where possible. Early warning systems will need to be evaluated for applicability; dependability and feasibility based on risk.

Appendix A: Facility Security Survey

APPENDIX B
From Web site of the National Infrastructure Protection Center
http://www.nipc.gov
Frequently Asked Questions
Section C: The Role of the Private Sector

C-1: What steps can the private sector take to manage the risks from the threat?
PDD 63 encourages governments and the private sector to map out joint strategies that do not rely on increased government regulation or non-funded government mandates. The PDD and the demands it places on the federal government are designed to complement the efforts of market forces responsible for developing and introducing more robust and secure information system technologies; to bring about global solutions to international problems; and to enable private sector owners and operators, in their own right, to achieve and maintain adequate security.

C-2: What are "best business practices," and how can their adaptation enhance the security posture of the Nation and its critical infrastructures?
"Best practices" are those generally accepted protocols, procedures, and practices that are voluntarily implemented, because they promote the continuity of business or reliability of service expected by the customer. "Best practices" are pursued, in part, to avoid the often heavy costs associated with industry regulation, but mainly because they are consistent with sound business principles and more readily effect in a positive manner the corporate bottom line; that is, profitability. While not cost-free, "best practices" are analogous to low-cost, prophylactic measures often employed by the practitioners of preventative medicine and therefore a way to make available more present-day, scarce funding for the likely complex and costly solutions demanded by tomorrow's unforeseen problems. Again, the medical analogy would be a change in diet today to avoid costly heart by-pass surgery tomorrow.

In the near term, one of the ways to quickly and effectively achieve a much higher level of protection from cyber threats is to raise the level of existing protection through the application of "best practices," in particular those "best practices" focused on security-related concerns. The pursuit of "best practices" by the users of information systems is consistent with the blurring of formerly clear distinctions between foreign (or national security) and domestic policies, an artificial distinction no longer likely to serve our interests well. Disruption of the services on which our economy and way of life depend could have significant effects, and if repeated frequently could seriously harm public confidence. In this post-Cold War era, these postulated disruptions to the public safety would not likely rise from an assault on our territory employing traditional military force; rather, those with hostile intent could seek to probe electronically where they perceive us to be most vulnerable; namely, in our reliance on information technology. Our overall national security, economic and public safety interests are wholly dependent on public and private infrastructures that, in turn are becoming less and less separate.

Moreover, as the threats to these interests are harder to differentiate from local criminals or foreign powers, and because the techniques of protection, mitigation, and restoration that reflect "best practices" focused on reducing inherent vulnerabilities are largely the same regardless of the source of the threat, we conclude that responsibility for infrastructure protection and assurance can no longer be delegated exclusively on the basis of who the attacker is or where the attack originates. Rather, the responsibility should be shared cooperatively among all of the players. The business term "best practices" aptly describes a key component of the Nation's new first line of defense that must be jointly constructed -- by both public and private sectors, together -- as we accept the growing fact that our public safety, as well as the timely and efficient employment of the more traditional expressions of our ability to defend our national interests overseas, increasingly depends on the continuous availability of civilian infrastructures, especially communications and transportation.

PDD 63 recommends a sector-by-sector cooperation and information sharing strategy. In general, these sector structures should be partnerships among the owners, operators, and appropriate government agencies, which will identify and communicate "best practices." The Department of Commerce's National Institute of Standards and Technology (NIST) and the Department of Defense's National Security Agency (NSA) have been asked to provide technical skills and expertise required to identify "best practices" and evaluate vulnerabilities in the information networks and associated control systems. Further, the sharing of information and techniques related to exploited vulnerabilities is also crucial. This should include exchange of data on the development and deployment of ways to detect, identify and prevent events, mitigate damage, quickly recover services, and eventually reconstitute the infrastructure.

One very effective "best practice" is incorporating a risk-management process, based on sound, quantitative, risk assessment methodologies. These methodologies would address risks associated with physical attacks, cyber attacks that could corrupt essential information or deny service, the possibility of cascading effects, and new levels of interdependency.

The following are suggested, immediate actions that infrastructure owners and operators should consider prior to the conduct of any formal risk assessment: first, isolate critical control systems from nonsecure networks by disconnecting the "critical control systems" from the those more routine, supervisory mechanisms connected directly to the Internet or by installing adequate fire walls; second, adopt proven procedures and policies for password control and protection, or install more modern authentication mechanisms; and, third, provide for individual accountability through protected action logs or their equivalent. As owners and operators exhibit basic yet prudent "best practices" such as the aforementioned immediate actions, they are also laying the foundation for later implementation of more in-depth, sophisticated risk assessment and management initiatives the private-public partnership called for in PDD 63 can provide if we, as a Nation, are to effectively manage the truly complex and interdependent set of infrastructures we have erected as indispensable parts of our society.

APPENDIX C
InfraGard from NIPC Web Site
InfraGard is an information sharing and analysis effort serving the interests and combining the knowledge base of a wide range of members. At its most basic level, InfraGard is a cooperative undertaking between the U.S. Government (led by the FBI and the NIPC) and an association of businesses, academic institutions, state and local law enforcement agencies, and other participants dedicated to increasing the security of United States critical infrastructures.

All InfraGard participants are committed to the proposition that a robust exchange of information about threats to and actual attacks on these critical infrastructures is an important element for successful infrastructure protection efforts.

The goal of InfraGard is to enable that information flow so that the owners and operators of infrastructure assets can better protect themselves and so that the United States government can better discharge its law enforcement and national security responsibilities.

Questions: If you would like more information about the InfraGard program, please contact your local FBI field office. If you would like additional information about the National Infrastructure Protection Center, please visit the NIPC Web site or inquire via e-mail at nipc@fbi.gov.

APPENDIX D
Additional Information
A) Industry Initiatives
Increasing concern about security is evidenced by initiatives sponsored by industry organizations such as the Electric Power Research Institute, government initiatives numerous books and articles being published in trade journals. Three initiatives are of particular interest for Public Utilities:

• In addition to the federal initiatives, industry organizations have also mapped out initiatives addressing security. The most notable and relevant to water production being the Electric Power Research Institute - (EPRI) Enterprise Infrastructure Security Initiative:
  This initiative includes the following topics, which are particularly relevant to the utility industry:
  • Security Primer - A discussion of basic security principles, nomenclature, protection techniques, references, etc.
    
  • DCS/PLC Primer - A presentation on the design, functionality and security issues related to these systems
    
  • SCADA/EMS (Energy Management Systems) Primer - A presentation on the design, functionality and security issues related to these systems
    
  • Guidelines - Addresses issues raised in Security Primers
    
  • Industry Strategy Paper - Addresses information sharing and reporting issues
    

According to EPRI, the only product published as of the date of this report from the EIS initiative is the Security Primer, order number 1000797. It will not be available for sale to non-members until late 2001 and will be priced at $2,500. EPRI products are generally only available to members upon publication. They are then made available for non-member distribution six to eighteen months after initial publication date. The remaining products should not be expected for non-member distribution until the end of 2001 at the earliest.

Additional information can be found on EPRI's Web site at http://www.epri.com.

• A research project funded by US Environmental Protection Agency (USEPA) in cooperation with the AWWA Research Foundation (AWWARF), which has contracted with Sandia Laboratories to develop a vulnerability assessment methodology.
  
• The Association of Metropolitan Water Agencies (AMWA) Critical Infrastructure Protection Advisory Group (CIPAG), which began meeting in January 2001. CIPAG includes representatives from industry and federal agencies and is providing input and support for a variety of services including:
  • An Information Sharing and Assistance Center for the water supply sector
    
  • Guidance documents that will outline what steps to take to protect a facility against attack, respond to attack and mitigate the consequences of an attack
    
  • Cooperative meetings for critical infrastructure sectors organized through the US Chamber of Commerce and the Critical Infrastructure Assurance Office (CIAO), a federal coordinating office
    

B) Summary of Security Best Practices
Best practices addressed in this memorandum encompass the following topic areas:

• Computer and network operation security
  
• Physical security
  

Computer Operation Security

• Internal threats are usually the main security challenge. The key to managing internal threats is understanding who did what and when they did it.
  
• Log-ins should be traceable and a strong password authentication process used.
  
• Suggest that Utilities consider fiber optics to each of its remote facilities. This would be the best solution in terms of bandwidth and security.
  
• Suggest taking a look at commercially available tools for monitoring computers and having a security team review activity on a weekly basis to verify that authorized users are using the system.
  
  

Physical Security

• A public utility's physical security measures may want to consider including gate card readers, motion detection alarms, perimeter intrusion monitoring, key locked remote facilities (inside of fence on metal cabinets) and closed circuit television monitoring.
  
• A good practice would be to tie authentication of physical presence.
  
• Public utility's remote sites can also be vulnerable during times of data communication failure when the remote system cannot communicate with an operations center. While these are rare and usually short-lived events, undetected intrusions could occur during these times. If a utility provides fiber optic links to each remote facility will provide a more secure communications for intrusions alarm notification. Other alternatives being considered by many utilities are remotely monitored CCTV using fiber optics communications media or digital phone services such as DSL.
  

C) Literature Searches
A number of periodicals publish articles on various aspects of computer and network security issues. Typical articles frequently available include:

• CIO Magazine: http://www.cio.com. See the March 1, 2001 issue Special Section Security accessible at http://www.cio.com/archive/030101/index_content.html for general information on security practices including "12 Keys for Locking Up Tight".
  
• Information Week: http://informationweek.com. See the July 10, 2000 issue feature article "The Threat from Within" accessible at http://www.informationweek.com/maindocs/index_794.htm for information and analysis of security practices in business.
  
• InfoWorld: http://www.infoworld.com. See the May 28, 2001 issue feature article "Biometrics: Security or Special Effect".
  

In addition security organizations provide good sources of information on best practices. These include:

• National Security Institute: http://www.nsi.org
  
• Computer Security Institute: http://www.gocsi.com
  
• National Infrastructure Protection Center: http://www.nipc.gov
  

Some other security system providers include (Web site information added):

• Foundation Software-auditing product
  
• e-Security, Inc.: http://www.esecurityinc.com. Real time security awareness and response software
• Pentasafe: http://www.pentasafe.com. Auditing and security software for AS400
  
• F Secure: http://www.f-secure.com. Data security software for mobile, distributed enterprise.