Richmond Water Plant Response

Yes, at the direction of Governor Youngkin, VDH, partnered with engineering firm Short Elliott Hendrickson, Inc. (SEH), to conduct its own investigation.  The final report was issued on April 16.  

Additionally, on January 23, VDH’s ODW issued an Initial Notice of Alleged Violation (NOAV) to the City of Richmond pursuant to Code of Virginia §12VAC5-590-110. 

On February 21, Richmond DPU responded to the Initial NOAV.

On March 11, ODW issued a letter to DPU and the Richmond mayor regarding Richmond's NOAV Response.

The water crisis was completely avoidable and should not have happened. The Department of Public Utilities (DPU) management and leadership made at least three critical errors in their operational decisions, which caused the water crisis. First, DPU operated the water treatment plant (WTP) in “Winter mode,” meaning that the WTP only used the overhead main power feed during the winter months as a cost saving measure. This decision unnecessarily created a single point of failure without adequate redundancy and was the exact wrong season to do it. DPU has stopped operating in this mode. The water crisis would never have happened if DPU had operated the plant in “Summer mode,” such that both the overhead and underground power feeds were supplying power to the WTP.

Next, DPU knew that flooding of critical electrical and pumping systems was a risk for decades; yet did not take appropriate actions to properly maintain critical back-up systems powering valves that must be shut to prevent flooding. The Uninterruptible Power Supply (UPS) battery back-up systems powering the valves were not properly maintained and were past their design lives. Operators reported that they knew the UPSs did not work; yet DPU did not address it. Had DPU properly tested, maintained, and replaced the UPS battery back-ups (every 3-5 years), then the water crisis would not have happened. Since the water crisis DPU has replaced the UPS battery back-ups and properly tested them.

Finally, DPU allowed the plant to become overly reliant on manual operation. The WTP should have more automatic operation. The back-up diesel generators were useless in the power failure because the back-up generators required manual start and were tied to the single point of failure, Switchgear 6, from “Winter mode” operation. Given the WTP’s need for significant manual operation, DPU did not have enough trained staff, specifically an electrician, at the WTP during a known weather event, even though the governor and mayor had issued emergency declarations. Had the back-up diesel generators been automatically available with both the overhead and underground main power feeds activated (“Summer mode”), then the water crisis would not have happened. DPU is in the process of improving automation at the WTP. The back-up diesel generators should be automatically operated by October 2025.

The three improper operational decisions were a symptom of institutional complacency. DPU staff had a culture of not reporting problems or not promptly repairing and maintaining equipment. Flooding was a known hazard at the WTP for decades; yet, there was institutional acceptance that problematic conditions such as possible flooding at the WTP were normal and could not be changed or fixed. DPU staff did not worry enough about these problems. DPU had not updated its emergency response plans for a power outage, did not have standard operating procedures to respond to a power outage, and had not practiced or trained DPU management or staff on emergency response awareness. DPU is updating its SOPs for emergency response and has trained staff on emergency response to a power outage.

Redactions are associated with this FOIA exemption: § 2.2-3705.2. Exclusions to application of chapter; records relating to public safety.

14. Information contained in (i) engineering, architectural, or construction drawings; (ii) operational, procedural, tactical planning, or training manuals; (iii) staff meeting minutes; or (iv) other records that reveal any of the following, the disclosure of which would jeopardize the safety or security of any person; governmental facility, building, or structure or persons using such facility, building, or structure; or public or private commercial office, multifamily residential, or retail building or its occupants:

a. Critical infrastructure information or the location or operation of security equipment and systems of any public building, structure, or information storage facility, including ventilation systems, fire protection equipment, mandatory building emergency equipment or systems, elevators, electrical systems, telecommunications equipment and systems, or utility equipment and systems.

VDH is planning a revised, second Notice of Alleged Violation (NOAV) based on the investigation. A NOAV is always required when the Office of Drinking Water believes a regulatory violation occurred. The NOAV outlines the specific observations for each regulation alleged to be in violation. The regulated party, in this case the City of Richmond (as owner and operator of the public water system), is provided with due process and an opportunity to either agree or disagree with the observations or the alleged violation. The NOAV also describes the Office of Drinking Water’s requested actions to eliminate or offset the alleged violation. In this specific situation, the City of Richmond must submit a corrective action plan and time to respond and eliminate the alleged violations.

The following regulations were identified in the NOAV previously issued to the City of Richmond, but only 12VAC5-590-360.A and 12VAC5-590-510.C were alleged to have been violated based on the facts known at that time.

• The Waterworks Regulations at 12VAC-590-350.A state, “Frequent assessments shall be made by the [waterworks] owner of the water supply and waterworks to locate and identify health hazards to the waterworks. The manner and frequency of making these assessments, and the rate at which discovered health hazards are to be removed, shall be the responsibility of the owner. Every effort shall be made by the owner, to the extent of his jurisdiction, to prevent the degradation of the quality of water supplies.”

• The Waterworks Regulations at 12VAC5-590-360.A state, “The [waterworks] owner shall provide and maintain conditions throughout the entirety of the waterworks in a manner that will assure a high degree of capability and reliability to comply with Part II (12VAC5-590-340 et seq.) of [the Waterworks Regulations]. This requirement shall pertain to the source water, transmission, treatment, storage, and distribution system facilities and the operation thereof. The owner shall identify and evaluate factors with the potential for impairing the quality of the water delivered to the consumers. Preventative control measures identified in Part II of [the Waterworks Regulations] shall be promptly implemented to protect public health.”

• The Waterworks Regulations at 12VAC5-590-450 state, “Waterworks operation comprises the constant oversight and management of the facilities and personnel. Consideration shall be given to such factors as the competency of personnel; water quality, including drinking water standards; water treatment plant maintenance and cleanliness; analytical laboratory control; and the operation and maintenance of the facilities, including water treatment plant equipment, distribution system equipment, and piping. As the complexity of the waterworks increases, so does the expertise and skill required of the operating staff.”

• The Waterworks Regulations at 12VAC5-590-461.B state, in part, “Operator requirements. The operation of all waterworks must rest in the hands of qualified staff. The number and qualifications of persons constituting the operating staff at a waterworks depend principally upon the capacity of the waterworks, the number of persons served by the waterworks, and the complexity of the treatment process or processes… 1. The operator attendance requirements specified in subsection C of this section are a minimum to protect the health of the consumer and safety of the operating staff. [VDH] may increase the required operating attendance when appropriate to protect human health.”

• The Waterworks Regulations at 12VAC5-590-470 state, “The waterworks shall be maintained in a clean and orderly condition.”

• The Waterworks Regulations at 12VAC5-590-480.E state, in part, “Process control instruments, monitors, gauges, and controllers, including reading, recording, and alarm features, required in Part III, Manual of Practice (12VAC5-590-640 et seq.), shall be maintained fully operational and calibrated in accordance with the manufacturer instructions.”

• The Waterworks Regulations at 12VAC5-590-505.A – B state, “A. The owner of a community waterworks (including consecutive waterworks) shall develop and maintain an emergency management plan for extended power outages. B. The plan shall be kept current and shall be retained at a location that is readily accessible to the owner in the event of an extended power outage.”

• The Waterworks Regulations at 12VAC5-590-510.C state, “All waterworks shall provide a minimum working pressure of 20 psigauge (psig) at all service connections.”

• The Waterworks Regulations at 12VAC5-590-725 state, in part, “The design of computers, including supervisory control and data acquisition (SCADA) systems if used to monitor and control water treatment and distribution system facilities, shall meet the following general requirements... 4. Waterworks pumps, chemical feeders, and other essential electrical equipment controlled through a SCADA or an automated control system shall have the capability for independent manual operation. Where a high degree of reliability is required, a backup control system shall be provided.”

• The Waterworks Regulations at 12VAC5-590-730 state, “A. An emergency management plan for extended power outages shall be developed for each community waterworks as specified in 12VAC5-590-505. B. Alternative power sources at all waterworks shall be considered in the design to maintain a minimum level of service during an electrical power outage.”

The city provided a response to the prior NOAV, which the ODW reviewed. ODW previously provided comments and recommendations to the city regarding its response to the first NOAV.

A Notice of Alleged Violation (NOAV) is a formal notification issued by VDH pursuant to Virginia Waterworks Regulations. A NOAV summarizes any alleged violations to the law or regulations, the facts about an incident, initial findings, and the expectations for correction.  

Upon receipt of the NOAV, the recipient has 30 days to respond and provide requested information as outlined in the NOAV. 

Pursuant to the Virginia Waterworks Regulations, the recipient may request an informal fact-finding proceeding (IFFP) if the recipient disagrees with the allegations in this initial notice or wishes to obtain a determination as to whether a violation has occurred.  The recipient has 15 business days upon receipt of the initial notice to inform VDH if it wishes to request an IFFP. 

Failure to take all the necessary corrective actions in a timely manner to voluntarily return the situation to compliance may result in enforcement action. 

ODW will have several division leads responsible for the critical follow-up work, including the Compliance & Enforcement Division; the Capacity Develop team, the Emergency Preparedness Coordinator, and the Richmond Field Office for additional technical support and inspections.

ODW will expect the City of Richmond waterworks owner to submit a corrective action plan and timeline to complete needed follow-up. ODW will closely monitor the corrective action plan and ensure that it is implemented.

ODW expects public trust to improve over time as the City of Richmond demonstrates its ability to provide safe and reliable drinking water over the long run.

Significant learning and understanding developed from the water crisis, which the city of Richmond waterworks will use to improve operations and oversight.

The City of Richmond waterworks is making noticeable improvements with its operations, policies, training, procedures, communications, and infrastructure. The city is developing a new culture for emergency preparedness and response.

Short Elliott Hendrickson, Inc. (SEH) is an engineering firm with more than 900 employees. Almost 60% of SEH’s work is focused on municipalities, including water supply, treatment, distribution, and storage. SEH serves metropolitan areas with more than 50,000 customers such as Minneapolis Waterworks, St. Paul Regional Water Services, Madison, Eau Claire, Des Moines, South Bend, the Western Virginia Water Authority, and the Rivanna Water & Sewer Authority, demonstrating SEH’s ability to evaluate and support large water systems. SEH has more than 15 years of experience with waterworks in Virginia. SEH had no conflicts of interest with the City of Richmond or the associated water crisis.

In the morning hours of Monday, Jan. 6, a winter weather-related power outage impacted operations at the City of Richmond’s water plant along the James River.    

Under normal circumstances, the drinking water treatment process begins by withdrawing water from the James River. The water is then pumped into the water plant, where it goes through several stages of treatment to remove bacteria, viruses, parasites, and other contaminants of concern. After the final treatment, safe and clean water travels to homes and businesses from the water treatment plant through storage reservoirs, elevated storage tanks, and hundreds of miles of distribution piping.  

Monday’s power disruption caused water to flood an area of the plant where pumps and electrical cabinets were located. Water production stopped because the water treatment plant’s “Supervisory Control and Data Acquisition System” (SCADA system) went offline. The SCADA system consists of hardware and software to monitor and adjust plant processes. Best information suggests that the flooding – to a critical operational area of the water treatment plant – happened when the SCADA system went offline. Without the SCADA system working properly, pumps and valves had to be manually operated. Additionally, pumps at the plant stopped working and needed repairs from the flooding. Electricians, pump mechanics, and a contractor for the SCADA system worked to bring these critical needs back online.  The City of Richmond has brought in many resources and experts to complete repairs as quickly as possible.  

On Tuesday, January 7, the SCADA system glitched a second time causing additional delays in water production. The Richmond City Department of Utilities (DPU) had additional pumps delivered and installed to replace failed pumps that needed repair. Troubleshooting, repairs, and other work continued to bring the water treatment plant back to normal operation. 

Four Henrico DPU staff were onsite helping at the water treatment plant, including two electricians. An engineer from Whitman Requardt and Associates was onsite providing technical assistance. The Hanover utilities director and the Chesterfield DPU director visited the water treatment plant and assessed operations. The VDH’s Office of Emergency Preparedness, the Virginia Department of Emergency Management (VDEM), and the Office of Drinking Water (ODW) assisted water distribution via water tanker trucks for rapid deployment to healthcare facilities as needed. VDH’s ODW had engineers and other staff monitoring 24/7 at the water treatment plant to help with regulatory questions, needs, and technical support.

ODW had engineers and other technical specialists at the water treatment plant 24 hours per day to ensure that the City of Richmond returned to compliance with applicable regulations that protect public health and drinking water as quickly as possible. ODW provided inspections, funding, compliance, and regulatory oversight of drinking water systems in Virginia. ODW also provided advice and technical support to help resolve ongoing issues while evaluating the causes of the event and ensure regulatory compliance. VDH completed an after-action review, timeline of events, learning lessons, compliance review, and other assessments. 

ODW performed numerous inspections from January 6 through January 11.  ODW has been at the water treatment plant routinely since then.  Before the water crisis, ODW’s engineering team completed an initial inspection on September 19, 2024 for a series of planned inspections.  ODW completed an investigation into the cause of the water crisis and the final report was released on April 16.  Inspections occur over time because of the size and complexity of the water system.  The last complete inspection of the facility happened in 2022 where the US EPA led an inspection of the facility with support from ODW.