FROM: Robert Thompson, General Manager
Originator: Wally Ritchie, Director of Finance
SUBJECT:
title
PLANT NO. 1 AND PUMP STATION INDUSTRIAL CONTROL SYSTEM NETWORK REFRESH
end
GENERAL MANAGER'S RECOMMENDATION
recommendation
RECOMMENDATION: Recommend to the Board of Directors to:
A. Approve a Purchase Order Contract to World Wide Technology, LLC for the purchase of Network Infrastructure Hardware for the Plant No.1 and Pump Station Industrial Control System Network Refresh using the OMNIA Cooperative Purchasing Agreement, Contract No. R210407, for a total amount not to exceed $1,516,873 (includes sales tax); and
B. Approve a contingency in the amount of $151,687 (10%).
body
BACKGROUND
Orange County Sanitation District’s (OC San) existing Plant No. 1 and Pump Station Industrial Control System (ICS) network infrastructure equipment has been in service for over 10 years. There are a total of 96 switches currently in production within Plant No. 1 and all Pump Stations. Standard procedure is to replace equipment prior to it going to end-of-life status but this equipment is already at its end-of-life and is no longer supported by the vendor. The Plant No.1 ICS network infrastructure is in dire need of an upgrade and the plan is to upgrade the entire network stack with a current series of modern switching hardware models that are vendor supported. The Plant No. 2 network infrastructure equipment was replaced two years ago as part of the J-117B project.
RELEVANT STANDARDS
• Protect OC San assets
• Ensure the public’s money is wisely spent
• 24/7/365 treatment plant reliability
• Maintain a culture of improving efficiency to reduce the cost to provide the
current service level or standard
PROBLEM
The existing Plant No. 1 and Pump Station ICS network infrastructure equipment has been in service for over 10 years. Running equipment that is at the end of its life cycle and no longer supported by the vendor poses significant risks. This equipment could fail at any time, and without the ability to source replacements immediately, business operations may be impacted. Additionally, there is a heightened risk of cybersecurity vulnerabilities, as the software running on the network switches is outdated and there is no remediation plan due to the lack of support. The entire Plant No. 1 and Pump Station ICS network infrastructure is currently facing this dilemma. Over the past year, Plant No. 1 has experienced multiple switch failures and degraded performance.
PROPOSED SOLUTION
The Cisco Catalyst 9000 series and Industrial Ethernet 3300 series provides reliable and secure connectivity, with high performance, low latency, and advanced security. The network equipment provided by World Wide Technology, LLC meets all of OC San’s ICS network performance and resiliency requirements.
TIMING CONCERNS
It is imperative to replace the Plant No. 1 and Pump Station ICS network infrastructure as soon as possible to avoid end-of-life equipment failure that can have a detrimental effect on the Plant No. 1 and Pump Station SCADA systems.
RAMIFICATIONS OF NOT TAKING ACTION
OC San’s network engineers have recently experienced an increase in equipment failures due to old and outdated hardware. If equipment failure occurs within the core switches, it could result in a complete network outage, halting business continuity and operations.
PRIOR COMMITTEE/BOARD ACTIONS
N/A
ADDITIONAL INFORMATION
Based on network hardware review, staff recommends approving the Purchase Order Contract to World Wide Technology, LLC using the OMNIA Cooperative Purchasing Agreement, Contract No. R210407. The term of this Purchase Order Contract will begin upon effective date of the Notice to Proceed.
CEQA
N/A
FINANCIAL CONSIDERATIONS
This request complies with authority levels of OC San’s Purchasing Ordinance. This item has been budgeted (Budget FY2024-25 and 2025-26, Section 8, Page 91, ICS Network Equipment Upgrade) and the budget is sufficient for the recommended action.
ATTACHMENT
The following attachment(s) may be viewed on-line at the OC San website (www.ocsan.gov) with the complete agenda package:
N/A