FROM: James D. Herberg, General Manager
Originator: Kathy Millea, Director of Engineering
SUBJECT:
title
PROCESS CONTROL SYSTEMS UPGRADES STUDY, PROJECT NO. SP-196
end
GENERAL MANAGER'S RECOMMENDATION
recommendation
RECOMMENDATION:
A. Approve a Master Services Agreement with ABB Inc. for the design, installation, implementation, and maintenance of a process control system;
B. Authorize staff to specify ABB Inc. as the sole source provider for equipment, materials, software, and services for control systems on current and future projects;
C. Ratify the addition of ABB Inc. to the list of pre-approved Original Equipment Manufacturers for procurements under $100,000 for equipment, materials, software, and services for control systems.; and
D. Authorize the General Manager to approve and execute the 3-way Escrow Intellectual Property Agreement (Exhibit “E” to the Master Services Agreement with ABB Inc.) with Escrow Agent NCC Group Software Resilience (NA) LLC, at a cost not to exceed $100,000, in a form approved by General Counsel, for escrow setup and validation.
body
BACKGROUND
The Supervisory Control and Data Acquisition (SCADA) system provides essential process monitoring, control, and archiving of process data, including data required for regulatory compliance. The SCADA system provides automatic control and allows operators to interactively monitor and control the process, electrical and central generation facilities from centralized areas, assuring that the systems are performing properly and that alarm conditions can be monitored and responded to immediately.
The existing SCADA system utilizes programmable logic controllers (PLCs) and a Human-Machine-Interface (HMI) system consisting of networks, servers, workstations, software, and a data archiving system referred to as a historian.
The existing HMI software package is no longer supported by any vendor and requires emulation software to use Windows-based workstations. This system will become increasingly difficult to service and maintain with time because there are only a few people outside of the Orange County Sanitation District (OC San) who can manage that software to support the current and future needs of OC San.
Furthermore, the existing make and model of the PLCs used at OC San facilities has reached the end of commercial availability and requires the selection of a new model for future designs. OC San has purchased sufficient spare parts for system maintenance; however, a long-term solution is required. OC San is currently using the latest PLC model of the existing manufacturer as an interim solution.
Process Control Systems Upgrades Study, Project No. SP-196, was created to determine the requirements for the new SCADA system, to develop a scope of work to replace the existing HMI and perform a multi-phase procurement to evaluate and select a vendor platform to replace the existing SCADA system. Replacement of the existing HMI will occur under Process Control Systems Upgrades, Project No. J-120. In February 2018, the Board of Directors approved a contract with Stantec to assist OC San with this work.
RELEVANT STANDARDS
• 24/7/365 treatment plant reliability
• Sustain 1, 5, 20-year planning horizons
• Maintain a proactive asset management program
PROBLEM
The existing SCADA system is at the end of its useful life and requires replacement to continue to provide a reliable system for OC San Operations and Maintenance staff. The selection and sole sourcing of a new vendor for HMI software and controller hardware is required to provide a long-term integrated solution that minimizes staff training and spare part inventory.
PROPOSED SOLUTION
Approve a Master Services Agreement (MSA) with ABB Inc. for the design, installation, implementation, and maintenance of a control system for a term of 15 years with renewals up to 25 years. Specific equipment and services will be procured by Task Orders associated with each project.
Through a competitive procurement process, OC San staff concluded that ABB Inc. provided the best value, and the pricing was fair and reasonable. This Agreement requires that ABB submit evidence that OC San receives the lowest unit price for products, items, components, and services for the duration of the MSA.
To protect OC San, the MSA requires that the intellectual property and materials be held using an Escrow Agreement in a three-way contract between OC San, ABB, Inc. and Iron Mountain as the escrow agent. OC San is currently negotiating the Escrow Agreement terms and conditions with Iron Mountain.
The Escrow Agreement enables OC San to fully use and benefit from the intellectual property in the event that ABB, Inc. is unable or unwilling to perform under the terms of the MSA. The intellectual property includes files and source code for software, hardware and software configuration files for the SCADA Platform, and all tools, devices, software, and hardware required for the repair and reproduction of these materials. The costs for this Escrow Agreement include an initial setup fee, validation, and annual maintenance costs.
TIMING CONCERNS
The existing HMI and PLC systems have been in service for over 20 years and have become obsolete. Finding programmers with the appropriate training to support the HMI system is becoming difficult. The existing HMI needs to be replaced as a complete system.
RAMIFICATIONS OF NOT TAKING ACTION
It will become increasingly difficult to support and maintain the HMI software due to staff and consultant retirements in the next 10 years. Replacement parts and repair of the existing PLC hardware will not be available. OC San will continue to use the latest PLC hardware from the current manufacturer.
ADDITIONAL INFORMATION
On February 19, 2019, OC San issued a Request for Qualifications (RFQ) from firms who are manufacturers and experts in the design, development, installation, and start-up of SCADA platforms and who can guarantee the long-term product viability, system performance, and reliability of the SCADA system. On April 16, 2019, the following six firms submitted Statement of Qualifications (SOQs) in response to the RFQ:
• ABB Inc. (ABB)
• AVEVA Software LLC (AVEVA)
• Emerson Process Management Power & Water Solutions, Inc. (Emerson)
• Honeywell
• Rockwell Automation
• Tihedral - VTScada
The SOQs included a number of mandatory capabilities and features, which only the following three firms indicated that they could satisfy. These three firms advanced to the proposal phase:
• ABB Inc. (ABB)
• AVEVA Software LLC (AVEVA)
• Emerson Process Management Power & Water Solutions, Inc. (Emerson)
On January 14, 2020, OC San issued a Request for Proposals (RFP) to the three qualified firms to (1) further evaluate the technical merit of the proposed SCADA platforms and project delivery approach, (2) establish long-term guaranteed prices for software licenses, labor, and equipment, (3) establish a fixed price for Project No. J-120, and (4) establish for comparative purposes a reference cost to replace all of the control system hardware and software.
On June 16, 2020, the proposals were received and evaluated in accordance with the evaluation process set forth in OC San’s Purchasing Ordinance by a pre-selected Evaluation Team consisting of the following OC San staff: Project Manager, Project Engineer, two Engineering Supervisors, one Engineering Manager, and one Maintenance Manager. The Evaluation Team also included three non-voting technical representatives and three non-voting representatives from the Contracts Administration Division.
The following evaluation criterion were described in the Request for Proposals (RFP) and used to determine the most qualified Proposer:
|
CRITERION |
WEIGHT |
|
Technology |
50% |
|
Price |
40% |
|
Project J-120 Delivery Response |
10% |
Initial scores were developed for the proposals and there was a natural break in rankings between Emerson and the other proposers, ABB and AVEVA. Based on this, OC San moved forward in the trials with only Emerson.
From October 12 through October 30, 2020, OC San conducted technical trials with Emerson. During the trials, Emerson’s system failed to meet five separate mandatory RFQ requirements. For this reason, OC San eliminated Emerson from further consideration.
On November 9, 2020, ABB was informed that OC San would proceed to the trial phase with ABB. The trials were conducted between February 8 and February 19, 2021, and ABB successfully demonstrated that their system met all of the requirements.
On March 17, 2021, the Evaluation Team conducted an interview with ABB as part of the procurement and evaluation process. At the conclusion of the interview, the Evaluation Team determined that ABB would provide the most complete SCADA system at the most competitive price and thereby will provide the best value for OC San.
Due to the elimination of Emerson the ABB and AVEVA price proposal scores were updated which resulted in the final scoring as presented in the table below:
|
Proposer |
Non-Disclosure Agreement Response (Pass/Fail) |
Price Proposals (Max.40 Points) |
Project Delivery Response (Max. 10 Points) |
Technology Response (Max. 50 Points) |
Total Score (Max. 100 Points) |
|
ABB |
Pass |
40 |
7 |
32 |
79 |
|
AVEVA |
Pass |
25 |
6 |
34 |
65 |
|
Bidder |
Price Proposal |
|
ABB Inc. |
$48,436,793 |
|
AVEVA Software, LLC. |
$78,984,524 |
A notice was sent to all the vendors on April 1, 2021, informing them of the intent of OC San to negotiate with ABB Inc. for the MSA.
Review of Unit Prices and Agreement:
Staff reviewed the unit prices for the MSA. In addition, discussions were held to clarify MSA exceptions identified by ABB Inc. during the RFP process. ABB Inc. agreed to the Terms and Conditions of the MSA. This was further validated by staff from Stantec, the consultant retained to support OC San through this project.
Based on the above, staff recommends approving the Agreement to ABB Inc.
CEQA
The project is exempt from CEQA under the statutory exemptions set forth in CEQA Guidelines Section 15262. A Notice of Exemption was filed on March 1, 2018.
FINANCIAL CONSIDERATIONS
This request complies with the authority levels of OC San’s Purchasing Ordinance. The required funding for all but the Escrow Agreement will be obtained from the projects where the equipment, material, and services are utilized, or in the appropriate operating budget allocations.
The initial costs for the Escrow Agreement would be funded under Process Control Systems Upgrades Study, Project No. SP-196 (Budget Update, Fiscal Year 2021-2022, Appendix A, Page A-8) and the budget is sufficient for the recommended action. On-going costs for the Escrow Agreement would be funded under the Repairs and Maintenance line item for the Administrative Services Department (Budget Update Fiscal Year 2021-20221, Page 39) and the available funding is sufficient for the recommended action.
ATTACHMENT
The following attachment(s) may be viewed on-line at the OC San website (www.ocsan.gov) with the complete agenda package:
• Master Service Agreement
• Draft Exhibit “E” Escrow Agreement
• Presentation
GR:dm:gc