Modern wastewater uses the cloud

Formally organised in 1834, Waterford Township is located geographically in the centre of Oakland County, Michigan, USA, and is home to over 72,000 residents. It is known regionally for its 34 lakes, from which it earns its name.

In the public utilities industry, however, Waterford is known for its leadership and persistent innovation in water and wastewater management.

With 600 km of water main and 592 km of sewer, water management in Waterford is no small task. The Department of Public Works (DPW) operates and maintains 19 production wells, 3 storage tanks, 11 treatment plants and 63 sewer lift stations. To run all this, it invested years ago in integrating its core applications, including geographic information systems (GIS), asset management systems (AMS), enterprise content management (ECM) and SCADA, all of them sharing data to enable seamless operations.

That system has delivered much value over the years, but nothing lasts forever. And in the last few years, Russell Williams, director of public works, and Frank Fisher, engineering superintendent, started on a project to upgrade core infrastructure. It was an opportunity, not just to avoid obsolescence, but to improve scalability and fault tolerance.

“We attended a conference in Chicago about the release of Opto22’s EPIC in 2018,” Williams said. “We were talking about it on the ride back and said, ‘If this does what it’s supposed to, then it changes the whole layout of everything.’” They were particularly intrigued by MQTT and the way its report-by-exception behaviour promised to improve the efficiency and stability of their infrastructure.

At the time, Waterford’s system used a polling program to request updates from a network of Opto 22 M4 RTUs that communicated through licensed RF transmitters. DPW had already begun replacing these with SNAP-PAC-S2 controllers and industrial cellular modems that communicated through a private internet tunnel. But with MQTT on groov EPIC, they saw the potential to reduce the complexity of the system further.

“We have many lift stations that will spend most of their time sitting,” Williams explained. “So why transfer data all the time?” And with no dependence on a central polling program, they could eliminate a potential point of systemic failure.

“It just looks too simple. You’ve got to question it,” Williams recalled. But, willing to test the premise, Williams and Fisher purchased three groov EPIC units to play with over the summer, and soon they had the evidence in hand.

“We disconnected a controller and within a millisecond the system reported the failure. It really is that easy: change a variable and it shows up in the broker, then on your mobile phone,” Fisher said.

Waterford DPW has since begun rolling out groov EPIC controllers. Since the groov EPIC can communicate directly with cloud-based systems, Waterford chose to host its MQTT broker (Ignition from Inductive Automation) on Amazon Web Services (AWS). Each EPIC uses a CA-signed client SSL certificate to establish a secure broker connection over a cellular modem configured to block any traffic not originating from a trusted IP address. Additionally, with MQTT’s device-originating connections, the groov EPIC's device firewall in each lift station can be closed entirely to outside connections.

With that foundation, Waterford DPW made other infrastructure changes that have put it at the forefront of municipal utilities management. For instance, their Ignition MQTT server is hosted in an Ohio, USA, data centre, but if there are any issues, in 30 minutes Fisher can have the entire system up and running on a snapshot of the same server hosted in an Oregon data centre.

In time, they will likely set up full server redundancy. Meanwhile, they continue to run the legacy SCADA system (on upgraded hardware) in parallel from the local department office. It still manages about a third of the system and will stay up until they have fully migrated to Ignition.

Williams recognises that Waterford is leading digital transformation in the public sector. “I was at a FEMA training not that long ago, and they were adamant about not having an internet connection on your SCADA system,” he said. “But everything we are looking at will be more secure than we could do from the office, because then you make a building a single point of failure.”

With data coming across much faster, and groov EPIC providing spare data processing capacity, Williams and Fisher expect to dramatically increase the connectivity of the system. By integrating residential meter data, for instance, they could help the system stay balanced against demand. And if they can talk other agencies and neighbouring counties into sharing data, they see the potential to build an advanced warning system that would improve their reaction time to system disturbances.

At this point, Waterford DPW has converted 36 lift stations and four treatment plants to the new system, with more to come throughout 2021.