Tips On Overcoming Challenges With Converged Networks

Edited by Leslie Langnau, Managing Editor

The trend of converging industrial networks with enterprise networks is encountering several challenges including lack of consistent guidance and recommendations and legacy systems. Use of the Converged Plantwide Ethernet system can resolve many of these challenges.

As manufacturing companies expand their global operations to take advantage of new opportunities and reduce operational costs, they must continually increase connectivity between industrial automation and control systems (IACS) and business systems to:

• Ensure consistent quality and performance across global operations
• Balance manufacturing with demand
• Improve and meet regulatory compliance

The goal is to gain more flexible and agile manufacturing operations in order to respond to rapidly changing market conditions.

Industrial automation and control system (IACS) manufacturers are currently falling short of these objectives. The solution is better access to information. With a constant flow of data, engineers and managers can develop more efficient ways to connect globally with suppliers, employees, and partners, and therefore, more effectively meet the needs of end customers.

To obtain more access to information, manufacturers are converging their IACS networks with their enterprise networks. The challenges to a successful convergence, though, include:

• Reliability—Global integration challenges a system’s ability to deliver consistent access to data while making the manufacturing environment flexible. Issues that must be addressed include security, availability, and asset use because IACS equipment is mission-critical, and efficiency is important to remain competitive.

• Cost—Legacy IACS can be difficult to integrate with the enterprise and can be costly to operate due to the multiple networks in use that require management, training, integration, gateways, spares, and so on.

• Product design integration—Limited access to local subject-matter experts constrains collaborative manufacturing.

• Service integration—Incorporating real-time plant productivity and operational data into manufacturing execution systems (MES), customer relationship management (CRM), supply chain management (SCM), and other enterprise resource planning (ERP) systems restrict and constrain the ongoing move to service-oriented architectures.

• Partner connections—With an aging and decreasing workforce and increased manufacturing complexity, manufacturers are trying to find ways to leverage relationships with IACS vendors to support their plant floor applications.

These challenges are pushing manufacturers to adopt standard Ethernet and IP network technologies throughout the manufacturing environment. But, challenges here have slowed the adoption.

One challenge has been the lack of consistent guidance and recommendations that are relevant to both IT and control engineers. Another challenge is that some IACS vendors continue to promote legacy or application-specific IACS networking technologies.

Modern, full-duplex, switched Ethernet networks offer real-time performance, including latency, jitter, and packet-loss avoidance that meet or exceed the needs of IACS applications while offering better benefits than the older field-bus networks they replace. In addition, these modern networks have mature and tested technologies to safely secure the network and the systems they interconnect beyond what are available for the older field-bus networks.

IACS network environments have evolved over the years, driven by design features not specific to industrial Ethernet but to networking for the IACS in general. These features, which are viewed as best practices are:

• Industrial characteristics
• Interconnectivity and interoperability
• Real-time communication, determinism, and performance
• Availability
• Security
• Manageability
• Scalability

IACS end-devices and network infrastructure are located in harsh environments that require compliance to environmental specifications such as IEC529 (ingress protection) or NEMA specifications.

The physical layout of the IACS equipment impacts the network topology. Unlike IT networks, which are largely redundant star topology networks, IACS networks require the use of bus, linear, star and ring topologies. In plants with long manufacturing lines, or equipment with long runs and interconnected operations (such as a printing press), it is often not feasible or cost-effective to use a redundant star topology. In manufacturing environments, the costs of cabling are high. Given these cost considerations, many manufacturers choose to implement a ring topology rather than a redundant star topology where network resiliency is a requirement. In many cases, the IACS network uses a combination of topologies, with large rings connecting multiple star-based Cells/Areas.

IACS network devices can interoperate using standard, common protocols at Layer 7 (application), but often some gateway device/service is needed to perform an application layer translation.

One solution is a Converged Plantwide Ethernet (CPwE) design, which is based upon the use of CIP as the common application layer protocol for IACS network interoperability employing EtherNet/IP as the IACS network.

The TCP/IP protocol suite with the CIP application layer protocol helps ensure that IACS devices, regardless of vendor, communicate and work together. Additionally, conformance testing from organizations such as the ODVA certifies that EtherNet/IP devices from various vendors communicate and interoperate.

The TCP/IP standards outline a range of features and functions. Therefore, in theory, the concepts, recommendations and implementations CPwE specifies should be applicable to a range of other vendor’s devices and solutions.

Key considerations in achieving real-time communications include the following:

• Number of switches, routers, and amount of traffic in the Layer 2 network, all of which affects latency and jitter.
• Ratio of LAN switch ports to uplink switch ports based on traffic loads and patterns. Typically, this means using 10/100 Mbps for IACS devices and 10/100/1000 Mbps for uplinks.
• Use of Internet Group Management Protocol (IGMP) to manage the efficient delivery of multicast traffic.
• Use of quality-of-service (QoS) parameters to meet the real-time requirements of various traffic flows.

Key availability considerations include:

• Creating alternative data communication paths, regardless of physical layout. Risk profile, opportunity cost, culture, and other variables determine how much and to what level redundant paths are required.
• Eliminating single points of failure with critical operations, which could include the use of dual-power supplies, alternate routes for redundant media, and redundant IACS network infrastructure, such as routers, switches, and firewalls.
• Using advanced network resiliency and convergence techniques to improve availability, such as EtherChannel/LACP, Multiple Spanning Tree Protocol (MSTP), Flex Links, and Hot Standby Routing Protocol (HSRP).
• Although a redundant star topology offers the best convergence capabilities, consider alternative ring recovery techniques when configured in a ring topology.
• Use routing protocols such as EIGRP or OSPF to achieve high availability.

Connecting an IACS network to the enterprise network exposes the IACS application and enterprise network to the security risks of the Internet. Mitigating these risks may be more difficult and more critical in the IACS than in the enterprise network because of the higher requirement for availability in an IACS and the sensitivity of these systems to different disruptions. Of the three security properties of confidentiality, integrity, and availability, IACS applications are primarily concerned with availability and integrity. Many of the applications that IACS networks support cannot be stopped or interrupted without serious physical damage or loss of productivity with measurable financial damage.

Building secure and reliable IACS networks using Ethernet and IP has been a challenge. But specific security principles of the CPwE architecture can help ensure secure transmissions. These principles are:

• Control data flows between different IACS levels (ACLs, firewall rules, and so on).
• Prevent direct communication between IACS and enterprise applications.
• Restrict real-time manufacturing data to the IACS network.
• Restrict enterprise access to the mirror version or copies of IACS data to the DMZ (also referred to as perimeter networking).
• Authenticate and authorize user access based on the level within the IACS network and the role (read/read-write/local/remote/vendor/partner).
• Control rogue access to switches inside the IACS network (port level MAC-address controls, administratively shutdown unused ports, etc).
• Control which IACS devices can be plugged into the switch (for example, port security, DHCP snooping).
• Detect and mitigate malicious traffic originating from infected devices that are plugged into the IACS network.
• Detect and mitigate malicious traffic originating from the corporate IT network.
• Secure connectivity for remote access to IACS devices.
• Use DMZ design options based on costs and levels of security and redundancy required.
• Limit rogue network communication activity from impacting networking devices (set root bridge, SNMP capabilities, and so on).
• Regarding data and services in the DMZ, connection initiation should originate from either the Manufacturing or Enterprise zone and terminate in the DMZ. Connections originating from the DMZ should be exceptions.
• Document and define policy and risk appropriate for the environment.

The above are provided as principles, with the understanding that customers may choose to make exceptions.

The communication-messaging model in IACS environments has loose ties to traditional client-server or peer-to-peer IT models. Unlike the typical IT environment, standard Ethernet and IP IACS network communications have different patterns, loads, and frequencies. Standard IACS network communications are also driven by status polling between devices, cyclic data transfer, or change of state message patterns. The different requirements of the layers have led key IACS providers to define a variety of communication models, including OSI Layers 1 to 7 networking protocols.

IACS networks using standard Ethernet and IP have a common core. This includes the physical transmission technology (Ethernet, Layer 1), the bus access method (Ethernet, Layer 2), the Internet Protocol (IP, Layer 3), the TCP and UDP protocols (Layer 4), and other standard protocols such as Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), and the Simple Network Management Protocol (SNMP). All these are established within IT and are implemented to varying degrees, unchanged in IACS applications.

The goal of an Ethernet and IP IACS network is to ensure that the application layer protocol of choice, assuming it is based on standard Ethernet and IP is supported to meet the operating constraints of the IACS application.

CIP is a messaging protocol that defines how different IACS network devices, systems, and applications come together to form an IACS application. It is an application-layer protocol (OSI Layers 5 to 7). EtherNet/IP extends the application of Ethernet TCP/IP to the IACS for CIP applications.

CIP is a connection-based protocol and offers two main types of messaging: Explicit-informational and Implicit I/O. The protocol specifies a set of objects and services used to develop an IACS network. CIP is implemented at the application layer of four networks: CompoNet, DeviceNet, ControlNet, and EtherNet/IP.

The important aspects of the CIP implementation of EtherNet/IP are the various types of messaging that are used and how they are implemented in standard Ethernet TCP/IP.

Other key technical considerations for EtherNet/IP implementations include the following:

• The producer-consumer model specifies that “producers” of I/O data communicate through UDP unicasts or multicasts. The consumers (for example, controllers) typically respond with UDP unicast messages. Where multicast is chosen for use, Cisco, Rockwell Automation and the ODVA recommend the application of IGMP to manage the multicast traffic flow.
• EtherNet/IP implementations have traditionally been unable to route multicast traffic since the time-to-live field in the IP packet is set to 1. Although updated CIP EtherNet/IP specifications (CIP Specifications version 1.3, Volume 2 EtherNet/IP Adaptation of CIP, December 2006) call for this limit to be removed, several recommendations are based on the implementation of TTL=1 because the routing of multicast traffic requires a more complex set of protocols and considerations to be applied.
• By the current EtherNet/IP standard, a multicast group is created for each EtherNet/IP adapter that “produces” information, and for each “produced” tag (shared piece of data) established by a controller. EtherNet/IP specifies an algorithm to establish the multicast address and the commands to join and leave multicast groups. Current EtherNet/IP multicasting is based on IGMP version 2, although there are devices (producers) that may still be based on IGMP version 1. IGMP version 1 devices should function in a version 2 environment. This was not tested in the CPwE solution.
• Depending on the device producer, options may be enabled to configure whether the traffic generated by the device is unicast or multicast. This allows more flexibility in Cell/Area zone design and provides a means to manage the number of multicast groups within a Cell/Area zone.

This material is excerpted from a report, Converged Plantwide Ethernet Design and Implementation Guide, Rockwell Automation and Cisco. The authors of this report are: Paul Didier, Industry Solutions Architect, Enterprise Systems Engineering, Cisco Systems, Fernando Macias, Technical Marketing Engineer, Enterprise Systems Engineering, Cisco Systems, James Harstad, Senior Program Manager, Enterprise Systems Engineering, Cisco Systems, Rick Antholine, Commercial Engineer, Commercial Engineering, Rockwell Automation, Scott A. Johnston, Network & Security Services Consultant, Rockwell Automation, Sabina Piyevsky, Team Leader, Global Sales and Marketing – Commercial Engineering, Rockwell Automation, Mark Schillace, Sr. Commercial Engineer, Global Sales and Marketing – Commercial Engineering,Rockwell Automation, Gregory Wilcox, Networks Business Development Manager, Rockwell Automation, Dan Zaniewski, Senior Commercial Engineer, Rockwell Automation, Steve Zuponcic, Marketing Architect, Rockwell Automation

Rockwell Automation
www.rockwellautomation.com

Cisco Systems Inc.
www.cisco.com

>>Reprint Information<<

---

Filed Under: CONNECTIVITY • fieldbuses • networks • gateways
Tagged With: rockwell, cisco
 

---