Egress security platform solution for provincial networks of China Broadnet

I. Introduction

With the in-width and in-depth integration of telecom, radio and TV, and Internet, China Broadcasting Network Co., Ltd. (China Broadnet) has obtained the Basic Telecommunication Business Operation Permit and 5G commercial license issued by the Ministry of Industry and Information Technology of the People's Republic of China in May 2016 and June 2019, respectively. Its core services have transformed from traditional cable television to fixed-line broadband services and 5G mobile communication services. With an extensive user basis in cable television, the broadband user scale of China Broadnet has exceeded 40 million within several years. With the implementation of the Broadband China strategy, the industrial development of UHD videos and the continuous implementation and promotion of national and industrial strategies such as "New Infrastructure Construction" in the future, the traffic of intranet and extranet services of China Broadnet is poised for rapid growth.

The development of China Broadnet's broadband services is restrained by the high inter-network settlement expenses of the three major operators. Although the Ministry of Industry and Information Technology of the People's Republic of China issued the Notice on Adjusting the Inter-network Settlement Policy of Internet Trunk Networks in February 2020 and the three major operators reduced the inter-network settlement expenses of Internet trunk networks of China Broadnet by over 30%, the estimated expense calculated based on the annual traffic of 100 Gb still teaches CNY 56 million. Therefore, making efficient use of the egress line resources and optimizing the resource allocation of egress lines while ensuring the broadband experience for users is particularly important for radio and television operators. In addition, as a late comer in the broadband service market, China Broadnet is facing fierce competition from the three major operators. Therefore, China Broadnet needs to enhance the user experience and value-added services of its broadband services to stand out from peers.

II. Current situation and trend of provincial network egresses of China Broadnet

As the Internet broadband services of China Broadnet growth rapidly, the current provincial network egresses of China Broadnet in different provinces have the following characteristics in traffic, lines and services, bringing challenges to the construction of the provincial network egress security platform of China Broadnet.

Large network traffic and rapid growth: The egress bandwidth of provincial networks of China Broadnet is generally above 100 Gb/s with an annual average growth rate of over 20%. Among them, the proportion of HD and UHD video data is increasing day by day with concurrent session demands from over 100 million Internet terminals of various types.

Multi-type egress lines: At present, there are many sources of egress lines for China Broadnet's provincial networks, including traditional operators' egresses, secondary operators' egresses, and Internet ICP egresses. The rental cost of egress lines is high and the main services vary between different egress lines.

Integrated bearing of multiple services: It integrates the services for public users or group customers such as broadband Internet access, broadband television, 5G mobile communication and IoT services. Different service applications and different users may select different egresses.

Service communication during the transition to IPv6: During the transition from IPv4 to IPv6, IPv4 and IPv6 will coexist for a long period and IPv4/IPv6 communication demands are required by the intranet and extranet of China Broadnet.

III. Key requirements for the egress security platform for provincial networks of China Broadnet

Based on the service development characteristics of China Broadnet's provincial networks, the key requirements for the egress security platform for provincial networks include:

Improving the broadband service experience of users: A good broadband experience for users is the basis for reducing complaints and continuously developing users. In terms of broadband experience, the egress lines make a significant difference. The traditional routing and scheduling mode based on user source IP does not meet the routing and scheduling needs of different applications. China Broadnet urgently needs an application-based refined intelligent routing and scheduling technology to improve the user experience of its broadband services.

Improving the utilization of egress lines and optimizing the resource allocation of egress lines: The rental cost of China Broadnet's egress lines is high. Therefore, China Broadnet needs to optimize the utilization of existing egress line resources to improve the efficiency and effect of resource utilization through refined intelligent routing and scheduling. China Broadnet should optimize the configuration of egress lines based on the resource utilization statistics in order to accommodate the expansion of egress lines due to future service growth.

Ensuring the normal communication between IPv4 and IPv6 in intranet and extranet of China Broadnet during the transition to IPv6: The security platform needs to ensure the normal communication between IPv4 and IPv4, IPv4 and IPv6, IPv6 and IPv6 in intranet and extranet of China Broadnet during the transition to IPv6.

Meeting the network supervision requirements: The security platform can record and retain the Internet network address and internal network address used by users. The logs need to be retained for at least six months for audit and traceability by relevant departments.

Innovative value-added services for group users: Radio and television operators have been exploring group customer businesses for innovation as a key source to increase revenue, avoid homogeneous competition and realize business transformation. Besides, the Internet egress security service, as an innovation direction, will provide differentiated value-added services for group customers.

Network address translation (NAT), routing and scheduling under the traffic of N*100 Gb/s: The entire platform design needs to meet the requirements of routing, scheduling, NAT, and NAT log collection performance under high-concurrent scenarios with the great traffic of N*100 Gb/s in the next few years.

IV. H3C security egress platform solution for provincial networks of China Broadnet

Figure 1 Architecture of security egress platform solution for provincial networks of China Broadnet

With the extensive experience in serving radio and television operators, H3C has put forward the security egress platform solution for provincial networks of China Broadnet based on the in-depth analyses of the current development situation and trends of the businesses of radio and television operators as well as their service demands. This solution provides application-based intelligent routing and customer-based security value-added services for group customers. It can facilitate radio and television operators to increase the utilization and optimize the allocation of egress line resources, creating value-added security services for group customers.

The overall architecture of the solution is shown in the figure above. M9000, an integrated gateway dual-node cluster, is deployed at the egress side of the trunk network. The cluster connects with the core routers of provincial networks through N*100 Gb/s links, and deploys optical splitters on trunk links to copy the trunk traffic to the TAP copy splitter (6520X-G). Then, 6520X-G forwards the corresponding traffic to the user behavior management system (ACG1000-XE), full-traffic threat probes, and DDoS detection devices through the N*10 Gb/s links based on the copying and shunting strategies. At the physical bypasses of core routers, the core routers are connected with the DDoS scrubbing device, firewall (including IPS and AV), ACG user behavior management platform, management platform, log system, DNS, and situational awareness platform.

The core devices of this solution include:

Figure 2 Core network element devices

Integrated gateway dual-node cluster M9000: It mainly implements link load balancing (LLB) and NAT44/NAT64 functions of the egresses. A single device supports up to 14 slots. A single LLB service card has a 100 Gb/s throughput and nearly 100-million-level concurrent connection capabilities. In the cluster mode, it can be expanded to the throughput above the TB level and the concurrent connection above the billion level.

TAP copy splitter 6520X-G: It mainly provides copying and splitting functions. A single device has 48 × 10 Gb/s + 2 × 40 Gb/s + 4 × 40/100 Gb/s interfaces, supporting the copying and splitting in scenarios above 400 Gb/s.

Application management system ACG1000-XE: It mainly implements the identification, management and control of user applications. A single device has a 10 Gb/s application identification capability, while that of the cluster mode has no upper limit.

The technical features of this solution are as follows:

1) Application-based intelligent routing

Figure 3 Principle of application-based intelligent routing and scheduling

The traditional egress routing mode of provincial networks mainly includes two scheduling modes based on the user source IP and ISP. The routing and scheduling mode based on user source IP schedules the user traffic to the specified egress line according to the user source IP address, while the routing and scheduling mode based on ISP schedules the egress traffic to the corresponding ISP egress line according to the ISP to which the server resources requested by the user belong. Both the above two scheduling modes have the problem of serious load imbalance or even blocking of the egress lines due to the tidal effect of the client or the server. The routing and scheduling modes unable to distinguish applications cannot satisfy the performance requirements of specific applications on egress lines.

To solve the above problems, H3C proposed a solution that adopts application-based routing to improve the user experience of broadband services and used link-based resource statistics to realize the refined operation of line resources. In this solution, with the application-based intelligent routing technology, customers can schedule the traffic of different users and applications to the suitable egress line to improve the user experience of broadband services; realize refined scheduling by combining traffic application and line status to make full use of the resources of various egress lines; and rationally expand the capacity of the lines or provide decision-making supports for the construction of CDN local storage based on the usage of various egress lines and the changes in service traffic demands, optimizing the resource allocation of egress lines. The specific process is as follows:

(1) Health status check of the link: The egress gateway simulates real user behaviors to initiate HTTP/HTTPS requests on DNS and Internet link by link, and determines the health status of each link according to the response.

(2) Health status check of the NAT resource pool: Egress gateway simulates real user behaviors to use the IP address in the NAT address pool as the source address to detect extranet services (such as DNS, Baidu, and WeChat), and determines the availability of the IP address in the NAT resource pool according to the responses.

(3) Traffic mirroring and application identification: Based on the hashing algorithm of the source address, the TAP splitter mirrors the trunk traffic collected by the optical splitter to the user behavior management device ACG in batches. ACG identifies the traffic of N*100 Gb/s in the whole network at 10 Gb/s, and obtains the mapping relationship between the applications using network traffic and the destination IPs.

(4) Application identification synchronization: ACG periodically synchronizes APP-IP entries and APP classification structure to LLB.

(5) Routing policy configuration: The network administrator formulates an application routing policy based on the characteristics of the link resource demands for different applications and users as well as the service application attributes of all egress links.

(6) Application identification and scheduling:

When the client initiates a service request, LLB matches the application scheduling policy according to the APP-IP entry identification, schedules the user traffic to the egress link with high priority, and detects the blocking of links in real-time.

When the high priority link is blocked, LLB schedules new service requests to the low priority link.

(7) Link traffic statistics and operation: LLB collects link traffic information in real-time and sends it to the management platform, and the management platform completes the traffic statistics and display of all links, providing data support for rational allocation of egress link resources.

2) NAT-based IPv4/IPv6 network interconnection

Figure 4 Principle of NAT-based IPv4/IPv6 network interconnection

As shown in the figure above, NAT64/NAT44 based on the integrated gateway dual-node cluster M9000 meets the normal interconnection requirements between IPv4 and IPv6. The specific service process is as follows:

(1) IPv6 and IPv6 interconnection: The network device supports the direct interconnection of the IPv6 protocol stack.

(2) IPv4 and IPv4 interconnection: Socket-based (IP address + port number) NAT44 dynamic address translation translates a large number of intranet IPv4 addresses to a small number of extranet IPv4 addresses, and translates different intranet IPv4 addresses and ports to extranet IPv4 addresses and ports.

(3) IPv4 and IPv6 interconnection:

Intranet IPv4 actively communicates with IPv6: IPv4 and IPv6 static mapping is configured in the egress gateway to realize the interconnection between IPv4 and IPv6 through NAT64.

Intranet IPv6 actively communicates with IPv4: Socket-based (IP address + port number) NAT64 dynamic address translation translates a large number of intranet IPv6 addresses to a small number of extranet IPv4 addresses.

3) NAT log retention meets the requirements of network security supervision

Figure 5 Principles of NAT log retention and audit traceability

As shown in the figure above, the distributed log system realizes NAT log collection, storage, audit and traceability with the large traffic of N*100 Gb/s and high concurrency of N*1 million events per second (EPS). The specific service process is as follows:

(1) NAT data collection: Egress gateway cluster M9000 actively sends NAT logs to the log system through the syslog protocol in real-time to collect NAT logs. The log system adopts the distributed architecture, which can be extended linearly and can meet the log collection requirements in the future under scenarios with the traffic of N*100 Gb/s and high concurrency.

(2) Data processing: The log system classifies and normalizes the collected NAT logs, converts the logs of different types and expression modes into a unified log format, and uniformly stores the NAT logs after the conversion. Logs are retained for six months, meeting the data retention requirements of the network supervision department. Besides, the log system can help customers conduct audit trails, investigate and collect evidence, and deal with emergencies through rapid retrieval of the massive NAT logs.

4) Value-added operation based on user identification

Figure 6 Principle of value-added operation based on user identification

As shown in the figure above, based on the user identification technology, the platform provides group customers with differentiated value-added services, including high-quality network services, comprehensive security threat detection, protection services, and statistics and operation services. The specific service process is as follows:

(1) Customer traffic identification, security detection and protection: The platform identifies the customer traffic based on the IP addresses. The TAP splitter identifies the traffic of group customer services that requires security protection/analysis, and copies and captures them to the full-traffic threat probes and DDoS detection devices respectively. The core router redirects the traffic of group customers according to the policy-based routing based on the IP address, realizing security protection.

(2) DDoS detection and scrubbing: When the DDoS detection device detects an attack, it sends the IP traction message of the host being attached to the DDoS cleaning device. The scrubbing device issues a BGP policy-based routing to the core router to redirect the abnormal traffic to the scrubbing device for cleaning. After scrubbing, the traffic is re-injected back to the core router.

(3) Statistics and operation: The platform collects the traffic used to complete services and the security logs through the full-traffic threat probe and log system, and sends them to the situation awareness platform. Based on the security big data and AI technology, it analyzes statistics, identify risks and predicts trend of the service traffic and security events of the group customers.

5) High reliability and scalability of the egress platform

Figure 7 High reliability and scalability of the egress platform

The high reliability and scalability of the security egress platform solution for provincial networks of China Broadnet are reflected in the high reliability and scalability design of network element devices and network architecture.

The egress gateway adopts the highly reliable deployment mode of the dual-node cluster. A single device adopts the orthogonal CLOS architecture that separates control, data and forwarding. The main processing units, switches, interfaces, services, PSUs and fans are redundant. A single device supports up to 14 slots and can be expanded to the throughput above the TB level and the concurrent connection above the billion level in cluster mode.

The platform architecture separates message application and analysis from the scheduling, forwarding and processing of the traffic, which are implemented by different sub-systems. For example, ACG identifies the type of traffic application and LLB processes and forwards the traffic based on the application type. The processing performance of the subsystems can be adjusted as required to match the scale of the current network services.

Bypass identification and detection should be conducted on the group customer traffic to ensure the security of services. In the solution, the traffic in trunk links is mirrored to the dedicated TAP copy splitter through the optical splitter. A single server can copy the traffic of the splitter at 400 Gb/s. At the same time, the solution reduces the performance consumption of the egress gateway or the core router due to traffic mirroring and copying. Besides, the bypass method supports escape. A TAP device failure will not affect the services of the current network.

Based on the scalable architecture design and high-performance network element devices, this solution can meet the requirements of NAT, intelligent routing and traffic processing for the traffic of N*100 Gb/s in the future, and can also adapt to the value-added services of group customers with the traffic of above 100 Gb/s.

V. Conclusion

The egress security platform solution realizes the refined operation of broadcast resources at radio and television egress links, improves the use efficiency and effectiveness of bandwidth resources, optimizes the user experience of intranet broadband services, and provides extensive and in-depth network security value-added services for group customers, realizing resource efficiency improvement, service optimization and open-source services.