Proactive Network Maintenance (PNM) has taken the cable industry by storm over the past five years, and while most operators agree they would like to add these groundbreaking capabilities to their toolbox, the best path to achieving a successful implementation is less clear.
In this article, we will examine this challenge and give real-life examples of the journey taken by a few companies in pursuit of this goal.
Most cable operators seeking PNM solutions have two primary business objectives. They are:
Customer quality of experience/churn reduction:
It is more difficult than ever to keep customers; poor quality of experience is the number one reason for leaving
Find HFC problems before customers are impacted
Fix problems faster once customers are affected
Don't unnecessarily visit a customerís home or business (even just once)
Increased return on opex investment: Operators face too much waste in current plant maintenance and repair practices
Too many unproductive truck rolls
Too much time spent finding versus fixing issues
Targeted vs. scheduled preventative maintenance
Challenges to achieving desired results
On the surface, the concept of using pre-equalization coefficients to detect, correlate and localize plant weaknesses otherwise invisible to traditional techniques is solid, but as with most things in the HFC world the devil is in the details. CableLabs' mathematical equations to convert pre-eq coefficients into frequency responses per-CPE and correlation concepts are a solid foundation, but getting from these building blocks to a usable and sustainable tool has been the stumbling block. We've observed the following challenges:
Processing, interpreting and displaying results so an average tech can understand and act on them
Scaling from a manual, single-use proof of concept tool to one available to techs enterprise-wide
Acquiring instantaneous feedback from the toolset to verify effectiveness of fix before closing ticket
Training users, plus launching, supporting, evolving and maintaining the tool over the long run
Operator success stories:
Truck rolls are a fact of life in the cable industry. No matter what improvements are made in HFC maintenance tools, loose connectors won't tighten themselves and cracked hardline wonít self-heal. Many customers pursuing a PNM tool or system primarily want to ensure that rolling a truck has a productive result: In other words, they want to make sure a customer-impacting problem is resolved or an issue which certainly would have soon caused a customer impact is fixed. No more driving around all day to find the real source of a problem -- or worse yet not find it.
Benefits of PNM implementations include: Improved home vs. plant dispatch accuracy
Don't dispatch multiple service techs to customer homes when the root cause of their issues is a single impairment in the hard plant
Less non-productive tweaking of the plant
Identify, locate and address the true root cause of impairment instead of driving around and adjusting amps to mask symptoms of impairment.
Proactively fix tomorrow's problems
Service techs ensure home wiring is clean before leaving after a service call and maintenance techs address hidden plant weaknesses while servicing other HFC issues in outside plant.
One Latin American cable MSO needed to address unproductive truck rolls. Much of their field force was subcontracted, and it was billed per truck roll regardless of whether an issue was addressed or not. The following is an example of an issue that would commonly lead to multiple truck rolls before finding and fixing the root cause.
Using the PNM, techs could clearly see where to send a truck; once there, they quickly found the problem. After replacing the broken pin, they immediately refreshed their display and verified the effectiveness of the fix before bucketing down and closing the ticket.
What are the potential savings if 10 percent of issues are like the one above for maintenance techs? Instead of rolling trucks three or four times to fix "the tough 10-percenters," they can kill them on the first shot with a PNM tool.
Result: The efficiency gains achieved by maintenance/service techs alone result in <1-year payback on PNM tool investment.
Clusters of customers canceling services due to poor broadband that stems from a shared but hard-to-find plant impairment can often lead operators to pursue PNM systems. Most operators have a small percentage of tough issues -- yet even just a few issues would require weeks and many truck rolls to finally find and fix the true root cause.
Universally the troubleshooting process for these tough 10 percent problems tends to make a bad situation even worse for customers. By constantly tweaking amps to mask symptoms, operators create more problems when the plant heats up or cools down. Yet swapping tap face plates and amp modules causes sporadic service outages -- and still doesn't address the real problem.
Customers on these nodes finally lose patience and switch providers in large numbers.
A North American operator suffered an issue which contributed to intermittent performance. Pre-equalization can generally compensate for most impedance mismatches, but this effectively hides them until temperature changes or other issues emerge, worsening the condition until the digital cliff is overrun, and service is impacted.
The node below was frequently on the MSO's "Top 5 Worst Nodes List" but the source of the problem was not obvious using traditional tools and methods.
Using its PNM tool, the operator quickly identified the suspect span and the source of the issue, calculated the distance-to-fault and independently verified the issue. Addressing this may not have
an immediate impact on common metrics such as MER or code word errors, but the extra pre-equalization margin provided makes the plant more robust against environmental changes and the customer impact (and churn!) that results.
Customer churn is typically in the 3 percent to 7 percent range for cable operators globally. Roughly half of this is avoidable (quality/performance driven) and the rest purely cost/promotion-driven. PNM tools/systems are ideally suited to help quickly find and fix the true root causes of issues that lead to avoidable churn, often before impacting the customer.
2 percent avoidable churn annually
10 percent-- conservative estimate of reduction in avoidable churn with PNM tool/system
Using that simple formula, it's apparent that PNM pays for itself within one year based solely on avoidable churn savings. Drilling down: $1,000 x 2 percent Avoidable Churn x 10 percent Reduction = $2/Customer Annually Assuming 300,000 Subs from Above Result = $600,000 Annual Savings Result: Avoidable churn savings alone result in <1-year payback.
In addition, service providers typically see a significant reduction in unnecessary CPE replacements post-implementation. Customer satisfaction improves too. After all, customers are never happy about technician visits, which disrupt their day and mean admitting a stranger into their home. If the site visit does not address their problem, dissatisfaction is multiplied (and may be accompanied by social media complaints).
Other financial considerations
While the initial purchase of a PNM system may appear to be the lion's share of the overall investment, there are other costs to consider that will vary widely from system to system. Employee training should be considered; an overly complex tool may have a sizable learning curve to overcome for before an operator achieves payback. If the application is not an extension to an existing toolset, the service provider must then consider the internal costs to administer yet another new system.
Also, be cognizant of how the tool will be supported. Who will you call with questions about usage, when CMTS/CPE firmware changes break the tool, compatibility issues surface with new CPE introductions, or security issues emerge? Think ahead about how the tool will evolve to refine existing capabilities and introduce new ones. If a tool is not supported, and a year later degrades to the point of being shelfware, your entire investment is wasted.
It is also critical to select a vendor-neutral system -- one that will work equally well with data from all network equipment vendors. As networks evolve and as operators deploy new technologies such as DOCSIS 3.1 and remote PHY, likely from multiple vendors, a system from a neutral source ensures consistency in physical and virtual test solutions regardless of underlying network architecture or service provision equipment vendor(s) chosen. Proprietary systems may at first appear easier to buy as part of a network equipment bundle, but operators should consider the long-term limitations of being locked into a single ecosystem.
PNM capabilities have gained acceptance globally as a mainstream addition to MSO toolsets for early adopters, and the process for selecting, launching and maintaining a system is not without risk. Careful selection of a PNM system best matching an MSOís need, with proper implementation and support, can result in superior payback financially and continue to provide both tangible and intangible benefits for years to come.
— Jim Walsh, Marketing Manager, Viavi Solutions. Exclusive to UBB2020. Follow Viavi on Twitter @ViaviSolutions, on Facebook or LinkedIn.