Troubleshooting like a ninja
If troubleshooting was easy, everyone would be successful at it. There’s an art to UC troubleshooting and monitoring, and with the right tools and training, you can effectively combat your UC issues.
Where to start
✓ Check your server health
✓ Check the conference call history
✓ Check inflight conference calls
✓ Check network capacity
✓ Check endpoints
Getting more advanced
Servers running hot
Check the infrastructure metrics such as CPU, memory, services (if applicable) for your monitored devices. Overutilization of UC devices can cause performance issues in your application. Setting up proactive alerting to let you know via email or SNMP trap when infrastructure metrics are about to hit critical is key for you to proactively monitor your environment.
Voice quality
All major vendors allow you to report on voice quality metrics from Avaya, Cisco and Microsoft to SBC vendors like Oracle, AudioCodes and Sonus. Being able to historically search and review troublesome calls is critical for finding out the root cause of voice quality issues. Searching by user or extension in a performance management tool like Prognosis allows you to view calls a user or extension made and diagnose by looking at degradation factions like packet loss, jitter, latency. If you have an SBC in the environment, you can also view the call path from end to end with VQ360.
Download a PDF copy of the UC troubleshooting guide for Pros
Use a network troubleshooting feature set like IR Path Insight:
- If you’re using Avaya or Skype for Business, you can view the network hops that a call has taken. The Network Hop visual diagram shows the latency between each hop. You can click on these router/switches in the diagrams to get plain english root cause analysis of that network interface. For Cisco, even though network hop data is not reported by the vendor, you can still go into the network troubleshooting link in Prognosis Web UI, and review problematic interfaces.
- If you’re having voice quality issues, it is always a good idea to check if QoS (DSCP) is enabled on your network. This indicates whether the voice is being prioritized correctly over other traffic. Path Insight comes built in with a synthetic call simulator that allows you to run a test to indicate exactly which router or switch is stripping or not configured for QoS.
Top tips
Cross off the basics
Tech support always starts by asking “Have you unplugged it and plugged it back in?” UC troubleshooting is no different, so start with the basics like verifying that all end points are connected correctly, whether the network is down, and the state of your server health.
Trust your instincts
If you have a hunch on where a problem started, investigate accordingly until you can rule it out or find the root cause.
Stay curious!
Have you ever been reviewing analytics or pulling reports and something completely off topic catches your attention from the corner of your eye? Our advice: stay curious. See what is causing you to pause and squint at the screen; you might just solve a problem or prevent one from occurring.
Traits of a black belted troubleshooter
- Tenacious
- Outside the box thinker
- Curious
- Competitive
- Persistent
- Attention to detail
- Patient
- Stars Wars Fan
01 | Measure the impact and optimize
Determining the right UC metrics to measure
When managing a complex UC environment, a fundamental requirement is an overview of the current state of the complete ecosystem and an effective alerting system that identifies when issues are occurring, or ideally before they occur. This requires a dashboard that provides an end-to-end visual representation and drill-downs to aid fast troubleshooting.
Measuring QoS: MOS or the R-Factor?
The Quality of Service (QoS) of voice and video is usually the most reliable indicator of the health of the entire UC environment and generally appears as a metric on CIO dashboards. It can be measured by Mean Opinion Score (MOS) – a subjective measure using a 1 to 5 scale, where 5 is excellent, or measured by the R-Factor, which uses a scale of 1 to 100, with 100 being excellent. MOS takes into account the CODEC being used as well as network measures such as packet loss, latency and jitter.
From MTTR to MTTI
Mean Time to Repair (MTTR), the speed at which problems can be resolved, is another common metric. However, a commonly held view is that 80% of MTTR is Mean Time to Identify (MTTI). Once the issue has been identified, a resolution is often quick to follow.
Averages Ignore Outliers
Subjective measures like these are a generalization and only encapsulate certain elements of a call. So, despite a high MOS, users may still be experiencing issues such as insufficient sound levels. Scores are also calculated on the mean data, so can ignore outlying issues: they can miss an important issue such as a high percentage of low scores coming from a concentrated group of affected users.
Real-time Monitoring and Alerts
UC performance management tools should monitor QoS in real time and alert IT Operations to issues as they are occurring.
Testing Inside and Out
Proactive testing of UC environments using agents that generate synthetic calls internally, combined with an external testing service automatically placing real calls into the organization, will identify whether there are issues with incoming and outbound calls.
Tying UC metrics back to the business objectives
The demands of the business on IT Operations has significantly increased in recent years. It’s easy to forget that at a minimum, all end users expect voice calls to just work. This was much easier to achieve using dedicated phone lines, but when calls are routed over the same network as all other data, issues are bound to occur. Add to this the network bandwidth demands of video traffic and application sharing and you have significant complexities. Those who take the time to plan and have the foresight to include ongoing monitoring and troubleshooting as a prerequisite to a UC deployment are most likely to have satisfied users within the business.
IT Operations leaders need to work closely with the business to negotiate a service-level agreement that balances the criticality of the business with the cost to deliver the service. For example, it might be acceptable to experience the occasional downtime in some areas of the business but not in others, such as the customer contact center.
When determining what matters most to the business, IT Operations leaders should look to the strategic objectives set by the CEO and how these relate to the individual business units that IT must serve.
Aligning IT metrics to business metrics should not be a ‘set and forget’ process, as requirements invariably change over time.
Turn UC insight into golden nuggets
Too often, insights get overlooked. We look to reports and analytics for answers, forgetting insights are garnered when we dig deeper into the analytics. But what sets an insight and golden nugget apart? Let’s say you get 10 actionable insights from a month of troubleshooting your unified communications. It’s likely only one of them holds the key – or trigger – to a big opportunity or threat. This is a golden nugget.
Uncover every implication the golden nugget could have. Be prepared to offer an action plan on how best to deal with it – or better still – how you dealt with it for the good of the company. Clearly illustrate the impact of your actions. For example, showcase the productivity gained in a monetary value, or the financial savings made, or the potential future earnings – these will all pique the interest of the C-suite.
What’s the difference between analytics and insights?
Analytics are a collection of data whereas insights are the pieces that we extract from the data that are actionable. You can do something about insights and utilize them to help your business.
What makes the C-suite pay attention to unified communications?
When communication channels are down, it can be devastating, whether direct or indirect. The C-suite wants to understand what may take the business or UC down, which is often revealed in the nuggets.
What does a golden nugget look like in real life?
Days before Black Friday, a large U.S. financial company’s system crashed. Prognosis quickly identified the golden nugget that delivered the root cause of the issue and why it crashed. With Prognosis they implemented an early warning system of threshold alerts, enabling them to prevent crashes in the future. They could then sail through the peak sale season without any problems.
02 | Become a champion for digital transformation
While UC metrics and insights can help you transform your UC environment, the move to digital is transforming how companies operate and compete. Digital transformation involves the innovative application of a technology such as cloud, mobility, collaboration, analytics, artificial intelligence (AI) or virtual reality (VR) to improve or create a new process, product or experience. Once this process or product is created or improved, it will ultimately drive value.
The connection between digital transformation and UC
Often, when considering digital transformation, people jump to thinking about technologies like mobility, AI, VR or Internet of Things (IOT). While these technologies play a part in digital transformation initiatives, Nemertes Research found that most companies with successful digital transformation initiatives have focused on foundational technologies first such as unified communications and collaboration, network and cloud infrastructure. Why? Because companies can’t effectively build and execute on digital transformation initiatives if they can’t collaborate! This is why there is such a close connection between high-performing unified communications and successful digital transformation.
Digital transformation gaining momentum
While some companies are trying to digitally transform to leapfrog or stay ahead of the competition, others are transforming to survive. Technology is driving disruption and change in every industry and most companies need to keep evolving to stay in the game. Other common reasons to digitally transform include:
- Increase efficiencies
- Generate more profit
- Meet customer and staff expectations
- Stay competitive
Wearing the digital transformation cape
Digital transformation doesn’t just happen. It requires a concerted effort and people need time and space to work through it. While it might start as a side project, serious transformation can quickly snowball.
Digital transformation is often driven by a member of the C-suite, typically a CTO, COO or CEO. Alternatively, a champion can emerge from anywhere within an organization; someone who sees an opportunity and is relentless in transforming what’s within their power
While initiatives led from the top may have a greater scope, the success of transformation isn’t measured by the status of leader; anyone can assume the position. Transforming one area of the business successfully can open doors to do the same elsewhere.
A study by Nemertes Research discovered that UCC correlates with digital transformation success. In fact, companies with successful digital transformation initiatives invest in UCC 72% more than unsuccessful companies.
Unified communications teams are perfectly positioned to become digital transformation champions. Through optimization of UC, not only are they ensuring that UC is reliable; they’re improving efficiencies, helping employees be more productive, speeding up the implementation of new technologies, and fronting strategic changes that reduce operating costs. They also have deep knowledge of emerging collaboration technologies and how their application can deliver measurable business improvement.
Ignoring digital transformation is no longer an option
Organizations not looking to new technologies to improve their efficiencies and profitability are seemingly stuck in a time warp and will undoubtedly get left behind. They also have deep knowledge of emerging collaboration technologies and how their application can deliver measurable business improvement.
Ask the Expert
Irwin Lazar
Irwin Lazar is VP and Service Director of Nemertes Research.
Q: Is digital transformation here to stay or is it simply a passing trend?
A: So long as technology continues to advance, then digital transformation is here to stay.
Digital transformation projects have no defined end-date, they continue to evolve as technology moves forward.
Q: How well do you think companies are undertaking digital transformation initiatives?
A: Our 2017-18 Enterprise Digital Transformation Study showed that 75% of organizations are planning or have already implemented a digital transformation strategy. However, just 23% of those who have an active effort underway have achieved success. Unsuccessful organizations struggle with a lack of participation and executive buy-in, as well as limited budgets, and an inability to determine demonstrable return on investment for digital transformation projects. They also often lack a collaboration environment conducive to speedy decision making and the ability to develop ideas and turn them into actions.
Q: From your perspective, what role does UC play in digital transformation?
A: UC plays two key roles in digital transformation. First, it provides the fundamental capabilities to effectively collaborate for both internal workers as well as with partners, customers and suppliers. If organizations can’t collaborate, they can’t develop and implement successful digital transformation strategies. Secondly, UC is itself a digital transformation technology. UC, especially when integrated into business process workflows and applications, can improve internal operations enabling organizations to reduce costs and create new business process efficiencies.
Q: What’s the risk of ignoring digital transformation?
A: The biggest risk of ignoring digital transformation is being disrupted by competitors that use emerging technologies to improve speed and agility, and to deliver new capabilities and services to the market. An extreme case of ignoring digital transformation is the video rental companies who no longer exist because they failed to see that the Internet would enable streaming of video content.
Q: What advice would you give to organizations trying to digitally transform?
A: We recommend developing a brainstorming and ideation process that looks at emerging technologies to develop options for leveraging them to improve existing business processes or to deliver new customer-facing services. This requires executive, IT, and line of business buy-in, and the development of a business case that shows a tangible return on investment at least six months after project launch. But before embarking on a digital transformation strategy, IT leaders should ensure that they have adequate tools in place for collaboration (internally and externally).
Q: Can anyone wear the digital transformation cape?
A: While anyone can be a digital transformation leader and evangelist, our research shows that the most successful organizations get the majority of their ideas out of IT, and that IT-business coordination, vision, and relationship building are all key success factors.
03 | Energizing your UC team and elevating productivity
Whether your team is focused on a UC cloud migration or digital transformation initiatives, they need to work like a well-oiled machine.
Let your big brains do big brain stuff
You hire advanced IT and network specialists for a reason – and it’s unlikely for Apple earbud troubleshooting. So why do they end up fielding those kinds of issues?
Your most expensive IT headcount is hired to work on strategic projects; they’re the brains behind innovation. You need to free up your big brains for big brain work. Let them be strategic. Studies show that constant switching and interruption-driven work is not good for productivity. If your staff is manually handling problems that could be solved through technology, you’re wasting brain power.
Free up engineers’ time to work on the stuff you hired them for – not fixing menial issues.
Creating happier teams
UC teams often must deal with painful, annoying, day-to-day frustrations such as network issues, endpoint faults, overloaded servers and more. What’s a good way to reduce the pain? Provide your teams with the right tools to effectively address systemic issues.
Ask the expert
Adam Geffner is the Principal Architect at Asurion and the Winner of the IR Summit Visionary Award in 2017. We asked Adam for his advice on boosting productivity within a UC team.
Q: What’s the best way to cut down on monitoring and troubleshooting hours in UC?
A: Know your tools and have efficient and practical processes around how to use them to triage and troubleshoot issues in the environment. Prognosis is a great tool, but only if the people looking at it know how to use it and drill down efficiently.
We also just recently deployed Prognosis integration with our Service Desk ticketing system, so tickets on major or critical alerts are automatically opened and assigned. Our telephony teams will get notified of Major and Critical alerts as they happen, but not necessarily on the small stuff (noise) like minors and warnings.
We have a resource who reviews those alerts while checking system health from Prognosis daily, and can bubble up any minor alerts worth noting. This reduces the noise/frequency of tickets that the Ops teams process, so they can focus on higher priorities while ensuring nothing gets missed.
Q: How do you make sure your UC dashboards are fit for purpose?
A: We’ve modified the dashboard display to ensure only systems deployed are visible in the navigation pane. We’ve customized the visible default fields that display in many dashboards to show only relevant data. We also have members create and use both custom key displays as well as purpose built mashups.
Q: How could Operations Managers be wasting employee time?
A: Treating or testing for symptoms rather than identifying and treating root cause. I once had a director who would test trunks for possible issues by having much of his staff make lots and lots of test calls, and report any bad calls and see if they matched customer complaints. Highly inefficient. If you don’t have an established process to deal with a particular type of issue, managers need to be open to their team’s ideas on efficient and logical ways to troubleshoot.
Some Ops Managers who have been in telephony for years may be treating an issue with an outdated approach, or one that’s no longer applicable. What worked or got them here today may not be what works or gets them where they need to be going forward. Be open to and rely on the technical expertise of your staff for suggestions; don’t just hammer down to your staff what you think is right, as it may be dated, inefficient or ineffective.
Q: What are the best ways to energize your UC team?
A: Reward and recognition. Tout their accomplishments in staff calls or department meetings. Send shout-out emails to the department that praise them for work on a project or tough issue. Show them that the praise is real, and recognized, and how it’s made a difference by saving money (by preventing lost revenue). They do make a difference and they need to know it.
Task them to come up with ways to improve efficiency and effectiveness on issues. Have a common issue that takes a little effort to resolve each time? Ask a person or small subset team to analyze the common contributing root causes. How can they help prevent these issues going forward? How can they use Prognosis to alert more quickly so they can be more proactive in resolving? Can they write an efficient set of triage steps for others to follow to most effectively RCA&R the issues, and add that to a training or runbook? Give them responsibility to improve the job for themselves and others.
Give them the latitude and approval for additional training, and conferences. A company willing to spend money on external trainings speaks volumes and the employee gets to expand their skillset and share with others, so it’s a win-win for employee and company.
Download a PDF copy of the UC troubleshooting guide for Pros
Unified Communications helps enhance communication within your company to better support collaboration and productivity. To read more about UC and how it can work for your business, read the rest of our guide series here:
