It’s been a bit of a quiet year so far filled with routine maintenance, but as we all know, even when things appear to be routine, there’s always the possibility you’re going to run into something entirely out of the ordinary course of troubleshooting. That’s the position I found myself in recently with a client. The client has Office 365 with Skype for Business Online, and they’ve been chugging away without issue for a little while now.
The Strange Issue with Logging into Skype for Business
This client recently hired a couple of new employees and part of the onboarding was shuffling older computers to the new employees and issuing new computers to the employees whose computers became hand-me-downs. That’s when they noticed a problem. The new employees would log into their computers, build a local profile, and everything would work … everything, that is, except for Skype for Business. To make matters worse, this issue popped up on the new computers as well. The user would log in, build a new local profile, and be unable to log into Skype for Business. After some asking around, we found out that it wasn’t happening consistently for all the new employees, or even for all the employees with new machines.
Digging In
Let’s see if I can successfully walk you through the corkscrew that is the symptoms of this issue. Bob had a computer. He could log in and use Skype for Business and he was happy. Steve was hired and management said, “Hey, Bob, you’ve been here a while, we’re going to give you a new computer and give yours to Steve!” Bob is happy to get a new computer and from the start, things look good — he can log in and do his job, but then the Skype for Business client pops up an error and says that the server is unavailable. In the meantime, Steve gets Bob’s old computer, logs in, everything looks good … and then he sees the same problem in Skype for Business that Bob just saw: “Server is unavailable.”
Now to add to the wonderful tapestry of confusion: Bob can log into his old computer using his credentials, use Steve’s credentials to log in to Skype for Business, and it works. Now that’s weird. Same computer, just different local profile. I’ve provided a table below to give a bit more visual representation of the issue to simplify the confusion:
By this time is when I got called in by the client. Time to roll up my Mirazon sleeves and dig in. First things first, I pulled some client logs and looked at what was going on in the background.
Here’s the first thing I saw in the UccApilog:
Well, admittedly, that was a new one for me. I’d never seen an issue with Skype for Business Online provisioning a certificate before.
Let’s take a look at the corresponding message:
So, we’re getting a 401 Unauthorized from the CertProvisioningService in Skype for Business Online. I’ll spare you most of the troubleshooting details prior to contacting Office 365 support, but suffice it to say we tried DNS, OU and GPO, rebuilding local profiles, disabling and re-enabling licensing, uninstall and reinstall of the client and the Office Suite, disabling antivirus, tethering the computer to an outside internet connection … We tried a lot of things. It was time to call in support.
Hi, Office 365 Support
Office 365 is unfortunately a bit of a black hole from a troubleshooting standpoint, so once you’ve exhausted everything from a client side standpoint, you have to engage Microsoft support to move forward. Over the course of the troubleshooting, we tried a bunch of things, retried things I’d already done, edited registry entries, and eventually exhausted what that level of support could do. They recommended I open a ticket with desktop client support. I opened a ticket with them, and after an hour of reviewing the issue, they kicked it back to Office 365 support.
A quick aside here. After the initial notification that there was an issue, we identified multiple users across multiple clients of ours that were experiencing a similar issue. In working with the original Office 365 support, we resolved all but two of them, but unfortunately, no single fix worked for any of the other affected users, so we had to start over essentially at square one with each user for this case study in question.
Back with Office 365 support, we escalated the issue to Tier 2 and we were back to the troubleshooting dance, and after hours of frustration, I’m pleased to announce the issue has been resolved, and I’m here to tell you how we did it so that hopefully, should you find yourself with the same issue, you can avoid the heartache, late nights, and frustration that I went through.
The Fix
All of the things that comprise the fix are things that we had previously already tried. Somehow, individually, they didn’t fix it, but by their powers combined … well, they didn’t form Captain Planet, but they did fix the issue on more than just one user’s computer.
- Disable Antivirus.
For our clients, it’s Trend Micro. Shut down the agent, screenshot the service startup types so you know what to set them back to, and then disable all the services associate with the antivirus.
- Delete SFB/Lync profile folders.
Browse to c:\users\<username>\appdata\local\microsoft\office\16.0\lync and delete all the sip_ folders and as much of the tracing folder as you can. Make sure you have exited completely out of the Skype for Business client, otherwise you won’t be able to delete some of the files that you need to delete.
- Delete Credentials from Credential Manager.
Open up Control Panel, and open Credential Manager:
Once that’s open, choose Windows Credentials, go to Generic Credentials, and remove EVERYTHING from under Generic Credentials:
Delete Registry Keys.
Open up Regedit, backup your registry (just in case — always a best practice whenever deleting registry keys), browse to hklm\software\microsoft\ and delete the IdentityCRL and, if it exists, MSOIdentityCRL keys. If you get an error that prevents you from deleting one of these keys, run regedit as administrator. The next place you need to go is HKCU\software\microsoft and again delete the IdentityCRL and MSOIdentityCRL keys.
From an admin command prompt, IPConfig /flushdns.
- Remove Certificates from Certificate Store.
Run certmgr.msc and delete any certificates under Personal that are issued by “Communications Server”. Also, if there are any certificates under LyncCertStore, remove those as well.
Now you’ve done all the prep work. Open up that client and magically, you’re able to log in. Why this combination works like chocolate and peanut butter, I don’t know. As I said before, we had tried each of these steps before individually, testing after each one, in some cases, re-installing afterwards. There’s just something magical about these items together that makes it click.
If you run into this issue, let me know if these steps work. I’ve been able to replicate the results on two users. Unfortunately, or fortunately, depending on how you look at it, I’ve run out of broken users to test this on, and given the wildly erratic results I’ve had with the other solutions, I’m highly interested to know if this consistently fixes this problem with others. Due to my limited pool of test subjects, I have not been able to test out various combinations of these steps, so again, if you run into this and have the time to mix and match, please let me know what works and what doesn’t.
Drop us an email if you worked through this or need help! You can also call us at 502-240-0404 and get a real, life human on the phone to get started.