Lync 2013 Response Groups Not Responding

By December 11, 2014Lync, Microsoft

microsoft auditRecently I wrote about an internal Lync Server CMS replication issue we were experiencing. While we were fixing that issue, we also had a report come in that our Response Groups were not working properly.  Hooray! When it rains, it pours I suppose.

We use Lync RGS for simple Hunt Groups – like Tech Support, and Purchasing, and Sales, etc. – all inbound phone calls (other than private DIDs) come into Lync, then to an Exchange Auto Attendant – and then to Response Groups.  They are pretty important to us.

We are a managed services provider (MSP), so having our Tech Support Response Group Queue down is not good — really not good.  Let’s see what’s going on, shall we? Warning: there are a lot of screen shots below.  I apologize in advance for all the scrolling you’re about to do.

Symptom 1: When you called a response group directly, you get a fast busy.

Symptom 2: When you called our main number and Exchange AA picks up and you transferred to a Response Group, you got a “call cannot be completed” and got sent back to the AA.

Symptom 3: When you called a Response Group while tracing S4 in OCSLogger/Snooper – you get a nice generic 26017 entry

  • ms-diagnostics-public: 26017;reason=”Internal server error.”

Symptom 4: When you restarted the Response Group Service (RTCRGS), it consistently pushed errors related to WorkflowRuntime and Contact Objects.

Event 31028: LS Response Group Service – WorkflowRuntime / CommitWorkBatch

Lync 2013 Response Groups Not Responding

Event 31067: LS Response Group Service – Match Making / Contact Object

Lync 2013 Response Groups Not Responding

And another curious “Information” log entry in the middle of those…

Event 31035: LS Response Group Service – Application Endpoint

Lync 2013 Response Groups Not Responding

That doesn’t make sense.  At all.

I started by focusing on the 31067 error and begun looking at the RGS Application Endpoints in the Lync Shell.

Lync 2013 Response Groups Not Responding

Then I verified those Application Contact Objects existed in ADSI Edit – and they did.

Lync 2013 Response Groups Not Responding

And…

Lync 2013 Response Groups Not Responding

So that’s annoying.

Let’s go back to the 31035 Event. I traced all the RGS items in OCSLogger/Snooper while I restarted the server and found this curious log:

TL_ERROR(TF_COMPONENT) [2]1C38.2C98::11/16/2014-02:45:12.980.00000457 (RgsHostingFramework,AepManager.StartConnectingAep:aepmanager.cs
(1144))(0000000002D826E8)[Exit] – Could not establish AEP AEP address=[sip:RtcApplication-77d9d0db-f566-4ed7-85b7-c03996c65e80@mirazon.com], exception=[System.InvalidOperationException: An application endpoint with the same uri already exists on the CollaborationPlatform.

Okay. That’s the RGS Presence Watcher. Interesting.

Another interesting log entry just above it:

TL_ERROR(TF_COMPONENT) [1]1C38.216C::11/16/2014-02:44:11.371.00000238 (RgsHostingFramework,AepManager.StartConnectingAep:aepmanager.cs
(1144))(0000000002D826E8)[Exit] – Could not establish AEP AEP address=[sip:RtcApplication-77d9d0db-f566-4ed7-85b7-c03996c65e80@mirazon.com], exception=[System.InvalidOperationException: The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly.

Huh…Performance Counters.  I reached out to some friends and they asked me to look at the Registry Entry for the Windows Workflow Foundation entries.

Specifically, I was asked about HKLM – System -> CurrentControlSet -> Services -> WWF 3 and WWF 4 like below:

Lync 2013 Response Groups Not Responding

You’ll notice that under PerIniFile it references PerfCounters.ini. That’s what it says now. Before, it said “PerfCounters_d.ini” (with the underscore d) which is odd.

At my friends’ recommendation, I reloaded the performance counters in both WWF 3.0 and 4.0 folders.

Lync 2013 Response Groups Not Responding

And the following log entries appear:

Lync 2013 Response Groups Not Responding

And:

Lync 2013 Response Groups Not Responding

Okay.  I waited and took a deep breath.  And I restarted Response Groups – RTCRGS again.

Lync 2013 Response Groups Not Responding

Hooray! You could see RGS Stop and Start, and the Application Endpoints were created without error.

Lync 2013 Response Groups Not Responding

All is well.

So, what was the problem? I guess the Response Group problem was related to Performance Counters – specifically the Windows Workflow Foundation performance counters. Reloading those fixed whatever it was that was cause the RGS Errors at the top of this blog post.

Are you experiencing trouble with Lync? Email us or give us a call at 502-240-0404 so we can help!