PointBridge Blogs - Jeff Schertz

By: Jeff Schertz Posted: July 2, 2009 at 11:02 AM

Byron Spurlock has a blog article that briefly talks about the different topologies for Edge servers in R2 but I wanted to go into a little more detail and highlight a few seemingly small, but important changes introduced in R2 that sort of flew in under the radar as all the other neat features in R2 took center-stage.

Pre-R2 Topologies

Firstly, anyone versed in OCS should know that when deploying an Edge Server you could select what roles you wanted to install, with a few limitations. The following table shows the different Edge Server configurations supported by OCS 2007 pre-R2 as described in the Edge Server Deployment Guide.

Topology	Description
Consolidated Edge Topology	Access Edge Server, Web Conferencing Edge Server, and A/V Edge Server are collocated on a single computer.
Single-Site Edge Topology	Access Edge Server and Web Conferencing Edge Server are collocated. The A/V Edge Server is on a separate computer.
Scaled Single-Site Edge Topology	Computers with a Web Conferencing and Access Edge Server role collocated on them are load balanced. Two or more A/V Edge Servers are each installed on separate computers and load balanced.
Multiple Site with a Remote Site Edge Topology	Primary Site: Computers with a Web Conferencing and Access Edge Server role collocated on them are load balanced. Two or more A/V Edge Servers are each installed on separate computers and load balanced. Each Remote Site: One or more Web Conferencing Edge Server are installed on a dedicated computer. The A/V Edge Server is installed on a dedicated computer.

The different Edge roles were allowed to be installed on different physical hardware, resulting in many different (and potentially confusing) scenarios. Basically you could have (1) a dedicated server for each role, (2) separate servers with the Access Edge and Web Conferencing together with another server hosting the A/V Conferencing role, or (3) all three Edge roles on the same system. Then you could install multiple arrays of these different scenarios. Lost yet? Yeah, me too.

Microsoft acknowledged that this flexible, albeit overly complex set of configurations was rarely used in practice. Nearly every enterprise level deployment we performed prior to R2 simply used one or more Consolidated Edge Server. The requirement to break up SIP and Media traffic on dedicated systems was never important enough to warrant separating all those roles out as usually physical space, available IP addresses, and cost were the driving factors.

Enter Release 2

The supported Edge topologies for OCS 2007 R2 have been greatly simplified by changing one key requirement: Only Consolidated Edge Servers are supported. In fact, the deployment wizard (whether run from the Standard Edition or Enterprise Edition media) only allows for the installation of all components. There is no longer an option to choose which Edge role(s) to install on the server. Regarding support, the R2 documentation states the following:

Supported Topologies
Each Edge Server always runs all three Edge Server services: Access Edge service to validate external users and enable instant messaging (IM), Web Conferencing Edge service to enable external users to join on-premises meetings, and A/V Edge service to enable the sharing of audio and video with external users, and enable Desktop Sharing with external users.
You can configure one or more Edge Servers on your network, according to your performance needs. Three Edge Server topologies are supported: single consolidated edge topology, scaled consolidated edge topology, and multiple-site consolidated edge topology.

What is important to clarify is that all three different topologies still utilize Consolidated Edge Servers. By having all three Edge roles in a single server you simply add additional hosts to create and expand a pool of servers as bandwidth requirements increase. The previously supported Consolidated and Single-Site scenarios are affectively merged into the same scenario now, leaving just 3 unique topologies.

For example, here’s the diagram of the Scaled Consolidated Edge Topology that shows a pool of consolidated Edge Servers:

So the simple take-away is that with R2 forget what you know about previous Edge role collocation and just think that there is now only one type of Edge Server.

Multiple Access Edge Roles

Ok, so now you may be saying to yourself “I thought multiple Access Edge servers weren’t supported in a multi-site topology” and you’d be correct. If you are deploying multiple consolidated Edge Server in different sites that all support users for the same SIP domain (read: single DNS SRV record for Automatic Configuration) then the primary site’s Access Edge Server roles should be configured to handle external client login. The other server’s Access Edge roles will still be deployed and configured with unique FQDNs, but they will not be used and basically sit idle. The remote site’s Edge server’s will handle Web Conferencing and A/V traffic though, which is considerable more bandwidth intensive the basis SIP traffic that the primary site handles.

A/V Edge with NAT

As Matt and others have previously discussed NAT works when used on the A/V Edge Role in R2, the different support statements throughout the documentation can be read in ways that might seem confusing.

The Security section makes a this statement:

If the edge server is a single consolidated edge server, Office Communications Server 2007 R2 allows the use of NAT for all three edge services.

The first time I read that I assumed (thinking back to what I knew was true with pre-R2 topologies) that it meant the NAT limitation hinged on the word ‘consolidated’ not realizing that the keyword was actually ‘single’.

Meanwhile the Operations portion of the R2 documentation has a clearer definition that better calls attention to the fact that a single server versus multiple servers is where the limitation on NAT stands.

To allow the external IP address to be translated from a public IP (NAT), select the External IP address is translated by NAT check box. This option is valid for a single A/V Edge service deployment. Multiple A/V Edge services that are deployed behind a load balancer do not support network address translation (NAT).

So, I hope this helps clear up any confusion people might have on the change in Edge topologies.

OCS Dial-In Conferencing: Unable to Join Conference Calls

By: Jeff Schertz Posted: June 25, 2009 at 1:51 PM

In keeping with this month’s apparent theme of troubleshooting Live Meeting and Audio Conferencing problems for external users, I ran into yet another weird one. This time we have a pretty basic Office Communications Server 2007 R2 deployment with Enterprise Voice using a NET VX1200 media gateway with Cisco Call Manager 4.1. All OCS features are deployed and working with best practices followed for nearly every piece of the puzzle; no cutting any corners. The latest round of OCS patches have been applied and everything is looking good.

That was until I started testing Dial-In Conferencing. I found that OCS users were able successfully join audio conferences using either a dynamic Conference ID from a scheduled meeting or their own personal Conference IDs, but only when they joined from the link within an email/meeting.

But if an OCS user attempted to manually dial the Conferencing Attendant number, or if a PBX or PSTN phone called directly into the same number, they were unable to join the meeting. The Conferencing Attendant would accept the given ID as valid and then attempt to transfer the call into the meeting, at which point would fail as the attendant responded with:

"Sorry, but i can't seem to connect you to your conference right now. Please try your call again later. Goodbye."

The caller would then be immediately disconnected. I did find out that after restarting the server the very first attempt to join any conference would actually work, but then all subsequent attempts would fail regardless of where the call was dialed from. Whether other OCS users were already connected to the meeting (using the link) or no one was connected, inbound callers would always fail. And it didn't matter if callers attempted to join anonymously (via meeting passcode) or authenticated (using their personal PIN).

Something was definitely wrong; I triple-checked the OCS configuration and event logs but nothing was out-of-place. So I walked through each of these three scenarios with debug logging enabled:

Test Scenarios
- Joined OCS User A to scheduled Audio conference using link in meeting invite = Success
- Joined OCS User B to conference by manually dialing '6789' in the Search bar in OC = Failure
- Joined PSTN Caller using external DID '+13125556789' = Failure

Recorded trace data on FE server for following components:
- AcpMcu
- AvMcu
- CAAServer
- CASServer
- S4
- SIPStack

After going through the AvMcu log with Microsoft PSS the following error was located at the time when callers were unable to join the meeting and were disconnected:

TL_ERROR(TF_COMPONENT) [3]117C.1554::06/11/2009-16:00:38.162.0003a54f (AvMcu,UserMediaManager.CreateEndpointAndStreamsCallback:usermedia.cs(821))( 00000000027630E1 )[UserMediaManager]{sip:a4eecd6e49dd404488af679e8e8a1a29@anonymous.invalid} MP CreateEndpointAndStreams exception. System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.Rtc.Internal.Sip.TLSListener.SignString(String signString, String& hashAlg, String& signAlg)

The highlighted section in the line above caught our attention as I had already seen a previous issue earlier that week at the same client which ended up being related to their internal certificate. They had deployed an internal Windows 2008 Enterprise CA but had elevated the signing algorithm to SHA2 256, above the default SHA1 value. But the affected server in that case was the only Windows Server 2003 system in the domain, all others (including the OCS servers) were running on Server 2008 which natively supports that higher level. But because the failure seemed to be internal as the conference service couldn’t handling moving calls between it;s own services (from the lobby to a meeting) I had a hunch it might still be related to the hash level.

Here we can see in the details of the certificate that the Signature Algorithm is indeed sha256RSA:

A quick verification of the General Properties on the Windows 2008 Enterprise Certification Authority show that the Hash algorithm used on the root CA is also using sha256:

Since the internal CA here is only configured to sign certificates using SHA2 we went out to RapidSSL to request a thirty-day free trial certificate to temporarily use on the Front-End Server. Here we can see that the certificate is using the more common SHA1 for it’s Signature Algorithm:

After applying the new certificate to OCS and selecting it in IIS I rebooted the server for good measure. Initially the problem appeared to be resolved as I was able to dial directly into an audio conference from an internal PBX phone, followed by a PSTN phone. The second caller was actually added to the conference successfully (yeay!) but in doing so the first caller was immediately booted out of the room (boo!). And when I dialed in from a third phone, you guessed it, the second caller was promptly disconnected.

I looked at the OCS Event Log on the Front-End server showed a whole bunch of new error messages that had not been there before, describing lots of MCU errors, which would explain the failure to join additional parties. Details on the error messages can be found in this TechNet blog.

Turns out that the new certificate was (partially) to blame as it’s Issuing CA’s certificate is not configured by default in Windows Server in a way that is supported by OCS. By checking the new certificate's path we see that the Equifax Secure Certificate Authority is the issuer used by the RapidSSL free certificate:

By locating that certificate in the Third-Party Root Certification Authorities\Certificates folder in the Local Computer store and viewing the properties we can see that by default the certificate is only enable for a specific sub-set of purposes. To resolve the MCU errors in OCS it needed to be enabled for all purposes.

Once that change was made and the services are restarted, the MCU event log errors stopped appearing and all parties were all to join Dial-In Audio Conferences regardless of where and how they connected to the service. This proved that the previous certificate was the culprit and that the higher level of encryption on the signature was causing the validation problems.

Microsoft is currently looking into this and have successfully reproduced the issue in a lab. Once their debugging is completed I’ll update this blog posting with details on whether a hotfix or KB article is released in response.

A/V Edge Authentication Connection Errors

By: Jeff Schertz Posted: June 22, 2009 at 8:11 AM

Just a quick note regarding an error I recently ran across. A client was experiencing problems with Dial-In Conferencing after a recent deployment and during troubleshooting the issues I ran across this pair of errors in the Front-End server’s OCS event log:

OCS Audio-Video Conferencing Server
Event ID 32018
“The Audio-Video Conferencing Server encountered an error when requesting credentials from the A/V Edge Authentication Service.”

OCS Protocol Stack
Event ID 14502
”A significant number of connection failures have occurred with remote server…”
8007274D
80072746

The IP address of the the ‘remote server’ described in ID 14502 was actually the IP address on the Edge server’s internal interface. So after checking the the common reasons for server-to-server communications issues (filtered ports on firewalls, incorrect or missing DNS entries, or invalid certificates) everything appeared to check out. All other features were working, as internal to external audio and video communications were tested multiple times in addition to Live Meeting and Desktop Sharing.

The A/V Authentication service was correctly configured on the Edge Server’s Interfaces tab on the default port of 5062, and from the Front-End server I was able to telnet directly to that port. Hmmmm…

Next step was to check the internal configuration and make sure that the Front-End services were attempting to go to the right place.

Bingo! The A/V Edge Servers entry under the Global Properties > Edge Servers tab had the correct FQDN but was mistakenly configured to use 443 instead of 5062, where the A/V Auth service was actually listening on the Edge’s internal IP.

Simple enough, just change it, right? That would be too easy. You can’t edit the current entry. And attempts to directly resolve it will be thwarted by one of two errors messages depending on if you attempt to delete the current entry and replace it with a new one, or simply try to add a second entry for the same FQDN with a different port value.

“The A/V Edge Server internal FQDN and A/V authentication port is currently assigned to a pool or server. Please check the Status Pane. Removing this entry may result in the failure of users to exchange media.”

“Trusted entry breaks the FQDN, Port or Version uniqueness constraint.”

Resolution

At this point we’ll need to un-assign the current value from a couple places in the OCS configuration so that the invalid entry can be removed.

First go to the Pool Properties under the Pool object and change the A/V Authentication Service to (None).

And then from the Mediation Server Properties also change the A/V Edge Server setting to (None).

Give Active Directory some time to replicate these configuration changes around, and we can go back and remove the original entry and replace it with the correct port number.

Revisit the Global Properties on the OCS Forest object and delete the existing A/V Edge Server entry with the invalid port listed.
Create a new entry with the correct values. (Depending on the time elapsed between now and the previous steps you may receive another “Trusted entry breaks FQDN…constraint” warning but it can be ignored at this point.)

After restarting the Front-End services those two errors should stop appearing in the event log and Front-End to A/V Authentication communications should now be working.

External Live Meeting Only with OCS

By: Jeff Schertz Posted: June 3, 2009 at 3:44 PM

5
Comments

One of the biggest cost-saving benefits of Office Communications Server 2007 has been the integrated Web Conferencing features, a.k.a. Live Meeting. Many companies currently pay for off-site host conferencing services like Microsoft's own hosted Live Meeting, or WebEx just to name a few. In the same way that R2’s Dial-In Conferencing can reduce costs by eliminating the need for a hosted phone bridge service, all versions of OCS can offer the same cost-cutting advantages by bringing Live Meeting services in house.

One example are organizations who already have a limited LCS deployment in-house but due to various internal political issues are not able to deploy IM services to the company as a whole, either inside or out. With the addition of a simple, yet robust OCS deployment it’s possible to configure OCS to support only Web Conferencing for external users.

The approach used for a specific OCS 2007 R2 deployment included:

A single Enterprise Edition Front-End Consolidated Server

No Hardware Load Balancer is required
No Director

A Standard Edition Consolidated Edge Server
- (2) Network cards for separate internal and external roles
- Separate, unique IP subnetworks for the internal and external interfaces
- A single firewall with separate ports and rule-bases for each Edge interfaces
- No Reverse HTTPS Proxy deployed (will be added at a later date to support shared content download)

Internal Deployment

The internal deployment follows the standard installation and configuration steps; nothing out of the ordinary is required here. The basic SRV and A DNS records are setup to support Automatic Configuration and a pool record (using the IP address of the Front-End server itself). Certificate deployment is standard. If migrating users from LCS then make sure to install a certificate on the LCS server (if not already deployed) so that MTLS server-to-server communications will work correctly between LCS and OCS services.

Once everything is installed then you’ll need to configure the Web Conferencing policies to allow users rights to start conferences and (if desired) allow anonymous users to join.

The one major caveat during server configuration is to make sure and do not leave the External Web Farm FQDN blank during the Enterprise Pool creation (or Standard Edition server configuration). Even though there will not be a reverse proxy configured and thus no connectivity into the internal Web Components IIS site, and thus no real need for a defined external URL, it wont work. Once everything is deployed external clients will simply not be able to login to Live Meeting as the lack of a defined external Web Farm FQDN causes OCS to ‘give up’ and basically not allow external Web Conferencing connections. External Communicator sign-in and IM/Presence will work, just not Live Meeting. In this example the internal FQDN was simply duplicated into the external field. You could probably put any string in there, valid or not. Just as long as it’s NOT blank.

External Deployment

As with anything in OCS, the external user access is where it can get tricky. Because this specific client was licensed for only Standard Edition a single consolidated Edge server was deployed, even though the A/V roles will not be utilized. This was done for two reasons, (1) licensing, and (2) future expansion. If they want to later open up external Communicator access and support features like Desktop Sharing, even though A/V sessions would still not be officially supported, the A/V functionality would need to be in place to get Desktop Sharing to work between internal and external clients. This point is worth reiterating. Desktop Sharing works via media communications.

The R2 Technical Reference details the Desktop Sharing Architecture explains that because the RDP sessions inherently run into the same challenges as media communications Microsoft decided to use A/V functionality in OCS, which already solved the previous issue. So basically Desktop Sharing is an RDP (Remote Desktop Protocol) session over SRTP (Secure Real-Time Protocol). The main difference is that Desktop Sharing uses TCP while standard media typically uses UDP, but the ports are the same.

But now back to our scenario; we only want Live Meeting supported externally, no audio, video, Communicator sign-in, and thus no Desktop Sharing. The reason I point this out is that when testing a deployment like this it would be a mistake to use Desktop Sharing to validate connectivity when it works completely different then Web Conferencing does.

First off, with any Edge deployment the firewall rules are most important. Let’s assume a best practice deployment with the following IP subnetworks:

And here are the IP addresses and FQDNs to be used throughout this article:

Location	Interface	Role	IP Address	FQDN
Perimeter	External	Access Edge	103.207.10.10	sip.domain.com
Perimeter	External	Web Conferencing	103.207.10.11	webconf.domain.com
Perimeter	External	A/V Conferencing	103.207.10.12	av.domain.com
Perimeter	Internal	Edge Private Interface	10.1.1.100	edgeserver.dmz.local
Internal		OCS Front-End Server	172.16.1.111	ocspool.domain.local

I’ve chopped down the Microsoft diagram to include only the communications we need to support external Live Meeting clients. The two inbound rules (4,6) from the Internet are pointing to the separate IP addresses for the Access Edge and Web Conferencing Edge. The internal communications (5,7) are all happening between the internal Front-End server and the perimeter Edge server.

Firewall Policy Rules:

Rule	Firewall	Direction	Source Port	Source IP	Destination Port	Destination IP	Description
4	External	Inbound	Any	Any	443	103.207.10.10	Live Meeting sign-in and SIP signaling
6	External	Inbound	Any	Any	443	103.207.10.11	Live Meeting content sharing
5	Internal	Outbound	Any	172.16.1.111	5061	10.1.1.100	Communications from Front-End Server to Edge
5	Internal	Inbound	Any	10.1.1.100	5061	172.16.1.111	Communications from Edge to Front-End Server
7	Internal	Outbound	Any	172.16.1.111	8057	10.1.1.100	Live Meeting content sharing

Since the A/V Edge role is deployed as part of a consolidated Standard Edition server, it must be configured during setup or the wizard will not advance. Clearly there will be no external access to the A/V Edge and Authentication roles, so an internally-issued certificate will be fine here as the Access Edge and Web Conferencing roles would get third-party certificates (Entrust, Digicert, etc). The nice thing about this configuration is if a later pilot rollout of external A/V is requested all that is needed is a certificate and some firewall changes and everything is already set to go.

Testing

Once all the rules are setup and the Edge server is deployed and configured each can be tested via telnet. From the internal Front-End server you should be able to telnet to both ports 5061 and 8057 on the Edge’s internal IP. You can also perform the same test from the Edge server to 5061 on the internal Front-End server. Additionally any public host should be able to telnet to 443 on either the sip or webconf IP addresses to verify the external rules as well.

Connecting directly to the port via telnet will both confirm that the firewall rule is correctly defined and that the service is active and listening.

A failed connection to any of the ports would be pretty clear:

C:\>telnet 10.1.1.100 5061

But a successful connection to either ports 5061 and 443 would not be as obvious as nothing appears to happen. The blinking cursor indicates a connection to the port; there is just no interaction:

C:\>telnet 10.1.1.100 5061

Meanwhile a successful connection to the PSOM service on port 8057 at the Edge internal interface would at least spit out a few characters:

C:\>telnet 10.1.1.100 8057

Internal Configuration

By default web conferencing is not enabled in a fresh installation. There are a couple ways to turn it on, depending on if you want everyone in the organization to have the same features and permissions, or if you need to have separate groups of users with different policies.

The simplest approach is to just edit the Default Policy and enable Web conferencing and any other desired options.

Alternatively you can elect to choose from different set policies for each SIP-enabled user account. The built-in policies offer different levels of permissions but can each be customized, as well as removed or replaced by other new policies.

External Configuration

Now even though external Communicator support is not part of this deployment, user accounts must still be SIP-enabled in order to schedule and initiate Live Meeting sessions and thus invite attendees. If Communicator must still be blocked, then limiting the ability to install the Communicator application on corporate systems is one approach. Another would be configuring global policy settings to limit the types of communications (block IM, peer-to-peer audio, etc) in the client.

But for this deployment all OCS features would be used internally, so only external user login was undesired. In order to first allow access to Live Meeting externally the Edge Server properties must be configured to allow any external access.

Edge Server Properties > Access Methods

The next step is to enable rights on each user account for remote user access. This inherently also grants the user the ability to sign-in to Communicator remotely; these rights cannot be separated. So if only anonymous access to Live Meeting sessions is desired for all remote users, whether they are corporate employees or a third-party, then this step can be skipped. But since most deployments would want corporate users to be able to sign-in as themselves to conferences regardless of their location that permission must be enabled on each account.

User Account Properties > Communications Additional Options:

Changing Passwords on OCS Service Accounts

By: Jeff Schertz Posted: May 29, 2009 at 8:53 AM

3
Comments

Whether troubleshooting a problem or performing regular maintenance it is sometimes necessary to reset the passwords on the service accounts used by OCS.

Best Practice

Microsoft’s recommendation is to configure each service account to never have their passwords expire, but some organizations may have policies limiting or completely banning that type of approach.

In most cases it is best practice to retroactively disable password expiration on each Active Directory User Account used by OCS, as the setup wizard does not automatically configure this when creating the accounts.

A Standard Edition deployment will only have created RTCService and RTCComponentService accounts (assuming the default names), while an Enterprise Edition deployment would also add the RTCGuestAccessUser. If a Communicator Web Access server is deployed, then an additional service may also exist in the domain if

For each of these accounts, configure the “Password never expires” setting on the Account tab using the Active Directory Users & Computers management tool.

The Hard Way

But if the service account passwords must adhere to a corporate policy, then it’s key to monitor the Office Communications Server log on the Front-End server for any Warning events with ID 122296 as these events will alert administrators to the number of remaining days before the password expires (0 in the example shown below).

If the expiration event occurs, the services themselves will still continue to run, and even start correctly. So rebooting the server won’t flush out the root cause easily as one would expect the services to fail on start with a error about ‘invalid credentials’. That would be too easy.

Instead what will appear in the log are errors (12295) relating to the the services failing to verify the validity of the service account password, with a specific error code for expiration (0x800708C2).

From an end-user standpoint Office Communicator will still function. IM Conversations and telephony will still work, Live Meeting still works, all the major stuff still looks fine. But users may report that they can no longer expand any distribution lists within their OC Contact List, and others may notice an alert explaining that Office Communicator cannot download the corporate address book. The root cause of those issues are that the IIS application pool also runs under the RTCComponentService identity, which is effectively broken.

Resolution

Now whether you’ve caught the issue before anything expired or stumbled across this article in a panic the resolution is basically the same: change the passwords, update the services, and restart. (Of course if they haven’t expired you can simply set the accounts to not expire and leave the existing password, but then you would have stopped reading this article after the first few paragraphs :P ).

Begin by setting a new, strong password on each service account:

RTCService
RTCComponentService
RTCGuestAccessUser (Enterprise Edition only)

Then stop each of the OCS Service accounts on any domain-joined servers (Front-End, Mediation, etc) and update the service account’s password from the Log On tab in the properties.

Restart each service and check the event log, it should look much nicer than the mess of warnings and errors previously seen.

But wait, there’s more…

I imagine that most administrators would easily get through these steps without even searching online for assistance. The event logs, when found, are pretty clear about the problem and resetting the services are pretty simple.

But Office Communicator still won’t work correctly, yet. All users (internal or external) may be re-prompted for credentials after signing-in, with a message indicating it’s for access to the corporate address book. Also distribution group expansion will still be broken, but now with a different error. Beforehand the client would show a description immediately below the group saying “Cannot perform this action, and the cause is unknown”, but now the description would say that the user wasn’t granted sufficient rights to use that feature. Better, but not all better.

The missing link is that credentials also need to be updated within IIS on the Web Components server, and Communicator Web Access, if deployed.

Web Components Server

Open up the Internet information Services (IIS) Manager and expand the Application Pools.

View the Advanced Settings for the LSGroupExpAppPool entry. Highlight Identity under the Process Model heading and hit the ellipses button to the right to edit the current Application Pool identity.