harbar.net component based software & platform hygiene

SharePoint Central Administration: High Availability, Load Balancing, Security & General Recommendations

Print | posted on Friday, December 26, 2008 7:31 PM

I keep getting sandbagged by folks on the topic of the SharePoint Central Administration (SPCA) application, and there is still considerable confusion about how SPCA should be best deployed within a farm topology, how to make it “Highly Available” and “Secure”. Most of the queries are around what I do in my deployments and what recommendations I have for SPCA. Therefore this article covers these topics along with some additional discussion and general recommendations.

Most of this information is available elsewhere but alas in a disparate and confusing manner. The idea of this article is to consolidate these SPCA related tips into one reference guide and offer some additional explanation which you may find useful (and that I can point people to when being cornered!).

I assume you are all familiar with SPCA and therefore I won’t spend time babble speaking about what it is for and so on. I will point out though that SPCA is pretty important (!!) in the grand scheme of things SharePoint, so it deserves plenty of consideration in your farm topology planning. I will use the classic ‘Medium Server Farm’ topology in the examples below, however the concepts are valid for larger environments as well (those being the context for most of the questions I’ve received).

 

This short update is to clarify the situation following discussions with Microsoft Premier Field Engineers on the matter.

Running Central Administration on more than one server in the farm is 100% supported, and indeed a recommended best practice.

Load Balancing Central Administration is 100% supported. And even if it wasn’t it wouldn’t matter as you can simply take load balancing out of the equation and hit one of the machines directly.

Implementing Kerberos Authentication for load balanced Central Administration is 100% supported.

Implementing Central Administration on Port 80 or 443 is 100% supported.

If you are told by someone that the above approaches are unsupported, they are incorrect. Please contact me via this blog and I will follow up with you.

[UPDATE 13/09/2010] Whilst this article is about SharePoint 2007, it is also entirely valid for SharePoint 2010.


Running Central Administration on more than one server in the farm.

One of the most common issues I see when performing health checks is that SPCA is only installed on one server in the farm, most commonly the ‘Index’ or ‘Application’ server. This is not unreasonable as it is the approach promoted by Microsoft for the classic ‘Medium Server Farm’, consisting of one ‘Application’ server, and two ‘Web Front End’ servers. This model has been around forever (Solution Accelerator for Intranets) and is tried, tested and proven.

A key design driver for this model with respect to the hosting of SPCA on the Application server is that access to SPCA will be “round the back”, on the inter-server LAN only from a small number of authorized administrators.

However deploying this model causes a couple of problems and/or risks:

  1. When crawls are taking place, SPCA response times and usability can tank.
  2. If the Application server goes offline or otherwise dies you have no SPCA.

Of course in event of a disaster with the Application server a different server can be provisioned with SPCA by SPCA itself if you could get to it (Operations… Services on Server), running the Configuration Wizard or STSADM, but this is far from ideal as you will likely need quick access to SPCA to help fix the problem at hand!

The solution to these problems is to deploy SPCA to more than one server in your farm. This is possible despite a bunch of erroneous information out there, including a MCS Consultant’s blog “explaining” topologies, stating that SPCA only runs on one machine, which is simply the default configuration.

[UPDATE, 18/05/2009] The blog post mentioned above has now been corrected.

This is what the Advanced button is all about in the SharePoint Configuration Wizard (SPCW). When we join a server to the farm we can use this screen to also provision SPCA onto that server.

SPCW1 SPCW2

If we go back to our ‘Medium Server Farm’ topology, the sensible place therefore to install SPCA is onto both of the ‘Web Front Ends’. Once we have done this each machine hosts SPCA, as detailed in the diagrams below.

 MediumFarm Topology 

At this point we have a couple machines running SPCA. Because of the frankly bizarre way SharePoint deals with addressing for SPCA each server is listening on the specified port, but only the primary server (the first machine we installed SPCA on) will be used. If we attempt to access the second server directly, it will redirect us to the first one! This is not the configuration we require.

[UPDATE, 27/12/2008] The redirect only takes place is we request the “top level” URL, e.g. http://mossweb2:8080/. If we request http://mossweb2:8080/default.aspx the redirect will not occur.

 

 


Correcting the Central Administration AAM Collection

The cause of the redirect is by default, the Public URL for every Internal URL is that of the first server, as shown below.

AAMsBefore

We can fix this easily by editing the Public URLs so that instead of having two entries within the AAM Collection in the Default Zone (which logically is incorrect) we have two separate Zones, which can then have different URLs.

This is done by clicking the Edit Public URLs button within the Central Administration AAM Collection page, and then adding the second server’s URL into one of the Zones (Intranet recommended):

PublicZoneURLs

Once we hit Save our AAMs are as we wish:

AAMsAfter

Now if we access http://mossweb2:8080 from a web browser we won’t be redirected. We have two “independent” SPCA instances and we can choose which one to use simply by changing the URL in browser’s address bar.

[Documented previously on the From the Field blog]

[UPDATE, 27/12/2008] Making such modifications will break the second (and third, forth etc) instance when using Kerberos Authentication. SharePoint can only configure Authentication on a URL in the Default Zone. Therefore if we wish to use Kerberos, we should *NOT* make these changes, and workaround the issue by using the full address (e.g. http://mossweb2:8080/default.aspx) when requesting SPCA. More details in the Kerberos section below.

 

 


Modifying the CentralAdministrationURL Registry key.

Having our AAMs in order doesn’t help if we use the SharePoint v3 Central Administration shortcut on our servers. This shortcut is not simply a URL but rather runs PSConfigUI.exe to determine the URL of SPCA and then open a browser window with that URL. This is a good thing in most scenarios as our SPCA may have been moved. PSConfigUI determines the URL by checking a registry key.

The problem here is that this is only designed to support a standard configuration, with only one SPCA URL. On every server, the URL will be the URL of the first server on which SPCA was installed. In our scenario, if we are logged on to MOSSWEB2 and click the shortcut it doesn’t make sense to hop across to MOSSWEB1.

We can fix this easily by editing the CentralAdministrationURL string value at HKLM\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\12.0\WSS on each machine hosting SPCA, to be the local SPCA instance.

Registry

Now for both of the Web Front End servers we will go to the local SPCA. Our ‘Application’ server of course can only have one value.

[Documented previously on William Baer’s blog]

 

 


Providing “Highly Available” SPCA using Network Load Balancing

At this point we now have resilience for our SPCA as it’s deployed on more than one server and we can easily switch the instance we wish to use. However an extremely common request is for “High Availability”. I personally don’t like that term as there’s way more to it than meets the eye. What people generally mean is they want to load balance SPCA.

Despite what you might read elsewhere, it is possible to load balance SPCA and it’s really easy. There’s nothing clever going on with SPCA and in fact it’s generally simpler than other Web Applications as the SPCA IIS Web Site doesn’t include any Host Headers in it’s bindings.

The first thing we need is a DNS A Record (not an Alias), e.g. spca.sharepoint.com for the URL we wish to use to access SPCA and a Virtual IP Address (VIP) for that to listen on.

Don’t ask me why Microsoft calls VIPs “Cluster IP Addresses”, I have no clue!

Assuming our Network Load Balancing cluster is already setup and includes our VIP, we then go ahead and set up a NLB Port Rule for the port we are running SPCA on (e.g. 8080). We should use Single Affinity for performance reasons (there is no benefit to "true” load balancing in this scenario).

PortRule

The next step is to add an additional Public URL within the Central Administration AAM Collection for our new URL (this will prevent Render Failed errors in Web Parts and other mishaps whilst using this URL).

AAMsNLB

We can now browse to http://spca.sharepoint.com:8080/ to use SPCA and if one machine goes down, the application will still be available without us needing to change anything. You can use another load balancer product if you wish, but remember the Public URL is required.

We could now update the registry key to point to this new URL and regardless of which SharePoint machine we are on we will access SPCA through NLB.

 [UPDATE, 27/12/2008] Making such modifications will break the load balanced URL when using Kerberos Authentication. SharePoint can only configure Authentication on a URL in the Default Zone. Therefore if we wish to use Kerberos, we should *NOT* make these changes, and workaround the issue by adding the Load Balanced URL in the default zone. More details in the Kerberos section below.

 


Running Central Administration on the standard Port

A lot of folk are annoyed that the SPCW doesn't allow port 80 to be used for SPCA. There are a couple of reasons for this. Firstly the bogus security by obscurity recommendation (running on a high port), is still persuasive even if it’s only actual security benefit is defence in depth.

Secondly, the IIS Default Web Site uses port 80 and you can’t have multiple sites on the same port unless Fixed IPs or Host Headers are used. If not, only one site at a time on a given port can be started. This second reason is much more important, as if port 80 was enabled in the SPCW, there’d be a ton of broken deployments right out of the gate, by default.

However if we have intelligent security architects at our company and wanted to use the standard port we can.

The first thing we need to do is to stop the Default Web Site using the IIS Manager on each machine running SPCA.

Then we use STSADM to set the port used by SPCA:

stsadm –o setadminport –port 80

 

Doing this will change the bindings in IIS and modify the AAMs, we also need to repeat the steps in Correcting the Central Administration AAM Collection section to correct the Public URLs as shown below.

AAMsPort80

If we are doing load balancing, we would also of course need to modify our NLB port rule so that it is load balancing traffic on port 80 and add our additional Public URL (http://spca.sharepoint.com).

AAMsPort80NLB

Don’t forget to also modify the Registry key which will be reset by the setadmin port operation.

 


Running Central Administration over SSL.

So far all of our access to SPCA has been over HTTP, which means plain text over the network. In lots of places SPCA asks us for service account credentials which are then passed back to the server when we hit OK or Save. These credentials also go over the wire in plain text. These credentials could therefore potentially be disclosed, to prevent this we need to implement Transport Layer Security.

There are two ways to do this – and remember we hopefully trust our network, it’s ours after all in most cases :). Firstly and the best option by far in terms of performance and security is to use IP Sec which is built into Windows. This is vastly preferable, but unfortunately not many people seem to have this together with the requisite infrastructure configured. The second more common option is to use HTTP (SSL).

This requires us to request a SSL certificate for each name our SPCA application is running under (e.g. MOSSWEB1 & MOSSWEB2). We would then add the cert for our server (one per server) to the Machine Certificate Store. IIS7 makes this process much less error prone as we can do all of this from the IIS Manager and don’t have to open the Certificates MMC Snap in. This is all done from the Server Certificates item within the Server Home section.

SSL1

We then need to add an IIS binding for HTTPS and select an SSL Certificate. We do this by selecting the SPCA Site within the Connections pane and clicking Bindings in the Actions pane. On the Site Bindings dialog, click Add…

SSL2

Our next step is to require SSL which we configure from the SSL Settings item within the SPCA Site Home.

SSL3

Once our certificates are installed and SSL is configured, we use STSADM to set the port used by SPCA and enable SSL:

stsadm –o setadminport –ssl –port 443

This step tells SharePoint to use SSL and will also remove the HTTP binding from IIS.

Again, we would need to repeat the steps in Correcting the Central Administration AAM Collection section to correct the Public URLs.

AAMsSSL1

Don’t forget to also modify the Registry key which will be reset by the setadmin port operation.

If we are doing load balancing, we need a single certificate for the load balanced URL (e.g. http://spca.sharepoint.com), which is applied to both servers.

With SSL we can’t do both direct server access and load balancing unless we use a wildcard certificate and fully qualified names for direct server access.

Once our certificates are installed, we use STSADM to set the port used by SPCA and enable SSL:

stsadm –o setadminport –ssl –port 443

We would also need to add an NLB Port Rule to load balance our SSL port

Again, we would need to repeat the steps in Correcting the Central Administration AAM Collection section to correct the Public URLs.

AAMsSSL2

Don’t forget to also modify the Registry key which will be reset by the setadmin port operation.

We don’t need to use 443 for SSL but it would be silly to use a non standard port! However bear in mind that multiple SSL sites using Host Headers (yes, it is possible) requires use of a wildcard SSL Certificate. If we are planning to host other SSL Web Applications we should consider using an alternate port for SPCA.

Either way, using individual server names or a load balanced URL, once SSL is configured we will no longer see the security warning on pages where credentials are submitted.

SSLmain

 

 


Isn’t coming in over the Public LAN a risk?

At this point we’re coming in over the front end, Public LAN and many people see this as a risk. It isn’t really, assuming we have configured the membership of the Farm Administrators group correctly and optionally configured Transport Layer Security. Another concern is that this traffic is also going to degrade the end user Web Application performance, this again is a bit of a myth as SPCA shouldn’t be used heavily by large numbers of users.

Consider the counterpoint that allowing access “in the back” is more secure. This of course is security by obscurity again and is this case is completely moot.

If we are really paranoid we can use IIS to further restrict access to SPCA to a defined list of client IP Addresses using the IP Address Restrictions feature.

Note that in IIS7 this feature (IpRestrictionModule) is neither installed by default, or by the MOSS SP1 installer and must be added using Add Role Services within Server Manager.

 

 


Right, so what about Kerberos?

Configuring SPCA to use Kerberos is very straightforward and the same as for any other Web Application. I have separate resources on Kerberos available here.

One of my key recommendations here is to configure using NTLM first. In other words, don’t select Kerberos in the SPCW. It’s not about how great your build scripts are, its about having a working deployment with the least amount of hassle. By configuring as NTLM first, you can be sure that SPCA is working and then setup Kerberos. This way you’re not in a situation where it is difficult to diagnose what is at fault, SPCA or Kerberos.

In a nutshell the process is:

  1. Setup SPN(s). Don’t forget to include the port in the SPN if you are using non standard ports. Note you *do not* need to configure delegation on any computer or user accounts for SPCA. SPCA does not include any functionality which requires Kerberos Delegation.
  2. Configure Kerberos in SPCA or using STSADM
  3. Optionally set Negotiate “only” using ADSUTIL.VBS.
  4. Configure Kernel Mode Authentication (if using IIS7).

 

 [UPDATE, 27/12/2008]

If you wish to use Kerberos with multiple SPCA instances (and optionally Load Balancing) you *MUST NOT* make the modifications to the AAM Collection as described in the earlier section.

Making such modifications will break the second (and third, forth etc) instance when using Kerberos Authentication. SharePoint can only configure Authentication on a URL in the Default Zone.

Take the examples below, our first server (which is the URL in the Default Zone) displayed the RSS Feed from a remote site (which requires Kerberos). The second server cannot display the Web Part and it displays the usual “doesn’t support authenticated feeds” error.

CAKerb1 CAKerb2

If we attempt to configure the second instance to use Negotiate in SPCA or STSADM the command will fail. We can’t work around it by attempting the configuration locally on each machine either. It simply doesn’t work!

KerbError

Therefore if we wish to use Kerberos, we should *NOT* make the AAM changes and stick with the default AAM configuration:

KerbAAMs

If we want to also use Load Balancing with Kerberos, we need to enter an additional Internal URL for our load balanced URL:

KerbAAMsNLB

Now we can configure each URL to use Kerberos, which we must do with STSADM as SPCA is mightily confused with our configuration and can only set the authentication for a single URL!

stsadm -o authentication -url http://mossweb1:8080 -type Windows –usewindowsintegrated
 

stsadm -o authentication -url http://mossweb2:8080 -type Windows –usewindowsintegrated
 
stsadm -o authentication -url http://spca.sharepoint.com:8080 -type Windows
                                                                 -usewindowsintegrated

 

Now we have SPCA using Kerberos correctly on both instances (and optionally Load Balancing).

CAKerb3 CAKerb4

However our redirect is still a problem. The way to work around this issue is to update the Registry key to use the full address, with the default.aspx bit (e.g. http://mossweb2:8080/default.aspx).

This is the best you can get. Watch out however thou, if you click the Central Administration link in the very top left hand corner, this is a link to ‘/’ so you will redirected to the Public URL (mossweb1). The Home tab link doesn’t do this.

 


Other Recommendations

  1. Don’t let the SPCW pick a random high port
    If you wish to keep a high port for SPCA, stick with something sensible like 8080. I like 8080 as it’s contiguous with other default ports in the product (Document Conversions uses 8081 and 8082 by default) and it’s also a semi-standard from back in the day when admin ports like 88 and 8080 were common.
  2. Don't use DNS CNames (Aliases)
    You should only use Aliases when you need one. In addition they cause problems with certain clients (IE6) and Kerberos. Oh and did I mention you shouldn’t use an alias when it’s not an alias you’re after? :)
  3. Windows Firewall on Windows 2008 
    Remember this guy will get in the way of your inter-server communications, including requests to SPCA on high ports. You will need to allow your SPCA ports through. Another in depth post on how to configure the Windows Firewall for a SharePoint Farm is coming soon.
  4. Internet Explorer Trusted Zone
    Ignore the crazy install guides – don’t use them. Don’t be adding SPCA URLs into the Trusted Zone! It’s a totally bogus, single server recommendation. Use the Intranet Zone instead which will send credentials automatically and means you don’t need to turn off Internet Explorer Enhanced Security Configuration.
  5. AAMs are not backed up
    The only way to back up your AAM configuration is to document it. The only way to restore your AAM configuration is to re-implement it based upon that documentation. This is not specific to SPCA but a general problem (one of many!) with AAMs.
  6. Save hassle by running the SPCW first on a machine you want to host SPCA.
    Tidying up later is always a hassle as the SPCW will not change this. If you install SPCA on the Application server and later “move” it, the registry key will always be the application server URL.
  7. Use SETADMINPORT and AAMs with caution!
    In addition changing the admin port or AAM Internal URL, will modify the registry key:
    1. Running STSADM –o SETADMINPORT will reset the key back to the original server with the port specified, on all servers in the farm. It will also reset the AAM URLs back to the original ‘redirect’ configuration.
    2. Modifying the first AAM Internal URL will set the key to that value, on all servers in the farm

 


Summary

So there you have it. A bunch of stuff about Central Administration. Hopefully some of it will be useful, I am sure there are things I have overlooked and/or the odd mistake, and if so I will update in the future.

I managed to get all the way to the end before saying “Best Practice” – this area is a classic example of where there are no across the board best practices, only design compromises. For example using load balancing for SPCA is only a best practice if certain business requirements demand it. Like most other things in SharePoint you need to take a pragmatic, holistic view based upon the entire solution space and then choose what is best for your deployment.