Earlier today a Twitter conversation amongst some SharePoint people including my good buddies Todd Klindt and Rick Taylor took place on the subject of the infamous “loopback fix”. I promised to do a follow up post here to clear up some misconceptions about this subject with respect to SharePoint.
What is the issue?
Windows Server 2003 SP1 introduced a loopback security check. This feature is obviously also present in Windows Server 2008. The feature prevents access to a web application using a fully qualified domain name (FQDN) if an attempt to access it takes place from a machine that hosts that application. The end result is a 401.1 Access Denied from the web server and a logon failure in the event log.
Unfortunately 401.1 is not really helpful as this error code means there is a problem with the user credentials. Of course, the HTTP spec doesn’t know about security features in a vendor’s implementation so there can’t be a HTTP error code for such a feature. This can lead to much banging of the head on the desk. It’s one of numerous causes of the 401.1 which are nothing to do with invalid credentials (e.g. attempting to use Kernel Mode Authentication with domain account in IIS7).
What this means is that when you browse a SharePoint Web Application which uses a fully qualified domain name from a WFE in the farm you will get a 401.1. This is very annoying on a development box, or when testing locally, or in other SharePoint specific scenarios (more on those later).
OK, so what’s the big deal here?
Microsoft call this a security feature. Spence calls this a security fix! There are many exploits which attempt to attack via reflection – i.e. pretending to be local as to bypass constraints. This setting should have been in the box since Windows NT4, but it wasn’t. Microsoft have done the right thing and addressed the problem based on customer feedback and exploits. They have fixed a hole and further tightened the attack surface of a Windows server. Good Job. Anyone who cares about host security or platform hygiene knows it makes sense.
But it breaks my SharePoint, dang it!
Yup, it does. But only if you are attempting to access a Web Application from a server hosting it (i.e. locally). You shouldn’t be doing that. Well, you shouldn’t be doing it in production. If you or your company admins are testing locally you have bigger problems that a pesky security fix.
The trouble is there are also scenarios where this fix will break normal operations of SharePoint.
- Search Indexing.
If you are hosting the WSS Web Application Service on your Indexer for the purposes of having a “Dedicated Crawl Front End” and avoiding a network hop. This is common in small scale “Medium Server Farms”. Because the Indexer is crawling itself, the crawl log will fill up with 401s and your content won’t get indexed. - Web Application “Warm Ups”.
If you are running a scheduled task or timer job to hit the Web Application to avoid the start up lag after an application pool recycle, the “warm up” will fail with a 401. - Custom Code using SharePoint Web Services.
If you have custom code, either in SharePoint or out with it that leverages SharePoint Web Services (such as using the ExcelService API) these requests will fail with a 401.
Okay, so get to the point what should I do?
Microsoft’s KB Article 896861 details two workarounds. One is to disable the Loopback Check entirely – and this is commonly promoted as the thing to do on all your SharePoint Servers. The second is to add a list of addresses to exclude from the check. Both of these are accomplished by means of a registry key in the LSA hive.
So which one should you use? The answer, of course is, it depends.
If you are working on a development environment or on just a single MOSS box – go for it - disable it completely. You need to debug and test locally and it’s likely you don’t know what addresses you will use ahead of time. I as a matter of course disable the check as part of my sysprep build for all my development and test machines. I never hit the problem because my base image is all sorted as I want it. I recommend you do the same.
However, for production environments, DO NOT DISABLE this feature. You are unpicking a serious security check of the OS. If that environment underwent a security audit by a competent security engineer, it would be flagged. You should add a list of addresses you wish to exclude. This makes your scenario work whilst retaining the security check (which is important if you are handing over the environment to your customer’s admins who may decide to browse the interwebz from the console :)).
It’s not that big of a deal to figure out which addresses you need to exclude, and on what machines you need to apply the change. If you can’t do this quickly, you don’t understand your topology or services.
This change can of course be applied by Group Policy, so should you need to do it on a bunch of boxes (e.g. five WFEs with custom code using SharePoint Web Services) the overhead is avoided. Of course this assumes you are using GPOs to manage your SharePoint Servers (you should be). GPOs are also useful if you need to add additional addresses later on.
Another advantage of using the list, is the change does not require a reboot of the server to take effect, just a restart of the IISAdmin service.
Of course this is a lifecycle issue and like anything else, you should weigh the security risk versus additional complexity on a per implementation basis to make the right call. And more important than this, is you need to document the configuration.
In addition as a consultant or otherwise, if you are hitting 401s that don’t make sense – check this, along with Kernel Mode AuthN and Local Intranet Zone before anything else. Just like everyone else this problem has had me scratching my head for a couple hours. So remember this issue when diagnosing the dreaded 401!
Conclusion
The loopback check is a good thing, not a bad thing, if you care about security and platform hygiene. Do not disable this feature in production. Feel free to disable it in development/test environments. Todd is also planning to spend some time on this subject in his next netcast.
I greatly enjoyed my recent trip to Houten, Netherlands to present at the SDN event. Many thanks to the SDN crowd and those that attended my sessions and of course the SharePint event afterwards. It was a very impressive event in terms of organisation and the other great speakers and content.
Here are the slide decks from my three sessions as promised.
All of the content was updated for the SDN event, with Mythbusters including new details on a topic I will be expanding on more in the future – Agile Farms. It was the Kerberos sessions that have changed the most however, with more complete coverage and including updates to deal with some of the increasingly common misinformation out there on this subject. The sessions are now much more balanced, and enough time on each subtopic is possible. Of course, the real value is in the demonstrations, which make up over 40 minutes of each session (and they all worked!). I am considering screen casts of these demos in the near future.
I will be posting a couple of short follow ups on Kerberos related topics which came up in discussion during the event shortly, namely Excel Services and Search.
Once again, a big shout out to the SDN people for organising such a great event! I even managed to snag me some Stroopwafels :)
One of the most interesting areas of late in the SharePoint world has been the rapid explosion in it’s adoption within the Education sector here in the UK. Accordingly, the fine folks over at the Learning Gateway User Group are holding a one day Conference on July 15 in Birmingham.
The event takes place at the stunning Belfry – near where I used to live many moons ago, and features some top notch speakers and SMEs. If you are working in the education space, this is a must attend event. In addition, even if you are not in education, the material presented will be of great value, especially if you are in public sector or are getting started with your SharePoint deployment. Some great topics are being presented with many best practices and tips and tricks. On top of all this, it’s a great price and only £150 per delegate. Perfect for the times.
Check out more information over at the Learning Gateway Conference web site.
I will be speaking at the Software Development Network event in Houten, Netherlands on June 26th. SDN is a really strong community in the Netherlands and it is a great privilege to be selected as a speaker there amongst some other great talent.
I will be presenting the following sessions:
You can find out more information about the event over on the SDN site. See you there!
The excellent guidance over on TechNet regarding DR farms using Log Shipping has been updated with additional details for SP2 and some excellent coverage of pre-requisites, fundamental planning considerations and performance. Essential reading for SharePoint administrators (even if you haven’t implemented Log Shipping in your solution).
Configure disaster recovery across SharePoint farms by using SQL Server log shipping
For those that have attended any of my recent sessions on SharePoint Web Content Management you will have already seen the Adventure Works Travel (AWT) sample site. This sample has now (finally) been released into the wild, and you can grab it at www.mssharepointdeveloper.com. Also available are a number of training modules which I contributed towards. On the right hand side, swivel the navigation to the SharePoint on the Web section to access the training and the sample.
The AWT sample provides excellent guidance on building publishing sites using Office SharePoint Server’s WCM features. I know a few folks won’t thank me for mentioning it, but if you remember MCMS, this is like Woodgrove, but this time it’s pretty decent :) Go check it out!
[UPDATE] To all those trying to grab the AWT installer, it’s not there anymore so please don’t send me anymore mail, or comments! Pretty please! :) I don’t know why it was removed and I don’t know when it will be back. I will post an update once it’s back. Apologies.
[UPDATE] Now available at http://code.msdn.microsoft.com/spotw/Release/ProjectReleases.aspx?ReleaseId=2702
On a number of occasions I’ve been asked for my “cheat sheet” for setting up multiple SharePoint Web Applications using SSL and listening on Port 443. Whilst this might seem a straightforward task there are a few key things about SSL which are not well enough known, and surprise surprise, SharePoint makes life a little more difficult due to it’s total ignorance of correct addressing. This article is basically a tidied up version of the cheat sheet, along with a little bit of further explanation. This is primarily so I have something to point people to when asked about the procedure.
The steps detailed below show how to setup two Web Applications, but of course you can rinse and repeat if you need more than two.
Multiple SSL Web Applications on Port 443
Recently a bunch of people have asked me about the supportability of the approaches to providing “high availability” for SharePoint Central Administration I detailed in my article SharePoint Central Administration: High Availability, Load Balancing, Security & General Recommendations.
This short update is to clarify the situation following discussions with Microsoft Premier Field Engineers on the matter.
Running Central Administration on more than one server in the farm is 100% supported, and indeed a recommended best practice.
Load Balancing Central Administration is 100% supported. And even if it wasn’t it wouldn’t matter as you can simply take load balancing out of the equation and hit one of the machines directly.
Implementing Kerberos Authentication for load balanced Central Administration is 100% supported.
Implementing Central Administration on Port 80 or 443 is 100% supported.
If you are told by someone that the above approaches are unsupported, they are incorrect. Please contact me via this blog and I will follow up with you.
At the end of last month, James Petrosky the acting Program Manager of the SharePoint Certified Master program introduced the first SharePoint MCMs over on the Master Blog, which was reposted by Dave Pae over on the SharePoint Team Blog. Since then the MCM has come up in a number of conversations on various blogs, in their comments and elsewhere. It is apparent from these that there remains some unfortunate misconceptions about the MCM and this post is an attempt to address them.
For whatever reason many assumptions about the program have been made, which are just plain wrong. I guess me and a bunch of other people who have blogged about MCM haven’t been particularly articulate, or perhaps people just don’t want to read and digest, so here goes…
1. The MCM is a 100% technical certification.
This should really be obvious from the name. The Microsoft Certified Master is all about technical competence. 100% about technical competence. 0% about anything else. The key word is not ‘Master’, but ‘Certified’. The MCM has absolutely nothing to do with giving people a distinction. You are either good enough technically to be an MCM, or you are not. That’s it. Simple.
2. The MCM for SharePoint 2007 is a quality certification.
It ain’t no badge. I just got thru doing it. It’s hard. It’s fair. You have to know an enormous amount of stuff, and you must be experienced in deploying SharePoint. Wannabes have no chance, no matter how good their intentions. So many people have dismissed the certification, and this is a joke. Again, the key is Certification, you are validated that you really know your stuff by the best in the business. Not people that talk a good game, but those that actually do it for real in the real world. You must know what you are doing. You cannot pass by virtue of your employer, personal relationships or anything else not related to your competence with the product. You are either good enough, or not. And that goes both for getting in the program as well as for actually achieving the certification.
3. There is no relationship with the MVP program or any other community initiative
Microsoft sponsored or otherwise. MCM is a certification, MVP is an award for community contribution. Once again I have to state that MVP does not mean technical competence. Sure a lot, if not most, of the SharePoint MVPs happen to be technically excellent, but MVP is just a award Microsoft gives to show appreciation for community efforts. Of course there are significant benefits to being one, and many choose to use it as a commercial lever. MCM is not about having a badge for people who can’t be an MVP for whatever reason. MCM is a certification, not a badge.
Of course both programs have their place both in terms of community and in terms of addressing the real problem of SharePoint skills in the field. However the bottom line is that SharePoint is still new, and it takes time for field readiness to be achieved. It’s really a little annoying that this misconception even exists – it doesn’t with respect to the SQL and Exchange MCM programs. Why? Because those communities are much more mature.
Another point here is it is entirely reasonable that the majority of MCMs will probably not be active in the community. Why? Well because they spend their time actually deploying big ass SharePoint, as opposed to talking about doing it on the Interweb, or at large events. Many MCMs will be names you’ve never heard of, and that’s a good thing.
4. Let’s get it straight. MCM is not just for Microsoft people.
This is the one which is being totally misunderstood. I’m gonna lay out the facts, nice and simple. I’m gonna give you real numbers.
So far there have been two SharePoint MCM “rotations”, alpha and beta. The RTM rotation takes place in June.
The alpha delivery was for Microsoft employees only. It would be pretty stupid to have the first rotation offered externally to paying customers! Imagine the abuse they would get! It’s about making sure the thing is decent, and nothing is perfect first time.
- The alpha delivery had 12 attendees
- 3 alpha students passed on the first attempt
- 5 alpha students (so far) passed following retakes
The beta delivery was for Microsoft people AND “Partners” (Microspeak for non MS employees)
- The beta delivery had 16 attendees
- The beta delivery had 9 MS employees
- The beta delivery had 7 non-MS employees
- 6 beta students passed on the first attempt
- 4 beta students who just so happened to be MS employees passed on the first attempt
- 2 beta students who just so happened to be non-MS employees passed on the first attempt
- [UPDATE 11/06/09] 2 beta students passed following retakes
It’s that straightforward. MCM has nothing whatsoever to do with recognizing people for anything other than their technical competence with SharePoint. Being a MS employee has zero relevance whatsoever. Going forward you can expect to see the numbers become even more “balanced” but the bottom line is why does anyone care? Being good enough is the only thing that matters for MCM, where you work is completely irrelevant.
[UPDATE, thanks to Mike Walsh] It’s worth noting that the details listed on the SharePoint Team Blog are a combination of those who passed over the course of the two rotations, alpha and beta.
5. Being a softie has zero relevance to selection
Another key misconception is that it is “easier” to be allowed to attend, selected (and pass) if you are a MS employee. Actually the opposite is true. Vast majority of the MS people going through the program are from the field, either MCS or PFEs. I know all about working in the field for Microsoft. It’s about one thing, utilization. Just like any other Professional Services organization. You think it’s easy for these people to get their manager to allow them to have three weeks of zero utilization and associated expenses? Maybe you’ve heard about the two rounds of layoffs recently? Come on. Think about it. It isn't easy. It’s easier for the likes of me to be able to sit this sort of qualification.
As for selection… To get selected you must meet the pre-requisites and pass a selection interview by the MCM people. Again, where you work for has zero relevance. Sure, if you work at MS, and are good enough there’s a reasonable chance you will either be known or be “recommended” but that’s it. Microsoft has like 70,000 employees, it ain’t no golf club.
And being a MS employee won’t help your chances much in terms of success either. There is an urban myth that if you work for MCS you have “access” to secret MS only mojo that makes you loads better. Absolute 100% total hogwash. Take it from me, I know this is not the case. Of course being an employee gives you access to some things, and tools or what not, but those won’t really help in terms of MCM success.
[UPDATE] Being a Microsoft employee undoubtedly has significant benefits, after all you work for the company that makes the software you are being tested on. However the reality of MCM is that it doesn’t help *that* much. Again, either you are good enough to be pass or you are not.
Conclusion
Hopefully the five points above will help address misconceptions about the SharePoint MCM. I’ve no doubt whatsoever some still won’t “get it” or believe the facts detailed. The bottom line is that time will prove the value and real story of the SharePoint MCM just as it has for the other MCM programs.
As you’ve probably noticed it’s been a little quiet here of late. Sorry about that. I’ve been pretty busy with various things. One of those things was undertaking the beta rotation (R2) of the Microsoft Certified Master for SharePoint 2007 out in Redmond WA. Many people have asked me about the experience and so this post is an attempt to distill my impressions on the program. Please note that at the time of writing I do not know if I have passed – more on that later.
[Update] Shortly after writing this post (I keep things in the hopper for a bit before publishing, kinda like a delay on sending emails) I was notified I had achieved the Certified Master. It's an honour to be part of this small group of SharePoint Masters.
I’m not going to spend time here covering the basics of the MCM and what it’s all about, you can find that information at my previous post: Microsoft Certified Master for SharePoint. Before I get started on another one of my ramblings I cannot disclose any technical details about the contents of the course or the exams so this post will concentrate on the overall experience.
For three weeks, including the weekends, sixteen of the best SharePoint professionals on the planet were imprisoned in a room on Microsoft’s campus in Redmond to learn from the best of the best. Three weeks is a long time to commit to something like this, I certainly haven’t done anything remotely similar since getting started in this business some 15 years back.
All of the students were already SharePoint experts. R2 was the first delivery to include non Microsoft employees, and we had a good mix of folks from all over the world and all with different experiences and perspectives. Given the rigorous candidate selection process a certain amount of arrogance is to be expected and desired. However that arrogance got slapped into shape good and proper within a couple days, as we got into the material and the hours.
In a word, MCM is brutal. That’s a good thing. Over the three weeks the sheer breadth of what we covered was incredible, and it is well recognized that there was a lot of stuff not covered. Obviously SharePoint is a giant product, but the way the course was structured was pretty much spot on the money. I wasn’t expecting this at all, in fact I was very dubious going in although trying to keep an open mind.
First of all, they actually did things in the right order and covered in depth core IA and Infrastructure topics in the first week, with a heavy emphasis on storage. Then we dove into feature areas, and lastly development. There were a couple pieces which were kinda out of “order” but this was mainly due to logistics.
We got started everyday at 8am sharp with 45 minutes or so for lunch and a finish (of class) at around 6pm. Most people would stay in the room and work on labs or further study until much later. Generally I was hitting the sack at around midnight. There is no time for anything but MCM during these three weeks! If you are not fully committed, you won’t succeed, it’s that simple.
As for logistics, they were first rate. Astonishing! I’ve been to plenty of things on campus and generally time is loose and a lot of things go haywire. Not the MCM. Sure there were a few rough edges, but credit is due to James Petrosky, the acting Program Manager for pulling off a significant feat here.
I stayed at a corporate apartment close to campus which made things somewhat easier to remain focused purely on MCM. It had a dodgy Comcast cable box which wouldn’t work with Windows at all, so I spent most of the time whilst in the place running Mac OS! Interestingly social computing also played a part in my experience. Yes, I know, I can hear you giggling in the back, but it’s true. A number of peeps in the big house were following my updates and in addition I received incredible moral support from friends and colleagues on the socialnet wishing me luck and telling me to”kill it” and so forth. Amazingly another actual use for the socialnet!
Back to the course – the instructors were awesome – every last one of them. They ain’t no PowerPoint jockeys. Each of them was a SME in their field(s) and imparted a tremendous amount of knowledge. For the most part they did not teach to the exams – which from my point of view was great. Regardless of my success or otherwise the three weeks would be entirely worthwhile just for the information picked up over the time.
Even better than that however, was how much the instructors validated my points of view. Often times in the crazy world that is SharePoint consulting and deployment I find myself thinking, “Is it just me, or is everyone just plain crazy?”, the stuff I hear partners and customers warbling about is bizarre. MCM validated that I am not crazy, at least not when it comes to SharePoint :). Probably the single best technical thing about MCM is spending three weeks with true peers (both students and instructors) – for me that is incredibly rare and precious.
Every student had access to a personal blade with decent specs to work through the lab exercises and also try various things out. A few minor points aside this worked really well. In addition, there were various group exercises during the course.
OK, so what about the exams? There’s three 2 hour written tests (like CBTs). These were hard. These are not like those silly MCTS exams, these are decent questions with decent answers. They were really good. You cannot pass these if you don’t know what you are doing, it’s that simple. I would also suggest that if you don’t know what you are doing going in the door, the classroom instruction won’t make you ready for them.
And then the Qualification Lab, an all day (on the last day, a Saturday) hands on exam with numerous scenarios which you need to implement on a personal blade. 12 hours pointing, typing and clicking. It’s again brutal. I had to bail a little early to catch a flight to London and was basically drained completely. And I’m a SharePoint guy who actually implements stuff!
Now, it wasn’t all work. There were various social things organized such as Go Karting, various dinners, visits from Product Managers, socials with the Product Group and visits to interesting places on campus. However these were the exception, most peeps spent the evenings either working on labs or reviewing the “pre-reading” material. Interestingly the “pre-reading” was updated shortly before our rotation to include a whole ton more stuff. I loaded this all up on my Sony Reader, which was very useful and saved me from still looking at a damn laptop screen 12 hours into the day.
Another interesting element of the MCM for SharePoint was the human dynamics at play during the three weeks. No reservations whatsoever, all the other students were kick ass SharePoint people, and also just plain kick ass people. How the dynamics ebbed and flowed was inspiring.
Here are some tips if you are considering the MCM for SharePoint:
- Eat well – it might sound obvious, but don’t fool yourself you can make up for it later. It’s three weeks! Don’t be starving and certainly don’t be feasting. You need to be sharp!
- Sleep well – again obvious, but don’t think coffee can save you – it’s three weeks! You need mind food. Although you may dream of things SharePoint !!!
- Get some exercise – at least a little bit – get outside and/or do something else to de-stress. You will get stressed or wound up as I prefer to call it. Redmond isn't great weather wise, so take your chance when it comes.
- Read the pre-reading, then read it again. The single best resource for MCM. Don’t assume you know it all, you don’t!
- Leverage your classmates and enjoy yourself. Don’t take it completely seriously. That way madness lies.
- Know what you know, know what you don’t know, and never confuse the two. The unofficial motto of a Microsoft Certified Master. Be confident, but humble.
So that’s it – a little recap of my MCM experience. Right now, I’m actually finding it hard to get motivated back into regular work after such an intense, overwhelming, emotional and exhausting three weeks.
Don’t underestimate the MCM for SharePoint. It ain’t for the faint hearted. Don’t assume you have what it takes, you’re existing badges don’t matter a damn. I have no reservations in recommending it, if you are SharePoint professional it’s the single best way to make a significant step up in terms of your skills. However, you have to be ready.
Without a shadow of a doubt, the MCM for SharePoint is the finest technical training I have ever undertaken. By a very long way, and this includes some well known “security” training. Props to Per and all the MSL dudes, James, Michal and all the instructors for delivering an outstanding program to help address the current state of field readiness for SharePoint.
A shout out to my fellow R2 Students and all the Instructors, hoping to see you all soon. And apologies for bailing early on the last day, I sure coulda used a couple drinks!
[Update] I'd also like to point you to these other excellent posts by some of my fellow R2 prisoners:
[Via Mirjam] This has been around for ages, but I never saw it originally. It includes tenants of the right approach to blogging. Some of my favorites include:
Blogs that are purely self-promotional can quickly fizzle out. Be outwardly focused, rather than self-focused.
Find other bloggers in your niche that you respect and work with them rather than against them.
Get away from the computer once in a while and do something recreational or creative.
Essential reading for the SharePoint blogosphere, which of late has been loosing the plot a little bit.
The 7 Habits of Highly Effective Bloggers