Building a SharePoint Platform That Scales

Editor's Notes

• #3: SCOTT
• #4: DAN
• #5: DAN
• #8: So a lot of the time we’re working on our design and reducing our requirements into fashionable
• #9: MSTechEdShane Young and Todd Klindt held a session called “What could go wrong?”So what could go wrong with logical architectures and taxonomies?What could go wrong if you skip your logical architecture or your planning to make your system effective for your end user?Should the logical architecture and design planning be skipped because you have a vital requirement for SharePoint to be deployed immediately, your system will begin to see issues and degradation unfortunately at both a technical and information worker level.Typically the first thing to go is any sensibility of governance as to how sites are provisioned, created and permissions applied to them. Sometimes what I like to call the out of the box model is used where everything is in a single site collection and security groups become fairly painful and troublesome.Next the issue of those security groups and permissions become pain points as users learn about inheritance – something I’m sure that some of you have witnessed first hand.Then there’s the idea that the information, while readily available, cannot be found very easily by the end user – something perhaps to think about while constructing the underlying architecture of the system.Search isn’t working properly – the shared service provider or providers as the case may be isn’t setup to scope the appropriate web application(s) and users search results begin to show them data that wasn’t necessarily intended for them.And my personal favorite outcome of what could go wrong – the system begins to drag because it wasn’t planned with the purpose in mind. One of my previous clients intended to put terabytes upon terabytes of data into their SharePoint system to house their ISOs, executables and everything else that their help desk could want – needless to say after changing the allowed upload file types and upload sizes on the 2007 based system on an underlying 2003 infrastructure it became painfully aware that their content databases were bloated. Search wasn’t too happy either indexing the first few hundred gigabytes of data.
• #10: I’m sure that everyone here has seen the pinwheel of SharePoint… this is my crafty word art rendition.So as with any SharePoint implementation, figuring out the vision behind the system is fairly important. Some systems are simply like my former client, a document workspace. For other clients I’ve worked with, they’re more interested in the centralized point for collaboration to run their workflows and view KPIs of their data – relying on the collaboration capabilities of a centralized workspace and document management to handle their documents as they press through the system.When scoping for Enterprise Search, it’s a capability that sometimes is forgotten about and requires a little planning as well on the infrastructure side to make sure that there’s space for both query indices to be pushed forward as well as to ensure that the documents being placed in the system are understood size wise – Search stops reading your documents at 16 MB and requires a registry fix to direct your crawler to parse more.So thinking forward, during your requirements gathering and analysis of your client’s system, start thinking through what the considerations for a System Vision is… will it be a highly collaborative system? Or will it be full of applications and workflows leveraging SharePoint lists and document libraries? Is the intent to be more involved with content deployment and the WCM side of SharePoint for an internet facing site that will require additional security controls? Is SharePoint just being used to replace a static information portal that has WCM capability? Is it being looked at to replace Shared Drives?So remember, what are you trying to meet?
• #11: What works best for your organization?If you look at the SharePoint wheel there are six areas that they attempt to accomplish, are you trying to bite them all off at the same time or are you working to separate them out and accomplish them one at a time.Consider the context - What may work wonderfully for a small organization – task management for instance with users in groups, rolling up data to a top level site, may not work so well when you span it out across the enterprise. Think about investigating Project Server or something along these lines to handle a true task management system.
• #12: SharePoint as a Service – is this what you’re looking for? The McDonald’s like approach, I’d like a site collection with a wiki, and a web appication… similar to Microsoft BPOS, or 1and1 Hosting, or SharePointHosting.com
• #13: Client that wants office integration, but also wants to use ADFS, but also wants to use an LDAP – there will be compromises and tradeoffs that have to be assessed.So what are we going to talk about? You might say, “You’re skipping to the chase” and in some respects you’re right, but I also like to catch a vision as to what we’re working toward before actually trudging out.So stepping into the design side of things, remembering the context of your environment is the key thing. During your initial assessment and design phase of a project, you’ll make considerations, define tradeoffs, perform analysis of the user population and functional community that you’re working with and probably end up with a list of compromises that need to be made that need to be made with your stakeholders during a decision briefing.For instance, are you hosting everything locally at a single site, or are you using a distributed architecture, or are you merely buying the service of SharePoint from a hosting provideWhat capabilities are sought after?What are the environment limitations?Are you building into the cloud?
• #14: So you’re pulling together your entire vision, you’re gathering your requirements, the scope is growing and growing… and you feel like you’ve got something like this.Do you feel like you’re trying to chomp off more than you can deliver? Take a step back.Build out small and scale.On a side note, a friend of mine thought that I actually made this80/20 rule – 80% of users should be using 20% of the functionality. Don’t worry about the 20% that want to use 80%!
• #15: This ends up impacting the end user navigation and a slew of other things.Incorporating how your end user thinks will define that taxonomy which will drive the logical architectureWhat defines a logical architecture?Why is a logical architecture important?How can you really make use of a logical architecture?What does a logical architecture consist of and look like?What is a logical architecture?The purpose of investing in logical architectureis to tackle some of the complex and interrelated design questions that are most common when standing up your SharePoint environment.Hopefully, it will serve its purpose in that once you’ve got your logical architecture defined, you’ll be able to better facilitate with other engineers and developers. Additionally, as you build out your logical architecture and environment you’ll be able to more easily see what interfaces will need to be connected to. Furthermore, for those of you that live in a cube farm while you’re on client site, it’s always nice decorate with a large print off from a plotter to hang on your wall.Why is a logical architecture important?You’ve invested in the design, you know the logical data flow and integration between systems and you can now more easily map this to your taxonomy and to your physical hardware. In fact, you can better plan your hardware requirements based off of a logical architecture once you’ve got it on paper rather than blindly ordering hardware that looks really cool on the Dell web site.How can you really make use of a logical architecture? Does it need to be updated?Well, I think that question is somewhat rhetorical at this point, it’s an important artifact that will help you to better justify your scalability needs as well as resource needs when integration tasks come down the line.What does a logical architecture look like?Next slide
• #16: What is a logical architecture?The purpose of investing in logical architectureis to tackle some of the complex and interrelated design questions that are most common when standing up your SharePoint environment.Hopefully, it will serve its purpose in that once you’ve got your logical architecture defined, you’ll be able to better facilitate with other engineers and developers. Additionally, as you build out your logical architecture and environment you’ll be able to more easily see what interfaces will need to be connected to. Furthermore, for those of you that live in a cube farm while you’re on client site, it’s always nice decorate with a large print off from a plotter to hang on your wall.Why is a logical architecture important?You’ve invested in the design, you know the logical data flow and integration between systems and you can now more easily map this to your taxonomy and to your physical hardware. In fact, you can better plan your hardware requirements based off of a logical architecture once you’ve got it on paper rather than blindly ordering hardware that looks really cool on the Dell web site.How can you really make use of a logical architecture? Does it need to be updated?Well, I think that question is somewhat rhetorical at this point, it’s an important artifact that will help you to better justify your scalability needs as well as resource needs when integration tasks come down the line.What does a logical architecture look like?Next slide
• #17: It provides a logical flow to some extent of how a user enters a SharePoint instance, all the way to what site they’re accessing to the content database that is serving up the data.Web Zones – Default, Intranet, Internet, Extranet, CustomZone Polices – Deny All, Allow All, Specific groups – this is how you can blend CALs, deny all except a group of users on a particular web application or site.Authentication Providers – Integrated Windows Authentication with NTLM or Kerberos? Forms Based through an LDAP, a SQL server? A custom membership provider?Application Pools – Are you using domain accounts, local accounts to host your SharePoint instance? This is what the web application runs as. If it doesn’t have access to talk to a domain controller if you’re using domain accounts, your SharePoint instance will essentially be in the dark.Web Applications – The true way that a user really interfaces with SharePointMySites, your Collaborative sites, your Secure Content Authoring and Publishing for web content managementSite Collections – The bread and butter of SharePoint taxonomy
• #19: Available from MicrosoftYou’ll noticed that the key difference is the SSP which provides for MySites, Search, Personalization and the WCM publishing aspect of things.
• #21: When talking about taxonomy with clients, I typically talk to them about how to group things together to make best use of their site structure – organizations like your human resources, accounting, facilities, operations, etc… and also by the programs, projects, cross functional groups, and so on and so forth. More often than note we find a need for developing a taxonomy that provides for initiatives to live outside of the taxonomy of a site collection.Taxonomy of dataTaxonomy or organizationTaxonomy of functional groupsTaxonomy is can be defined as a classification schemes of like things, defining a structure of knowledge and information, providing a common language for users to better navigate.So remember, while the taxonomy of a system may be something that is planned by a technical resource, we’re working to make it available for our end users to be able to find their data most easily.Remember that we’re humans, most of the time dealing with folks that aren’t always as technical as we are. Structuring a taxonomy to be as user friendly as possible is key, especially if the user base hasn’t gone through some semblance of training is a must.
• #22: So what’s a site taxonomy?Well, a site taxonomy is composed of web applications – you may just have one such as wsshacker.net.Or you may have several, where you’ve got names such as operations.wsshacker.net, teams.wsshacker.net, my.wsshacker.net where you’ve split our content and taxonomy, and potentially functionality through the use of CNAMES in DNS. This is pretty easy to do but requires that you do a little more coordination with your DNS management team or with your service provider.Another option you might choose is the way of wsshacker.net/operations, wsshacker.net/teams, wsshacker.net/myAs you can see, all of these are completely fine, each has there benefit, each as their limitation. If you’re going to scale out extremely large where you need to partition things out, I’d recommend going the way of defined CNAMES with separated out web applications. If you’re probably going to maintain a small organization’s system with limited taxonomy changes, I’d recommend going the way of a single URL with nested site collections (the draw back being that you must create them manually).
• #23: So what’s a site taxonomy?Well, a site taxonomy is composed of web applications – you may just have one such as wsshacker.net.Or you may have several, where you’ve got names such as operations.wsshacker.net, teams.wsshacker.net, my.wsshacker.net where you’ve split our content and taxonomy, and potentially functionality through the use of CNAMES in DNS. This is pretty easy to do but requires that you do a little more coordination with your DNS management team or with your service provider.Another option you might choose is the way of wsshacker.net/operations, wsshacker.net/teams, wsshacker.net/myAs you can see, all of these are completely fine, each has there benefit, each as their limitation. If you’re going to scale out extremely large where you need to partition things out, I’d recommend going the way of defined CNAMES with separated out web applications. If you’re probably going to maintain a small organization’s system with limited taxonomy changes, I’d recommend going the way of a single URL with nested site collections (the draw back being that you must create them manually).
• #24: Web ApplicationsSite CollectionsSitesNested Site CollectionsExcluded Paths.net 2 assists greatly as it will actually pick up virtual folders and exclude them from the path use if they’re there – something that you couldn’t do with SharePoint Portal Server 2003, at least not easily.
• #25: 1 – Content types and site columns, there are boundaries that you probably don’t want to exceed.2 – System constraints and performance degradation as your content databases become enormously large. Admin DR / Search / End User ViewsWhy not just search for everything?First – Again, constraints with the number of files that reside in the document library.Second – File permissions become even more of a hassle, consider if you have a fairly large organizationThird – Authoritative data, what’s the correct copy. SharePoint unfortunately still maintains separate copies of files – no single instance, you’re increasing your storage requirements.
• #27: What about permissions? How do they affect taxonomy?Inheritance of permissions is a big reason to keep users all within a single site collectionIt’s also a reason to break things out – you’ll end up with 50 user groups that are all distributed throughoutBreaking inheritance, copies the site permissions that are existing already – if you have a huge user base, this could be an issue when you’re cleaning folks outWe had an issue with Inheritance and breaking it, primarily with Project Server
• #28: What’s that really mean? Basically the logical architecture is the model to define your SharePoint architecture, what it looks like and how it operates.
• #29: From the overarching stakeholders, gather what they are looking to do, whether it’s organizationally based, functionally, application or process basedAggregate the capabilities and begin specing out how they match to SharePoint sites and site collectionsDetermine the overarching taxonomy of sites and permissions – use Mark Miller’s Mind Map site collection builder as a model available from End User SharePointPrototype the taxonomy on a microcosm scale to ensure that it will work and that the security permissions that are in place will be usableValidate the taxonomy in a virtual environment perhaps, showing them the navigation, the overarching flow of the taxonomyImplement the taxonomy on a macrocosm level of the web application(s) and begin end user deploymentsHighly recommend Mark Miller’s tool to allow for site collection deployment buildouts.
• #30: Scope Creep… your taxonomy might start off looking like this… and end up looking like this and require you to go out and buy a migration tool or put your administrators feet to the fire.Logical Architecture and Taxonomy should begin near the beginning of the project. Without a logical architecture in place, it’s difficult to create the different mappings that determine your taxonomy and furthermore your integration into other systems.You’ll end up with something where you’ve got no rhyme or reason to your taxonomy, publishing sites mixed with team sites, mixed with site collections and random templates used across the board.While the Microsoft project plan doesn’t explicitly say Logical Architecture, it speaks to many facets of it.
• #31: There is nothing that states to build a logical architecture, but there is significant planning – all of which would get rolled into your logical architecture.While this project plan isn’t “everything” that you’re going to do – for small implementations you might not have quite as much, but for larger organizations you might go above and beyond this.
• #33: What will the system be doing?Dependent on it’s purpose determines what the roles installed may be.How big will the system be?Will you have lots of flat site collections or several deep site collectionsHow will it be accessed?What’s the authentication method? Are you on an ID island?Are we dealing with a cross domain solution?Do we need federation? Can there be a trust in place?What will the usage levels be?Will you have lots of users going through and performing searches constantly? If so, you might determine that your logical architecture and roles are slightly different.SQL mirroring or clustering?More something to note in your logical architecture to assist your implementation engineer.
• #34: The surrounding infrastructure will impact your logical architecture and taxonomy significantly. If you have a system that has a gigabit backbone switch for the SharePoint enclave itself, but you’re working in a datacenter that still uses CAT-3, you’re going to be limited.Do you have a network boundary surrounding your system? Is there a Bluecoat proxy? A load balancer available? Split DNS available?Offline - Likewise, if you’re working for a commercial company that wants to be able to take data and replicate it, this will affect your taxonomy – perhaps you’ve got locked down information sites in your taxonomy because of it with a few sites that you take offline with Groove, Infonic, Outlook or Colligo.System Memory – if you’re running x64, then you should be able do quite a bit with your SharePoint system – there are limits to how much memory you can allocate to an individual worker process.Number of sites and site collections – rule of thumb is not to go greater than 50,000 site collections within a web application, and no greater than 50,000 sites below each site collection - ~250k sites. If you see your implementation getting larger than this, you’re going to need to branch out your sites and partition them separately.DNS – Do you have management? Can you create CNAMES / A RECORDS? Are you able to create MX records?Authentication Methods – Are you forced to use a third party LDAP that has groups available? Are you utilizing something for external users?PKI / SSL / Wildcard – Do you have certificate services available? Have you defined how you will utilize SSL?Network Interfaces – Are you able to do network load balancing with multicasting? Unicasting? Do you have a limited number of IP addresses?Storage – It’s not really a logical architecture consideration, except that you probably need to consider that if you don’t have a large storage array for your environment, you may not be able to scale your system up, thereby redefining what your system is actually used for.
• #36: Courtesy of Michael T. Noel @michaeltnoelcco.com
• #38: User Experience and process drives taxonomyTaxonomy drives the logical architecture and partitioningIn turn drives your hardware requirements