SharePoint Experts, Information Architects, Expert Witness

We provide consulting in a broad array of business and technology from architecture to design to deployment of global systems with a focus on surfacing data in the enterprise. Specialists in Microsoft, we are a premier provider of SharePoint Expertise (including 2016 and Office 365). We also provide Expert Witness/Legal Expert in eDiscovery, source discovery, patent infringement, piracy and more! We also have established SICG DLDS s.a. - our counterpart in Costa Rica that specializes in water systems (http://www.crwatersolutions.com) - Contact me direct: david_sterling@sterling-consulting.com or call 704-873-8846 x704.

Search This Blog

Thursday, January 31, 2013

SharePoint 2013 Event ID 2163

After installing SharePoint 2013, you might come across an issue where you see an Application Event ID 2163 start popping up. While it may seem innocuous, it actually will cause multiple problems down the road. 

The error is effectively telling you that the SharePoint Tracing Service is not able to write to the SharePoint log folder; when it cannot, it will write to the account local folder (under AppData). This means the service will not really work because the local user folder is not accessible to other SharePoint accounts and services.

The Event ID 2163 error message will be something like this:
Tracing Service failed to create the trace log file at location specified in SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\15.0\WSS\LogDir. Error 0x0: The operation completed successfully. . Traces will be written to the following directory: C:\Users\SPSERV~1\AppData\Local\Temp\.
In the System Application Event log, the message looks like this:


As you can see, the account that's having the problem is indicated. Sorting this all out is pretty simple - what it really means is simply that the account running the service (SPService as shown) doesn't have permissions to write to the SharePoint Log file directory. 

If you open up the Services (Start > Administrative Tools > Services or Start > Run... then enter services.msc and click OK) and locate the SharePoint Tracing Service, you can determine the account in use:


The confusing part about the message is that it indicates not the physical path to the log files but the Registry setting (HKEY_LOCALMACHINE/Software/Microsoft/Shared Tools/Web Server Extensions/15.0/WSS) - remember, by default, the SharePoint logs are located in c:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\Logs but if you are working in a production environment, the logs are probably (or should be) located elsewhere. You might already know where they are but if you don't, you can easily use the setting to find them - if you open the Registry editor (Start > Run... then enter regedit then click OK) then follow the path, you will see the "LogDir" setting (FYI don't make any changes here):


As you can see, this is the default setting - %CommonProgramFiles%\Microsoft Shared\Web Server Extensions\15\LOGS\ - that is, the default path mentioned above.

Using Windows Explorer, navigate to the location specified - (by default, c:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\). Right click on the folder and select Properties then click the Security tab:


As you can see, the account in question here (SPService) is a managed account and would NOT be a member of the Administrators group so by default, it has no access here. Click the Edit... button, add the account and give it Full Rights to this folder:


Click Apply then OK then close out the properties window (Side note: it is not a bad idea to add the SharePoint Farm account with full permissions here).

Once this is completed, a reboot is suggested but not necessary and may not be possible in a production environment; you can optionally use the Services (Start > Administrative Tools > Services or Start > Run... then enter services.msc and click OK) - right click on the SharePoint Tracing Service and select Restart:


Problem solved!!

About us:
Sterling International Consulting Group is a specialist in enterprise systems including architecture, governance, business continuity, and disaster recovery planning with over 28 years in IT and management consulting. SICG also has a dedicated practice in SharePoint Technologies and Microsoft Technologies. From planning to analysis to development to deployment, SICG has the experience and knowledge to understand the best for your business – find out why: sicg@sterling-consulting.com – www.sterling-consulting.com.
Keep up with the latest articles and tips around SharePoint from author & CEO David Sterling via http://www.sharepoint-blog.com and his personal blog site:http://davidmsterling.blogspot.com or contact him directly at david_sterling@sterling-consulting.com.

Sunday, January 27, 2013

SharePoint 2013 AppFabric Event ID 1000 and Event ID 1026


In working SharePoint 2013, you may come across an issue with the permissions with the AppFabric when it is first setup. You might notice the problem right away or you may not - usually, it becomes obvious when things aren't working (like creating a site fails, etc.).

Most notably, if you check the System Event Application log, you'll see these errors repeating over and over:

Event ID 1000
Faulting application name: DistributedCacheService.exe, version: 1.0.4632.0, time stamp: 0x4eafeccf
Faulting module name: KERNELBASE.dll, version: 6.1.7601.18015, time stamp: 0x50b8479b
Exception code: 0xe0434352
Fault offset: 0x0000000000009e5d
Faulting process id: 0x524
Faulting application start time: 0x01cdfcb5fdb726f2
Faulting application path: c:\Program Files\AppFabric 1.1 for Windows Server\DistributedCacheService.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 13afe590-68aa-11e2-9254-000c2950c2a6
And the second:
Event ID 1026
Application: DistributedCacheService.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.ApplicationServer.Caching.DataCacheException
Stack:
   at Microsoft.ApplicationServer.Caching.VelocityWindowsService.StartServiceCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

FYI: Side note here - this fix applies to ALL uses of AppFabric with this problem, NOT just SharePoint.

You'll notice a couple of things here as well - you will find that the AppFabric Caching Service (using Start > Administrative Tools > Services) is set to Automatic but it's stopped and if you try to start it, the above errors will occur and the service will stop (the same errors above repeated in the System Event Application log).

The problem here is that it effects a bunch of stuff in SharePoint; for example, creating new Site Collections will fail. Attempting to deploy web parts and solutions will fail too. As well, if you check the SharePoint ULS log (c:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\logs), you'll see some errors like this:

SPDistributedCacheClusterCustomProvider:: GetValue(...)
Fail to load running IIS site with the error Access is denied
Unknown SPRequest error occurred

Using some of the AppFabric PowerShell commands (open the AppFabric PowerShell via Start > All Programs > AppFabric for Windows Server > Caching Administration Windows PowerShell), you can check the status of the 'host' using Get-CacheHost. When this displays, you'll see that it indicates a status DOWN. Using other cache commands, such as Start-CacheCluster won't work either - they will simply timeout.

This was a tough one to track down - searching for the errors through Google/Bing and the rest, there are lots of possible solutions - few that drive to a solution. However, many indicated that the error is somehow permissions based - something also indicated in the SharePoint Log (i.e. the 'Access is denied' message). After many, many hours, we tracked down the final cause - it appears that when any one of the Application Pool accounts does not have proper access to either SQL or proper rights on the front end web server, this is the result.

To fix the problem, do the following:

To begin with, check your SQL Server settings - open SQL Sever Management Studio (08 or 12), expand the Server name, then expand Security. Find the account being used for the Application Pool, right click on it and select Properties.

Under the User Mapping tab (left side), verify that the it has rights to the database used for the Application Management Service database (you can find what that is called using Central Administration > Manage Service Applications) AND that it has been granted SPDataAccess rights (double check the rights, make sure SPReadOnly is NOT checked).

Do this for ALL application pool accounts.

Return to the server and try to start the AppFabric service again. If it does not work (in our case, it did not), it turns out that it is actually in the permissions settings on the Web Server itself.

Log in to the front end server and select Start then right click on Computer and select Manage (or Start > Administrative Tools > Server Manager).

Expand the options then expand Configuration then expand Local Users and Groups.

First, find the WindowsFabricAdministrators group and make sure that the SharePoint Farm account is a member - if it is not, add it.

Next, for each of the application pool accounts, make sure they are members of the following groups:

IIS_USRS
WSS_WPG
WindowsFabricAllowedUsers

The last one was the key - once the Application Pool accounts were added to that group, the AppFabric service started all by itself - no more errors!

UPDATE: 
It appears this error can also be caused by a quirk in AppFabric - it appears that this error can ALSO be caused if using a HOSTS file instead of DNS. If you are using the HOSTS file, you should make sure that the Fully Qualified Domain Name is used - i.e. <server>.<domain>. So if you have in the hosts file:

10.0.0.10 MyServer

It should be:

10.0.0.10 MyServer.Mydomain.com

UPDATE 1/14:
Note that the WindowsFabricAdministrators and WindowsFabricAllowedUsers do not appear automatically when installed by SharePoint. These ONLY appear if the App Fabric is installed separately.



About us:
Sterling International Consulting Group is a specialist in enterprise systems including architecture, governance, business continuity, and disaster recovery planning with over 28 years in IT and management consulting. SICG also has a dedicated practice in SharePoint Technologies and Microsoft Technologies. From planning to analysis to development to deployment, SICG has the experience and knowledge to understand the best for your business – find out why: sicg@sterling-consulting.com – www.sterling-consulting.com.
Keep up with the latest articles and tips around SharePoint from author & CEO David Sterling via http://www.sharepoint-blog.com and his personal blog site:http://davidmsterling.blogspot.com or contact him directly at david_sterling@sterling-consulting.com.




Friday, January 25, 2013

SharePoint 2013 Uninstall and AppFabric Woes


At some point either during development or setup of a farm, you might have to remove SharePoint 2013 and re-install. Like 2010, the SharePoint uninstall does so pretty cleanly but with some caveats.

For one (just like 2010), the SharePoint databases are not removed. This is so that you can remove a server from the farm without affecting others. However, if you are truly removing SharePoint (i.e. a single server or to do a re-install), you have to manually delete all of the DB’s created for the various services, etc. directly using SQL Server Management Studio. This includes the SharePoint Config database.

However, SharePoint 2013 also installs a number of other services within the SharePoint 2013 Products Preparation Tool – one of these happens to be the Microsoft AppFabric 1.1 for Windows Server. AppFabric was developed for Azure and is a set of integrated technologies that makes it easier to build, scale, and manage Web and composite applications that run on IIS. SharePoint 2013 makes use of these services (thus why required) through the Distributed Cache Service.

When removing SharePoint 2013, it can have an adverse impact on the AppFabric setup. SharePoint creates its own ‘Cache node’ to coincide with SharePoint. This directly affects the web.config files across the system and sites. It doesn't ALWAYS happen but when it does, it’s a real pain.

The problem comes in when you attempt to re-install SharePoint; it would appear that the AppFabric settings are not picked up during the new installation which unfortunately causes both to fail. You can get SharePoint installed, but you’ll see problems when you attempt to add Services like the Access Service. Unfortunately, SharePoint simply gives you one of those simplistic “error has occurred” pages and looking into the ULS log doesn’t give you much. In most cases, it will simply indicate ‘access is denied’ even when you are logged in as the Farm Administrator account.

If you do have this problem, the quick fix is here is to uninstall SharePoint then uninstall AppFabric 1.1 but of course, there is a gotcha. If you attempt to remove the AppFabric, it will indicate that it will not delete the existing configuration (i.e. settings in the web.config files).

So, I am not a big fan of messing with web.config files, especially with SharePoint (since it keeps the web.config in the DB anyway), it turns out Microsoft was wise enough to provide us with a UI tool to do this. Full details can be found under the MSDN article here: http://msdn.microsoft.com/en-us/library/ff637683(v=azure.10).aspx.

More useful are the cleanup instructions found here: http://msdn.microsoft.com/en-us/library/ff637741(v=azure.10).aspx – in this article, it includes a link to the AppFabric Uninstall Cleanup Tool – this is a small application that will do the cleanup for you and prevent having to manually edit anything.

So you might think you are all set – au contraire! Turns out that the AppFabric uninstall has a bug! Oh what fun!

So to sum it up, a proper uninstall/re-install (again, when you have this problem – if the AppFabric is working, no need for this) is as follows from start to finish:



1) Uninstall SharePoint 2013 from the web front end

2) On SQL Server, delete any databases that were created by the previous installation (or locally if using SQL Express)

3) Uninstall AppFabric 1.1 from the web front end

4) Use the AppFabric Uninstall Cleanup Tool to remove the web.config entries (again on the web front end server)

5) Reboot the web front end server

6) Login to the front end server and from the SharePoint 2013 Setup, run the SharePoint 2013 Products Preparation Tool – this will detect that the AppFabric is missing and try to install it again

7) When the process completes, it will ask for a reboot; on restart it will start up again – however here’s where it gets tricky:

     a) If the process runs through, starts up on reboot and simply finishes, you are fine.

     b) If the process starts up again and tries to install the AppFabric (again) then asks for a reboot, the problem is that the AppFabric uninstall didn’t complete. In effect, the Prep tool will continually run at startup and ask for a reboot every time (but AppFabric WILL NOT be installed). So when this happens, you have to do some additional steps to clear it out:

          i) First verify in the Prep Tool message window that there were no changes

          ii) Open the “Start” folder (All Programs > Start), right click on the Microsoft SharePoint 2013 Products Preparation Tool link and delete it (Or you can do a reboot and when the Prep tool restarts, click Cancel):



          iii) Reboot the server – the Prep tool should not restart

     c) Now back to why AppFabric has a problem – it turns out that in the background, the AppFabric installation is failing with an “MSI Error 1603 installing AppFabric 1.1” (you’d see that if you tried to install AppFabric by itself) – to fix this:


          i) Open the Registry Editor (Start > Run… > Regedit) (disclaimer: make a backup before you make changes – if you don’t know how, you should get someone that does!)

          ii) In the registry editor, find the ‘PSModulePath’ ** key HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Control/Session Manager/Enviroment/PSModulePath – you can search for PSModulePath but make SURE you have the path correct (there is more than one!)

          iii) Delete the PSModulePath key:



          iv) Reboot the server

          v) Login to the front end server and from the SharePoint 2013 Setup, run the SharePoint 2013 Products Preparation Tool – this should again detect that the AppFabric is missing and try to install it again – when complete it will request a reboot

          vi) On the restart, login again – this time the Prep tool should complete and when you click Finish will NOT require a reboot

** BTW: Kudos to Lucas Massena for finding the AppFabric 1.1 uninstall bug fix:  http://social.msdn.microsoft.com/profile/lucasmassena/?type=forum&referrer=http://social.msdn.microsoft.com/Forums/en/velocity/thread/561f3ad4-14ef-4d26-b79a-bef8e1376d64


8) When the Prep tool completes successfully, you can re-install SharePoint from scratch including creating new databases for the install


Good luck on this one!





 About us:
Sterling International Consulting Group is a specialist in enterprise systems including architecture, governance, business continuity, and disaster recovery planning with over 28 years in IT and management consulting. SICG also has a dedicated practice in SharePoint Technologies and Microsoft Technologies. From planning to analysis to development to deployment, SICG has the experience and knowledge to understand the best for your business – find out why: sicg@sterling-consulting.com – www.sterling-consulting.com.
Keep up with the latest articles and tips around SharePoint from author & CEO David Sterling via http://www.sharepoint-blog.com and his personal blog site:http://davidmsterling.blogspot.com or contact him directly at david_sterling@sterling-consulting.com.



SharePoint 2013 Application Pool Account User Profile


As was true with SharePoint 2010, it is very important that the account used for the SharePoint Application pools has a valid Domain User Profile to operate. This is particularly needed in 2013 as it can cause issues when deploying applications.

The error appears in the Event Application Log on the SharePoint front end server. Usually indicating that the account (the application pool account) was logged in with a temporary profile and that all changes will be lost when logged out (the account name is specified in the error).

Fixing this issue is usually easily done simply by logging into the SharePoint front end console (i.e. ON THE BOX) with the application pool account. Assuming all is well, a new profile folder is created in the c:\Users folder. If needed (though this doesn’t always work), you can add the account to the Remote Desktop Users group and try to use Remote Desktop to login that way (remember to remove that right after the fact!).

However, that does not always fix the problem – I’ve had cases where even after repeated login attempts, the error still shows. As well, when logging in with the account, Windows will show a quick ‘popup’ message that indicates it is using a temporary profile. Further, if you reboot the SharePoint server (or run an IISReset), you may still see the Event Application Log error when a SharePoint site is opened (i.e. the application pool is ‘logged in’).

First of all, changing the account or permissions won’t fix the problem – what it means is that there is a corrupt profile for that account. Second, the problem will not fix itself.

I’ve used many ways to correct the problem but found the “perfect” fix:

Quick disclaimer: Backup the registry before you edit it or work with someone that knows how to use it properly.

Start off using an administration account and repeat the following process for every server in the farm (note: I've had to do this on the Domain Controller too!):


a.       Open the C:\Users folder and look for a folder under the same name as the account (i.e. SPAppPool) – if you DO NOT see it, log out and go on to the next server

b.      Zip the folder up if you want to (though not necessary) and delete the folder

c.       Using Start > Run… to open a command line (and so you are running as administrator), enter REGEDIT and run it

d.      When the Registry Editor opens, open the following subkey: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList

e.      Under this subkey, you will see a list of the accounts – each Key is based on the “SID” (the unique ID) for each account profile:

f.        When you select each subkey, you will see the name of the account under the “ProfileImagePath” value – find the one that belongs to the application pool account and delete the entire key (right click on the S-1… key and select Delete) then close out the editor

When you have completed the above on every server in the Farm, do the following:
  1. Next log in to the SharePoint front end server using an Administrator account
  2. Click Start then right click on Computer and select Manage – this will open the Server Manager page (or click it in the tool bar if available)
  3. Expand Configuration then Local Users and Groups then expand Groups
  4. Right click on the Administrators group and select Add to Group – add the application pool account in question
  5. Log out of the server and log back in using the application pool account (it should take a few minutes); when the user profile is created and desktop setup, you should see the standard Server Manager page that pops up for Administrators (you can close that)
  6. Open Windows Explorer and open c:\Users – verify that the account you are logged in as has a new folder there (that indicates the profile was created successfully)
  7. Assuming all is well, log out and log back in with an Administrator account
  8. Click Start then right click on Computer and select Manage – this will open the Server Manager page (or click it in the tool bar if available)
  9. Expand Configuration then Local Users and Groups then expand Groups then the Administrators group
  10. Remove the application pool account in question
Note that if you added the account to the Remote Desktop group, you should remove it.


Once you have completed, you can verify that all is well:
  1. Login to the SharePoint front end using an Administrator/Farm account
  2. Close any open browser windows
  3. Clear the Event Application log (if you can)
  4. Use the Start > Run… command (so you are running as an administrator), enter iisreset and click OK
  5. Open the Central Administration site
  6. Check the Event Application log (refresh if you have to) and verify that the error is no more

Tuesday, January 22, 2013

SharePoint Error: Verify each User Profile Application has an associated Search Service Connection


In both SharePoint 2010 and 2013, you might come across this message in the Health Analyzer. Basically this indicates that one or more application doesn't have a Search Service Connection with it (most times it is the User Profile Service).

As it turns out, this may or may not be correct!

First of all, ‘Services’ are assigned to each application. This is what makes SharePoint “multi-purpose” and “multi-tenant” (just in case you don’t know, there can be different ‘groups’ of services for specific purpose – for example: common services (BDC, MMS, etc.) across two applications but separate search for each as in an intranet/extranet).

There is always a group of these called ‘default’. When a Service Application is created, it is either assigned as default or not. Other groups (considered ‘custom’) are assigned at the Application level when the app is created. So easiest to think of it this way – you can create as many (or few) services you need then spread them out across applications; each application can have the same or different settings.

To see these settings is simple – open your Central Administration Site, click on the Manage Web Applications link to open the management page:




Click the name of the application in question to select it (as shown, SharePoint – 80) then click Service Connections in the ribbon:

From here you can select the group – default is of course, ‘default’. Selecting [custom] allows you to create a new group where you can change which Services are available to this application. For example, you might not want Excel available:


So back to the error - Two conditions appear to generate it:

  1. Indeed there is not a Search Service assigned to the particular application – the fix is to simply add it. This is only typical if custom groupings have been used.
  2. One of:
a.       The Search Service was created and not assigned as default
b.      More than one Search Service was created and none are assigned as default
c.       The Search Service was rebuilt at some point and not assigned as default

If it is any one of #2 above, the error will remain until a Search Service is assigned to the default group.
Now comes in the problem – you can easily reassign services as you saw above, however, you CANNOT add a service to the default group without using PowerShell (thanks guys…).

Within PowerShell, you have the ability to add and remove Service assignments using the “Add-SPServiceApplicationProxyGroupMember” and “Remove-SPServiceApplicationProxyGroupMember” commands. However, there is a gotcha when trying to add to the default – in reality the ‘default’ group has no ‘identity’ name (it’s blank).

Rather than reproduce the results of two excellent posts (kudos to you both) – see the following two articles (note that the same steps work for both 2010 and 2013):

Associating Services to Custom Application Proxy Groups in SharePoint 2010: http://ekhanna.wordpress.com/tag/add-spserviceapplicationproxygroupmember/

And for dealing with how to use the Default Group: http://pinchii.com/home/tag/add-spserviceapplicationproxygroupmember/

Hopefully this will save you some time and headache!