SharePoint Experts, Information Architects, Expert Witness

SICG provides a broad array of business and technology consulting from architecture to design to deployment of global systems with a focus on surfacing data in the enterprise. We focus on the "How", not just the possible. Contact me direct: david_sterling@sterling-consulting.com or call 704-873-8846 x704.

Search This Blog

Sunday, January 27, 2013

SharePoint 2013 AppFabric Event ID 1000 and Event ID 1026


In working SharePoint 2013, you may come across an issue with the permissions with the AppFabric when it is first setup. You might notice the problem right away or you may not - usually, it becomes obvious when things aren't working (like creating a site fails, etc.).

Most notably, if you check the System Event Application log, you'll see these errors repeating over and over:

Event ID 1000
Faulting application name: DistributedCacheService.exe, version: 1.0.4632.0, time stamp: 0x4eafeccf
Faulting module name: KERNELBASE.dll, version: 6.1.7601.18015, time stamp: 0x50b8479b
Exception code: 0xe0434352
Fault offset: 0x0000000000009e5d
Faulting process id: 0x524
Faulting application start time: 0x01cdfcb5fdb726f2
Faulting application path: c:\Program Files\AppFabric 1.1 for Windows Server\DistributedCacheService.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 13afe590-68aa-11e2-9254-000c2950c2a6
And the second:
Event ID 1026
Application: DistributedCacheService.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.ApplicationServer.Caching.DataCacheException
Stack:
   at Microsoft.ApplicationServer.Caching.VelocityWindowsService.StartServiceCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

FYI: Side note here - this fix applies to ALL uses of AppFabric with this problem, NOT just SharePoint.

You'll notice a couple of things here as well - you will find that the AppFabric Caching Service (using Start > Administrative Tools > Services) is set to Automatic but it's stopped and if you try to start it, the above errors will occur and the service will stop (the same errors above repeated in the System Event Application log).

The problem here is that it effects a bunch of stuff in SharePoint; for example, creating new Site Collections will fail. Attempting to deploy web parts and solutions will fail too. As well, if you check the SharePoint ULS log (c:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\logs), you'll see some errors like this:

SPDistributedCacheClusterCustomProvider:: GetValue(...)
Fail to load running IIS site with the error Access is denied
Unknown SPRequest error occurred

Using some of the AppFabric PowerShell commands (open the AppFabric PowerShell via Start > All Programs > AppFabric for Windows Server > Caching Administration Windows PowerShell), you can check the status of the 'host' using Get-CacheHost. When this displays, you'll see that it indicates a status DOWN. Using other cache commands, such as Start-CacheCluster won't work either - they will simply timeout.

This was a tough one to track down - searching for the errors through Google/Bing and the rest, there are lots of possible solutions - few that drive to a solution. However, many indicated that the error is somehow permissions based - something also indicated in the SharePoint Log (i.e. the 'Access is denied' message). After many, many hours, we tracked down the final cause - it appears that when any one of the Application Pool accounts does not have proper access to either SQL or proper rights on the front end web server, this is the result.

To fix the problem, do the following:

To begin with, check your SQL Server settings - open SQL Sever Management Studio (08 or 12), expand the Server name, then expand Security. Find the account being used for the Application Pool, right click on it and select Properties.

Under the User Mapping tab (left side), verify that the it has rights to the database used for the Application Management Service database (you can find what that is called using Central Administration > Manage Service Applications) AND that it has been granted SPDataAccess rights (double check the rights, make sure SPReadOnly is NOT checked).

Do this for ALL application pool accounts.

Return to the server and try to start the AppFabric service again. If it does not work (in our case, it did not), it turns out that it is actually in the permissions settings on the Web Server itself.

Log in to the front end server and select Start then right click on Computer and select Manage (or Start > Administrative Tools > Server Manager).

Expand the options then expand Configuration then expand Local Users and Groups.

First, find the WindowsFabricAdministrators group and make sure that the SharePoint Farm account is a member - if it is not, add it.

Next, for each of the application pool accounts, make sure they are members of the following groups:

IIS_USRS
WSS_WPG
WindowsFabricAllowedUsers

The last one was the key - once the Application Pool accounts were added to that group, the AppFabric service started all by itself - no more errors!

UPDATE: 
It appears this error can also be caused by a quirk in AppFabric - it appears that this error can ALSO be caused if using a HOSTS file instead of DNS. If you are using the HOSTS file, you should make sure that the Fully Qualified Domain Name is used - i.e. <server>.<domain>. So if you have in the hosts file:

10.0.0.10 MyServer

It should be:

10.0.0.10 MyServer.Mydomain.com

UPDATE 1/14:
Note that the WindowsFabricAdministrators and WindowsFabricAllowedUsers do not appear automatically when installed by SharePoint. These ONLY appear if the App Fabric is installed separately.



About us:
Sterling International Consulting Group is a specialist in enterprise systems including architecture, governance, business continuity, and disaster recovery planning with over 28 years in IT and management consulting. SICG also has a dedicated practice in SharePoint Technologies and Microsoft Technologies. From planning to analysis to development to deployment, SICG has the experience and knowledge to understand the best for your business – find out why: sicg@sterling-consulting.com – www.sterling-consulting.com.
Keep up with the latest articles and tips around SharePoint from author & CEO David Sterling via http://www.sharepoint-blog.com and his personal blog site:http://davidmsterling.blogspot.com or contact him directly at david_sterling@sterling-consulting.com.




Post a Comment