SharePoint Experts, Information Architects, Expert Witness

We provide consulting in a broad array of business and technology from architecture to design to deployment of global systems with a focus on surfacing data in the enterprise. Specialists in Microsoft, we are a premier provider of SharePoint Expertise (including 2016 and Office 365). We also provide Expert Witness/Legal Expert in eDiscovery, source discovery, patent infringement, piracy and more! We also have established SICG DLDS s.a. - our counterpart in Costa Rica that specializes in water systems (http://www.crwatersolutions.com) - Contact me direct: david_sterling@sterling-consulting.com or call 704-873-8846 x704.

Search This Blog

Sunday, January 27, 2013

SharePoint 2013 AppFabric Event ID 1000 and Event ID 1026


In working SharePoint 2013, you may come across an issue with the permissions with the AppFabric when it is first setup. You might notice the problem right away or you may not - usually, it becomes obvious when things aren't working (like creating a site fails, etc.).

Most notably, if you check the System Event Application log, you'll see these errors repeating over and over:

Event ID 1000
Faulting application name: DistributedCacheService.exe, version: 1.0.4632.0, time stamp: 0x4eafeccf
Faulting module name: KERNELBASE.dll, version: 6.1.7601.18015, time stamp: 0x50b8479b
Exception code: 0xe0434352
Fault offset: 0x0000000000009e5d
Faulting process id: 0x524
Faulting application start time: 0x01cdfcb5fdb726f2
Faulting application path: c:\Program Files\AppFabric 1.1 for Windows Server\DistributedCacheService.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 13afe590-68aa-11e2-9254-000c2950c2a6
And the second:
Event ID 1026
Application: DistributedCacheService.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.ApplicationServer.Caching.DataCacheException
Stack:
   at Microsoft.ApplicationServer.Caching.VelocityWindowsService.StartServiceCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

FYI: Side note here - this fix applies to ALL uses of AppFabric with this problem, NOT just SharePoint.

You'll notice a couple of things here as well - you will find that the AppFabric Caching Service (using Start > Administrative Tools > Services) is set to Automatic but it's stopped and if you try to start it, the above errors will occur and the service will stop (the same errors above repeated in the System Event Application log).

The problem here is that it effects a bunch of stuff in SharePoint; for example, creating new Site Collections will fail. Attempting to deploy web parts and solutions will fail too. As well, if you check the SharePoint ULS log (c:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\logs), you'll see some errors like this:

SPDistributedCacheClusterCustomProvider:: GetValue(...)
Fail to load running IIS site with the error Access is denied
Unknown SPRequest error occurred

Using some of the AppFabric PowerShell commands (open the AppFabric PowerShell via Start > All Programs > AppFabric for Windows Server > Caching Administration Windows PowerShell), you can check the status of the 'host' using Get-CacheHost. When this displays, you'll see that it indicates a status DOWN. Using other cache commands, such as Start-CacheCluster won't work either - they will simply timeout.

This was a tough one to track down - searching for the errors through Google/Bing and the rest, there are lots of possible solutions - few that drive to a solution. However, many indicated that the error is somehow permissions based - something also indicated in the SharePoint Log (i.e. the 'Access is denied' message). After many, many hours, we tracked down the final cause - it appears that when any one of the Application Pool accounts does not have proper access to either SQL or proper rights on the front end web server, this is the result.

To fix the problem, do the following:

To begin with, check your SQL Server settings - open SQL Sever Management Studio (08 or 12), expand the Server name, then expand Security. Find the account being used for the Application Pool, right click on it and select Properties.

Under the User Mapping tab (left side), verify that the it has rights to the database used for the Application Management Service database (you can find what that is called using Central Administration > Manage Service Applications) AND that it has been granted SPDataAccess rights (double check the rights, make sure SPReadOnly is NOT checked).

Do this for ALL application pool accounts.

Return to the server and try to start the AppFabric service again. If it does not work (in our case, it did not), it turns out that it is actually in the permissions settings on the Web Server itself.

Log in to the front end server and select Start then right click on Computer and select Manage (or Start > Administrative Tools > Server Manager).

Expand the options then expand Configuration then expand Local Users and Groups.

First, find the WindowsFabricAdministrators group and make sure that the SharePoint Farm account is a member - if it is not, add it.

Next, for each of the application pool accounts, make sure they are members of the following groups:

IIS_USRS
WSS_WPG
WindowsFabricAllowedUsers

The last one was the key - once the Application Pool accounts were added to that group, the AppFabric service started all by itself - no more errors!

UPDATE: 
It appears this error can also be caused by a quirk in AppFabric - it appears that this error can ALSO be caused if using a HOSTS file instead of DNS. If you are using the HOSTS file, you should make sure that the Fully Qualified Domain Name is used - i.e. <server>.<domain>. So if you have in the hosts file:

10.0.0.10 MyServer

It should be:

10.0.0.10 MyServer.Mydomain.com

UPDATE 1/14:
Note that the WindowsFabricAdministrators and WindowsFabricAllowedUsers do not appear automatically when installed by SharePoint. These ONLY appear if the App Fabric is installed separately.



About us:
Sterling International Consulting Group is a specialist in enterprise systems including architecture, governance, business continuity, and disaster recovery planning with over 28 years in IT and management consulting. SICG also has a dedicated practice in SharePoint Technologies and Microsoft Technologies. From planning to analysis to development to deployment, SICG has the experience and knowledge to understand the best for your business – find out why: sicg@sterling-consulting.com – www.sterling-consulting.com.
Keep up with the latest articles and tips around SharePoint from author & CEO David Sterling via http://www.sharepoint-blog.com and his personal blog site:http://davidmsterling.blogspot.com or contact him directly at david_sterling@sterling-consulting.com.




14 comments:

Sebastian Sageder said...

Thank you David for your great post. We experience the same errors in our farm. But I cannot find the groups WindowsFabricAdministrators and WindowsFabricAllowedUsers. Can you give me some advice which permissions these groups have and when these groups are created?

Anonymous said...

Unfortunately, none of these solutions worked for me. However, I solved it using the following method:

Grant-CacheAllowedClientAccount -Account "DOMAINNAME\username"
then
Remove-SPDistributedCacheServiceInstance
then
Add-SPDistributedCacheServiceInstance

Reference: http://msdn.microsoft.com/en-us/library/ff921031(v=azure.10).aspx
and
http://msdn.microsoft.com/en-us/library/ff921012(v=azure.10).aspx

This is for SPF2013. Hope this helps

Anonymous said...

Unfortunately, none of these solutions worked for me. However, I solved it using the following method:

Grant-CacheAllowedClientAccount -Account "DOMAINNAME\username"
then
Remove-SPDistributedCacheServiceInstance
then
Add-SPDistributedCacheServiceInstance

Reference: http://msdn.microsoft.com/en-us/library/ff921031(v=azure.10).aspx
and
http://msdn.microsoft.com/en-us/library/ff921012(v=azure.10).aspx

This is for SPF2013. Hope this helps

Chris Grist said...

Thanks Anon, your solution worked for me!

Anonymous said...

None of the solutions worked for me. So I had to export the cache configuration by Export-CacheClusterConfig -File c:\CurrentClusterConfig.xml, edit it manually by setting the host name to FQDN and reimport it Import-CacheClusterConfig -File c:\CurrentClusterConfig.xml. That finally fired the service up.

Miki said...

Hi,

I am having the same trouble as you had. I've tried all the steps mentioned and i could not solve it. Can you tell me how did you import the modified config file with the new server name in it?

I can do the export step
Then i can cahnge the old server name for the new one
When i try to import the config file using import-cacheclusterconfig i get the next error:
import-cacheclusterconfig : Referencia a objeto no establecida como instancia
de un objeto.
En línea: 1 Carácter: 1
+ import-cacheclusterconfig -file E:\sharepoint\AppFabric\Cache
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Import-AFCacheClusterConfigur
ation], NullReferenceException
+ FullyQualifiedErrorId : System.NullReferenceException,Microsoft.Applicat
ionServer.Caching.Commands.ImportAFCacheClusterConfigurationCommand

Does anybody know how can i import back the modified config file?

Thanks in advance

David M. Sterling said...

It looks like something is wrong with your config; like the format of the file or some unknown character. You can always try to remove AppFabric manually and re-install.

Anonymous said...

Commenting out all rows in hosts file made AppFabric service available again.
Thank you!

Anonymous said...

I have a problem with AppFabric in the Web Server (it is a virtual machine which run on a Production Server ). The problem is that when I run Get-CacheHost i get: Service Status: UNKNOWN (Even ID 1000 and 1026)

However, when i load that same virtual machine in the Testing Server and run Get-CacheHost, the Service Status: UP

I re-installed the virtual WEB server from scratch (Win2008 R2 Cd) in the production environment and still says UNKNOWN.
Any idea thanks for your help.

By Edino

David M. Sterling said...

Getting an unknown status - "Watchman" supplied additional information:

In case it helps for somebody else.
After trying almost everything even to re-install windows from scratch. I still got the Unknown status. The solution, i got it from this link: [http://www.codeproject.com/Questions/288275/NET-runtime-error|leo://plh/http%3A*3*3www%2Ecodeproject%2Ecom*3Questions*3288275*3NET-runtime-error/PnaF?_t=tracking_disc] //////////////---------
This doesn't seem to be .Net Framework issue. This error might be due to some important system dlls missing in your computer. Try A repair tools to fix the issue, 39453.html[^]
//////////////---------

I run cCleaner Professional on the Hyper V server. It is working now

joan said...

Hi I am receiving the following error message when re-joining to the workflow farm. I have checked and I also do not see the windowsfabricallowedusers group and I understand this is due to app fabric that was installed with SharePoint, but how do I get the groups now that I already have SharePoint installed? See error message and please respond asap.
[Verbose] [2016-07-10 11:08:18 AM]: Validating input and configuration parameters.
[Progress] [2016-07-10 11:08:18 AM]: Validating input and configuration parameters.
[Verbose] [2016-07-10 11:08:19 AM]: Installing auto-generated certificate.
[Progress] [2016-07-10 11:08:19 AM]: Installing auto-generated certificate.
[Verbose] [2016-07-10 11:08:22 AM]: Granting 'Log on as Service' privilege to the run as account.
[Progress] [2016-07-10 11:08:22 AM]: Granting 'Log on as Service' privilege to the run as account.
[Verbose] [2016-07-10 11:08:22 AM]: Windows Fabric configuration started.
[Progress] [2016-07-10 11:08:22 AM]: Windows Fabric configuration started.
[Error] [2016-07-10 11:08:22 AM]: System.Management.Automation.CmdletInvocationException: Group WindowsFabricAllowedUsers not found. ---> System.UnauthorizedAccessException: Group WindowsFabricAllowedUsers not found.
at Microsoft.ServiceBus.Commands.WinfabDeploymentHelper.UpdateWindowsFabricGroups(String runAsAccount, Boolean add)
at Microsoft.ServiceBus.Commands.AddSBHost.ConfigureWinfabService(String runAsAccount, String clusterManifestFilePath)
--- End of inner exception stack trace ---
at System.Management.Automation.PowerShell.EndInvoke(IAsyncResult asyncResult)
at Microsoft.ServiceBus.ConfigWizard.CommandletHelper.InvokePowershell(Command command, Action`3 updateProgress)
at Microsoft.ServiceBus.ConfigWizard.ProgressPageViewModel.AddSBNode(FarmCreationModel model, Boolean isFirstCommand)

Anonymous said...

Please advise re-joining an existing farm is throwing errors:

[Verbose] [2016-07-10 11:08:18 AM]: Validating input and configuration parameters.
[Progress] [2016-07-10 11:08:18 AM]: Validating input and configuration parameters.
[Verbose] [2016-07-10 11:08:19 AM]: Installing auto-generated certificate.
[Progress] [2016-07-10 11:08:19 AM]: Installing auto-generated certificate.
[Verbose] [2016-07-10 11:08:22 AM]: Granting 'Log on as Service' privilege to the run as account.
[Progress] [2016-07-10 11:08:22 AM]: Granting 'Log on as Service' privilege to the run as account.
[Verbose] [2016-07-10 11:08:22 AM]: Windows Fabric configuration started.
[Progress] [2016-07-10 11:08:22 AM]: Windows Fabric configuration started.
[Error] [2016-07-10 11:08:22 AM]: System.Management.Automation.CmdletInvocationException: Group WindowsFabricAllowedUsers not found. ---> System.UnauthorizedAccessException: Group WindowsFabricAllowedUsers not found.
at Microsoft.ServiceBus.Commands.WinfabDeploymentHelper.UpdateWindowsFabricGroups(String runAsAccount, Boolean add)
at Microsoft.ServiceBus.Commands.AddSBHost.ConfigureWinfabService(String runAsAccount, String clusterManifestFilePath)
--- End of inner exception stack trace ---
at System.Management.Automation.PowerShell.EndInvoke(IAsyncResult asyncResult)
at Microsoft.ServiceBus.ConfigWizard.CommandletHelper.InvokePowershell(Command command, Action`3 updateProgress)
at Microsoft.ServiceBus.ConfigWizard.ProgressPageViewModel.AddSBNode(FarmCreationModel model, Boolean isFirstCommand)

David M. Sterling said...

Are you attempting to install Workflow Manager?

David M. Sterling said...

ASSUMING you are installing on the SAME BOX as SharePoint? IF So, uninstall workflow manager. Remove the workflow service from SharePoint Central Admin, double check to ensure your permissions on accounts (make sure the Farm Account is LOCAL ADMINISTRATOR on the system), make sure all account permissions are correct on SharePoint AND SQL Server. Reboot, TEMPORARILY add the account used for your Application Pool to the LOCAL ADMINISTRATOR group. Check the logs for any errors and make sure you fix them first. Create a group (if you haven't already) for the Workflow Administrators and Workflow Users) adding the SharePoint Farm Account to the workflow administrators group and other SP accounts to the workflow users group. Create a new empty site to be the Workflow site (a subsite), install Workflow Manager using GROUPS and NOT an individual account - assuming that works, check logs for errors. If all well, use the Configuration wizard to add the workflow service. Join the farm and you should be good.