AnsweredAssumed Answered

AlfrescoRuntimeException: GetModelsDiff return stats is 404

Question asked by callen on Dec 29, 2012
Latest reply on Jan 8, 2013 by romschn
I have been running 4.2.b since 12/2/12 with no problems. Then yesterday 12/27/12 share refused to allow users to log in. (The server is restarted automatically each night)

The log  seems to indicate an error with the solr index, but I really do not have enough knowledge to diagnose. I have copied the alf_data to another machine, and the problem is duplicated on the 2nd machine, which would seem to indicate the error is not related to the machine or environment, but within the alf_data.

Please help!! – I will post a copy of the log. – We are shut down at the the time.

2012-12-28 19:51:56 Commons Daemon procrun stdout initialized
Listening for transport dt_socket at address: 8000
2012-12-28 19:52:30,511  WARN  [handler.component.SpellCheckComponent] [localhost-startStop-1] No queryConverter defined, using default converter
2012-12-28 19:52:39,011  WARN  [handler.component.SpellCheckComponent] [localhost-startStop-1] No queryConverter defined, using default converter
2012-12-28 19:54:18,011  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'sysAdmin' subsystem, ID: [sysAdmin, default]
2012-12-28 19:54:18,448  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'sysAdmin' subsystem, ID: [sysAdmin, default] complete
2012-12-28 19:55:35,526  INFO  [extensions.webscripts.TemplateProcessorRegistry] [localhost-startStop-1] Registered template processor Repository Template Processor for extension ftl
2012-12-28 19:55:35,542  INFO  [extensions.webscripts.ScriptProcessorRegistry] [localhost-startStop-1] Registered script processor Repository Script Processor for extension js
2012-12-28 19:56:15,714  INFO  [domain.schema.SchemaBootstrap] [localhost-startStop-1] Connecting to database: jdbc:postgresql://localhost:5432/alfresco, UserName=alfresco, PostgreSQL Native Driver
2012-12-28 19:56:15,714  INFO  [domain.schema.SchemaBootstrap] [localhost-startStop-1] Schema managed by database dialect org.hibernate.dialect.PostgreSQLDialect.
2012-12-28 19:56:25,698  INFO  [domain.schema.SchemaBootstrap] [localhost-startStop-1] No changes were made to the schema.
2012-12-28 19:56:29,870  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'Search' subsystem, ID: [Search, managed, solr]
2012-12-28 19:56:30,729  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'Search' subsystem, ID: [Search, managed, solr] complete
2012-12-28 19:56:31,886  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'thirdparty' subsystem, ID: [thirdparty, default]
2012-12-28 19:56:33,807  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'thirdparty' subsystem, ID: [thirdparty, default] complete
2012-12-28 19:56:33,807  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'OOoDirect' subsystem, ID: [OOoDirect, default]
2012-12-28 19:56:36,776  WARN  [alfresco.util.OpenOfficeConnectionTester] [localhost-startStop-1] An initial OpenOffice connection could not be established.
2012-12-28 19:56:36,823  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'OOoDirect' subsystem, ID: [OOoDirect, default] complete
2012-12-28 19:56:36,839  INFO  [repo.admin.ConfigurationChecker] [localhost-startStop-1] The Alfresco root data directory ('dir.root') is: C:\Alfresco\alf_data
2012-12-28 19:56:36,964  INFO  [admin.patch.PatchExecuter] [localhost-startStop-1] Checking for patches to apply …
2012-12-28 19:56:40,323  INFO  [admin.patch.PatchExecuter] [localhost-startStop-1] No patches were required.
2012-12-28 19:56:40,370  INFO  [repo.module.ModuleServiceImpl] [localhost-startStop-1] Found 3 module(s).
2012-12-28 19:56:40,761  INFO  [repo.module.ModuleServiceImpl] [localhost-startStop-1] Starting module 'alfresco-mm-repo' version 0.1.5.9.
2012-12-28 19:56:40,979  INFO  [repo.module.ModuleServiceImpl] [localhost-startStop-1] Starting module 'org.alfresco.module.vti' version 1.2.
2012-12-28 19:56:41,104  INFO  [repo.module.ModuleServiceImpl] [localhost-startStop-1] Starting module 'org.alfresco.integrations.google.docs' version 2.0.0.
2012-12-28 19:56:41,182  WARN  [repo.module.ModuleServiceImpl] [localhost-startStop-1] A previously-installed module 'org_alfresco_module_wcmquickstart' (version 1.0) is not present in your distribution.
2012-12-28 19:56:41,198  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'fileServers' subsystem, ID: [fileServers, default]
2012-12-28 19:56:44,182  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'Authentication' subsystem, ID: [Authentication, managed, alfrescoNtlm1]
2012-12-28 19:56:44,886  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'Authentication' subsystem, ID: [Authentication, managed, alfrescoNtlm1] complete
2012-12-28 19:56:45,151  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'fileServers' subsystem, ID: [fileServers, default] complete
2012-12-28 19:56:45,151  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'imap' subsystem, ID: [imap, default]
2012-12-28 19:56:46,339  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'imap' subsystem, ID: [imap, default] complete
2012-12-28 19:56:46,339  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'email' subsystem, ID: [email, outbound]
2012-12-28 19:56:46,636  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'email' subsystem, ID: [email, outbound] complete
2012-12-28 19:56:46,636  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'email' subsystem, ID: [email, inbound]
2012-12-28 19:56:47,120  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'email' subsystem, ID: [email, inbound] complete
2012-12-28 19:56:47,120  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'googledocs' subsystem, ID: [googledocs, default]
2012-12-28 19:56:47,401  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'googledocs' subsystem, ID: [googledocs, default] complete
2012-12-28 19:56:47,401  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'Subscriptions' subsystem, ID: [Subscriptions, default]
2012-12-28 19:56:47,479  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'Subscriptions' subsystem, ID: [Subscriptions, default] complete
2012-12-28 19:56:47,495  INFO  [repo.usage.UserUsageTrackingComponent] [localhost-startStop-1] Disabled - clear non-missing user usages …
2012-12-28 19:56:48,011  INFO  [repo.usage.UserUsageTrackingComponent] [localhost-startStop-1] Found 0 users to clear
2012-12-28 19:56:48,011  INFO  [repo.usage.UserUsageTrackingComponent] [localhost-startStop-1] … cleared non-missing usages for 0 users
2012-12-28 19:56:48,011  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'Synchronization' subsystem, ID: [Synchronization, default]
2012-12-28 19:56:48,370  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'Synchronization' subsystem, ID: [Synchronization, default] complete
2012-12-28 19:56:49,417  ERROR [web.context.ContextLoader] [localhost-startStop-1] Context initialization failed
org.alfresco.service.cmr.repository.InvalidNodeRefException: Node does not exist: workspace://SpacesStore/50e3a132-62f0-45e3-832e-e090f1d749e6(null)
   at org.alfresco.repo.node.db.DbNodeServiceImpl.getNodePairNotNull(DbNodeServiceImpl.java:178)
   at org.alfresco.repo.node.db.DbNodeServiceImpl.getChildAssocs(DbNodeServiceImpl.java:1780)
   at sun.reflect.GeneratedMethodAccessor277.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:601)
   at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
   at org.alfresco.repo.tenant.MultiTNodeServiceInterceptor.invoke(MultiTNodeServiceInterceptor.java:105)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
   at $Proxy7.getChildAssocs(Unknown Source)
   at sun.reflect.GeneratedMethodAccessor277.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:601)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy35.getChildAssocs(Unknown Source)
   at org.alfresco.repo.security.authority.AuthorityDAOImpl.isAuthorityContained(AuthorityDAOImpl.java:1003)
   at org.alfresco.repo.security.authority.AuthorityDAOImpl.isAuthorityContained(AuthorityDAOImpl.java:1012)
   at org.alfresco.repo.security.authority.AuthorityDAOImpl.getContainingAuthoritiesInZone(AuthorityDAOImpl.java:802)
   at org.alfresco.repo.security.authority.AuthorityServiceImpl.getContainingAuthoritiesInZone(AuthorityServiceImpl.java:551)
   at org.alfresco.repo.site.SiteServiceImpl.listSitesImpl(SiteServiceImpl.java:1040)
   at org.alfresco.repo.site.SiteServiceImpl.listSites(SiteServiceImpl.java:934)
   at org.alfresco.repo.site.SiteServiceImpl.listSites(SiteServiceImpl.java:911)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:601)
   at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
   at net.sf.acegisecurity.intercept.method.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:80)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
   at org.alfresco.repo.security.permissions.impl.ExceptionTranslatorMethodInterceptor.invoke(ExceptionTranslatorMethodInterceptor.java:46)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
   at org.alfresco.repo.audit.AuditMethodInterceptor.invoke(AuditMethodInterceptor.java:159)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
   at org.alfresco.repo.transaction.CheckTransactionAdvice.invoke(CheckTransactionAdvice.java:47)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
   at org.alfresco.repo.transaction.RetryingTransactionAdvice$1.execute(RetryingTransactionAdvice.java:64)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:433)
   at org.alfresco.repo.transaction.RetryingTransactionAdvice.invoke(RetryingTransactionAdvice.java:67)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
   at $Proxy89.listSites(Unknown Source)
   at org.alfresco.repo.site.SiteServiceBootstrap$1.doWork(SiteServiceBootstrap.java:63)
   at org.alfresco.repo.security.authentication.AuthenticationUtil.runAs(AuthenticationUtil.java:529)
   at org.alfresco.repo.site.SiteServiceBootstrap.onBootstrap(SiteServiceBootstrap.java:59)
   at org.springframework.extensions.surf.util.AbstractLifecycleBean.onApplicationEvent(AbstractLifecycleBean.java:56)
   at org.alfresco.repo.management.SafeApplicationEventMulticaster.multicastEventInternal(SafeApplicationEventMulticaster.java:209)
   at org.alfresco.repo.management.SafeApplicationEventMulticaster.multicastEvent(SafeApplicationEventMulticaster.java:180)
   at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:303)
   at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:911)
   at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:428)
   at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:276)
   at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:197)
   at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:47)
   at org.alfresco.web.app.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:63)
   at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4791)
   at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5285)
   at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
   at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
   at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
   at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:618)
   at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:963)
   at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1600)
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
   at java.util.concurrent.FutureTask.run(FutureTask.java:166)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
   at java.lang.Thread.run(Thread.java:722)
2012-12-28 19:57:03,557  WARN  [alfresco.util.OpenOfficeConnectionTester] [DefaultScheduler_Worker-2] Error trying to query Open Office version information. OpenOffice.org's ConfigurationRegistry not implemented in this version of OOo. This should not affect the operation of OOo.
2012-12-28 19:57:03,604  INFO  [alfresco.util.OpenOfficeConnectionTester] [DefaultScheduler_Worker-2] The OpenOffice connection was re-established.
2012-12-28 19:58:02,682  INFO  [extensions.webscripts.TemplateProcessorRegistry] [localhost-startStop-1] Registered template processor freemarker for extension ftl
2012-12-28 19:58:03,136  INFO  [extensions.webscripts.ScriptProcessorRegistry] [localhost-startStop-1] Registered script processor javascript for extension js
2012-12-28 19:58:03,136  INFO  [extensions.webscripts.TemplateProcessorRegistry] [localhost-startStop-1] Registered template processor freemarker for extension ftl
2012-12-28 19:58:03,182  INFO  [extensions.webscripts.ScriptProcessorRegistry] [localhost-startStop-1] Registered script processor javascript for extension js
2012-12-28 19:58:17,745  INFO  [extensions.webscripts.DeclarativeRegistry] [localhost-startStop-1] Registered 339 Web Scripts (+0 failed), 352 URLs
2012-12-28 19:58:17,745  INFO  [extensions.webscripts.DeclarativeRegistry] [localhost-startStop-1] Registered 8 Package Description Documents (+0 failed)
2012-12-28 19:58:17,745  INFO  [extensions.webscripts.DeclarativeRegistry] [localhost-startStop-1] Registered 0 Schema Description Documents (+0 failed)
2012-12-28 19:58:21,120  INFO  [extensions.webscripts.AbstractRuntimeContainer] [localhost-startStop-1] Initialised Spring Surf Container Web Script Container (in 17910.336ms)
2012-12-28 19:58:21,167  INFO  [extensions.webscripts.TemplateProcessorRegistry] [localhost-startStop-1] Registered template processor freemarker for extension ftl
2012-12-28 19:58:21,214  INFO  [extensions.webscripts.ScriptProcessorRegistry] [localhost-startStop-1] Registered script processor javascript for extension js
2012-12-28 19:58:25,011  ERROR [quartz.core.JobRunShell] [SolrTrackerScheduler_Worker-3] Job Solr.CoreTracker-archive threw an unhandled Exception:
org.alfresco.error.AlfrescoRuntimeException: 11280001 GetModelsDiff return status is 404
   at org.alfresco.solr.client.SOLRAPIClient.getModelsDiff(SOLRAPIClient.java:1036)
   at org.alfresco.solr.tracker.CoreTracker.trackModels(CoreTracker.java:1787)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1126)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:472)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
2012-12-28 19:58:25,042  ERROR [quartz.core.ErrorLogger] [SolrTrackerScheduler_Worker-3] Job (Solr.CoreTracker-archive threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: org.alfresco.error.AlfrescoRuntimeException: 11280001 GetModelsDiff return status is 404]
   at org.quartz.core.JobRunShell.run(JobRunShell.java:227)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
Caused by: org.alfresco.error.AlfrescoRuntimeException: 11280001 GetModelsDiff return status is 404
   at org.alfresco.solr.client.SOLRAPIClient.getModelsDiff(SOLRAPIClient.java:1036)
   at org.alfresco.solr.tracker.CoreTracker.trackModels(CoreTracker.java:1787)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1126)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:472)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   … 1 more
2012-12-28 19:58:25,042  ERROR [quartz.core.JobRunShell] [SolrTrackerScheduler_Worker-2] Job Solr.CoreTracker-alfresco threw an unhandled Exception:
org.alfresco.error.AlfrescoRuntimeException: 11280000 GetModelsDiff return status is 404
   at org.alfresco.solr.client.SOLRAPIClient.getModelsDiff(SOLRAPIClient.java:1036)
   at org.alfresco.solr.tracker.CoreTracker.trackModels(CoreTracker.java:1787)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1126)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:472)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
2012-12-28 19:58:25,073  ERROR [quartz.core.ErrorLogger] [SolrTrackerScheduler_Worker-2] Job (Solr.CoreTracker-alfresco threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: org.alfresco.error.AlfrescoRuntimeException: 11280000 GetModelsDiff return status is 404]
   at org.quartz.core.JobRunShell.run(JobRunShell.java:227)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
Caused by: org.alfresco.error.AlfrescoRuntimeException: 11280000 GetModelsDiff return status is 404
   at org.alfresco.solr.client.SOLRAPIClient.getModelsDiff(SOLRAPIClient.java:1036)
   at org.alfresco.solr.tracker.CoreTracker.trackModels(CoreTracker.java:1787)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1126)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:472)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   … 1 more
2012-12-28 19:58:25,870  INFO  [web.site.EditionInterceptor] [http-apr-8080-exec-4] Unable to retrieve License information from Alfresco: 404
2012-12-28 19:58:30,589  ERROR [quartz.core.JobRunShell] [SolrTrackerScheduler_Worker-3] Job Solr.CoreTracker-archive threw an unhandled Exception:
org.alfresco.error.AlfrescoRuntimeException: 11280002 GetModelsDiff return status is 404
   at org.alfresco.solr.client.SOLRAPIClient.getModelsDiff(SOLRAPIClient.java:1036)
   at org.alfresco.solr.tracker.CoreTracker.trackModels(CoreTracker.java:1787)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1126)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:472)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
2012-12-28 19:58:30,589  ERROR [quartz.core.ErrorLogger] [SolrTrackerScheduler_Worker-3] Job (Solr.CoreTracker-archive threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: org.alfresco.error.AlfrescoRuntimeException: 11280002 GetModelsDiff return status is 404]
   at org.quartz.core.JobRunShell.run(JobRunShell.java:227)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
Caused by: org.alfresco.error.AlfrescoRuntimeException: 11280002 GetModelsDiff return status is 404
   at org.alfresco.solr.client.SOLRAPIClient.getModelsDiff(SOLRAPIClient.java:1036)
   at org.alfresco.solr.tracker.CoreTracker.trackModels(CoreTracker.java:1787)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1126)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:472)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   … 1 more

here is the catalina log: (slightly different time period)

Dec 28, 2012 12:18:30 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load org.bouncycastle.jce.provider.symmetric.AES$ECB.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
   at java.security.Provider$Service.getImplClass(Provider.java:1279)
   at java.security.Provider$Service.newInstance(Provider.java:1237)
   at javax.crypto.Cipher.chooseProvider(Cipher.java:845)
   at javax.crypto.Cipher.init(Cipher.java:1348)
   at sun.security.ssl.CipherBox.<init>(CipherBox.java:175)
   at sun.security.ssl.CipherBox.newCipherBox(CipherBox.java:208)
   at sun.security.ssl.CipherSuite$BulkCipher.newCipher(CipherSuite.java:467)
   at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:507)
   at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:485)
   at sun.security.ssl.CipherSuite.isAvailable(CipherSuite.java:190)
   at sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:342)
   at sun.security.ssl.SSLContextImpl.getDefaultCipherSuiteList(SSLContextImpl.java:300)
   at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:576)
   at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:446)
   at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:140)
   at org.alfresco.encryption.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:168)
   at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
   at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
   at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
   at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
   at org.alfresco.httpclient.AbstractHttpClient.executeMethod(AbstractHttpClient.java:130)
   at org.alfresco.httpclient.AbstractHttpClient.sendRemoteRequest(AbstractHttpClient.java:106)
   at org.alfresco.httpclient.HttpClientFactory$HttpsClient.sendRequest(HttpClientFactory.java:364)
   at org.alfresco.solr.client.SOLRAPIClient.getModelsDiff(SOLRAPIClient.java:1033)
   at org.alfresco.solr.tracker.CoreTracker.trackModels(CoreTracker.java:1787)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1126)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:472)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)

Dec 28, 2012 12:18:30 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load org.bouncycastle.jce.provider.symmetric.AES$ECB.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
   at java.security.Provider$Service.getImplClass(Provider.java:1279)
   at java.security.Provider$Service.newInstance(Provider.java:1237)
   at javax.crypto.Cipher.chooseProvider(Cipher.java:845)
   at javax.crypto.Cipher.init(Cipher.java:1348)
   at sun.security.ssl.CipherBox.<init>(CipherBox.java:175)
   at sun.security.ssl.CipherBox.newCipherBox(CipherBox.java:208)
   at sun.security.ssl.CipherSuite$BulkCipher.newCipher(CipherSuite.java:467)
   at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:507)
   at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:485)
   at sun.security.ssl.CipherSuite.isAvailable(CipherSuite.java:190)
   at sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:342)
   at sun.security.ssl.SSLContextImpl.getDefaultCipherSuiteList(SSLContextImpl.java:300)
   at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:576)
   at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:446)
   at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:140)
   at org.alfresco.encryption.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:168)
   at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
   at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
   at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
   at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
   at org.alfresco.httpclient.AbstractHttpClient.executeMethod(AbstractHttpClient.java:130)
   at org.alfresco.httpclient.AbstractHttpClient.sendRemoteRequest(AbstractHttpClient.java:106)
   at org.alfresco.httpclient.HttpClientFactory$HttpsClient.sendRequest(HttpClientFactory.java:364)
   at org.alfresco.solr.client.SOLRAPIClient.getModelsDiff(SOLRAPIClient.java:1033)
   at org.alfresco.solr.tracker.CoreTracker.trackModels(CoreTracker.java:1787)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1126)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:472)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)

Dec 28, 2012 12:18:30 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load org.bouncycastle.jce.provider.symmetric.AES$ECB.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
   at java.security.Provider$Service.getImplClass(Provider.java:1279)
   at java.security.Provider$Service.newInstance(Provider.java:1237)
   at javax.crypto.Cipher.chooseProvider(Cipher.java:845)
   at javax.crypto.Cipher.init(Cipher.java:1348)
   at sun.security.ssl.CipherBox.<init>(CipherBox.java:175)
   at sun.security.ssl.CipherBox.newCipherBox(CipherBox.java:208)
   at sun.security.ssl.CipherSuite$BulkCipher.newCipher(CipherSuite.java:467)
   at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:507)
   at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:485)
   at sun.security.ssl.CipherSuite.isAvailable(CipherSuite.java:190)
   at sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:342)
   at sun.security.ssl.SSLContextImpl.getDefaultCipherSuiteList(SSLContextImpl.java:293)
   at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:576)
   at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:494)
   at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:313)
   at org.apache.tomcat.util.net.jsse.JSSESocketFactory.acceptSocket(JSSESocketFactory.java:182)
   at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:216)
   at java.lang.Thread.run(Thread.java:722)

Dec 28, 2012 12:18:30 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load org.bouncycastle.jce.provider.symmetric.AES$ECB.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
   at java.security.Provider$Service.getImplClass(Provider.java:1279)
   at java.security.Provider$Service.newInstance(Provider.java:1237)
   at javax.crypto.Cipher.chooseProvider(Cipher.java:845)
   at javax.crypto.Cipher.init(Cipher.java:1348)
   at sun.security.ssl.CipherBox.<init>(CipherBox.java:175)
   at sun.security.ssl.CipherBox.newCipherBox(CipherBox.java:208)
   at sun.security.ssl.CipherSuite$BulkCipher.newCipher(CipherSuite.java:467)
   at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:507)
   at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:485)
   at sun.security.ssl.CipherSuite.isAvailable(CipherSuite.java:190)
   at sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:342)
   at sun.security.ssl.SSLContextImpl.getDefaultCipherSuiteList(SSLContextImpl.java:293)
   at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:576)
   at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:494)
   at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:313)
   at org.apache.tomcat.util.net.jsse.JSSESocketFactory.acceptSocket(JSSESocketFactory.java:182)
   at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:216)
   at java.lang.Thread.run(Thread.java:722)

Dec 28, 2012 12:43:30 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load org.bouncycastle.jce.provider.symmetric.AES$ECB.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
   at java.security.Provider$Service.getImplClass(Provider.java:1279)
   at java.security.Provider$Service.newInstance(Provider.java:1237)
   at javax.crypto.Cipher.chooseProvider(Cipher.java:845)
   at javax.crypto.Cipher.init(Cipher.java:1348)
   at sun.security.ssl.CipherBox.<init>(CipherBox.java:175)
   at sun.security.ssl.CipherBox.newCipherBox(CipherBox.java:208)
   at sun.security.ssl.CipherSuite$BulkCipher.newCipher(CipherSuite.java:467)
   at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:507)
   at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:485)
   at sun.security.ssl.CipherSuite.isAvailable(CipherSuite.java:190)
   at sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:342)
   at sun.security.ssl.SSLContextImpl.getDefaultCipherSuiteList(SSLContextImpl.java:300)
   at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:576)
   at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:446)
   at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:140)
   at org.alfresco.encryption.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:168)
   at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
   at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
   at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
   at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
   at org.alfresco.httpclient.AbstractHttpClient.executeMethod(AbstractHttpClient.java:130)
   at org.alfresco.httpclient.AbstractHttpClient.sendRemoteRequest(AbstractHttpClient.java:106)
   at org.alfresco.httpclient.HttpClientFactory$HttpsClient.sendRequest(HttpClientFactory.java:364)
   at org.alfresco.solr.client.SOLRAPIClient.getModelsDiff(SOLRAPIClient.java:1033)
   at org.alfresco.solr.tracker.CoreTracker.trackModels(CoreTracker.java:1787)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1126)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:472)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)

Outcomes