AnsweredAssumed Answered

Upgrade 4.2c mt to 4.2d fails

Question asked by ebogaard on Sep 9, 2013
Latest reply on Sep 10, 2013 by ebogaard
When testing the upgrade of 4.2c multi tenant to 4.2d, it fails.
It seems that most SQL-patches are succesfully applied. During startup, I get only two few unusual WARNs:

 2013-09-09 11:57:33,433  WARN  [admin.patch.AbstractPatch] [AbstractPatch Processor for patch.calendarAllDayEventDatesCorrectingPatch1] AbstractPatch Processor for patch.calendarAllDayEventDatesCorrectingPatch: Failed to process entry "tenantx.com".
org.alfresco.service.cmr.repository.InvalidNodeRefException: Failed invocation: ReflectiveMethodInvocation: public abstract boolean org.alfresco.service.cmr.repository.NodeService.hasAspect(org.alfresco.service.cmr.repository.NodeRef,org.alfresco.service.namespace.QName) throws org.alfresco.service.cmr.repository.InvalidNodeRefException,org.alfresco.service.cmr.dictionary.InvalidAspectException; target is of class [org.alfresco.repo.node.db.DbNodeServiceImpl]
        at org.alfresco.repo.tenant.MultiTNodeServiceInterceptor.invoke(MultiTNodeServiceInterceptor.java:144)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
        at com.sun.proxy.$Proxy9.hasAspect(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor247.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
        at com.sun.proxy.$Proxy28.hasAspect(Unknown Source)
        at org.alfresco.repo.node.MLPropertyInterceptor.getPivotNodeRef(MLPropertyInterceptor.java:324)
        at org.alfresco.repo.node.MLPropertyInterceptor.invoke(MLPropertyInterceptor.java:174)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.alfresco.repo.lock.mem.LockableAspectInterceptor.invoke(LockableAspectInterceptor.java:112)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.alfresco.repo.node.NodeRefPropertyMethodInterceptor.invoke(NodeRefPropertyMethodInterceptor.java:212)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
        at com.sun.proxy.$Proxy9.getProperties(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor251.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:196)
        at com.sun.proxy.$Proxy9.getProperties(Unknown Source)
        at org.alfresco.repo.rule.RuleServiceImpl.getRule(RuleServiceImpl.java:678)
        at org.alfresco.repo.rule.RuleServiceImpl$2.doWork(RuleServiceImpl.java:1168)
        at org.alfresco.repo.rule.RuleServiceImpl$2.doWork(RuleServiceImpl.java:1165)
        at org.alfresco.repo.security.authentication.AuthenticationUtil.runAs(AuthenticationUtil.java:546)
        at org.alfresco.repo.rule.RuleServiceImpl.executePendingRule(RuleServiceImpl.java:1164)
        at org.alfresco.repo.rule.RuleServiceImpl.executePendingRulesImpl(RuleServiceImpl.java:1119)
        at org.alfresco.repo.rule.RuleServiceImpl.executePendingRules(RuleServiceImpl.java:1092)
        at org.alfresco.repo.rule.RuleTransactionListener.beforeCommit(RuleTransactionListener.java:57)
        at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:737)
        at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:717)
        at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:683)
        at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
        at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
        at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:469)
        at org.alfresco.repo.batch.BatchProcessor$TxnCallback.run(BatchProcessor.java:756)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)
Caused by: org.alfresco.service.cmr.repository.InvalidNodeRefException: Node does not exist: workspace://SpacesStore/380f9ea7-f49b-491c-ba76-6f3a504112b2(null)
        at org.alfresco.repo.node.db.DbNodeServiceImpl.getNodePairNotNull(DbNodeServiceImpl.java:189)
        at org.alfresco.repo.node.db.DbNodeServiceImpl.hasAspect(DbNodeServiceImpl.java:1018)
        at sun.reflect.GeneratedMethodAccessor247.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.alfresco.repo.tenant.MultiTNodeServiceInterceptor.invoke(MultiTNodeServiceInterceptor.java:140)
        … 45 more



2013-09-09 11:57:33,454  ERROR [admin.patch.AbstractPatch] [localhost-startStop-1] AbstractPatch Processor for patch.calendarAllDayEventDatesCorrectingPatch: 1 error(s) detected. Last error from entry "tenanty.com"
org.alfresco.service.cmr.repository.InvalidNodeRefException: Failed invocation: ReflectiveMethodInvocation: public abstract boolean org.alfresco.service.cmr.repository.NodeService.hasAspect(org.alfresco.service.cmr.repository.NodeRef,org.alfresco.service.namespace.QName) throws org.alfresco.service.cmr.repository.InvalidNodeRefException,org.alfresco.service.cmr.dictionary.InvalidAspectException; target is of class [org.alfresco.repo.node.db.DbNodeServiceImpl]
        at org.alfresco.repo.tenant.MultiTNodeServiceInterceptor.invoke(MultiTNodeServiceInterceptor.java:144)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
        at com.sun.proxy.$Proxy9.hasAspect(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor247.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
        at com.sun.proxy.$Proxy28.hasAspect(Unknown Source)
        at org.alfresco.repo.node.MLPropertyInterceptor.getPivotNodeRef(MLPropertyInterceptor.java:324)
        at org.alfresco.repo.node.MLPropertyInterceptor.invoke(MLPropertyInterceptor.java:174)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.alfresco.repo.lock.mem.LockableAspectInterceptor.invoke(LockableAspectInterceptor.java:112)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.alfresco.repo.node.NodeRefPropertyMethodInterceptor.invoke(NodeRefPropertyMethodInterceptor.java:212)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
        at com.sun.proxy.$Proxy9.getProperties(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor251.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:196)
        at com.sun.proxy.$Proxy9.getProperties(Unknown Source)
        at org.alfresco.repo.rule.RuleServiceImpl.getRule(RuleServiceImpl.java:678)
        at org.alfresco.repo.rule.RuleServiceImpl$2.doWork(RuleServiceImpl.java:1168)
        at org.alfresco.repo.rule.RuleServiceImpl$2.doWork(RuleServiceImpl.java:1165)
        at org.alfresco.repo.security.authentication.AuthenticationUtil.runAs(AuthenticationUtil.java:546)
        at org.alfresco.repo.rule.RuleServiceImpl.executePendingRule(RuleServiceImpl.java:1164)
        at org.alfresco.repo.rule.RuleServiceImpl.executePendingRulesImpl(RuleServiceImpl.java:1119)
        at org.alfresco.repo.rule.RuleServiceImpl.executePendingRules(RuleServiceImpl.java:1092)
        at org.alfresco.repo.rule.RuleTransactionListener.beforeCommit(RuleTransactionListener.java:57)
        at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:737)
        at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:717)
        at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:683)
        at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
        at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
        at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:469)
        at org.alfresco.repo.batch.BatchProcessor$TxnCallback.run(BatchProcessor.java:756)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)
Caused by: org.alfresco.service.cmr.repository.InvalidNodeRefException: Node does not exist: workspace://SpacesStore/380f9ea7-f49b-491c-ba76-6f3a504112b2(null)
        at org.alfresco.repo.node.db.DbNodeServiceImpl.getNodePairNotNull(DbNodeServiceImpl.java:189)
        at org.alfresco.repo.node.db.DbNodeServiceImpl.hasAspect(DbNodeServiceImpl.java:1018)
        at sun.reflect.GeneratedMethodAccessor247.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.alfresco.repo.tenant.MultiTNodeServiceInterceptor.invoke(MultiTNodeServiceInterceptor.java:140)
        … 45 more


After server setup, I get a list of missing nodes (which is untrue):


2013-09-09 11:59:07,176  ERROR [node.ibatis.NodeDAOImpl$MySQL] [http-bio-8443-exec-4] ALF-12358: Deleted node - removed child assoc: 17315
2013-09-09 11:59:07,219  ERROR [node.ibatis.NodeDAOImpl$MySQL] [http-bio-8443-exec-4] ALF-12358: Deleted node - removed child assoc: 17316
2013-09-09 11:59:07,245  ERROR [node.ibatis.NodeDAOImpl$MySQL] [http-bio-8443-exec-4] ALF-12358: Deleted node - removed child assoc: 17318
2013-09-09 11:59:07,262  ERROR [node.ibatis.NodeDAOImpl$MySQL] [http-bio-8443-exec-4] ALF-12358: Deleted node - removed child assoc: 17319
2013-09-09 11:59:07,287  ERROR [node.ibatis.NodeDAOImpl$MySQL] [http-bio-8443-exec-4] ALF-12358: Deleted node - removed child assoc: 17320
2013-09-09 11:59:07,319  ERROR [node.ibatis.NodeDAOImpl$MySQL] [http-bio-8443-exec-4] ALF-12358: Deleted node - removed child assoc: 48815
2013-09-09 11:59:07,344  ERROR [node.ibatis.NodeDAOImpl$MySQL] [http-bio-8443-exec-4] ALF-12358: Deleted node - removed child assoc: 46041
2013-09-09 11:59:07,364  ERROR [node.ibatis.NodeDAOImpl$MySQL] [http-bio-8443-exec-4] ALF-12358: Deleted node - removed child assoc: 46048
2013-09-09 11:59:07,389  ERROR [node.ibatis.NodeDAOImpl$MySQL] [http-bio-8443-exec-4] ALF-12358: Deleted node - removed child assoc: 17862
2013-09-09 11:59:07,405  ERROR [node.ibatis.NodeDAOImpl$MySQL] [http-bio-8443-exec-4] ALF-12358: Deleted node - removed child assoc: 46148
2013-09-09 11:59:07,423  ERROR [node.ibatis.NodeDAOImpl$MySQL] [http-bio-8443-exec-4] ALF-12358: Deleted node - removed child assoc: 45976



After that, Alfresco just keeps on trying connecting and disconnecting to the DB-server (MariaDB), between 100 and 200 times a second. Which gives the following error:
 2013-09-09 11:59:15,178  ERROR [post.lookup.PostLookup] [DefaultScheduler_Worker-5] Exception during update of posts: 
org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Cannot open connection
        at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:596)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:371)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:335)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:273)
        at org.alfresco.util.transaction.SpringAwareUserTransaction.begin(SpringAwareUserTransaction.java:411)
        at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:436)
        at org.alfresco.repo.lock.JobLockServiceImpl.doWithRetry(JobLockServiceImpl.java:500)
        at org.alfresco.repo.lock.JobLockServiceImpl.getLockImpl(JobLockServiceImpl.java:454)
        at org.alfresco.repo.lock.JobLockServiceImpl.getLock(JobLockServiceImpl.java:218)
        at org.alfresco.repo.lock.JobLockServiceImpl.getLock(JobLockServiceImpl.java:209)
        at org.alfresco.repo.activities.post.lookup.PostLookup.acquireLock(PostLookup.java:789)
        at org.alfresco.repo.activities.post.lookup.PostLookup.execute(PostLookup.java:203)
        at org.alfresco.repo.activities.post.lookup.PostLookupJob.execute(PostLookupJob.java:50)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
Caused by: org.hibernate.exception.JDBCConnectionException: Cannot open connection
        at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:74)
        at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
        at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29)
        at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:426)
        at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
        at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119)
        at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57)
        at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1326)
        at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:555)
        … 14 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Too many connections
        at sun.reflect.GeneratedConstructorAccessor524.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        at com.mysql.jdbc.Util.getInstance(Util.java:386)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4190)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4122)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:927)
        at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1709)
        at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1252)
        at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2486)
        at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2519)
        at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2304)
        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
        at sun.reflect.GeneratedConstructorAccessor522.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
        at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:346)
        at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
        at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
        at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1158)
        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
        at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
        at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
        at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
        … 19 more


I had no problem using 4.2d 'from scratch' and after upgrading and 4.2c single tenant environment.

Outcomes