Problems with solr when migratin from Alfresco 5.2 to 6.2

[iohann95]

I'm having serious troubles trying to migrate my data from alfresco 5.2 to 6.2.

Everything works but when the reindex is happening, all sort of errors happen, the one that are blocking me this time is:

solr6_1                  | 2021-08-26 03:15:01.896 WARN  (qtp868964689-12) [   ] o.a.s.h.a.LukeRequestHandler Error getting file length for [segments_1x]
solr6_1                  | java.nio.file.NoSuchFileException: /opt/alfresco-search-services/data/archive/index/segments_1x
solr6_1                  |      at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
solr6_1                  |      at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
solr6_1                  |      at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
solr6_1                  |      at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
solr6_1                  |      at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
solr6_1                  |      at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
solr6_1                  |      at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
solr6_1                  |      at java.base/java.nio.file.Files.size(Files.java:2381)
solr6_1                  |      at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
solr6_1                  |      at org.apache.lucene.store.NRTCachingDirectory.fileLength(NRTCachingDirectory.java:128)
solr6_1                  |      at org.apache.solr.handler.admin.LukeRequestHandler.getFileLength(LukeRequestHandler.java:615)
solr6_1                  |      at org.apache.solr.handler.admin.LukeRequestHandler.getIndexInfo(LukeRequestHandler.java:588)
solr6_1                  |      at org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:348)
solr6_1                  |      at org.apache.solr.handler.admin.StatusOp.execute(StatusOp.java:48)
solr6_1                  |      at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
solr6_1                  |      at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:388)
solr6_1                  |      at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
solr6_1                  |      at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
solr6_1                  |      at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:748)
solr6_1                  |      at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:729)
solr6_1                  |      at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:510)
solr6_1                  |      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
solr6_1                  |      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
solr6_1                  |      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
solr6_1                  |      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
solr6_1                  |      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
solr6_1                  |      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
solr6_1                  |      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
solr6_1                  |      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
solr6_1                  |      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513)
solr6_1                  |      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
solr6_1                  |      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
solr6_1                  |      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
solr6_1                  |      at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
solr6_1                  |      at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
solr6_1                  |      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solr6_1                  |      at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
solr6_1                  |      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solr6_1                  |      at org.eclipse.jetty.server.Server.handle(Server.java:539)
solr6_1                  |      at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
solr6_1                  |      at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
solr6_1                  |      at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
solr6_1                  |      at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
solr6_1                  |      at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
solr6_1                  |      at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
solr6_1                  |      at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
solr6_1                  |      at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
solr6_1                  |      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
solr6_1                  |      at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
solr6_1                  |      at java.base/java.lang.Thread.run(Thread.java:829)

After that error the reindex seems to stop.

Thats a strange error to happen since I didn't brought the index from the old version since it's not compatible.

I'm using the alfresco community docker installer:

https://github.com/Alfresco/alfresco-docker-installer

[iohann95]

Sorry for the double post but now i'm also getting this error several times:

solr6_1                  | 2021-08-26 03:34:31.427 ERROR (org.alfresco.solr.AlfrescoCoreAdminHandler@63a5e46c_Worker-2) [   ] o.a.s.h.RequestHandlerBase java.lang.NullPointerException
solr6_1                  |      at org.apache.solr.update.UpdateLog.lookup(UpdateLog.java:968)
solr6_1                  |      at org.apache.solr.handler.component.RealTimeGetComponent.process(RealTimeGetComponent.java:219)
solr6_1                  |      at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:296)
solr6_1                  |      at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
solr6_1                  |      at org.alfresco.solr.SolrInformationServer.executeQueryRequest(SolrInformationServer.java:3387)
solr6_1                  |      at org.alfresco.solr.SolrInformationServer.getTrackerInitialState(SolrInformationServer.java:1415)
solr6_1                  |      at org.alfresco.solr.tracker.AbstractTracker.getTrackerState(AbstractTracker.java:332)
solr6_1                  |      at org.alfresco.solr.tracker.AbstractTracker.track(AbstractTracker.java:227)
solr6_1                  |      at org.alfresco.solr.tracker.ActivatableTracker.track(ActivatableTracker.java:97)
solr6_1                  |      at org.alfresco.solr.tracker.TrackerJob.execute(TrackerJob.java:60)
solr6_1                  |      at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
solr6_1                  |      at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
solr6_1                  |
solr6_1                  | 2021-08-26 03:34:31.429 ERROR (org.alfresco.solr.AlfrescoCoreAdminHandler@63a5e46c_Worker-2) [   ] o.a.s.SolrInformationServer Got no response from a tracker initial state request.
solr6_1                  | 2021-08-26 03:34:31.437 ERROR (org.alfresco.solr.AlfrescoCoreAdminHandler@63a5e46c_Worker-12) [   ] o.a.s.h.RequestHandlerBase java.lang.NullPointerException
solr6_1                  |      at org.apache.solr.update.UpdateLog.lookup(UpdateLog.java:968)
solr6_1                  |      at org.apache.solr.handler.component.RealTimeGetComponent.process(RealTimeGetComponent.java:219)
solr6_1                  |      at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:296)
solr6_1                  |      at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
solr6_1                  |      at org.alfresco.solr.SolrInformationServer.executeQueryRequest(SolrInformationServer.java:3387)
solr6_1                  |      at org.alfresco.solr.SolrInformationServer.getTrackerInitialState(SolrInformationServer.java:1415)
solr6_1                  |      at org.alfresco.solr.tracker.ShardStatePublisher.getTrackerState(ShardStatePublisher.java:123)
solr6_1                  |      at org.alfresco.solr.tracker.AbstractTracker.track(AbstractTracker.java:227)
solr6_1                  |      at org.alfresco.solr.tracker.TrackerJob.execute(TrackerJob.java:60)
solr6_1                  |      at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
solr6_1                  |      at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
solr6_1                  |

[angelborroy]

How are you deploying Alfresco?

[iohann95]

I'm using the community docker installer: https://github.com/Alfresco/alfresco-docker-installer

And also the alfresco s3 adapter (version 1.2.1)

Regards,

Iohann

[angelborroy]

That should be a problem with Docker Volumes binding for SOLR. Review that section in project documentation in order to be sure that everything is working fine.

[iohann95]

I don't know... solr is able to reach the folder. The index sometimes grows to about 2gb but then it  hangs. Looking at Solr Admin it seems to always hang at aroung 130k docs in the index.

I even installed another machine with the same configurations, but without data migration, and it works just fine...

[angelborroy]

How many RAM is assigned to your SOLR Docker service?

[angelborroy]

Did you try increasing RAM for this service?

This may be a cause of your system running consistently but failing from 130k docs.

[iohann95]

I try a few hours ago:

    solr6:
        build:
          context: ./search
          args:
            SEARCH_TAG: $SEARCH_CE_TAG
            SOLR_HOSTNAME: solr6
            ALFRESCO_HOSTNAME: alfresco
            ALFRESCO_COMMS: none
            CROSS_LOCALE: "true"
        mem_limit: 4096m
        environment:
            #Solr needs to know how to register itself with Alfresco
            SOLR_ALFRESCO_HOST: "alfresco"
            SOLR_ALFRESCO_PORT: "8080"
            #Alfresco needs to know how to call solr
            SOLR_SOLR_HOST: "solr6"
            SOLR_SOLR_PORT: "8983"
            #Create the default alfresco and archive cores
            SOLR_CREATE_ALFRESCO_DEFAULTS: "alfresco,archive"
            SOLR_JAVA_MEM: "-Xms3584m -Xmx3584m"
            SOLR_OPTS: "
                -XX:NewSize=2048m
                -XX:MaxNewSize=2048m
            "
        volumes: 
            - ./data/solr-data:/opt/alfresco-search-services/data  

Same result... in solr admin, "archive" core shows around 134k documents indexed. "alfresco" core, shows around 9k

just a note: my previous message is missing: i didn't deleted it.

[iohann95]

Solr used 100gb while indexing and took 2 days crashing sometimes, but eventually it commited the whole tlog and now the index is much smaller.

Now it's working fine...