Java Heap Space error in alfresco.log

[june_cataquez1]

Hi Alfrescians,

I would like to ask some assistance, I've encountered a 'Java Heap Space' error in alfresco.log and I'm not quite sure how to fix it, I've done some research and it's confusing to choose which solution to pick. I would like to hear from you (Alfrecians) the correct approach to my problem. Please see attached file. Thanks in advance.

Best Regards,

June

[afaust]

A heap space error can occur for any number of reasons: exceptionally high (concurrent) load, load of an extremely large data set, caches configured too large or even leaks of memory structures. There is no universal advice for dealing with heap space errors - it always requires to pinpoint the cause / trigger and deal with the specific issues.

In your case there appeared to be a longer duration of subsequent heap space errors beginning on 11:35 until ~14:00 when the server was restarted. This points to either a long running process that retains too much memory, a leak or some bad configuration regarding cache sizes / heap segment sizes.

It would be great if you could provide your JAVA_OPTS and any changes you made to alfresco-global.properties

[june_cataquez1]

Hi Sir,

Please correct me if I'm adding things that's not helpful in Alfresco performance.

I added something in my alfresco-global.properties:

_____________________________________________________________________________________
###############################
## Common Alfresco Properties #
###############################

dir.root=/opt/alfresco-community/alf_data

alfresco.context=alfresco
alfresco.host=10.128.60.108
alfresco.port=8080
alfresco.protocol=http

share.context=share
share.host=10.128.60.108
share.port=8080
share.protocol=http

### database connection properties ###
db.driver=org.gjt.mm.mysql.Driver
db.username=alfresco
db.password=alfresco
db.name=alfresco
db.port=3306
db.host=localhost
db.url=jdbc:mysql://localhost:3306/alfresco?useUnicode=yes&characterEncoding=UTF-8
# Note: your database must also be able to accept at least this many connections.  Please see your database documentation for instructions on how to configure this.
db.pool.max=275
db.pool.validate.query=SELECT 1

# The server mode. Set value here
# UNKNOWN | TEST | BACKUP | PRODUCTION
system.serverMode=UNKNOWN

### FTP Server Configuration ###
ftp.port=21

### RMI registry port for JMX ###
alfresco.rmi.services.port=50500

### External executable locations ###
ooo.exe=/opt/alfresco-community/libreoffice/program/soffice.bin
ooo.enabled=false
ooo.port=8100
img.root=/opt/alfresco-community/common
img.dyn=${img.root}/lib
img.exe=${img.root}/bin/convert

jodconverter.enabled=false
jodconverter.officeHome=/opt/alfresco-community/libreoffice
jodconverter.portNumbers=8100

### Initial admin password ###
alfresco_user_store.adminpassword=2d20d252a479f485cdf5e171d93985bf

### E-mail site invitation setting ###
notification.email.siteinvite=false

### License location ###
dir.license.external=/opt/alfresco-community

### Solr indexing ###
index.subsystem.name=solr4
dir.keystore=${dir.root}/keystore
solr.host=localhost
solr.port.ssl=8443

### Allow extended ResultSet processing
security.anyDenyDenies=false

### Smart Folders Config Properties ###
smart.folders.enabled=false

### Remote JMX (Default: disabled) ###
alfresco.jmx.connector.enabled=false

###Outbound mail
mail.host=10.190.1.30
mail.port=25
mail.username=dgranados
mail.password=geodude_78

###Inbound mail
email.server.enabled=true
email.server.port=1025
email.server.domain=demo.alfresco.org
email.server.allowed.senders=.*

##Cache
cache.node.nodesSharedCache.tx.maxItems=250000
cache.node.nodesSharedCache.maxItems=2500000

__________________________________________________________________________________

Also, here is my setenv.sh

__________________________________________________________________________________

# Load Tomcat Native Library
LD_LIBRARY_PATH=/opt/alfresco-community/common/lib:$LD_LIBRARY_PATH

JAVA_HOME=/opt/alfresco-community/java
JRE_HOME=$JAVA_HOME
JAVA_OPTS="-XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Djava.awt.headless=true -Dalfresco.home=/opt/alfresco-community -XX:ReservedCodeCacheSize=256m $JAVA_OPTS "
JAVA_OPTS="-Xms2000M -Xmx6000M  $JAVA_OPTS " # java-memory-settings
export JAVA_HOME
export JRE_HOME
export JAVA_OPTS
export LD_LIBRARY_PATH

Best Regards,

June
                

[afaust · Accepted solution]

For what reason did you set -XX:ReservedCodeCacheSize? I guess this is something you have found on the Internet somewhere but in my experience this is never necessary / relevant.

The biggest issue and a potential cause of a heap space error might be the setting of

cache.node.nodesSharedCache.tx.maxItems=250000
cache.node.nodesSharedCache.maxItems=2500000

This increases that particular cache to 10 times the default size (20x for the tx cache). Based on previous memory analysis I did, each entry may be roughly between 500 and 1000 bytes in size, so allowing for 2.5 million entries would mean the shared cache can grow up to ~2.3 GiB - and each user operation may handle up to the same in its transactional cache. Provided you really do have that many nodes in your system (at least 2.5 million) this could explain the heap space error. Even though 2.3 GiB only amounts to 1/3 of the -Xmx value, it can block a large enough chunk of the "old generation" region. When large amounts of additional objects need to be transferred from "new generation" there might not be enough space left...

For what reason did you set this value so high? Do you have any custom development in place that processes that many nodes?

[june_cataquez1]

Hi Sir,

Thanks for the info, the reason I put that there is that I'm afraid for the future transactions the users are about to make.I just see it on some random blogs and followed it. Thanks for making it clear to me, I should ask here more often than researching and doing things alone. I will erase this: cache.node.nodesSharedCache.tx.maxItems=250000
cache.node.nodesSharedCache.maxItems=2500000

And this will be a better start for me.

Best Regards,

June

[june_cataquez1]

Hi Sir,

After doing what you suggested, java heap space error once again occurs in the system.

Here's the fragment of the error:

2017-02-06 11:41:43,756 ERROR [org.springframework.extensions.webscripts.AbstractRuntime] [http-apr-8080-exec-2] Exception from executeScript - redirecting to status template error: 0106175617 Wrapped Exception (with status template): 0106702924 Failed to execute script 'classpath*:alfresco/extension/templates/webscripts/alvex/datalists/datalist-search.post.json.js': 0106702923 Java heap space
org.springframework.extensions.webscripts.WebScriptException: 0106175617 Wrapped Exception (with status template): 0106702924 Failed to execute script 'classpath*:alfresco/extension/templates/webscripts/alvex/datalists/datalist-search.post.json.js': 0106702923 Java heap space
        at org.springframework.extensions.webscripts.AbstractWebScript.createStatusException(AbstractWebScript.java:1138)
        at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:171)
        at org.alfresco.repo.web.scripts.RepositoryContainer$3.execute(RepositoryContainer.java:505)
        at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:457)
        at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:580)
        at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:649)
        at org.alfresco.repo.web.scripts.RepositoryContainer.executeScriptInternal(RepositoryContainer.java:421)
        at org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:301)
        at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:378)
        at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:209)
        at org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:132)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:61)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
        at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2466)
        at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2455)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.alfresco.scripts.ScriptException: 0106702924 Failed to execute script 'classpath*:alfresco/extension/templates/webscripts/alvex/datalists/datalist-search.post.json.js': 0106702923 Java heap space

Best Regards,

June

[afaust]

Do you have a heap dump for this out of memory error? Without that you will not be able to analyse this further / in detail. The cache settings I pointed too were just one of the possible causes or contributing factors - out of memory errors can be triggered by countless other constellations.

[june_cataquez1]

Hi Sir,

Regarding heap dump, is this a reliable link? Create and Understand Java Heapdumps (Act 4) - codecentric AG Blog : codecentric AG Blog . I will try adding heap dumps to know the causes of these out of memory errors.

Best Regards,

June