Search Services - Failed to initialize keystore

[FBoros]

I'm trying to install Alfresco Search Services using this guide.

I've generated secure keys for SLL communitacion with this tool and then moved them to <SOLR_HOME>/keystore.

Then I've made configurations in:

<SOLR_HOME>/templates/rerank/conf/solrcore.properties:

# solrcore.properties - used in solrconfig.xml
#

enable.alfresco.tracking=true

#
#These are replaced by the admin handler
#
#data.dir.root=DATA_DIR
#data.dir.store=workspace/SpacesStore
#alfresco.stores=workspace://SpacesStore

#
# Properties loaded during alfresco tracking
#

alfresco.host=localhost
alfresco.port=8080
alfresco.port.ssl=8443
alfresco.baseUrl=/alfresco

#alfresco.index.transformContent=false
#alfresco.ignore.datatype.1=d:content
alfresco.lag=1000
alfresco.hole.retention=3600000
# alfresco.hole.check.after is not used yet
# It will reduce the hole checking load
alfresco.hole.check.after=300000
alfresco.batch.count=5000
alfresco.recordUnindexedNodes=false

# encryption

# none, https
alfresco.secureComms=https

# ssl, default values
# keystore and trustore files are not provided by default
alfresco.encryption.ssl.keystore.type=JCEKS
alfresco.encryption.ssl.keystore.location=/opt/alfresco-community-201911/alfresco-search-services/solrhome/keystore/ssl-repo-client.keystore
alfresco.encryption.ssl.keystore.passwordFileLocation=
alfresco.encryption.ssl.truststore.type=JCEKS
alfresco.encryption.ssl.truststore.provider=
alfresco.encryption.ssl.truststore.location=/opt/alfresco-community-201911/alfresco-search-services/solrhome/keystore/ssl-repo-client.truststore
alfresco.encryption.ssl.truststore.passwordFileLocation=

# Default Tracker
alfresco.cron=0/10 * * * * ? *
alfresco.corePoolSize=8
alfresco.maximumPoolSize=-1
alfresco.keepAliveTime=120
alfresco.threadPriority=5
alfresco.threadDaemon=true
alfresco.workQueueSize=-1
alfresco.commitInterval=2000
alfresco.newSearcherInterval=3000
#ACL tracker configuration
#alfresco.acl.tracker.cron=
#alfresco.acl.tracker.corePoolSize=
#alfresco.acl.tracker.maximumPoolSize=
#alfresco.acl.tracker.keepAliveTime=
#alfresco.acl.tracker.threadPriority=
#alfresco.acl.tracker.threadDaemon=
#alfresco.acl.tracker.workQueueSize=
#Content tracker config
#alfresco.content.tracker.cron=
#alfresco.content.tracker.corePoolSize=
#alfresco.content.tracker.maximumPoolSize=
#alfresco.content.tracker.keepAliveTime=
#alfresco.content.tracker.threadPriority=
#alfresco.content.tracker.threadDaemon=
#alfresco.content.tracker.workQueueSize=
#Metadata tracker config
#alfresco.metadata.tracker.cron=
#alfresco.metadata.tracker.corePoolSize=
#alfresco.metadata.tracker.maximumPoolSize=
#alfresco.metadata.tracker.keepAliveTime=
#alfresco.metadata.tracker.threadPriority=
#alfresco.metadata.tracker.threadDaemon=
#alfresco.metadata.tracker.workQueueSize=

# HTTP Client

alfresco.maxTotalConnections=200
alfresco.maxHostConnections=200
alfresco.socketTimeout=360000

# SOLR caching

solr.filterCache.size=256
solr.filterCache.initialSize=128
solr.queryResultCache.size=1024
solr.queryResultCache.initialSize=1024
solr.documentCache.size=1024
solr.documentCache.initialSize=1024
solr.queryResultMaxDocsCached=2048

solr.authorityCache.size=128
solr.authorityCache.initialSize=64
solr.pathCache.size=256
solr.pathCache.initialSize=128

solr.ownerCache.size=128
solr.ownerCache.initialSize=64

solr.readerCache.size=128
solr.readerCache.initialSize=64

solr.deniedCache.size=128
solr.deniedCache.initialSize=64

# SOLR

solr.maxBooleanClauses=10000

# Batch fetch

alfresco.transactionDocsBatchSize=500
alfresco.nodeBatchSize=100
alfresco.changeSetAclsBatchSize=500
alfresco.aclBatchSize=100
alfresco.contentReadBatchSize=100
alfresco.contentUpdateBatchSize=1000

# Warming

solr.filterCache.autowarmCount=32
solr.authorityCache.autowarmCount=4
solr.pathCache.autowarmCount=32
solr.deniedCache.autowarmCount=0
solr.readerCache.autowarmCount=0
solr.ownerCache.autowarmCount=0
solr.queryResultCache.autowarmCount=4
solr.documentCache.autowarmCount=512

solr.queryResultWindowSize=512

#
# TODO
#
# cross language support
# locale expansion
# logging check report ....
#
#

alfresco.doPermissionChecks=true

# Evaluate permissions as a PostFilter. Setting this to false will evaluate permissions as a Query.

alfresco.postfilter=true

#
# Metadata pulling control
#
alfresco.metadata.skipDescendantDocsForSpecificTypes=false
alfresco.metadata.ignore.datatype.0=cm:person
alfresco.metadata.ignore.datatype.1=app:configurations
alfresco.metadata.skipDescendantDocsForSpecificAspects=false
#alfresco.metadata.ignore.aspect.0=

alfresco.topTermSpanRewriteLimit=1000

#
# Suggestions
#
solr.suggester.enabled=true
# -1 to disable suggester build throttling
solr.suggester.minSecsBetweenBuilds=3600

#
# Limit the maximum text size of transformed content sent to the index - in bytes
#
alfresco.contentStreamLimit=10000000
#Shard setup
shard.method=DB_ID
#END of solrcore

<alfresco-search-services>/solr.in.sh:

# Settings here will override settings in existing env vars or in bin/solr.  The default shipped state
# of this file is completely commented.

# By default the script will use JAVA_HOME to determine which java
# to use, but you can set a specific path for Solr to use without
# affecting other Java applications on your server/workstation.
#SOLR_JAVA_HOME=""

# This controls the number of seconds that the solr script will wait for
# Solr to stop gracefully or Solr to start.  If the graceful stop fails,
# the script will forcibly stop Solr.  If the start fails, the script will
# give up waiting and display the last few lines of the logfile.
#SOLR_STOP_WAIT="180"

# Increase Java Heap as needed to support your indexing / query needs
#SOLR_HEAP="512m"

# Expert: If you want finer control over memory options, specify them directly
# Comment out SOLR_HEAP if you are using this though, that takes precedence
SOLR_JAVA_MEM="-Xms1g -Xmx1g"

# Enable verbose GC logging...
#  * If this is unset, various default options will be selected depending on which JVM version is in use
#  * For Java 8: if this is set, additional params will be added to specify the log file & rotation
#  * For Java 9 or higher: each included opt param that starts with '-Xlog:gc', but does not include an
#    output specifier, will have a 'file' output specifier (as well as formatting & rollover options)
#    appended, using the effective value of the SOLR_LOGS_DIR.
#
#GC_LOG_OPTS='-Xlog:gc*'  # (Java 9+)
#GC_LOG_OPTS="-verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails \
#  -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime"

# These GC settings have shown to work well for a number of common Solr workloads
#GC_TUNE="-XX:NewRatio=3 -XX:SurvivorRatio=4    etc.

# Set the ZooKeeper connection string if using an external ZooKeeper ensemble
# e.g. host1:2181,host2:2181/chroot
# Leave empty if not using SolrCloud
#ZK_HOST=""

# Set the ZooKeeper client timeout (for SolrCloud mode)
#ZK_CLIENT_TIMEOUT="15000"

# By default the start script uses "localhost"; override the hostname here
# for production SolrCloud environments to control the hostname exposed to cluster state
#SOLR_HOST="192.168.1.1"

# By default the start script uses UTC; override the timezone if needed
#SOLR_TIMEZONE="UTC"

# Set to true to activate the JMX RMI connector to allow remote JMX client applications
# to monitor the JVM hosting Solr; set to "false" to disable that behavior
# (false is recommended in production environments)
#ENABLE_REMOTE_JMX_OPTS="false"

# The script will use SOLR_PORT+10000 for the RMI_PORT or you can set it here
# RMI_PORT=18983

# Alfresco configuration. This file is automatically included by solr. You can define your custom settings here
SOLR_OPTS="$SOLR_OPTS -Dsolr.jetty.request.header.size=1000000 -Dsolr.jetty.threads.stop.timeout=300000 -Ddisable.configEdit=true"

# Anything you add to the SOLR_OPTS variable will be included in the java
# start command line as-is, in ADDITION to other options. If you specify the
# -a option on start script, those options will be appended as well. Examples:
#SOLR_OPTS="$SOLR_OPTS -Dsolr.autoSoftCommit.maxTime=3000"
#SOLR_OPTS="$SOLR_OPTS -Dsolr.autoCommit.maxTime=60000"
#SOLR_OPTS="$SOLR_OPTS -Dsolr.clustering.enabled=true"

# Location where the bin/solr script will save PID files for running instances
# If not set, the script will create PID files in $SOLR_TIP/bin
#SOLR_PID_DIR=

# Path to a directory for Solr to store cores and their data. By default, Solr will use server/solr
# If solr.xml is not stored in ZooKeeper, this directory needs to contain solr.xml
#SOLR_HOME=

# Solr provides a default Log4J configuration properties file in server/resources
# however, you may want to customize the log settings and file appender location
# so you can point the script to use a different log4j.properties file
#LOG4J_PROPS=/var/solr/log4j.properties

# Changes the logging level. Valid values: ALL, TRACE, DEBUG, INFO, WARN, ERROR, FATAL, OFF. Default is INFO
# This is an alternative to changing the rootLogger in log4j.properties
#SOLR_LOG_LEVEL=INFO

# Location where Solr should write logs to. Absolute or relative to solr start dir
SOLR_LOGS_DIR=../../logs
LOG4J_PROPS=$SOLR_LOGS_DIR/log4j.properties

# Enables log rotation, cleanup, and archiving during start. Setting SOLR_LOG_PRESTART_ROTATION=false will skip start
# time rotation of logs, and the archiving of the last GC and console log files. It does not affect Log4j configuration.
# This pre-startup rotation may need to be disabled depending how much you customize the default logging setup.
#SOLR_LOG_PRESTART_ROTATION=true

# Sets the port Solr binds to, default is 8983
#SOLR_PORT=8983

# Uncomment to set SSL-related system properties
# Be sure to update the paths to the correct keystore for your environment
SOLR_SSL_KEY_STORE=/opt/alfresco-community-201911/alfresco-search-services/solrhome/keystore/ssl-repo-client.keystore
SOLR_SSL_KEY_STORE_PASSWORD=keystore
SOLR_SSL_KEY_STORE_TYPE=JCEKS
SOLR_SSL_TRUST_STORE=/opt/alfresco-community-201911/alfresco-search-services/solrhome/keystore/ssl-repo-client.truststore
SOLR_SSL_TRUST_STORE_PASSWORD=truststore
SOLR_SSL_TRUST_STORE_TYPE=JCEKS
SOLR_SSL_NEED_CLIENT_AUTH=true
SOLR_SSL_WANT_CLIENT_AUTH=false

# Uncomment if you want to override previously defined SSL values for HTTP client
# otherwise keep them commented and the above values will automatically be set for HTTP clients
#SOLR_SSL_CLIENT_KEY_STORE=
#SOLR_SSL_CLIENT_KEY_STORE_PASSWORD=
#SOLR_SSL_CLIENT_KEY_STORE_TYPE=
#SOLR_SSL_CLIENT_TRUST_STORE=
#SOLR_SSL_CLIENT_TRUST_STORE_PASSWORD=
#SOLR_SSL_CLIENT_TRUST_STORE_TYPE=

# Settings for authentication
# Please configure only one of SOLR_AUTHENTICATION_CLIENT_CONFIGURER or SOLR_AUTH_TYPE parameters
#SOLR_AUTHENTICATION_CLIENT_CONFIGURER="org.apache.solr.client.solrj.impl.PreemptiveBasicAuthConfigurer"
#SOLR_AUTH_TYPE="basic"
#SOLR_AUTHENTICATION_OPTS="-Dbasicauth=solr:SolrRocks"

# Settings for ZK ACL
#SOLR_ZK_CREDS_AND_ACLS="-DzkACLProvider=org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider \
#  -DzkCredentialsProvider=org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider \
#  -DzkDigestUsername=admin-user -DzkDigestPassword=CHANGEME-ADMIN-PASSWORD \
#  -DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=CHANGEME-READONLY-PASSWORD"
#SOLR_OPTS="$SOLR_OPTS $SOLR_ZK_CREDS_AND_ACLS"

The command I use to run new instance:

./solr start -a "-Dcreate.alfresco.defaults=alfresco,archive -Dsolr.ssl.checkPeerName=false -Dsolr.allow.unsafe.resourceloading=true -Dssl-keystore.password=keystore -Dssl-keystore.aliases=ssl-alfresco-ca,ssl-repo-client -Dssl-keystore.ssl-alfresco-ca.password=keystore -Dssl-keystore.ssl-repo-client.password=keystore -Dssl-truststore.password=truststore -Dssl-truststore.aliases=ssl-alfresco-ca,ssl-repo,ssl-repo-client -Dssl-truststore.ssl-alfresco-ca.password=truststore -Dssl-truststore.ssl-repo.password=truststore -Dssl-truststore.ssl-repo-client.password=truststore" -f -force

But I get this error:

2021-02-04 14:32:35.710 INFO  (main) [   ] o.e.j.s.Server jetty-9.3.14.v20161028
2021-02-04 14:32:36.523 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 6.6.5-patched a77dde61f5121c726afb0e50a8e34ebba88534a4 - build - 2018-10-26 16:24:31
2021-02-04 14:32:36.524 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port 8983
2021-02-04 14:32:36.524 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: /opt/alfresco-community-201911/alfresco-search-services/solr
2021-02-04 14:32:36.540 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2021-02-04T14:32:36.525880Z
2021-02-04 14:32:37.760 INFO  (main) [   ] o.e.j.s.Server Started @3146ms
2021-02-04 14:32:47.543 WARN  (Thread-12) [   x:alfresco] o.a.s.c.Config XML parse warning in "solrres:/solrconfig.xml", line 1919, column 88: Include operation failed, reverting to fallback. Resource error reading file as XML (href='solrconfig_insight.xml'). Reason: Can't find resource 'solrconfig_insight.xml' in classpath or '/opt/alfresco-community-201911/alfresco-search-services/solrhome/alfresco'
2021-02-04 14:32:48.461 WARN  (Thread-12) [   x:alfresco] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [org.apache.solr.analysis.WordDelimiterFilterFactory]. Please consult documentation how to replace it accordingly.
2021-02-04 14:32:48.479 WARN  (Thread-12) [   x:alfresco] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.SynonymFilterFactory]. Please consult documentation how to replace it accordingly.
2021-02-04 14:32:50.088 WARN  (Thread-12) [   x:alfresco] o.a.s.h.c.AlfrescoSolrClusteringComponent No default engine for document clustering.
2021-02-04 14:32:50.307 WARN  (Thread-12) [   x:alfresco] o.a.s.c.Config XML parse warning in "solrres:/solrconfig.xml", line 1919, column 88: Include operation failed, reverting to fallback. Resource error reading file as XML (href='solrconfig_insight.xml'). Reason: Can't find resource 'solrconfig_insight.xml' in classpath or '/opt/alfresco-community-201911/alfresco-search-services/solrhome/archive'
2021-02-04 14:32:50.729 WARN  (Thread-12) [   x:alfresco] o.a.s.h.c.AlfrescoSolrClusteringComponent No default engine for document clustering.
2021-02-04 14:32:50.866 ERROR (searcherExecutor-21-thread-1-processing-x:alfresco) [   x:alfresco] o.a.s.c.SolrCore null:org.alfresco.error.AlfrescoRuntimeException: 01040000 Failed to initialize keystore:
   Location: /opt/alfresco-community-201911/alfresco-search-services/solrhome/keystore/ssl-repo-client.keystore
   Provider:
   Type:     JCEKS
        at org.alfresco.encryption.AlfrescoKeyStoreImpl.loadKeyStore(AlfrescoKeyStoreImpl.java:576)
        at org.alfresco.encryption.AlfrescoKeyStoreImpl.safeInit(AlfrescoKeyStoreImpl.java:543)
        at org.alfresco.encryption.AlfrescoKeyStoreImpl.<init>(AlfrescoKeyStoreImpl.java:110)
        at org.alfresco.httpclient.HttpClientFactory.init(HttpClientFactory.java:150)
        at org.alfresco.httpclient.HttpClientFactory.<init>(HttpClientFactory.java:145)
        at org.alfresco.solr.client.SOLRAPIClientFactory.getRepoClient(SOLRAPIClientFactory.java:193)
        at org.alfresco.solr.client.SOLRAPIClientFactory.getSOLRAPIClient(SOLRAPIClientFactory.java:174)
        at org.alfresco.solr.lifecycle.SolrCoreLoadRegistration.registerForCore(SolrCoreLoadRegistration.java:73)
        at org.alfresco.solr.lifecycle.SolrCoreLoadListener.newSearcher(SolrCoreLoadListener.java:44)
        at org.apache.solr.core.SolrCore.lambda$getSearcher$15(SolrCore.java:2249)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:830)
Caused by: java.io.IOException: Is a directory
        at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at java.base/sun.nio.ch.FileDispatcherImpl.read(FileDispatcherImpl.java:48)
        at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
        at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:245)
        at java.base/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:223)
        at java.base/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
        at java.base/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
        at java.base/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
        at java.base/java.io.InputStream.read(InputStream.java:213)
        at java.base/java.util.Properties$LineReader.readLine(Properties.java:500)
        at java.base/java.util.Properties.load0(Properties.java:416)
        at java.base/java.util.Properties.load(Properties.java:405)
        at org.alfresco.solr.SolrKeyResourceLoader.loadKeyMetaData(SolrKeyResourceLoader.java:64)
        at org.alfresco.encryption.AlfrescoKeyStoreImpl$KeyInfoManager.loadKeyMetaData(AlfrescoKeyStoreImpl.java:1031)
        at org.alfresco.encryption.AlfrescoKeyStoreImpl$KeyInfoManager.<init>(AlfrescoKeyStoreImpl.java:1013)
        at org.alfresco.encryption.AlfrescoKeyStoreImpl.getKeyInfoManager(AlfrescoKeyStoreImpl.java:401)
        at org.alfresco.encryption.AlfrescoKeyStoreImpl.loadKeyStore(AlfrescoKeyStoreImpl.java:566)
        ... 14 more

It basically says, that my defined path for keystore points to a directory instead of a file which makes no sense to me. I've tried to move the keystore file in other directories or use configuration with some random path, but I always get this same error so it seems that "alfresco.encryption.ssl.keystore.location" in "solrcore.properties" has no impact on this error.

I wonder if there is some wrong configuration in my files or some other problem, but I've double checked every step from the guide and I'm stucked now. I will be very grateful for any help. 

[afaust]

You are changing the property in the solrcore.properties within the rerank template. I assume you already have cores created from that template as actual configuration in your solrhome (e.g. <solrHome>/alfresco/conf/solrcore.properties), and any changes made to the template file after that configuration was created as a copy thereof will not take any effect. It is generally always more appropriate to change the actual configuration files instead of the template ones. So change the correct solrcore.properties in the <solrHome>/alfresco/.. and <solrHome>/archive/.. configuration paths, restart and check if that has taken the effect you wanted to see.

[jpotts]

Can you do:

keytool -list -keystore /opt/alfresco-community-201911/alfresco-search-services/solrhome/keystore/ssl-repo-client.keystore -storetype JCEKS

To successfully list the contents of your keystore?

Also, if your cores already exist (solrhome/alfresco and solrhome/archive) then you should also check the conf/solrcore.properties file in each of those in case they are not configured correctly.

[FBoros]

I've tried that, but with no success.

[FBoros · Accepted solution]

I noticed that "archive" and "alfresco" cores have separate "solrcore.properties files" and they are generated by "/solrhome/templates/rerank/conf/solrcore.properties". So I've tried both changing configuration in those separate files and also deleting archive and alfresco directories and configuring template config file. All that resulted in the mentioned error.

Sadly I can't try now what jpotts suggested, but if anyone will have the same problem as me, what helped me was to generate keystore using this command:

./run.sh -alfrescoversion community -alfrescoformat classic

*source: https://github.com/Alfresco/alfresco-ssl-generator

It also generates password properties files. Then I've set these respective properties in solrcore.properties to point to the location of those files:

alfresco.encryption.ssl.keystore.passwordFileLocation=
alfresco.encryption.ssl.truststore.passwordFileLocation=