AAAR_Extract Data Quality

[diwakarm]

Hi Franc,

Whatever the extraction I do, it seems that Nodes Staging Failed, I guess which in turn affects the Data Quality.

Do i have to enable any other properties from alfresco side?

####alfresco-property file#######

audit.enabled = true
audit.alfresco-access.enabled=true

AAAR.log attached,show few errors relating to getCounters.

Attached AAAR.log

What understanding am missing? Kindly teach me!

 

[fcorti · Accepted solution]

Hi Divakar,

Which Alfresco version are you using?
If you use an Alfresco version 5.2+, please use Alflytics instead of AAAR.

If you are using a previous version of Alfresco, did you install the AAAR AMP on the Alfresco side?
The error 400 says that the services is not there.
Please follow the documentation below.
Alfresco AMP to A.A.A.R. dashboards, reports and analysis. 

[diwakarm]

Franc,

Am using Alfresco 5.1 and i have installed the corresponding amp in it.

When i hit these URL's through by browser, I get the respective errors:

1.   https://dms-stage.htcindia.com:443/alfresco/service/AAAR/getCounters?counter=auditTrail&application=... 

{     "status" :    {     "code" : 400,     "name" : "Bad Request",     "description" : "Request sent by the client was syntactically incorrect."   },        "message" : "01140054 FATAL: database \"${db.name}\" does not exist",     "exception" : "org.springframework.extensions.webscripts.WebScriptException - 01140054 FATAL: database \"${db.name}\" does not exist",      "callstack" :    [           ""      ,"org.springframework.extensions.webscripts.WebScriptException: 01140054 FATAL: database \"${db.name}\" does not exist"       ,"com.fcorti.aaar.GetCountersWebScript.executeImpl(GetCountersWebScript.java:98)"       ,"org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:64)"       ,"org.alfresco.repo.web.scripts.RepositoryContainer$3.execute(RepositoryContainer.java:512)"       ,"org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:457)"       ,"org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:580)"       ,"org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:649)"       ,"org.alfresco.repo.web.scripts.RepositoryContainer.executeScriptInternal(RepositoryContainer.java:421)"       ,"org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:301)"       ,"org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:378)"       ,"org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:209)"       ,"org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:132)"       ,"javax.servlet.http.HttpServlet.service(HttpServlet.java:727)"       ,"org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)"       ,"org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)"       ,"org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)"       ,"org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)"       ,"org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)"       ,"org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:61)"       ,"org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)"       ,"org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)"       ,"org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)"       ,"org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)"       ,"org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)"       ,"org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)"       ,"org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)"       ,"org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)"       ,"org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)"       ,"org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)"       ,"org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:188)"       ,"org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)"       ,"org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2466)"       ,"org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2455)"       ,"java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)"       ,"java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)"       ,"org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)"       ,"java.lang.Thread.run(Thread.java:745)"     ],      "server" : "Community v5.1.0 (r122274-b3) schema 9,016",   "time" : "Feb 14, 2018 10:15:16 AM" }

2.   https://dms-stage.htcindia.com:443/alfresco/service/AAAR/getCounters?counter=workflowInstances 

{     "status" :    {     "code" : 400,     "name" : "Bad Request",     "description" : "Request sent by the client was syntactically incorrect."   },        "message" : "01140055 FATAL: database \"${db.name}\" does not exist",     "exception" : "org.springframework.extensions.webscripts.WebScriptException - 01140055 FATAL: database \"${db.name}\" does not exist",      "callstack" :    [           ""      ,"org.springframework.extensions.webscripts.WebScriptException: 01140055 FATAL: database \"${db.name}\" does not exist"       ,"com.fcorti.aaar.GetCountersWebScript.executeImpl(GetCountersWebScript.java:98)"       ,"org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:64)"       ,"org.alfresco.repo.web.scripts.RepositoryContainer$3.execute(RepositoryContainer.java:512)"       ,"org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:457)"       ,"org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:580)"       ,"org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:649)"       ,"org.alfresco.repo.web.scripts.RepositoryContainer.executeScriptInternal(RepositoryContainer.java:421)"       ,"org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:301)"       ,"org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:378)"       ,"org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:209)"       ,"org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:132)"       ,"javax.servlet.http.HttpServlet.service(HttpServlet.java:727)"       ,"org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)"       ,"org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)"       ,"org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)"       ,"org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)"       ,"org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)"       ,"org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:61)"       ,"org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)"       ,"org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)"       ,"org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)"       ,"org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)"       ,"org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)"       ,"org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)"       ,"org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)"       ,"org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)"       ,"org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)"       ,"org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)"       ,"org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:188)"       ,"org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)"       ,"org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2466)"       ,"org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2455)"       ,"java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)"       ,"java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)"       ,"org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)"       ,"java.lang.Thread.run(Thread.java:745)"     ],      "server" : "Community v5.1.0 (r122274-b3) schema 9,016",   "time" : "Feb 14, 2018 10:18:59 AM" }

3.    https://dms-stage.htcindia.com:443/alfresco/service/AAAR/getCounters?counter=workflowTasks

{     "status" :    {     "code" : 400,     "name" : "Bad Request",     "description" : "Request sent by the client was syntactically incorrect."   },        "message" : "01140056 FATAL: database \"${db.name}\" does not exist",     "exception" : "org.springframework.extensions.webscripts.WebScriptException - 01140056 FATAL: database \"${db.name}\" does not exist",      "callstack" :    [           ""      ,"org.springframework.extensions.webscripts.WebScriptException: 01140056 FATAL: database \"${db.name}\" does not exist"       ,"com.fcorti.aaar.GetCountersWebScript.executeImpl(GetCountersWebScript.java:98)"       ,"org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:64)"       ,"org.alfresco.repo.web.scripts.RepositoryContainer$3.execute(RepositoryContainer.java:512)"       ,"org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:457)"       ,"org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:580)"       ,"org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:649)"       ,"org.alfresco.repo.web.scripts.RepositoryContainer.executeScriptInternal(RepositoryContainer.java:421)"       ,"org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:301)"       ,"org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:378)"       ,"org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:209)"       ,"org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:132)"       ,"javax.servlet.http.HttpServlet.service(HttpServlet.java:727)"       ,"org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)"       ,"org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)"       ,"org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)"       ,"org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)"       ,"org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)"       ,"org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:61)"       ,"org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)"       ,"org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)"       ,"org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)"       ,"org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)"       ,"org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)"       ,"org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)"       ,"org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)"       ,"org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)"       ,"org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)"       ,"org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)"       ,"org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:188)"       ,"org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)"       ,"org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2466)"       ,"org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2455)"       ,"java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)"       ,"java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)"       ,"org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)"       ,"java.lang.Thread.run(Thread.java:745)"     ],      "server" : "Community v5.1.0 (r122274-b3) schema 9,016",   "time" : "Feb 14, 2018 10:20:02 AM" }

[fcorti]

Mmm... really strange.
Can you please check (and share) the error you find into the catalina.out on Alfresco?

The issue seems to be into the AMP on Alfresco side.

[diwakarm]

Franc,

 Am using Alfresco AMP AAAR-Alfresco-CE-v5.1--v4.5.amp's

There seems to be no error in Catalina.out other the below one,

Error getting input stream for dependency saikuWidget/SaikuWidgetComponent.js. Skipping..
java.io.IOException: Stream closed

[diwakarm]

Francesco Corti,

This GetCounters error  was some post related to the same issue,but i couldn't get anything out of it.

[diwakarm]

And on top of it.

dm_dim_folders and dm_dim_documents tables are empty in AAAR_DataMart.

[diwakarm]

Francesco Corti‌

I user

AAA4.5

Alresco 5.1

P-BA 7

P-DI 7.1

PSQL-10

OS- CentOS7

I Beleive this is the reason for empty document and folder details,

Init nodes staging - Starting entry [Get documents and folders in staging]
2018/02/16 12:34:34 - Get documents and folders in staging - Starting entry [Set variables]
2018/02/16 12:34:34 - Get documents and folders in staging - Starting entry [Clean REST staging for documents]
2018/02/16 12:34:34 - Get documents and folders in staging - Starting entry [Get json for documents modified after]
2018/02/16 12:34:34 - Get json for documents modified after - Loading transformation from repository [Get json for documents] in directory [/Alfresco/Staging/Utility/]
2018/02/16 12:34:34 - Get json for documents modified after - Using legacy execution engine
2018/02/16 12:34:34 - Get json for documents - Dispatching started for transformation [Get json for documents]
2018/02/16 12:34:34 - stg_rest_documents.0 - WARNING! Error handling in combination with batch processing is not fully supported on the used database because of driver limitations. Proceed with caution at your own risk.
2018/02/16 12:34:34 - stg_rest_documents.0 - Connected to database [AAAR_DataMart] (commit=1000)
2018/02/16 12:34:35 - Json input.0 - Finished processing (I=2, O=0, R=0, W=2, U=0, E=0)
2018/02/16 12:34:35 - JSON items extraction.0 - Optimization level set to 9.
2018/02/16 12:34:35 - alfresco_id.0 - Finished processing (I=0, O=0, R=2, W=2, U=0, E=0)
2018/02/16 12:34:35 - Set alfresco_id.0 - Finished processing (I=0, O=0, R=2, W=2, U=0, E=0)
2018/02/16 12:34:35 - stg_rest_documents.0 - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Because of an error, this step can't continue:
2018/02/16 12:34:35 - stg_rest_documents.0 - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : org.pentaho.di.core.exception.KettleDatabaseException:
2018/02/16 12:34:35 - stg_rest_documents.0 - Unable to roll back database transaction to savepoint
2018/02/16 12:34:35 - stg_rest_documents.0 - Server versions prior to 8.0 do not support savepoints.
2018/02/16 12:34:35 - stg_rest_documents.0 -
2018/02/16 12:34:35 - stg_rest_documents.0 - at org.pentaho.di.core.database.Database.rollback(Database.java:4599)
2018/02/16 12:34:35 - stg_rest_documents.0 - at org.pentaho.di.trans.steps.tableoutput.TableOutput.writeToTable(TableOutput.java:358)
2018/02/16 12:34:35 - stg_rest_documents.0 - at org.pentaho.di.trans.steps.tableoutput.TableOutput.processRow(TableOutput.java:125)
2018/02/16 12:34:35 - stg_rest_documents.0 - at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2018/02/16 12:34:35 - stg_rest_documents.0 - at java.lang.Thread.run(Thread.java:748)
2018/02/16 12:34:35 - stg_rest_documents.0 - Caused by: org.postgresql.util.PSQLException: Server versions prior to 8.0 do not support savepoints.
2018/02/16 12:34:35 - stg_rest_documents.0 - at org.postgresql.jdbc3.AbstractJdbc3Connection.rollback(AbstractJdbc3Connection.java:167)
2018/02/16 12:34:35 - stg_rest_documents.0 - at org.pentaho.di.core.database.Database.rollback(Database.java:4597)
2018/02/16 12:34:35 - stg_rest_documents.0 - ... 4 more
2018/02/16 12:34:35 - stg_rest_documents.0 - Finished processing (I=0, O=0, R=1, W=0, U=0, E=1)
2018/02/16 12:34:35 - Get json for documents - Transformation detected one or more steps with errors.
2018/02/16 12:34:35 - Get json for documents - Transformation is killing the other steps!
2018/02/16 12:34:35 - Get json for documents - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Errors detected!
2018/02/16 12:34:36 - JSON items extraction.0 - Finished processing (I=0, O=0, R=1, W=139, U=0, E=0)
2018/02/16 12:34:36 - Get json for documents - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Errors detected!
2018/02/16 12:34:36 - Get documents and folders in staging - Finished job entry [Get json for documents modified after] (result=[false])
2018/02/16 12:34:36 - Get documents and folders in staging - Finished job entry [Clean REST staging for documents] (result=[false])
2018/02/16 12:34:36 - Get documents and folders in staging - Finished job entry [Set variables] (result=[false])
2018/02/16 12:34:36 - Init nodes staging - Finished job entry [Get documents and folders in staging] (result=[false])
2018/02/16 12:34:36 - Init nodes staging - Finished job entry [Get REST custom types] (result=[false])
2018/02/16 12:34:36 - Init nodes staging - Finished job entry [Get REST documents and folders] (result=[false])
2018/02/16 12:34:36 - Init nodes staging - Finished job entry [Get alfresco connection params] (result=[false])
2018/02/16 12:34:36 - Get nodes - Starting entry [Batch log 'Nodes staging' failure]
2018/02/16 12:34:36 - Batch log 'Nodes staging' failure - Loading transformation from repository [Batch log] in directory [/Alfresco/Utility]
2018/02/16 12:34:36 - Batch log 'Nodes staging' failure - Using legacy execution engine
2018/02/16 12:34:36 - Batch log - Dispatching started for transformation [Batch log]
2018/02/16 12:34:36 - dm_logs.0 - Connected to database [AAAR_DataMart] (commit=1)
2018/02/16 12:34:36 - batch_end.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)

[diwakarm]

‌

It was the problem with the JDBC Postgres drivers in PDI updating it fixed my issue.

But still, Data Quality issue seems to persist.

[fcorti]

Hi Diwakar,

I was talking about the Alfresco's catalina out, not the Pentaho's catalina out.

If you still have issues with it, you can skip the get counter call, opening the Pentaho's jobs/transformations using spoon and modifying the sequence of tasks to avoid getCounter execution.

I hope this will help you.