AnsweredAssumed Answered

More than one ProcessEngine: reducing memory footprint by 10x using MyBatis shared configuration fields

Question asked by jkronegg on Jul 25, 2014
Latest reply on Aug 22, 2014 by jkronegg
Hi all,

I have a setup with about 50
ProcessEngine
, each connected to a different datasource (one for each company branch), using H2 file database (for testing).
Each
ProcessEngine
uses about 4 MB of memory, including about 3.6 MB for MyBatis
Configuration
(determined through heap dump analysis).
<!–break–>
Under the assumption that MyBatis
Configuration
is stable over time, it is a big waste of memory: each
ProcessEngine
may share the same configuration.

Studying MyBatis Configuration class further, it appears that most of the memory is taken by the following fields:
  • sqlFragments
    : the
    Configuration.getSqlFragments()
    's call hierarchy has been examinated and all calls come from XML parsers and configuration-related methods (i.e. no query or other runtime methods). Thus, reusing
    sqlFragments
    should be safe.
  • mappedStatements
    : the
    Configuration.getMappedStatements()
    's call hierarchy has been examinated and all calls come from XML parsers and configuration-related methods (i.e. no query or other runtime methods). One manipulated data "databaseId" suggests that there is something specific to the database in the configuration but this flag is not used in Activiti configuration. Thus, reusing
    mappedStatements
    should be safe.
  • resultMaps
    : I found a post from Clinton Begin (MyBatis's principal developper) on the MyBatis mailing list: "By definition the result mappings should be deterministic and consistent.  They shouldn't really change on a per-request basis.". Thus reusing
    resultMaps
    should be safe.
Based on these asumptions, I wrote the following code, which shares the three above MyBatis
Configuration
fields amongst all
ProcessEngine
's. It is called after calling the
ProcessEngineConfigurationImpl.buildProcessEngine()
:

    private static final Logger LOGGER = Logger.getLogger("MyLogger");

    /**
     * Shared ORM configuration objects.
     */
    private static Map<String, Object> sharedOrmConfigurationObjects = new HashMap<String, Object>();

    /**
     * Reduce the memory footprint of the underlying ORM configuration.
     * The ORM configuration elements are shared for each Activiti ProcessEngineConfigurationImpl provided.
     * This is an experimental feature.
     * @param wfc the Activiti configuration
     */
    private static void reduceOrmMemoryFootprint(final ProcessEngineConfigurationImpl wfc) {
        SqlSessionFactory ssf = wfc.getSqlSessionFactory();
        if (ssf != null) {
            Configuration configuration = ssf.getConfiguration(); // the configuration is null if called before wfc.buildProcessEngine()
            if (configuration != null) {
                shareConfigurationObject(configuration, "sqlFragments");
                // Implementation note: the Configuration.getSqlFragments()'s call
                // hierarchy has been examinated and all calls come from XML parsers
                // and configuration-related methods (i.e. no query or other runtime
                // methods). Thus, reusing sqlFragments should be safe.
                shareConfigurationObject(configuration, "mappedStatements");
                // Implementation note: the Configuration.getMappedStatements()'s
                // call hierarchy has been examinated and all calls come from XML
                // parsers and configuration-related methods (i.e. no query or
                // other runtime methods). One manipulated data "databaseId"
                // suggests that there is something specific to the database in
                // the configuration but this flag is not used in Activiti
                // configuration. Thus, reusing mappedStatements should be safe.
                shareConfigurationObject(configuration, "resultMaps");
                // Implementation note: found on the MyBatis mailing list:
                //
                //      "By definition the result mappings should be
                //       deterministic and consistent.  They shouldn't
                //       really change on a per-request basis.".
                //              Clinton Begin (principal developper)
                //
                // Source: <a href='https://groups.google.com/d/topic/mybatis-user/-RG2pgNEtfI/discussion' rel='no-follow'>link</a>
            }
        }
    }

    /**
     * Share a specific field of the MyBatis Configuration.
     * @param configuration the current configuration
     * @param sharedObjectFieldName the field to be shared
     */
    private static void shareConfigurationObject(final Configuration configuration, final String sharedObjectFieldName) {
        Throwable exception = null;
        try {
            // get the current object
            Field sqlFragmentsField = Configuration.class.getDeclaredField(sharedObjectFieldName);
            sqlFragmentsField.setAccessible(true);
            Object currentObject = sqlFragmentsField.get(configuration);

            // get the shared object field
            Object sharedObject = sharedOrmConfigurationObjects.get(sharedObjectFieldName);

            // replace current object with shared object if present
            if (sharedObject == null) {
                // first time => set the shared object
                // Implementation note: since this is the first time, we don't need to update the current object
                sharedOrmConfigurationObjects.put(sharedObjectFieldName, currentObject);
            } else {
                // we got a shared object => use it in the configuration
                sqlFragmentsField.set(configuration, sharedObject);
            }

        } catch (NoSuchFieldException e) {
            exception = e;
        } catch (IllegalArgumentException e) {
            exception = e;
        } catch (IllegalAccessException e) {
            exception = e;
        }
        if (exception != null) {
            LOGGER.log(Level.WARNING, "could not configured ORM with shared " + sharedObjectFieldName + " field; " + exception.toString(), exception);
        }
    }

This leads to huge memory footprint reduction: the first created
ProcessEngine
uses about 4 MB of memory and the next
ProcessEngine
s use 0.4 MB, a 10x factor reduction.

In order ensure that the MyBatis shared
Configuration
fields do not change after using the
ProcessEngine
, I used a test case which get each shared fields hashCode before and after using the
ProcessEngine
.

Do you see some issue in the idea/implementation ?

Thanks,
Julien

Outcomes