AnsweredAssumed Answered

HPC for  BPM /Business Rules -- Support in Activiti

Question asked by on Oct 9, 2012
Latest reply on Oct 16, 2012 by jbarrez
Dear All

I have been thinking implications of High performance computing in BPM specifically in the area of Business rules where data size and rule complexity might be significant. I feel the capability to run complex business rules on significant amount of data (in GB/TB) will make adoption of BPM in engine in a wide variety of enterprise application. 

I would like to know your thoughts in this area and if potentially  this kind of capability can be added in activiti

I have shared my thoughts at the following link:

Including the contents from the link:

High Performance Computing for BPM /Business Rules

In today's business environment speed and agility are key needs for a decision maker. If we look at applications within an enterprise almost all of them either supports some kind of business process (though they might not have been implemented using BPM platform ) or support core operations like ERP, SCM etc (which might be seen as combination of forecasting, optimization and complex business rules processing).

Looking at the  emerging trends in high performance computing the shift is towards platform managed distributed high performance computing (HDFS-Hadoop) and column based databases (Cassandra/ HBASE etc). Hadoop essentially takes business logic execution to distributed data  (instead of moving data , which might be significant in size to processing points) and manages fault tolerance/ replication in transparent manner. At the heart of Hadoop is HDFS and Map-reduce ; in short HDFS manages data files sharing in  fault tolerant manner  across nodes and map reduce is a programming paradigm which breaks a large data set in parallel steps into smaller , where these steps are carried out in parallel.

However looking closely , Map-reduce paradigm in its pure form is not very well suited for business process which comprises of complex set of business rules. Essentially codifying business process which contains both parallel and sequential steps where some of the steps contain complex business rules that handle significant amount of data requires business user's expertise to determine the nature of execution of these steps.The idea is to use a business users intelligence to determine:

    The steps which should be run in sequential manner, which sequential step would gain performance benefit if processed in parallel manner (using Map Reduce).
    which steps should be run in parallel manner not using Map reduce (in the same server or multiple servers).
     Which parallel steps should in terms be further broken down to parallely process data either using Map reduce or custom multi threading.

Taking the thought one step ahead, a sequential step which processes significant amount of data (say in Giga bytes and not in Tera Bytes) can be handled in the same hardware using custom multi threading logic and in- memory cache. And need not need complexity of Map reduce architecture , which introduces programming complexity and platform related latencies.

Hence the requirement for a BPM platform that could use similar paradigm as Map reduce/ HDFS along with traditional techniques of high volume data processing (like in memory data processing). One could visualize these as new set of controls (e.g Map reduce, In memory procoressing, Column DB adaptors) which can be modeled in a BPM process.

Please note, in the previous paragraph I have used Map reduce as a programming paradigm which enables parallel processing of data into smaller sets in each subsequent step, we might choose  to call it something else in this context. Its my humble request to invite comments from both BPM and HPC community to share their thoughts. I will be further sharing my thoughts in subsequent posts.