How to handle a race condition signaling an Execution

cancel
Showing results for 
Search instead for 
Did you mean: 
blezek
Active Member

How to handle a race condition signaling an Execution

Jump to solution

Hi,

  We start long-running workflows that wait for a signal.  When certain events happen, we signal the workflow by searching for the list of Executions waiting on the signal (and some other criteria).  This can lead to race conditions because the incoming events can come in short order.  Here is example code:

    ExecutionQuery query = runtime.createExecutionQuery().processVariableValueEquals("Key", key).signalEventSubscriptionName(signalName);

    executions.forEach(ex -> {

        logger.info("Signaling workflow: " + ex.getId() + " with " + signalName);

        ...

        runtime.signalEventReceived(signalName, ex.getId(), signalVars);

The problem is that the long-running workflow performs a few checks right after the signal is received, then goes back to waiting on the signal (see image).  Because the above code can be called rapidly, between the query step and the signalEventReceived, the long-running workflow can go from waiting on the signal to being busy with other things.  I do synchronize in the Java code so that only one signal is sent at a time.  

We have tried a combination of "asynchronous" settings on the workflow after the intermediate signal catching event (the "Classify Series" and "Log Message", before it loops back to wait for the next signal).

If the events come slowly enough, the workflow returns to the signal catching event and all works well.

Question: How can I make sure the long-running workflow returns to waiting for the signal before the next signal needs to be sent?

Help?

-dan

1 Solution

Accepted Solutions
gdharley
Intermediate

Re: How to handle a race condition signaling an Execution

Jump to solution

Dan,

you have to change your design to decouple the signal send and receive logic.

Put the signals onto a queue and then have the receiver loop through each signal/message in it's own time. Before "stopping" at the next "signal" look to see if there is already a message on the queue and process it immediately.

What you describe is a very typical issue when the processing time for signals can exceed the interval between received signals.

Greg

View solution in original post

2 Replies
gdharley
Intermediate

Re: How to handle a race condition signaling an Execution

Jump to solution

Dan,

you have to change your design to decouple the signal send and receive logic.

Put the signals onto a queue and then have the receiver loop through each signal/message in it's own time. Before "stopping" at the next "signal" look to see if there is already a message on the queue and process it immediately.

What you describe is a very typical issue when the processing time for signals can exceed the interval between received signals.

Greg

blezek
Active Member

Re: How to handle a race condition signaling an Execution

Jump to solution

Greg,  Yup, that's what I figured you'd say...  =)  We will document that signal are delivered "best effort" and that workflows should not depend on them being delivered.  The notification logic adds the new information into a workflow variable and continues.  That logic also locks to have a single thread, so we don't clobber the workflow variables.

Thanks much,

-dan