AnsweredAssumed Answered

Thread/Transaction-safe action rules for massive concurrency

Question asked by iru on May 25, 2017
Latest reply on Jul 21, 2017 by iru

Hi there,

 

Got a task that seems easy but is getting us lots of headaches.
Functionally speaking, what we need is to create a custom rule in a folder, that given a file that is being uploaded, moves it into another folder, based on its creation date.

 

  • File enters: Inbox
  • File is moved to:  Folderized / yyyy / MM / dd / ... (in production up to minutes, for example)


Rule must implement two subtasks

  1. create the folderized structure if not existing
  2. move the file into the folderized structure

____


Now, because this must work thread-safely, on a concurrent batch of file upload, it may happen that multiple threads enter the rule at the same time, and here start the problems.

 

  • We started using the synchronized word, in the upper method (ActionExecuterAbstractBase#executeImpl), inner methods, or blocks of code, but this does not seem to work because even if we synchronize, the transaction may have not been committed, hence, no task1 output is available to other threads. Added a sleep but cannot ensure that it's done anyway and fails sometimes.

 

  • We also tried the RetryingTransactionHelper (RTH) workaround exposed by Alexey Vasyukov in the lightning talk about 'Alfresco repo under concurrent write load', of this year's beecon, but got several problems too.

I've already asked how this helper is supposed to work when the flag 'requiresNew' is set to false, because it seems not to retry, or we may be misunderstanding something.
https://community.alfresco.com/message/815007-re-retryingtransactionhelper-learnings?commentID=815007#comment-815007

 

The problem with this utility is that it behaves differently on a postgres based server vs. mysql based server (we're using latest alfresco520,  mysql server-core 5.7, with 5.1.32 driver , Ubuntu 16; Updated  checked InnoDB for all tables for transacational management)

Updated Tested in SQLServer too, and gives different behaviour from previous ones too.

 

The approach that works fine in Postgres is having task 1 under a RetryingTransactionHelper (and new transaction), and don't go into task 2 until it's successfully performed.
Task 2 is not executed under a RTH.

 

However when we try this with mysql, fails. Guess this error is launched because task 1's output is still not fully operable (due to being made inside a transaction) to the parent transaction.

Updated- MySQL  (first task with RTH, second task not - NonRootNodeWithoutParentsException)
- src: https://pastebin.com/raw/DD0bWjW2
- action context: https://pastebin.com/CVWS9U9q
- error: https://pastebin.com/raw/gx2TkWxn

 

Updated-SQLServer (DataIntegrityViolationException)

- error: https://pastebin.com/raw/BZSTKDUX 

 

We also tried to insert the second task inside RTH.
With the requireNew flag set to true seems as if the node is not visible to the code within the transaction and when trying to move it, the node is not found.
Guess because it's a new transaction and does not know about the "working in progress node" (seems some AOP implementation)

Updated Clarify that this happens no matter the database provider, so it's nice to see it's coherent.

Updated (both tasks with RTH, with requiresNew - InvalidNodeRefException: Node does not exist)
- src: https://pastebin.com/CvrRSt3X
- action context: same as before

- error: https://pastebin.com/raw/yVNc8YWf

If we insert it with a requireNewFlag set to false, it gives the previous NonRootNodeWithoutParentsException error.

______

 

We've tried other approaches/combinations and got several kinds of errors, but tried to summarize everything as much as possible.

 

The only solution that has seemed to work (in any database provider, concurrently) is when rule is executed in background. However, this is not desired; we would like it to be sequential, so final user can know if the action has been correctly performed.

 

We could also have a cron-like task, that populates the folderized structure, but will lead to lots of empty folders if no data is uploaded and would have to purge it afterwards.

 

Any suggestions on how to approach this?


Would provide code, but its something quite simple using fileFolderService, both to create and move. We've  analysed the code that is in the default "MoveActionExecuter" action rule provided in the core of Alfresco, and its quite similar to ours.

 

Thanks in advance.

Outcomes