AnsweredAssumed Answered

Performance, large transaction

Question asked by buurd on Jun 10, 2010
Latest reply on Jun 16, 2010 by buurd
Hi!

I'm running a large importjob that are going to put about 100 000 documents into our Alfresco-installation. But there are some problems that makes me a little bit worried.

1) We add all files into import-maps, they are named with unique names from the old system
2) We add metadata-files from the old system
3) We have written a small javascript that reads the metadata-file and then find the file with the unique id and copy it to it's right location and adds metadata according to the metadata-file.

The problem we see is that when we run a batch with about 1500-2000 files that are going to be copied and get metadata the system locks up for a while. And if the batch is large enough we get warnings in the log about transactioncache being full and, perpaps we should have waited for longer than an hour, restart alfresco.

So we handled the problem, running smaller batches that doesn't cause that long lockups (1-2 minutes instead of 15-20 min). Boring work but accepable.

Now I tried to remove some of the spaces holding the original files. Each space with original files contains between 1000-2500  files. Now the warning about the transactioncache occurs.

For a system that say it can handle 100 000 000 documents this seems very strange. I can understand that some tasks might take some time but I really don't understand why I get lockups???

Now I hope that someone is going to say that this is a common error of configuration that easially can be fixed. I'm going to get a tough time explaining this problem otherwise.

Thanks in advance
Roland

Outcomes