Skip navigation
All Places > Alfresco Content Services (ECM) > Blog > Author: tpage

Alfresco Content Services (ECM)

4 Posts authored by: tpage Employee
tpage

Search Codebase Restructure

Posted by tpage Employee Jun 21, 2019

You may have noticed some changes recently in the SearchServices codebase.  Before we go into the details it might be interesting to consider the number of git repositories that we were previously working with:

Git repositories used by the Search Team
This diagram just shows the projects we were editing for the latest version of Search Services.

If you find it hard to track the compatibility between different versions of ACS and Search Services[1] then you can imagine how we felt with all the different versions of these different internal libraries.

 

In January we started the process of consolidating this code by merging the solrclient history into Search Services (SEARCH-1395). The solrclient project was originally intended to support Solr 4 and Solr 6, however we subsequently took the decision not to support Search Services with Solr 4. Consequently the effort of maintaining a library with a separate lifecycle became redundant.

We made a similar change to our enterprise search codebase too, merging the git repositories for Insight Engine, Insight JDBC and Insight Zeppelin.

 

The next step is for us was to merge the enterprise and community code into a single repository.

A major motivation for this is that our end-to-end tests are currently in a separate project and we would prefer to store them in the same repository (on the same branch) as the production code. This will allow us to simultaneously make changes on feature branches to our production and test code. We have preemptively merged our test codebases, and have ticket for the rest of the work which we are hoping to start in the near future.

 

However we want to keep the Search Services code open source and so we've created a two-way community mirror that pushes all our community changes to GitHub and allows us to accept community pull requests and incorporate them with the enterprise code.

 

The current project structure now looks like this and, as mentioned, we're planning to simplify it further.

 

One side-effect of this is that we've introduced some new root directories (search-services and insight-engine) and a new root pom file. Our community repository does not contain the insight-engine code, and so needs to be built from within the search-services module. We have added this directory to our history too, as this will make it easier to copy bug fixes between branches and perform code comparisons. However this means that if you have an existing clone of the project then you will need to create a fresh clone.

 

If you come across any problems with the new codebase then please let us know (or send us a pull request!)  We're looking forward to spending more time adding value to the product and less time managing releases of internal libraries.

 

[1] Compatibility information for ACS and Search Services is available from our docs site.

A few years ago Samuel wrote a blog post entitled “So, when is Alfresco moving to GitHub?” In it he presented a number of reasons why it was difficult to move our code from SVN to Git. Given the recent move of Share to GitHub, I thought it would be worth writing an update on the situation.

 

The vast majority of our production code is now in Git. ACS Repository, Activiti, Search Services, Records Management, Google Docs Module, Android App, iOS App, Share, ADF Components, … and that’s not including our enterprise code which is primarily within our hosted GitLab.

 

Some of this code has always been in Git, but much of it has been migrated from SVN. I migrated the Records Management codebase in 2015, and since then we’ve had a fairly continuous stream of migrations.* The obvious question is "What’s changed since Samuel’s post?"

 

The answer is that not very much has changed. The team decided not to leave older releases in SVN for exactly the reasons mentioned by Samuel. Cherry-picking changes from SVN to Git can be done with svn diff --git and git apply, but it’s a pain, and we make a lot of service pack changes in our products. The ACS repository codebase is large and although we’ve split out several modules into smaller Git repositories, there is still over 2Gb of history in the Git repo.

 

The primary reason for migrating to Git is the popularity - Git is seven times more popular than any other version control system. Some of the reasons for this are Git’s local and lightweight branches (leading to cleaner workflows), faster access to history (since it’s all stored locally) and smaller overall repository size (leading to faster access to remote commits). Some other reasons are historical – SVN has greatly improved its merging and has got rid of the need for a .svn directory per folder. However since GitHub is the most popular and largest open source host in the world we want to use Git to make it easy for users to access our code.

 

 

* We’ve had migrations in the past too, e.g. Share extras in 2012, but the last year has seen a concerted effort to migrate projects.

As promised in our last release post, RM 2.6.c follows hot on the heels of 2.6.b.  RM 2.6.c is compatible with Alfresco Platform 5.2.g and Share 5.2.f.

What's in 2.6.c?

Due to the short period and a focus on enterprise features, there isn't a huge amount that's new in this release. We have had a lot of updates to improve strings in our ten supported localisations.  We have also acted on feedback we received to improve one of our English UI strings (RM-5793).

We have included a few security fixes by updating RM to use the latest release of Aikau.

Our developer documentation has had some attention too, and we've started work on a new format of technical documentation.  We've also formalised our contribution guidelines.

What's coming up next?

As noted above RM 2.6.c has been tested with Alfresco 5.2.x, but we don't know of any compatibility issues with the latest community release of ACS. If you try it and find issues then we'd love to hear about it.  Compatibility with 6.0.x is on our backlog, but we don't expect to address it until RM 3.0.a (nb. our next community release is currently expected to be 2.7.a).

If you have any feedback about 2.6.c, or about what you'd like to see in the next release, then please let us know. You can use the comments below, or feel free to message me or one of the team.

Links

It’s been a while since our last release of Community RM, but 2.5.b is now available.

 

What's in 2.5.b

 

The main driver for this release is compatibility with Alfresco Community 201702 (i.e. Alfresco 5.2.x). This involved some work to integrate with the shiny new site creation page as well as a few less visible compatibility fixes. The release also contains a lot of bug fixes, particularly around retention schedules, audit and rules.

 

Create site dialog with RM 2.5.b

 

Internally the main innovation has been to create a build which just tests Community behaviour. We have a lot of tests for our Enterprise artefacts (which include the community code too), but in order to allow us to make Community releases more frequently we have set up a new build to look for regressions in our community code.

 

We have also set up a publicly accessible Travis build to make it easier for anyone to see the status the current code, as well as any pull requests they submit.

 

What’s coming up next?

 

On master you’ll see all our current community development, which includes a large focus on the first release of the governance services v1 REST API. It’s been through a couple of significant refactorings since we started work on it, but we think it’s nearing completion now. The current plan is to release it in 2.6.a, which should be in roughly a month if all goes well.

 

You might also have noticed a new module in the project – rm-benchmark. We’ve recently been investigating our performance at scale, and the first part of that is to benchmark the performance of our core capabilities. See RM-3953 for more details.

 

So please let us know what you think of 2.5.b and what you'd like to see in 2.6.a and beyond.

 

Links

 

Filter Blog

By date: By tag: