Skip navigation
All Places > Alfresco Process Services & Activiti (BPM) > Blog > 2018 > November
2018

Last week we did a huge amount of work on improving our CI/CD process over our Core and Cloud repositories. We are trying to automate as much as we can our build processes to keep adding fixes that can be consumed by our community and enterprise adopters. We are trying to expand our Acceptance and Conformance testing and getting ready to do a Beta4 release by the end of the week/early next week.

 

@CTI777 Missing id for BPMNElement: https://github.com/Activiti/Activiti/issues/2175, Update process instance properties via API: https://github.com/Activiti/Activiti/issues/2147

@almerico automation with new JenkinsX Prow version of the example repositories towards automated testing with acceptance tests. https://github.com/Activiti/Activiti/issues/2100 (WIP)

@igdianov Worked on the following issues and associated PRs:

@balsarori working on process variable mappings for UserTasks and ServiceTasks (connectors)

@constantin-ciobotaru finished modeling changed regarding json based model types + adding validation on export application + acceptance tests for exporting applications

@miguelruizdev fixed https://github.com/Activiti/Activiti/issues/2139 and https://github.com/Activiti/Activiti/issues/2153 issues. Also started to code bpmn-scenarios-set1 in the conformance scenarios.

@ryandawsonuk - holidays -

@erdemedeiros worked to make process definitions available inside the query service (https://github.com/Activiti/Activiti/issues/2149): completed the work to send the events from runtime bundle and handle the event at the query-service side (missing handle the events in audit).

@salaboy worked on upgrading Spring Boot 2.1 and Spring Cloud Greenwich M3 and working on pipelines improvements with @igdianov

 

Get in touch if you want to contribute: https://gitter.im/Activiti/Activiti7

We are looking forward to mentor people on the technologies that we are using to submit their first Pull Request ;)

Last week we worked quite hard to improve our library pipelines, acceptance tests and issues that we are targeting for Beta4. We welcomed 3 new (but well-known) members in our community effort @CTI777, @almericoand @igdianovwho are working full time in our Beta4milestone. We have closed 9 issues already and we hope to be able to tackle down most of what is still open in the following next two weeks.

 

@CTI777worked on admin suspend/resume end points and integration tests.

https://github.com/Activiti/Activiti/issues/2150, https://github.com/Activiti/activiti-cloud-runtime-bundle-service/pull/145

https://github.com/Activiti/Activiti/issues/2148(in progress)

@almericoworking on new the automation of the example repositories towards automated testing with acceptance tests.

https://github.com/Activiti/activiti-cloud-query/pull/47

Worked also on https://github.com/Activiti/Activiti/issues/2117 and https://github.com/Activiti/Activiti/issues/2151

@igdianovWorked on

@balsaroriWorking on variable mapping

@constantin-ciobotarudid some improvements on modeling regarding json based model types required in aps side

@miguelruizdevworked on the refactoring of acceptance tests

@ryandawsonukworked on validation of process variables at process start and making process variables easy to interact with as json for https://github.com/Activiti/Activiti/issues/2049. Presented together with@mteodori at a public meetup about our experiences with keycloak

@erdemedeirosworked to make process definitions available from query (https://github.com/Activiti/Activiti/issues/2149). Currently they are only available from Runtime Bundle. Experimented some problems because the spring cloud message was sent to early (work in progress).

@salaboyworking with @igdianovand @almericoon the automation of our example layer and overall improvements to our pipelines. Worked with @miguelruizdevto finish the Acceptance Tests refactoring and merged the changes in our new repositories. I did some work during the weekend to update the Gitbook to start preparing for Beta4 https://www.gitbook.com/book/activiti/activiti-7-developers-guide/

Get in touch if you want to contribute: https://gitter.im/Activiti/Activiti7

We are looking forward to mentor people on the technologies that we are using to submit their first Pull Request

Activiti_Sticker

Hi everyone, I am happy to announce that Activiti Core & Activiti Cloud Beta3 artifacts had been released. This is our first 2 weeks release sprint where we focused on closing some important issues. You can find the full list here: https://github.com/Activiti/Activiti/milestone/15?closed=1 

These release moved us one step forward to work on tasks related to guarantee that the services are all consistent and are working as expected. As a result from Beta3 we are now ready to work on Conformance and Acceptance Tests which will provide forward compatibility when we add more building blocks (and alternative blocks) to our core services.

There are no major changes on Beta3 which require migration from Beta2 so you are safe to upgrade and everything should just work. You can find the published Beta3 artifacts in Maven Central and the tagged Docker Images in Docker Hub.

Also our HELM charts (activiti-cloud-full-example version 0.4.0) were upgraded to use Beta3 Docker Images.

If you find any issues with Beta3 please create an issue here.
Or get in touch via Gitter

We will be aiming for Beta4by the end of November and you can track progress here

As mentioned before we will be working on stability and automating our test suites for the Cloud approach, this includes the generation, build and publishing of Docker Images and HELM charts.

This is a great time to get involved to help us with the Beta testing process, integrating with your favourite Open Source projects and  learning about the technology stack that we are using to build these amazing tools. If you are ready for a big challenge get in touch via Gitter

We are also hiring: 

Last week we worked quite hard to close some critical issues to be able to release Beta3, which is now work in progress. You can find the issues that we manage to close here: https://github.com/Activiti/Activiti/milestone/15?closed=1. Most of the work that we are planning for Beta4 is about Conformance Tests, Acceptance Tests and automation to make sure that our release cycles can include more value after each iteration. We are currently looking into automate our examples layer to maintain a stable environment where we can promote new versions of the libraries as explained here: https://salaboy.com/2018/11/05/activiti-cloud-ci-cd-approach-for-java-libraries-and-beyond/https://salaboy.com/2018/11/05/activiti-cloud-ci-cd-approach-for-java-libraries-and-beyond/

 

@igdianovworked on decorating events with BusinessKey for correlation

@constantin-ciobotaruworked on modeling side to adapt the ModelService to json based model types

@miguelruizdevworked on the first iteration of the acceptance tests main refactoring

@ryandawsonukAligned query and runtime bundle task variablesby copying process variables into task as a precursor to providing variable mappings. Fixed bugaffecting runtime task lists for users with no groups.

@erdemedeirosfix issue related to task updated event (https://github.com/Activiti/Activiti/issues/2129). Started thinking about the evolutions of admin APIs.

@salaboyworking on the first iteration of the conformance testing framework to check valid Core and Cloud scenarios. You can find the work in progress in this branch: https://github.com/activiti/activiti/tree/2128-salaboy-conformance-testsand issue https://github.com/Activiti/Activiti/issues/2128https://github.com/Activiti/Activiti/issues/2128

 

 

Get in touch if you want to contribute: https://gitter.im/Activiti/Activiti7

We are looking forward to mentor people on the technologies that we are using to submit their first Pull Request ;)

This blog post describes our approach to CI/CD for our core Java Libraries. This includes the Process Engine, new APIs, and new Spring Boot Starters for the Runtime and Modeling Services of our Cloud Native Building Blocks.

We strongly believe that these practices will enable us to scale our development process to build a stable set of libraries and services. Due the complex nature of the tools that we are building, having a stable base and a core set of practices to follow will enable us to sustain a constant pace of development and delivery.

We have separated Core Java Libraries from Spring Boot Starters due to the relationship with Spring Boot and Spring Cloud respectively. We have also separated the Runtime and Modeling Services due the nature of the changes in these projects and to make sure that they can be released independently.

The following sections describe the main conventions that we have adopted and how the flow of changes will be propagated during a normal development cycle.

Java Core Libraries

We have worked hard to make sure that we enforce consistency across repositories, but at the same time ensure that each repository can evolve independently from the others. We are working under the premise that a PR merged to the develop branch will end up with a new release for the artefacts contained in the repository.

activiti-core-repositories.png

For this reason we have switched to use fixed version releases instead of using SNAPSHOTS. Every PR merged into develop will cause the patch version (SemVersion: Major.Minor.Patch) to be incremented and released to our Alfresco Nexus repository.

We are also creating BoMs (Bill of Materials)in each repository and now each repository has the same relative parent. For Activiti Core related repositories that is Activiti Build.

In practice, this means that each repository will have a Parent Pom which is used parent for all the modules inside that repository and a BoM which is used to declare all the dependency management for consumers of the projects contained in the repository. Each Parent Pom is used to contain and propagate only maven plugins and to define the internal maven dependency management.

Even if the previous diagram looks simple, there are a lot of details that are important to achieve a truly CI/CD approach. Here are some of the highlights:

 

 

      • Parent Pom (pom.xml): this Parent Pom is used as Parent of all the Core Activiti Projects. For that reason, this Parent Pom includes all the shared Maven Plugin Configurations.
      • BoM (activiti-parent-dependencies/pom.xml): this parent BoM includes the dependency management of Spring Boot which is going to be shared by all the children.

 

 

 

 

 

 

 

 

    • Parent Pom: (pom.xml) only acts as aggregator, each children points to Spring Boot Parent for plugins and includes/import the Activiti Dependencies BoM

Other Conventions

We use properties (maven properties) to make references to artifact versions.

Each repository will have a property associated to it and the name of this property will follow the following convention: <{repo-name}.version> </{repo-name}.version>

Examples of this properties tag: <activiti-build.version>, <activiti-api.version>,<activiti.version>and <activiti-dependencies.version>. Notice that the repository names match the property text.

In each repository you will find:

 

Changes propagation and Continuous Delivery approach

While working with these projects we want to ensure that we don’t block streams of work to produce milestone releases. In other words, we don’t want to freeze repositories to do releases. We want to allow the constant evolution of each repository independently of the release plans.

In order to achieve this dynamic, the following diagram describes the "updatebot" configurations to generate PRs for the changes in each repository.

updatebot-changes-activiti-core.png

This flow shows that for every change (that generates a release) in the Activiti Buildrepository, all the children will receive a PR with the updated version after the change is verified. Due the nature of Activiti Build (just a parent pom defining plugins and BoM definition) the rate of change at the top level will be quite low.

At this point, Activiti API will generate a new version of itself and will create a PR on Activiti Core Common, Activiti (Core) and Activiti Dependencies updating its own version and its own BoM.

For these repositories, Activiti Dependencies (which only contains a BoM) acts as a version aggregator for consumers to use. Activiti Dependencies serves as gate to control, test and decide when to release a new version set for all these projects. Based on recent findings, we can also use the Maven Enforcer plugin to enforce version convergence (thanks to @igdianov). This enforcement can be monitored by the pipeline to only merge the PRs when the version convergence is valid. Once we manage to release Activiti Dependencies, our consumers (in this case Activiti Examples) will be notified that a new version is available for consumption which can be automatically merged.

Activiti Cloud Runtime: CI/CD approach for Spring Boot Starters

For Activiti Cloud Runtime Spring Boot Starters we have the following dependency chart

activiti-cloud-repos

For Activiti Cloud each repository has a single parent Activiti Cloud Build(arrows were removed from the previous image to reduce the complexity of the diagram). Following the same approach as Activiti Build, the parent only includes the relevant plugins that apply to all of its children. Activiti Cloud Buildalso defines a parent/master BoM which in this case defines the Spring Clouddependency (Finchley.SR1 at the time of writing this document).

An important detail to notice here, is that Activiti Cloud Build doesn’t depend on Activiti Dependencies. The main reason for this is that not all the Activiti Cloud building blocks depend on Activiti Core libraries, for that reason the Activiti Dependency is not introduced at this level.

For Activiti Cloud we also have an aggregator project called Activiti Cloud Dependencieswhich collects all the individual repositories BoMs and it is designed to be used by consumers of our services. Activiti Cloud Dependencies is the main link between the Spring Boot Starters world provided by Activiti Cloud and services that can be packaged as Containers (Docker images).

After Activiti Cloud Service Commonrepository all the children are independent so they can evolve without being blocked by their parents. For development purposes, developers changing one service can always reference the fixed version of the service to test the changes in live environments. In other words, every change in Activiti Cloud Service Commonor above will be propagated to all the children, but changes in the child will only promote the change to the aggregator (Activiti Cloud Dependencies BoM). This enables us to do A/B testing of new features without fully releasing a new Labeled Activiti Cloud Dependencies for external consumption.

The repositories covered by Activiti Cloud Spring Boot Starter projects are as follows:

Starting with the parent (shared) projects:

 

 

 

 

 

 

Depending on these projects are the Activiti Cloud Spring Boot Starters:

 

 

 

 

 

 

 

 

 

  • Activiti Cloud Dependencies (org.activiti.cloud.dependencies)

 

The Spring Boot Starters generated by these repositories are:

  • Activiti Cloud Audit @EnableActivitiAudit
  • Activiti Cloud Query @EnableActivitiQuery
  • Activiti Cloud Runtime Bundle @EnableActivitiRuntimeBundle
  • Activiti Cloud Connector @EnableActivitiCloudConnector
  • Activiti Cloud App Service @EnableActivitiAppService

This means that if you add Activiti Cloud Dependencies (Master Cloud BoM) you will be able to include any of these starters in your Spring Boot Application.

Activiti Core and Activiti Cloud dependencies

It is important to understand how things relate dependency wise to understand where changes will impact projects down the chain. The following chart represents the dependency of modules between Activiti Core and Activiti Cloud:

activiti-core-and-activit-cloud-rel.png

It is important to notice that Activiti Cloud Runtime Bundleis the only one consuming Activiti Core Dependencies. This means that Activiti Cloud Query Service, Activiti Cloud Audit Service and Activiti Cloud Connectors will be only impacted by changes in Activiti Core Common and Activiti Cloud Service Common.

It is also important to understand that Core Commonand Cloud Service Commonare covering cross cutting concerns such as Security, Tracing, Identity, Alfresco DBP conformance, Rest Docs, Tests, Metadata, etc. and these cross cutting concerns modules shouldn’t change as often as the service themselves.

Activiti Cloud Modeling: CI/CD approach

In the same way as Activiti Cloud Runtime, the Activiti Cloud Modeling repositories produce Spring Boot Starters related to the modeling backend services.
activiti-cloud-modeling.png

Spring Starters

  • Activiti Cloud Process Model Service @EnableActivitiProcessModelService
  • Activiti Cloud Apps Service @EnableActivitiCloudAppsService

Release Process, Example Docker Images and Acceptance Tests

Our release process only covers releasing Maven artifacts to Maven Central (Activiti Core and Activiti Cloud), but we do have a set of example services (which generate Docker Images) that need to be tagged and tested against the released artifacts.

The following diagram describes the repositories involved:

activit-cloud-examples.png

As you can see, Activiti Cloud Application services, and Activiti Cloud Runtime Acceptance Tests are all consumers of Activiti Cloud Dependencies BoM. This makes them perfect examples of how you should build your applications, how to include the aggregator BoM and how to configure the Starters for your Spring Boot Applications. They also include Docker Files, Kubernetes Descriptor Files and HELM Charts definitions.

Activiti Cloud Infrastructure is just used for the examples and these services doesn’t depend on the Starters or any Activiti Cloud Dependencies. We are just bundling them together for our own consumption (Integration Testing and Example Deployments).

Repositories involved in runtime examples which are consuming Activiti Cloud Dependencies master BoM:

Activiti Cloud Modeling examples follow the same approach:

activiti-cloud-modeling-examples

Modeling Repositories which creates docker images are:

Summing up

We are planning to extend our release procedures to include all these example layer which includes creating and tagging docker images as well as publishing HELM charts. While we know that this is a considerable amount of work, we believe that these practices and convention based approach will permeate to teams building solutions with these building blocks, ultimately providing a SaaS approach to deliver solutions.

Activiti Cloud HELM chartsare still the recommended way to deploy our components and the Activiti Cloud Full Example HELM chart deploys all the services as described in the previous sections.

If you have question, want to contribute to the project or want to make it better please get in touch via Gitteror Github Issues.

Last week was all about planning for the next iteration plus trying to solve some critical issues that we found while doing Beta2 release. Beta3 will come quite fast, we hope accelerate the release cycle for the Beta phase to make sure that we fix well known issues and we provide consumable artifacts for community users. If you find issues with Beta2please feel free to report, comment and vote issues here: https://github.com/activiti/activiti/issues 

 

 

@balsaroriworked on exposing form key to process definition and task #2082

 

@constantin-ciobotaruworked to adapt the APS modeling to new activiti Beta2 modeling zip structure

 

@miguelruizdevworked on the refactoring and the increase of coverage of the acceptance tests.

 

@ryandawsonukUpdated the gitbook with details of migration from beta1 to beta2 and updated charts README to clarify how to enable modeling. Ran through the slackbot demo together with @miguelruizdevand @erdemedeiroto define, raise and address a related set of issues on task visibility in query.

 

@erdemedeirosworked with @ryandawsonukto identify the cause of main issues in query service (https://github.com/Activiti/Activiti/issues/2092, https://github.com/Activiti/Activiti/issues/2093, https://github.com/Activiti/Activiti/issues/2103). Fixed those issues.

 

@salaboyworked on new scenarios conformance document, organizing the GitHub issues under Beta3 milestone, planning and creating a new stable set 7.0.55 for activiti-cloud-dependencies.

 

Get in touch if you want to contribute: https://gitter.im/Activiti/Activiti7

We are looking forward to mentor people on the technologies that we are using to submit their first Pull Request ;)