Development Practices

Contributing to the Simantics codebase

The development of the Simantics platform is coordinated by the Simantics Technical Board. The Technical Board consists of three to six members that are selected among the THTH Simantics Division member organisations by the Simantics General Meeting. The duties of the Simantics Technical Board are defined in the THTH Simantics Division Rules and Regulations.

The Simantics Technical Board shall:

1. prepare a proposal for the Management Board for the exact content of the new releases of Simantics
2. prepare a proposal for the Management Board for the content of the training sessions
3. update simantics.org website both public and member areas

All contributions made to Simantics modules go through the Simantics Technical Board. The Simantics Technical Board meetings are arranged every week. These meetings are documented in the Simantics ticket management system (Redmine).

Using version control

First things first, read through Subversion best practices.

Branching

In Simantics, only new major releases may contain breaking changes. Therefore in Simantics we strive to use a specialization of the Branch-When-Needed system, i.e. Branch-Every-Major-Change. This allows us to keep continuous builds (hudson) working with less disruption, and enables easier pre-release testing, migration and peer review.

The rules are:

• Users commit their day-to-day work on /trunk.
• Rule #1: /trunk must compile and pass regression tests at all times. Committers who violate this rule are publically humiliated.
• Rule #2: a single commit (changeset) must not be so large so as to discourage peer-review.
• Rule #3: if rules #1 and #2 come into conflict (i.e. it's impossible to make a series of small commits without disrupting the trunk), then the user should create a branch and commit a series of smaller changesets there. This allows peer-review without disrupting the stability of /trunk.
• Rule #4: if API is changed, old version in the trunk must be tagged, and version number of the trunk must be increased. If API changes break other trunk code, development must continue in its own branch. Non-incremental change in an ontology is counted as major change (prevents using existing databases); new branch must be created.

Pros

/trunk is guaranteed to be stable at all times. The hassle of branching/merging is somewhat rare.

The word stable here implies mainly API stability but also functional stability. Functional stability must be enforced through both automatic and manual testing. If there are no test cases, functional stability becomes really hard to enforce. Therefore testing is key.

Violating these rules results in reverting the breaking commits from SVN unless the problems can be resolved otherwise through developer communication.

Integrating Branches

We obviously do not want development to continue eternally in branches so that changes are never merged between branches or integrated back to trunk. Branches are usually private or shared between a group of developers that work on a particular task. The process for integration to trunk is as follows:

• The Simantics Technical Board (STB) and its weekly meetings will be the place for the developers to advertise and present their changes to the board.
• The following things need to be documented for the changes in some way:
  • The what and why for the changes
  • Migration instructions from previous version (trunk):
    • A direct patch to fix problems
    • or plain-text instructions
  • When integration is done, mail must be sent to Simantics developers list about the change.
  • ...

TODO more description

Versioning

In Eclipse/OSGi conventions, components (bundles, plug-ins) are versioned <major>.<minor>.<service>[.<qualifier>].

Our approach to component versioning is as follows:

• All components are in one of two states: incubation or production
• Incubation components: <major> = 0, API breakage is allowed between <minor> versions
• Production components: <major> ≥ 1, API breakage is allowed between <major> versions

Creating a new plugin

Before a release, the plugin is in incubation phase and there are no quality or other requirements. The repository https://www.simulationsite.net/svn/simantics-incubator may be used to share or co-develop the plugin with other developers, but the plugin must not be included in the features defined in https://www.simulationsite.net/svn/simantics repository.

If the new plugin is a result of refactoring some already existing plugin into two pieces or otherwise is closely related to an other plugin under development, the new plugin can be committed to the same branch of https://www.simulationsite.net/svn/simantics where the other plugin is developed in. Note that whenever this kind of piecing happens, it has the potential to affect the consistency of other components or products. In such cases the refactor should be carried out in branches, not in trunk.

Naming a plugin

If the plugin is intended to be part of the official Simantics release, its name should begin with org.simantics followed by a word that describes the plugin. If the functionality of the same domain is divided into multiple plugins, the name may have more parts. The following suffixes are used:

• .feature for feature projects.
• .product A plugin defining a product.
• <plugin>.ui Provides UI-functionality for <plugin>. Separating base functionality and UI (that depends on SWT) is usually recommendable so that headless applications do not have unnecessary dependencies.
  • Example: org.simantics.simulation, org.simantics.simulation.ui

NOTE! If the the UI is light (e.g. wizards, dialogs, renderers, few external dependencies) the UI code can reside in <plugin>, a separate <plugin>.ui is not required. In such a case, the UI dependencies (to plugins such as org.eclipse.swt) are optional.

Naming in feature projects

Features have two distinct names: project name and feature ID.

1. The project name is defined in the .project file. It shall contain the .feature suffix since the project name affects how Eclipse checks the project out into workspaces from version control systems.
2. The feature ID is defined in feature.xml and it shall not contain the .feature suffix.
   • Rationale: current Buckminster resource map (RMAP) definitions rely on this convention for feature project resolution. Having a strict convention keeps RMAPs more uniform and clean.
3. Feature ID's should not overlap plug-in ID's if at all possible.
   • Rationale: Buckminster commands will not be able to perform actions on features if a plug-in with the same ID exists.
   • Suggestion: if you can't think of a unique ID for your feature otherwise, append -feature to it.

This naming convention exists mainly because it works well with Buckminster.

Modifying a feature

When a plug-in is added to or removed from a feature, the following steps must be performed:

1. Increment the version number of the feature, primarily the service version number part.
2. Increment service version of all other features dependent on this feature
3. Fix .product files that refer to the re-versioned features

This is necessary most of all to keep Eclipse IDE happy and working properly with checked out plug-ins and the P2-based target platforms.

If these steps are not performed, Eclipse will keep using the target platform version of the modified feature not realizing the changes you've made to it. This results often in products failing to work properly or not starting at all.

Publising a standalone plugin

1. Ensure that the plugin satisfies the following quality requirements:
   • The plugin contains no compilation errors (against which dependencies???)
   • All public classes and interfaces are documented (javadoc)
   • Each public package contains a package-info.java containing package-related comments and annotations (see [1] chapter 7.4)
   • examples/
   • unittests/
   • Version information
   • (optional) Build Script (targets: clean, build, clean-build)
     • [name]_[version]-src.zip
     • [name]_[version].jar
     • [name]_[version].zip
       • doc/
         • manual.pdf
         • changelog.txt
       • examples/
       • unittests/
       • javadoc/ (optional)
       • [name]_[version]-src.zip
       • [name]_[version].jar
     • [name]_[version]-project.zip (Contains everything in workspace)
     • See Example: Non-OSGi
2. Create a homepage for the plugin in wiki (for example org.simantics.databoard) that contains:
   • Development documentation that is not written in javadoc (basic concepts, components etc.) [We should discuss more about what and how to document] and links to user documentation if the plugin contributes new user interfaces.
   • Change log
   • Plugin roadmap
   • Who are responsible for the plugin
   • (optional) Download to artifacts created by Build script
3. Commit the plugin to https://www.simulationsite.net/svn/simantics/<moduleName>/trunk
4. Create a tag of the plugin to https://www.simulationsite.net/svn/simantics/<moduleName>/tags/<version>
5. Inform other developers about the new plugin through the mailing list [simantics-developers@simantics.org]. See also Mailing Lists.

Including a new plug-in to existing features

Inclusion of a plug-in in a feature must always be carefully considered. Consider the following before including anything:

• Cohesion: Does the plug-in logically belong in the considered feature.xml?
• Dependencies: What does the new plug-in depend on? Which dependencies are optional? Do the dependencies of the feature's other plug-ins intersect naturally with those of the new plug-in?
• Permission: do not add anything without permission from the feature maintainer or send a patch and hope it will be merged.

It is highly recommended to have plug-ins included in a single feature only. Instead of including plug-ins in multiple features, prefer creating highly logical and cohesive features that can acyclically include each other. In general completely new functionality should be put in its own feature. If you are unable to decide where a plug-in should be included, ask yourself is the plug-in properly cohesive, are its dependencies minimal?

Note that it is not technically mandatory that each plug-in only resides in one feature - it is simply good practice and will most likely help in training people to use the platform and also make maintainance easier. Having multiple features including the same plug-in will not break builds, but will make the product structure more difficult to understand.

Modifying a feature may break:

1. Project sets
   • FIX: add a line in the related PSF file(s) for the new plug-in in all project sets that include the modified feature
2. Jenkins jobs that depend on the feature
   • See #Building Products for instructions on how to define headless product builds

Releasing a new version of a component

Here, a component stands for a plug-in, a set of plug-ins or a feature.

1. Increment version number(s), incl. MANIFEST.MF, plugin.xml, feature.xml, build.xml, homepage.
   • Bug fixes increment service number, API changes minor number and large releases major number.
2. Run test cases or test suite
3. Check java docs are compiled without errors. Read Javadocs
4. Update homepage
5. (optional) run build.xml and upload generated artifacts to homepage
6. Create SVN tag: trunk/ → tags/[version]
7. Inform other developers Simantics developers list.

Making refactorizations that affect other plugins

1. Plan refactorization beforehand with the maintainers of all affected plugins
2. Create a new branch for all affected plugins
3. Refactor, different possibilities
   • One person refactors all plugins
   • Every maintainer refactors his/her own plugins
     • Create rename scripts for other maintainers to help them
4. Check that everything works
5. If changes were large inform other developers in the mailing list
6. Commit plugins to trunk

Developing and maintaining a released plugin

Generally, people need to be able to commit their changes continuously to the SVN location dedicated for a component. Often committing is necessary for transferring their changes to another machine or just to keep their code safe. We can't allow developers committing API changes or broken temporary code into SVN locations that other projects are using - those need to be stable to some degree. Hence, the active development and API breaking must occur in SVN locations that are documented to be unstable and modifiable only by component maintainer(s). These locations shall be released to the general public according to the roadmap of the component. For every release, a tag shall be created. For every release external dependencies need to be documented: component name, version to use (tag). Releases are versioned. Version numbers are composed of three (3) segments: 3 integers and a string respectively named major.minor.service. Read [2] and [3].

1. the major segment indicates breakage in the API
2. the minor segment indicates "externally visible" changes
3. the service segment indicates bug fixes and the change of development stream (the semantics attached to development stream is new to this proposal, see below)

By default, the following SVN layout should be used for components:

component/

trunk/

• The main development branch, unstable by nature. This version should only be used by active developers of this component.

branches/

• Branches should be created:
  • when preparing for a release, for stabilizing the component while letting active development run along in trunk.
  • for working on bug-fix releases.
  • when working on larger changes, trying out things or working with other developers. These kinds of branches should either be integrated back to where they were branched from or removed as obsolete.

tags/

• Released versions of the component. Other developers and project-sets should be using these for development if the component isn't already deployed into the target platform or it is necessary to have the code in your workspace for some other reason.

Developers may initially feel that creating tickets for every measly issue is too time consuming but it is necessary to make any kind of change tracking possible for releases. Obviously during the initial development of a component the code changes a lot and all the time. This is exactly why we have an incubator for starting projects that does not impose these strict rules but is more like a playground where the developer can freely "hack around" disregarding all these policies. A good developer will get started on design, testing and documentation also right from the start. When the developer feels his component is ready for consumption of the general public and the component passes peer review, it shall be ceremonially transferred from incubation into the official project. Each official component should have documentation, unit tests and possibly examples or really good reasons for not having any.

Modifying the target platform

The target platform is a composition of features and plug-ins that contains all the bundles that you as a developer can build your code against when starting from a blank table. It is a mixture of the standard Eclipse RCP platform, other general Eclipse components and also some Simantics components.

It makes sense to deploy into the target platform only if your plug-ins:

• are general, i.e. reusable by many other components
• have little dependencies
• are stable, both by API and functionally

Deploying plug-ins into the target platform will most likely break:

1. Headless PDE builds that depend on the feature
   • Remove the deployed plug-ins from the plugins/ directory of the build structure to take the deployed plug-ins from the target platform
   • The target platform(s) used by the builds must be updated
   • TODO: link to instructions for general hudson and PDE build configuration
2. Project sets
   • FIX: remove lines matching the plug-ins that were moved into the target platform

Deploying plug-ins for the target platform

The Simantics target platform is a P2 repository located at

• http://www.simantics.org/download/_version_/sdk (w/o source)
• http://www.simantics.org/download/_version_/sdk-source (w/ source)

where _version_ is one of head, stable, 1.5.

The repository is built using the buckminster build definition projects at svn:releng/sdk/branches. Ultimately, these builds use the feature project at svn:workbench/trunk/org.simantics.sdk.site.feature/ to define which features and plug-ins are part of the platform.

To include new features or plug-ins into the platform trunk development version:

1. Make your code/data available as plug-ins and/or features in a public repository, e.g. the Simantics repository.
2. Ask the target platform maintainer to include your components in the platform resource maps (see Buckminster#Resource_Maps).
3. Include them in org.simantics.sdk.site feature include them into the platform builds.

Jenkins builds target-simantics-head-sdk and target-simantics-head will take care of building and publishing the trunk platform.

Deprecated instructions

TODO: these instructions are out-of-date for Simantics 1.4+ and the proper use of P2 repositories

Standard Procedure

1. Test your deployed components.
2. Read the instructions below and deploy your plug-ins and features as P2 repositories
3. Make your built P2 repository available on the internet and notify the target platform maintainer to grab your changes and update the official target platform.

There are two main ways of deploying for the target platform:

• deploying plug-ins and fragments directly
• deploying plug-ins and fragments through features

These steps can be performed using the Eclipse export wizard through the Plug-in Development/Deployable features and Plug-in Development/Deployable plug-ins and fragments wizards

When considering whether your component should have a feature or not, consider these points:

• Eclipse products/applications are always made up of either a set of features or a set of plug-ins. In order for someone to use your component in an product, either your features or your plug-ins need to be included somewhere in the product. If a product is using features and you don't provide one, the product either has to create a new feature and include it or include your plug-ins in his own features.
• Features can be used as a way to give a collective version all plug-ins of a component. The feature will identify the versions of its included plug-ins but these need not be identical to the feature version. Not all plug-ins have to change versions while a feature version changes each time a new release of the component is made.
  • IMPORTANT: The feature deployment build will produce a P2 repository containing content.jar, artifacts.jar and two directories, features/ and plugins/. The features-directory will contain the feature as JAR files which is correct when used as a P2 repository. But if you want to install your deployed features into the current target platforms, the feature jars must be unpacked with the JAR file names as the directory names' Don't unpack directly into the features directory, the feature JARs do not contain a common parent folder. For example, if you've built your feature with an ID my.feature and version 1.0.0, the build will produce a file my.feature_1.0.0.jar. The contents of this jar must be unpacked into features/my.feature_1.0.0/. If you copied the JAR files into the features directory and unpack them there, do not leave the JARs in the features directory. This will also break the features.

Building Products

The best way to currently build products is to use the Eclipse tool Buckminster for the job.

Pre-requisites

1. You have a product configuration (.product file) for your product.
2. Your product configuration is based on features, not plug-ins
3. Buckminster locally installed or a jenkins installation with buckminster available

Example

To create your own (headless) buckminster product build follow these steps:

1. Export svn:tutorials/trunk/com.acme.movie.product.site.feature as a template for making your own headless buckminster product build.
2. To customize for your purposes, the following contents need to be configured:
   1. feature.xml
      • Change ID, version, name, provider and everything else to match your purposes.
      • Include all the features your .product file depends on, nothing else.
   2. Resources.rmap
      • Define mappings for all resources (features and plug-ins) needed by your product. See Buckminster definitive guide for help on defining resource maps.
   3. .credentials.properties
      • Insert any credentials needed to access resources defined in your resource map.
   4. site.cquery
      • Modify cq:rootRequest name to match your site feature ID.
   5. jenkins.buckminster.script
      • Replace com.acme.movie.product.site with what ever is your site feature's ID.
      • Replace p2.profileName with a new profile name for your product.
      • Replace p2.rootId to match the uid attribute defined in your Product Configuration file (.product).
      • Set build.label to what ever you want your built product .zip files to be named like.

Using Continuous Integration (Jenkins)

1. Install jenkins (see simantics.org jenkins)
2. Create new free-style software project job with appropriate name
3. Configure the project:
   1. In the Source code management section, use your version control system of choice to checkout the ...product.site.feature you've created for your build in the previous section. For the com.acme.movie example the proper settings would be:
      • Select Subversion Modules
      • Repository URL: https://www.simantics.org/svn/simantics/tutorials/trunk/com.acme.movie.product.site.feature
      • Local Module directory (optional): com.acme.movie.product.site
      • Check-out strategy: Always check out a fresh copy
   2. In the Build section:
      1. Add new Run Buckminster build step
      2. Select a Buckminster 3.7 installation to use
      3. Copy the contents of jenkins.buckminster.script into the commands text box
   3. Last, you'll want to archive the vital artifacts produced by your build. In the Post-build Actions section:
      1. Check Archive the artifacts
      2. Set value of Files to archive to buckminster.output/com.acme.movie.product.site_*-eclipse.feature/*.zip (obviously changing the value according to the name of your product site feature).

That should do it. Further configuration of the build is up to you, i.e. scheduling, parsing warnings, and so on.

External links

• Buckminster Homepage
• Eclipse Buckminster, The Definitive Guide