Shutting down a JVM process gracefully

During the development of eosgi-e4-plugin, the following question was raised: How can we shut down the started OSGi container in the way that shutdown hooks are called?

I did a lot of searching, but could not find a satisfiable solution. Let’s see what I have found.

Executing kill command

There is the option to call kill command on Unix systems and send SIGINT or SIGTERM to the JVM. The problems with this:

  • I do not want to execute one more process just to kill another one if not necessary
  • There is no such command on Windows. There is no command in windows that can send SIGINT or SIGTERM to a process

Using JNA and Native calls

Another option is to use JNA. This works pretty well in Linux, but Windows is still a problem. One can call WM_CLOSE on every window of an application or send a Ctrl+C event to the console of the running process, but there is no solution if the running program neither has a console nor a window. That is the case when a process is started with Eclipse Launcher. The console of the application is redirected; therefore external process cannot send CTRL events to it.

Trick: Use Attach API and a dynamically installed Java Agent

When a programmer calls System.exit(int) or Runtime.getRuntime().exit(int) within a java process, it starts clean shutdown procedure where all of the shutdown hooks are called. The goal is to force a code snippet in the target JVM to call the exit method. How to do that?

With Java Attach API it is possible to connect to any JVM that runs on the same host. It is also possible to load a Java Agent into the connected JVM. What if the Java Agent calls the exit method? That is possible. I tested it and it works well.

To save time for others, I implemented the Java Agent as an OpenSource project and uploaded it to Maven Central. It is configurable. One can tell via the Agent arguments

  • what exit code should be used for the standard shutdown procedure
  • should the JVM shut down forcibly with a different exit code if the standard shutdown procedure does not end within a timeout? This can happen if a non-daemon thread is locked.

Feel free to use this technique and let us know if it worked for you!

Posted in Uncategorized

How do you use OSGi? (part 4)

A couple of weeks ago I sent a form to OSGi dev mailing list. I have started to share the summary of answers with the community. As there were many questions, I decided to split the content and publicize it as several posts. If you are interested, the previous part is here. This is the final part of the series.

Getting information about the container

In the last years, we worked with Blueprint, Declarative Services and we developed our component model. For Blueprint, we created a webconsole plugin; there was already one for DS and we also developed a webconsole plugin for our component model ECM.

Every time I had an issue, I opened webconsole. A bundle did not start? Let’s check it in webconsole. Is a service available? Let’s check it in webconsole. Do others work in the same way as I do? Is it good if we develop only webconsole plugins when we create a technology like ECM? I asked, and I was a bit surprised when I saw the answers:

osgi-survey-summary-infobundle

Others:

  • Cloud RTI has extensive consoles for that
  • Karaf console
  • Apache Karaf Shell has a lot of extra information.
  • custom webconsole
  • karaf shell (3x)
  • Liferay’s App Manager

Oops! I think it is time to start supporting Gogo Shell, too. We will do so in the future.

Deploying modules

While I found webconsole to be excellent to get information about the system, it was difficult to use it to deploy each bundle during development. When we moved to OSGi, we started with FileInstall. That is good for a start, but the IDE should support deploying the bundles with their dependencies in my opinion. I was interested how others solve with this task.

osgi-survey-summary-deploybundles

The first answer is “Development Environment deploys them”.

Other answers:

  • Cloud RTI based, or via Apache ACE
  • Home made launcher
  • Docker
  • Apache Karaf Feature Installer does it, usually the required bundles are grouped as features and deployed accordingly.
  • Karaf console
  • home-made fileinstall that doesn’t yell about duplicate bundles
  • Maven and bundle:watch functionality of Apache Karaf when container is running, Apache Karaf features otherwise
  • Automated IDE Construction via Tycho
  • exported runnable jar
  • karaf shell and features
  • Manually
  • Karaf features
  • maven
  • Apache Sling OSGi Installer
  • liferay-ide

Third party modules

I was interested, how others search for third party modules that have the required functionality. I typically find a technology by searching on google, download it from maven central and check if the MANIFEST headers exist. I had the idea that others might know better tricks, so I asked:

osgi-survey-summary-3rdparty

Others are:

  • Sling provisioning model
  • mostly manual, or a combination of the above, and we copy them into the project
  • search.maven.org, usually you’ll find that the “bundle” in question is or is not build with the maven-bundle-plugin, that’s usually good enough as indicator, if not I’m using bundles from the Apache ServiceMix Project
  • ServiceMix
  • mvnrepository.com
  • p2-maven-plugin
  • Index-based search across repos within NetBeans

What problems do developers face during working with OSGi?

My first years with OSGi was a torture for me. There was a new mystery every day that we had to solve. The learning curve was extremely high. However, I believed that one day we would benefit from the sacrifice we had made.

We had to learn which patterns and technologies do not work in OSGi. Magical patterns like AOP that we loved before became our biggest enemy.  It is getting much better and I must say that our development workflow is much more effective than it was before.

I was interested if others have issues like we had. I got the following answers:

Availability of OSGi metadata from source (or correctness when available), though it’s improving (thanks ServiceMix!). Class loader and caching.


A lot of third party code is tough to modularize


Digging for dependencies when some external libraries need to be repackaged + Widespread use of ThreadContext classloader and Class.forName


The tools are still poorly integrated, dealing with services/remote services WRT tooling is overly complicated


Third party libraries with poor support for OSGi


Training younger developers and importing non osgi compatible libraries


Disconnect between OSGi community.

Camp 1 – OSGi Alliance promotes BndTools, Eclipse, Declarative services.

Camp 2 – Open source Karaf/Aries – most tutorials/solutions/user forum based around Blueprint rather than DS, use Maven rather than BndTools as poor Karaf BndTools integration.

Camp 3 – others -> Everit, Amdatu, Osgiliath, SpringDM, Gemini Blueprint, CDI

How does a new person starting with OSGi know where to start given all these different options?

Coming from JEE or Spring background it is not as easy developing OSGi applications.

Maturity of OSGi technology e.g. only recently has a speced solution for DS with transactions/JPA/JDBC been implemented.

JEE/Spring offers a lot out of the box that is easy to use e.g security, easy integation with Camel, Web services etc things that would often been considered the bread and butter in the ‘Enterprise’ space.

Documentation/examples/tutorials lacking compared with say the Spring reference manuals


No technical problems, but coworkers often lost need help because there’s very few resources online to fix problems, because of the size of the community


Integrating external technologies (like eg. Vaadin for Web UIs) which are not (really) OSGi ready in terms of missing dependencies and requirements at runtime


Integration with EJB, framework state/persistence


Dynamic proxy based Spring AOP


No IDE Refactoring for changing service reference binding (lazy) to interface binding (eager) and vice versa. No IDE Templates for using service lookup where component binding are unavailable. No IDE Templates for common OSGI methods including annotations.
ConfigurationAdmin multiton management is cumbersome especially when updating existing configuration. ConfigurationAdmin should have plugable persistence so that configuration may come from somewhere else other than the bundle storage (i.e. a Database).
IDE assistance and proposals for LDAP filter string generation.
Using Junit Test classes as a component to do integration/system tests.


steep leaning curve, feel hard to debug what’s went wrong


DS service outjection would be nice …


Adaption of legacy code to dynamics, elimination of technical deptthe utter lack of useful documentation


Some 3rd-party libraries still don’t play nice (e.g. still use Class.forName). OSGi is still not as wide-spread – despite being so much better – in the heads of people as e.g. Spring is. Sometimes missed visualization of wiring and state to analyze deployment problems in complex setups.


osgi repository is not so good to use


Tough learning curve for new developers.


Reaching out to customers is much harder than using the Java EE keyword.

Wow! People have lot’s of things in mind. I do not know how it is with others, but it was very instructive for me to read answers.

We were facing some of the issues, and we made solutions for some of them. I will work hard on making these solutions available to others within the next weeks/months. Where contact address was provided together with the answer, and we might have a solution, I will get in touch.

Posted in Uncategorized

Pictures about Everit OSGi Eclipse plugin

We are close to the first release of the Everit OSGi Eclipse plugin that will:

  • will show the OSGi environments that are configured in eosgi-maven-plugin
  • lets the programmer start, stop and debug the OSGi environments
  • supports live-upgrade of the environment by updating all dependencies (also the ones that are opened in the workspace as projects) on the container

Here is a picture how you will see the OSGi environments in Eclipse:

eosgi-e4-plugin-projexplorer

If you want to get notified about the new release, follow us on twitter.

 

Tagged with: ,
Posted in Uncategorized

How do you use OSGi? (part 3)

A couple of weeks ago I sent a form to OSGi dev mailing list. I have started to share the summary of answers with the community. As there were many questions, I decided to split the content and publicize it as several posts. If you are interested, the previous part is here.

Preferred platforms

I was curious what platforms people use. We use only nature OSGi containers at the moment with self-selected modules. We set up the Dev, Test and Production server with a self-written maven plugin. However, this maven plugin could setup an environment based on a complex project, too (e.g.: bundling glassfish). It is time to check what others use, what we should support next:

osgi-survey-summary-platformsMust note that the third answer is Apache Sling / Adobe AEM and the 6th answer is Nature OSGi container.

List of other answers:

  • Equinox standalone
  • JBoss AS OSGi
  • Equinox
  • Eclipse Equinox/E4
  • Apache Felix
  • felix

I must say that I am a bit surprised. A percent of the OSGi related job advertisements look for Adobe AEM developers based on my research. Where were you Sling / AEM programmers?

Component model

I had the hypothesis that Declarative Services is the most preferred component model in the OSGi world. Seems that I was right:

osgi-survey-summary-componentmodel

Other answers were:

  • Eclipse e4
  • Apache Felix Dependency Manager (2x)
  • Plain Old OSGi Servicetrackers
  • Currently migrating from SpringDM to DS
  • Domino (Scala DSL)
  • Native OSGI
  • plain osgi
  • Everit Component Model

I must say, we have a winner :). And that is no surprise; Declarative Services has far the greatest concept from all I have seen (I used to develop Java EE and Spring and Blueprint based applications). I disagree with only one part of the concept, how it handles multi-cardinality references.

Application Server

There might be a bit of misunderstanding here. I was interested if anyone uses a Java EE server below OSGi. Nowadays most of the Java EE Application Servers are built on top of OSGi somehow (E.g.: Glassfish beginning version 3.0). Here are the answers:

osgi-survey-summary-appservers

Other section contained Karaf and Tomcat several times.

Preferred persistence technology

Persistence is a fascinating question in the OSGi world. As former Java EE and Spring developers, we new JPA. After switching to OSGi, we tried to push JPA into our new projects, too. I re-implemented the JPA chapter of the OSGi Enterprise specification from ground to get the best support. It seems that many other developers are in a similar situation that we were:

osgi-survey-summary-persistence

Other mentioned persistence technologies:

  • Jinq
  • Jackrabbit Oak
  • Our own solution: Information Grid
  • Mongodb
  • Cassandra
  • Straight JDBC and JdbcTemplate
  • Neo4j
  • Plain JDBC
  • Jinq
  • Hibernate (through a custom framework called ServiceBuilder)
  • File-System

After a year of struggling, we decided to look around. I analyzed all of the persistence related technologies I could find. I came up with the idea to use Liquibase with Querydsl. Since that, I feel that I came out from the sea of unexpected issues.

To all JPA users: I honestly suggest that you should look around! JPA and OSGi just do not fit together in concept, no matter how much of free time is sacrificed by enthusiastic geeks.

Web interface

I was interested what others use to generate web user interfaces. I saw so many trends in the last two decades but saw almost no satisfied developer. When OSGi comes into the picture, the question is even more interesting as many of the concepts will fail on top of it. As I said, there are many technology stacks. I wrote a couple of examples but expected to have more other answers than anything else. Here are the answers:

osgi-survey-summary-webui

The Other answers in details:

  • Vaadin (5x)
  • JSP., Sightly
  • react.js, scala.js
  • Angular (4x)
  • Our web UI is not written in OSGi and is mainly HTML/CSS/JS
  • JQuery, Knockout
  • Web Components
  • JSPs, Portlets, Freemarker, Velocity, Soy, XSLT, and many others
  • JSP, Metal.js, AlloyUI ,jQuery
  • Liferay’s MVCPortlet
  • Alloy UI
  • liferay mvc portlet (2x)
  • Custom (2x)

Wow! I must say this is fragmented. Probably there are more AngularJS + Rest users than others. One reason can be that enRoute gives samples for this solution.

For applications that have many static contents and forms, but not rich-client like behavior, I found server side templating a better choice. I had an idea how to do server-side templating. I did a search and found the project Thymeleaf  that had very similar syntax that I had in mind. The issue is that Thymeleaf has other features that do not fit into a modular concept, and the developers did not want to put the OSGi Manifest headers into their JAR files. Therefore, I reimplemented what I need in a very simple project: https://github.com/everit-org/templating-html.

Since we use this concept, the learning curve was greatly shortened and the code keeps simple. It is only about templating and rendering the HTML output. As for navigation, we only have project specific solutions yet.

Are there other useful technologies?

I do a lot of searching on the internet. I was interested if I missed something. Here is the list of answers:

  • bnd-maven-plugin, karaf-maven-plugin
  • Bnd & Bndtools
  • We’re already doing a little OSGi development tooling as part of https://sling.apache.org/documentation/development/ide-tooling.html . Maybe it would be interesting to collaborate in an open source environment at Apache.
  • Amdatu
  • bnd, Scala, Apache Felix dependencymanager.lambda
  • bnd command line
  • EventAdmin, HttpService, ECF Remote Services
  • gradle-karaf-plugin, gradle-bundle-plugin
  • just bndtools
  • Gson and Jackson, Flyway, Docker, …
  • The current Apache Karaf Maven Plugin which checks possible runtime requirement problems right at compile time for given Apache Karaf Features
  • Clirr reports
  • DS Annotation Builder
  • Bnd tools / enRoute
  • CM, Metatype, Baseline, bnd
  • Sublime Text, node
  • SSH/SCP, ConfigAdmin + FileInstall in Karaf
  • BND command line to introspect bundles
  • Eclipse
  • bnd tools, blade tools
  • The ones we implemented 😉

What’s left?

Part 4 will be the last in this series. The questions of bundle deployment, how to find the best solution, and the question of complaints are left. Stay tuned!

 

Tagged with: , , ,
Posted in Uncategorized

How do you use OSGi? (part 2)

A couple of weeks ago I sent a form to OSGi dev mailing list. I have started to share the summary of answers with the community. As there were many questions, I decided to split the content and publicize it as several posts. If you are interested, the first part is here.

In the last couple of months, I visited some companies. I had the idea that every OSGi developer uses Eclipse. I was surprised as I saw that many developers prefer InterlliJ even without direct OSGi support. I had to ask the following question to get a clear picture:

osgi-survey-summary-devenvThe second answer is Eclipse + BndTools. Sadly Google cuts the end of it down. There might be some overlappings as two developers responded Eclipse and Eclipse + BndTools together. There were some developers, who filled the other section with the following contents:

  • Command line
  • Vim
  • Looking at moving from Netbeans to Eclipse + BndTools

We are developing an Eclipse plugin at the moment to support starting, stopping and upgrading OSGi containers based on Maven configuration. We will check if we can develop an IntelliJ plugin as it seems to be very popular, too.

I was interested if people are satisfied with their IDE or they need other features. So I asked the following question:

What features do you miss in your development environment during your work?

People had lots of ideas for new features here:

  • Bndtools for IntelliJ would be nice, if 100% compatible
  • Most IDEs seem to ignore the fact that the real work is done by the CI; it’s fairly useless to integrate testing in an IDE if it cannot be scripted and run consistently on a CI server
  • IntelliJ has very poor support for OSGi compared to its Java EE and Spring integration
  • Stability. Eclipse crashes a lot and the plugis will break other plugins. I have lost maven capabilities on a number of instances and have to reinstall
  • Integration between BndTools and Karaf
  • a component diagram generated from DS or other component frameworks would be very nice to get a higher-level overview ; bndtools not used because only Eclipse and forces conventions on us (e.g cnf/) that leak into git, and our build.
  • Manifest Viewer and Diff
  • Dependency resolution at build time based on OSGI meta data
  • Debug Tooling and visualization for service/ component startup. IDE based Service Catalog for finding and using appropriate services with descriptions (meta data service) how to use them as a developer.
  • something that can ease for debugging
  • P2 Repo support in Bnd
  • realtime baseline error feedback
  • none
  • BndTools (the same developer answered that he/she develops with IntelliJ)
  • Looking for developer studio for IntelliJ 🙂

All of them are very valuable answers to me. However, I would like to highlight one:

a component diagram generated from DS or other component frameworks would be very nice to get a higher-level overview

I talked to other guys, and it seems that many of us want this feature. I wanted it, too. That is why I started to develop a WebConsole plugin for our component model. The latest version (that includes the graph) of the webconsole plugin is not released yet, but the majority of the work is done. Here is a screenshot in case you are interested:

osgi-survey-summary-ecmgraph

Part 3 will come within a couple of days. Stay tuned!

Tagged with: , ,
Posted in Uncategorized

How do you use OSGi? (part 1)

Two weeks ago I sent a form to OSGi dev mailing list. My goal was to have an idea how others use OSGi in their projects. In the last couple of years, we developed 100+ OpenSource modules, Maven plugins and released them on Maven Central. We found these libraries very useful in our projects, but I was wondering how others solve the same goals.

As there were several questions, I share the summary of the responses in series of blog posts. I will also add the following information to each questions:

  • Why I asked it?
  • What conclusion did I draw from the answers?

Number of answers

41 answers arrived, including mine. I guess only geeks are subscribed to the OSGi dev mailing list, and the opinion of those geeks represent the view of their close colleagues, too.

I asked if I can share their answer anonymously. Here is the summary of responses:

osgi-survey-summary-sharetheanswer

I do not think I have to comment on this one :).

Build tool fragmentation

There are several build tools for Java (ANT, Maven, Gradle, …), and all of them has some support for OSGi. I was interested if the community has already chosen the best one for OSGi development.

osgi-survey-summary-buildtool

The following tools appeared in the other section:

  • PDE
  • Tycho
  • Bndtools
  • Liferay Gradle plugin

I am pretty happy with these answers as they showed the same what my hypothesis was. There is no preferred build tool that would get 90% of the votes. Maven is more popular than Gradle at the moment despite the fact that Gradle is advertised by several core members of OSGi.

Missing features in Build tools

11 people answered the question about the missing features of build tools. Here is a list of answers:

  • P2 repo. support
  • nothing really (we mostly implement ourselves what we miss)
  • A simple/lightweight method for packaging and running automated tests in OSGi; existing solutions involve a lot of ‘useless magic’. I usually use a home made launcher.
  • Simplicity
  • More granular compilation and hot deployments (using Karaf)
  • I think they do everything but they are hard to use and don’t cooperate very well. An example is I have a bnd build configured for bndrun and resolution but that doesnt work with my maven build terribly well as I still have to have a workspace. I also like to use eclipse but some others don’t so it would be nice if they were ide agnostic.
  • we use maven with maven-bundle-plugin and not bnd (reasons: bnd being too OSGi centric, bad IntelliJ support, lacking extra maven features); we miss: proper packaging (maven assemblies vs OSGi resolved distributions); maven’s lack of flexibility ; maven’s lack of automated semantic versioning at releases.
  • Easy unit tests with DS/SCR w/o container
  • none
  • Fast deployment, hot code replacement
  • Ant based Plugin SDK which we has till 6.2 though we have gradle based liferay workspace but it has not fullfledged features.

Build tool plugins of Everit?

We have a Maven plugin, too, that solves the following issues (other developers mentioned some of them):

  • Generating OSGi environments and adding all maven dependencies to them
  • The OSGi environments are described in ZIP files, and they are also maven artifacts. Therefore, any company can make a custom environment (e.g.: packaging a Glassfish or Liferay as an environment)
  • Starting the environments during the integration-test phase of maven to run tests that are implemented in one or more of the dependencies (or in the project that specifies the environment)
  • Hot replacement of the modules with a maven build if the environment is already started

If you have not used it till now, do not start using it now. Version 4.0.0 will come within weeks and besides that it will lots of new features, and it will be incompatible with the 3.x branch. I will write a detailed blog post about the new version and its usage when it released.

Development Environments

The next chapter about the questions of Development Environments will come within a couple of days. Stay tuned!

Tagged with: , , , ,
Posted in Uncategorized

Everit Email Modules are available on maven-central

Introduction

I am happy to announce that three new modules are available to use that simplify the job of sending emails:

  • email-api: Interfaces and classes to handle emails
  • email-javamail: Java Mail (JSR 919) based implementation of email-api
  • email-javamail-ecm: ECM components to use email sending functionality easily within OSGi environments

Programmers can use the API to:

  • Easily create emails with TEXT or HTML content or both
  • Add inline images to the HTML content
  • Add attachments
  • Send emails
  • Send multiple emails with on one channel (faster processing)

Why did we need another API?

We investigated the available APIs in Java and found the following:

  • It is very difficult to use javax.mail
  • Constructing emails and sending email functionality are not clearly separated in commons-email. By not having the separation, it is not possible to build solutions where the same interface can be used to send or persist emails in a queue
  • Spring needs lots of dependencies that do not work well within a modularized environment

 

Posted in Uncategorized