ALU

jvm

The JVM Advent Calendar: JCP.NEXT, Streamlining the JCP Program

Java Community Process (JCP) version 2.11, with a focus on streamlining the JCP program takes effect December 14th.

This blog post is on the latest JCP.Next effort to streamline the JCP program JSR lifecycle in response to feedback from the Java development community and the faster release cadence introduced in 2017.

Original Link

Apache Cayenne 4.0 —What It Is All About

Didn’t have much time for writing lately. Still felt the topic of Apache Cayenne 4.0 is important enough to say a few words, even though this is coming to you months after the final release. So here is a belated overview of the release scope and its significance. Enjoy!

As you may know, Cayenne is a Java ORM, i.e. an intermediary between your Java objects and the database, that keeps object and relational worlds in sync and lets you run queries and persist data. The current stable version of Cayenne is 4.0 and it was released in August 2018.

Original Link

The JVM Advent Calendar: How to Make Awesome Puzzlers With Java 12

Java 12 provides, in experimental form, a switch expression and new forms of the switch and break statements. There is a profusion of new syntax and semantics for constructs that may find little use-except, of course, for authors of puzzlers and certification exam questions for whom this is a wonderful gift. If you enjoy Java puzzlers and would perhaps like to create some yourself, read on.

The Java 12 Expression Switch

Java 12 introduces an expression switch — a version of switch that is an expression, not a statement. Here is a simple example:

Original Link

How to Create Modular Jars That Target a Java Release Before 9

If you’re a library author, you probably want to see your library used in a large number of applications. One way to increase the number of applications that can include your library is to make it compatible with older Java versions. At the same time, you may consider modularizing your library in order to make it attractive for applications that take full advantage of the Java Platform Module System (JPMS).

However, the JPMS is implemented only by Java 9 and newer versions. So, how can you build a library that is both modularized and compatible with Java versions before 9?

Original Link

Difference Between JDK Vs. JRE Vs. JVM

Today, we will discuss the introduction to JDK vs JRE vs JVM.

What Is the Java Development Kit (JDK)?

The JDK stands for Java Development Kit used for developing Java applets and apps. It is basically a software development environment. JDK includes Java Runtime Environment (JRE), which happens to be a loader/interpreter, a compiler called (javac), a document generator (Javadoc), an archiver (jar), and other tools required in development.

Original Link

Getting Started With Kotlin and Maven [Video]

When discussing Kotlin and Maven, conversations always start with the boring yet utterly important question: how do you create a Kotlin project? In this episode, we will look at how to create one from scratch with Maven, including a small Hello World example at the end. This video is highly recommended for all future episodes. Enjoy!

Original Link

JVM Ecosystem Survey: Why Devs Aren’t Switching to Java 11

Last week, Oracle’s Java Magazine and Synk released the JVM Ecosystem Report. This survey talked to over 10,000 developers across the globe about their choice of JVM languages, platforms, tools, processes, and applications.

This report shows that 88 percent of developers are still using Java 7 or 8 in their main application, with 8 percent using Java 9 and 10. Since Java 11 is the most recent version of the JDK, this brings up the question: why aren’t developers switching to more recent versions?

Original Link

Lessons From the JVM Ecosystem Report

Last week, Java Magazine and Snyk released an interesting survey report called the JVM Ecosystem Report. Zeroturnaround and JRebel have done similar surveys in previous years. The Eclipse Foundation also completed a Java EE survey earlier this year. These types of surveys always have their own biases, but I found that they provided an interesting snapshot into the developer community.

In this case, this report provides a good snapshot into the tools Java developers are using. It is definitely worth reading the entire report. Here are five things I found interesting from this year’s report.

Original Link

Java Microservices in the Cloud with Zing and Kubernetes

Many applications are now being developed using a microservices architecture, moving from monolithic designs to ones built from well defined, isolated components. Although this is a good starting point, how these individual services work together to deliver the required functionality requires orchestration. Kubernetes is an open-source container orchestration system for automating deployment, scaling and management of such applications.

In this on-demand webinar, we’ll look at how the Zing JVM from Azul can be deployed into different clouds to enable Java-based microservice applications using Kubernetes to take advantage of the performance enhancements it provides.

Original Link

JEP 181 Incompatibility: Dealing With Nesting Classes

JEP 181 is a nest-based access control introduced in Java 11, and it deliberately introduced an incompatibility with previous versions. This is a good example explaing that being compatible with prior versions of Java is not a rule carved into stone, but it rather is to keep the consistency and steady development of the language. In this article, I will look at the change with an example that I came across a few years ago and how Java 11 makes life easier and more consistent in this special case.

Java backward compatibility is limited to features, but not to behavior.

Original Link

10 Effective Tips on Using Maven

Maven is — without a doubt — the most popular build automation tool for software projects in the Java ecosystem. It has long replaced Ant, thanks to an easier and declarative model for managing projects, providing dependency management and resolution, well-defined build phases such compile and test, and support for plugins that can do anything related to building, configuring, and deploying your code. It is estimated to be used by 60 percent of Java developers in 2018.

Over the years, a number of usage scenarios and commands turned out to be quite useful for me when working on Maven-based projects. Here are a few usage tips that help in using Maven more effectively. There are many more, and one can obviously learn something new every day for a specific use case, but these are the ones I think can be commonly applied. The focus here is on aspects like command line usage, troubleshooting a certain issue, or making repetitive tasks easier. Hence, you won’t find practices like using dependencyManagementto centralize dependencies, which are rather basic anyway and more used in initially composing a POM.

Original Link

Meet Us at Oracle Code One and EclipseCon Europe

Next week, two important Java conferences are happening at the same time: Oracle Code One (previously Java One) in San Francisco, California, and EclipseCon Europe in Ludwigsburg, Germany.

The Open Liberty team is going to be out in full force — manning booths, giving talks, and most importantly, handing out cool swag, like Open Liberty t-shirts!

Original Link

Factory Pattern in Kotlin

Kotlin is one of the most competitive JVM languages. It has been created by JetBrains and it keeps rising. Google recently endorsed Kotlin as an officially supported language for Android development, enhancing Kotlin’s profile, but it also applies to server-side development. For example, Kotlin works smoothly with Spring Boot.

The factory method pattern is one of the creational patterns of the well known “Gang of Four” design patterns. In the factory pattern, we can create objects by calling a factory method rather by calling a constructor. As a result, we don’t need to specify the class of the created object. It provides different approaches to code for interface rather than implementation, and it makes our code less coupled and more extendable.

Original Link

Troubleshooting Java Applications With Arthas

Arthas is a Java Diagnostic tool open sourced by Alibaba. Arthas can help developer trouble-shoot production issues for Java applications without modifying your code or restarting your server.

Let’s start with a simple demo to see how we can use Arthas to trouble-shoot this Java program on-the-fly.

Original Link

Why Kotlin Is the Future of Android App Development?

It is an exciting time to be an Android developer. A big reason behind this excitement is the official support that Google has extended to the programming language Kotlin. Kotlin gives developers the features they asked for. It is a statically-typed programming language that can run on the Java Virtual Machine. It is an open source, general purpose and pragmatic computer programming language that combines both the object-oriented and functional programming features within it. Kotlin was developed in 2011 by the programmers of a software development company called JetBrains in Russia and was then augmented by several open source developers.

Kotlin is relevant today because of two reasons. It has been developed as a solution to the problems that Android developers have faced over a period of time. Therefore, it answers most of the main issues that surfaced in Java, providing developers with inter-operability, safety, clarity, and tooling support.

Original Link

How to Create java.awt.Color From String Representation

When using the Text-IO library, you can configure the text terminals via properties files. Colors can be specified using standard HTML names (e.g. orange, chocolate, crimson), hex strings (e.g. #ee5533, #e53, 0xEE5533), rgb format strings (e.g. rgb(217,33,119), rgba(112,25%,50%,0.9)), or hsl format strings (e.g. hsl(240,90%,70%), hsla(270,0%,100%,0.7)).

How does Text-IO create colors from these string representations? Until recently, the implementation used the static method web(String colorString) provided by javafx.scene.Color. This method supports all the above mentioned formats and returns a javafx.scene.Color instance. Text-IO needs a java.awt.Color, but creating one from the returned javafx.scene.Color is straightforward.

However, I was not happy with this solution. It made my code depend on JavaFX, although Text-IO doesn’t actually use JavaFX. And, starting with JDK 11, JavaFX is no longer part of the JDK.

So, I decided to write a little library named AWT Color Factory for creating java.awt.Color  objects from string representations. The simplest way to do this is to copy the code of the javafx.scene.Color.web(String colorString) method and adapt it to create  java.awt.Colorinstances.

But, what about licensing? JavaFX is released under the GPL, so my library must use the same license. Text-IO is, instead, released under the Apache-2.0 license. Is it then allowed to use my library? Yes, it is. The license under which JavaFX iwas released is actually not GPL but the GPL 2 with Classpath exception. The same applies to my library. This is very important, because the Classpath exception clause gives you permission to include the AWT Color Factory library in your executable code, regardless of the license under which your software is distributed.

Example Usage

Color c1 = ColorFactory.valueOf("firebrick");
Color c2 = ColorFactory.valueOf("#aa38e0");
Color c3 = ColorFactory.valueOf("0x40A8CC");
Color c4 = ColorFactory.valueOf("rgba(112,36,228,0.9)");
Color c5 = ColorFactory.web("forestgreen", 0.7);
Color c6 = ColorFactory.web("hsl(270,90%,70%)", 0.8);

AWT Color Factory is available in JCenter and Maven Central.

Maven

<dependency> <groupId>org.beryx</groupId> <artifactId>awt-color-factory</artifactId> <version>1.0.0</version>
</dependency>

Gradle

compile 'org.beryx:awt-color-factory:1.0.0'

The library requires Java 7 or newer. The jar artifact is modularized under the name org.beryx.awt.color.

Starting with version 3.2.0, Text-IO uses the AWT Color Factory library and no longer depends on JavaFX.

Original Link

Class Sharing in Eclipse OpenJ9: How to Improve Memory, Performance (Part 2)

Memory footprint and startup time are important performance metrics for a Java virtual machine (JVM). The memory footprint becomes especially important in the cloud environment since you pay for the memory that your application uses. In this tutorial, we will show you how to use the shared classes feature in Eclipse OpenJ9 to reduce the memory footprint and improve your JVM startup time.

Runtime Bytecode Modification

Runtime bytecode modification is a popular way of instrumenting behavior into Java classes. It can be performed using the JVM Tools Interface (JVMTI) hooks (details can be found here). Alternately, the class bytes can be replaced by the class loader before the class is defined. This presents an extra challenge to class sharing, as one JVM may cache instrumented bytecode that should not be loaded by another JVM sharing the same cache.

However, because of the dynamic nature of the OpenJ9 Shared Classes implementation, multiple JVMs using different types of modification can safely share the same cache. Indeed, if the bytecode modification is expensive, caching the modified classes has an even greater benefit, as the transformation only needs to be performed once. The only provision is that the bytecode modifications should be deterministic and predictable. Once a class has been modified and cached, it cannot be changed further.

Modified bytecode can be shared by using the modified= sub-option to -Xshareclasses. The context is a user-defined name that creates a logical partition in the shared cache into which all the classes loaded by that JVM are stored. All JVMs using that particular modification should use the same modification context name. They all load classes from the same shared cache partition. Any JVM using the same shared cache without the modifiedub-option finds and stores vanilla classes as normal.

Potential Pitfalls

If a JVM is running with a JVMTI agent that has registered to modify class bytes and the modified sub-option is not used, class sharing with other vanilla JVMs or JVMs using other agents is still managed safely, albeit with a small performance cost due to extra checking. Thus, it is always more efficient to use the modified sub-option.

Note that this is only possible because the JVM knows when bytecode modification is occurring because of the use of the JVMTI API. Redefined and retransformed classes are not stored in the cache. JVM stores vanilla class byte data in the shared cache, which allows the JVMTI ClassFileLoadHook event to be triggered for all classes loaded from the cache. Therefore, if a custom class loader modifies class bytes before defining the class without using JVMTI and without using the modified sub-option, the classes being defined are assumed to be vanilla and could be incorrectly loaded by other JVMs.

For more detailed information on sharing modified bytecode, see here.

Using the Helper API

The Shared Classes Helper API is provided by OpenJ9 so that developers can integrate class sharing support into custom class loaders. This is only required for class loaders that do not extend java.net.URLClassLoader, as those class loaders automatically inherit class-sharing support.

A comprehensive tutorial on the Helper API is beyond the scope of this article, but we will provide a general overview. If you’d like to know more details, you can find the Helper API implementation on GitHub.

The Helper API: a Summary

All the Helper API classes are in the com.ibm.oti.shared package. Each class loader wishing to share classes must get a SharedClassHelperobject from a SharedClassHelperFactory. Once created, the SharedClassHelperbelongs to the class loader that requested it and can only store classes defined by that class loader. The SharedClassHelpergives the class loader a simple API for finding and storing classes in the shared cache. If the class loader is garbage collected, its SharedClassHelperis also garbage collected.

Using the SharedClassHelperFactory

The SharedClassHelperFactoryis a singleton that is obtained using the static method com.ibm.oti.shared.Shared.getSharedClassHelperFactory(), which returns a factory if class sharing is enabled in the JVM; otherwise, it returns null.

Using the SharedClassHelpers

There are three different types of SharedClassHelper that can be returned by the factory. Each is designed for use by a different type of class loader:

  • SharedClassURLClasspathHelper: This helper is designed for use by class loaders that have the concept of a URL classpath. Classes are stored and found in the shared cache using the URL classpath array. The URL resources in the classpath must be accessible on the filesystem for the classes to be cached. This helper also carries some restrictions on how the classpath can be modified during the lifetime of the helper.
  • SharedClassURLHelper: This helper is designed for use by class loaders that can load classes from any URL. The URL resources given must be accessible on the filesystem for the classes to be cached.
  • SharedClassTokenHelper: This helper effectively turns the shared class cache into a simple hash table — classes are stored against string key tokens that are meaningless to the shared cache. This is the only helper that doesn’t provide dynamic update capability because the classes stored have no filesystem context associated with them.

Each SharedClassHelperhas two basic methods, the parameters of which differ slightly between helper types:

  • byte[] findSharedClass(String classname...) should be called after the class loader has asked its parent for the class (if one exists). If the findSharedClass() does not return null, the class loader should call the defineClass()on the byte array returned. Note that this function returns a special cookie for the defineClass(), not actual class bytes, so the bytes cannot be instrumented.
  • boolean storeSharedClass(Class clazz...) should be called immediately after a class has been defined. The method returns true if the class was successfully stored and false otherwise.

Other Considerations

When deploying class sharing with your application, you need to consider factors such as security and cache tuning. These considerations are briefly summarized below.

Security

By default, the shared caches are created with user-level security, so only the user that created the shared cache can access it. For this reason, the default cache name is different for each user so that clashes are avoided. On UNIX, there is a sub-option to specify groupAccess, which gives access to all users in the primary group of the user that created the cache.

In addition to this, if there is a SecurityManager installed, a class loader can only share classes if it has been explicitly granted the correct permissions. Refer to the user guide here for more details on setting these permissions.

Garbage Collection and Just-in-time Compilation

Running with class sharing enabled has no effect on class garbage collection (GC). Classes and class loaders are still garbage collected, just as they are in the non-shared case. Also, there are no restrictions placed on GC modes or configurations when using class sharing.

It is not possible to cache just-in-time (JIT) compiled code in the class cache. The AOT code in the shared cache is also subject to JIT compilation, and it affects how and when a method is JIT’ed. In addition, the JIT hints and profile data can be stored in the shared cache. You can use options Xscmaxjitdata and -Xscminjitdata to set the size for shared cache space for such JIT data.

Cache Size Limits

The current maximum theoretical cache size is 2GB. The cache size is limited by factors such as available system memory, available virtual address space, available disk space, etc. More details can be found here.

Example

To practically demonstrate the benefits of class sharing, this section provides a simple graphical demo. The source and binaries are available on GitHub.

The demo app works on Java 8 and looks for the jre\lib directory and opens each JAR, calling Class.forName() on every class it finds. This causes about 16,000 classes to be loaded into the JVM. The demo reports on how long the JVM takes to load the classes. This is a slightly contrived example, but it effectively demonstrates the benefits of class sharing. Let’s run the application and see the results.

Class-loading Performance

1. Download JDK with OpenJ9 from the Adopt OpenJDK project or pull from the docker image.

2. Download shcdemo.jar from GitHub.

3. Run the test a couple of times without class sharing to warm up the system disk cache, using the command in Listing 11:

Listing 11. Warming up the Disk Cache

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -Xshareclasses:none -cp shcdemo.jar ClassLoadStress

When the window in Figure 1 appears, press the button. The app will load the classes.

Figure 1. Press the button

Figure 1

Once the classes have loaded, the application reports how many it loaded and how long it took, as Figure 2 shows:

Figure 2. Results are in !

Figure 2

You’ll notice that the application probably gets slightly faster each time you run it; this is because of operating system optimizations.

4. Now, run the demo with class sharing enabled, as Listing 12 illustrates. A new shared cache is created. You can specify a cache size of about 50MB to ensure that there is enough space for all the classes. Listing 12 shows the command line and some sample output.

Listing 12. Running the Demo With Class Sharing Enabled

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -cp shcdemo.jar -Xshareclasses:name=demo,verbose -Xscmx50m ClassLoadStress
[-Xshareclasses persistent cache enabled]
[-Xshareclasses verbose output enabled]
JVMSHRC236I Created shared classes persistent cache demo
JVMSHRC246I Attached shared classes persistent cache demo
JVMSHRC765I Memory page protection on runtime data, string read-write data and partially filled pages is successfully enabled
JVMSHRC168I Total shared class bytes read=1111375. Total bytes stored=40947096
JVMSHRC818I Total unstored bytes due to the setting of shared cache soft max is 0. Unstored AOT bytes due to the setting of -Xscmaxaot is 0. Unstored JIT bytes due to the setting of -Xscmaxjitdata is 0.

You can also check the cache statistics using printStats , as Listing 13 shows:

Listing 13. Checking the Number of Cached Classes

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -cp shcdemo.jar -Xshareclasses:name=demo,printStats Current statistics for cache "demo": Cache created with: -Xnolinenumbers = false BCI Enabled = true Restrict Classpaths = false Feature = cr Cache contains only classes with line numbers base address = 0x0000000011F96000
end address = 0x0000000015140000
allocation pointer = 0x000000001403FF50 cache size = 52428192
softmx bytes = 52428192
free bytes = 10874992
ROMClass bytes = 34250576
AOT bytes = 1193452
Reserved space for AOT bytes = -1
Maximum space for AOT bytes = -1
JIT data bytes = 28208
Reserved space for JIT data bytes = -1
Maximum space for JIT data bytes = -1
Zip cache bytes = 902472
Data bytes = 351648
Metadata bytes = 661212
Metadata % used = 1%
Class debug area size = 4165632
Class debug area used bytes = 3911176
Class debug area % used = 93% # ROMClasses = 17062
# AOT Methods = 559
# Classpaths = 3
# URLs = 0
# Tokens = 0
# Zip caches = 5
# Stale classes = 0
% Stale classes = 0% Cache is 79% full Cache is accessible to current user = true

5. Now, start the demo again with the same Java command line. This time, it should read the classes from the shared class cache, as you can see in Listing 14.

Listing 14. Running the Application With a Warm Shared Cache

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -cp shcdemo.jar -Xshareclasses:name=demo,verbose -Xscmx50m ClassLoadStress
[-Xshareclasses persistent cache enabled]
[-Xshareclasses verbose output enabled]
JVMSHRC237I Opened shared classes persistent cache demo
JVMSHRC246I Attached shared classes persistent cache demo
JVMSHRC765I Memory page protection on runtime data, string read-write data and partially filled pages is successfully enabled
JVMSHRC168I Total shared class bytes read=36841382. Total bytes stored=50652
JVMSHRC818I Total unstored bytes due to the setting of shared cache soft max is 0. Unstored AOT bytes due to the setting of -Xscmaxaot is 0. Unstored JIT bytes due to the setting of -Xscmaxjitdata is 0.

You can clearly see the significant (about 40 percent) improvement in class load time from figure 3. Again, you should see the performance improve slightly each time you run the demo because of operating system optimizations.

Figure 3. Warm cache results

Image title

There are a few variations you can experiment with. For example, you can use the javaw command to start multiple demos and trigger all loading classes together to see the concurrent performance.

In a real-world scenario, the overall JVM startup time benefit that can be gained from using class sharing depends on the number of classes that are loaded by the application. A HelloWorld program will not show much benefit, whereas a large web server certainly will. However, this example has hopefully demonstrated that experimenting with class sharing is very straightforward, so you can easily test the benefits.

Memory Footprint

It is also easy to see the memory savings when running the example program in more than one JVM.

Below are four VMMap snapshots obtained using the same machine as the previous examples. In Figure 4, two instances of the demo have been run to completion without class sharing. In Figure 5, two instances have been run to completion with class sharing enabled, using the same command lines as before.

Figure 4. Two instances of demo with no class sharing

Figure 4-1

Image title

Figure 5. Two instances of demo with class sharing enabled

Image title

Figure 5-2

The share cache size is 50MB in the experiment, so the Mapped Files size of each instance in Figure 6 is 50MB more (56736KB – 5536KB) compared to Figure 5.

You can clearly see that the memory usage (Private WS) when shared classes are enabled is significantly lower. A saving of about 70MB Private WS is achieved for 2 JVM instances. More memory saving will be observed if more instances of the demo are launched with class sharing enabled. The test results above are obtained on a Windows 10 laptop with 32GB RAM, using an Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz.

We perform the same memory footprint experiment on a Linux x64 machine as well. Listing 15 shows the result of two JVM instances with no class sharing and Listing 16 shows the result of two JVM instances with class sharing enabled.

Looking at the results, RSS does not show much improvement when class sharing is enabled. This is because the whole shared cache is included in RSS. But, if we look at the PSS, which counts only half of the shared cache to each JVM (as it is shared by 2 JVMs), there is a saving of about 34MB.

Listing 15. Footprint on Linux With Class Sharing Disabled

pmap -X 9612
9612: xa6480_openj9/j2sdk-image/jre/bin/java -cp shcdemo.jar ClassLoadStress
Address Perm … Size Rss Pss Referenced Anonymous Swap Locked Mapping … ======= ======= ===== ======== ========= ==== ==== 2676500 118280 106192 118280 95860 0 0 KB
pmap -X 9850
9850: xa6480_openj9/j2sdk-image/jre/bin/java -cp shcdemo.jar ClassLoadStress
Address Perm … Size Rss Pss Referenced Anonymous Swap Locked Mapping … ======= ======= ===== ======== ========= ==== ==== 2676500 124852 112792 124852 102448 0 0 KB

List 16. Footprint on Linux With Class Sharing Enabled

pmap -X 4501
4501: xa6480_openj9/j2sdk-image/jre/bin/java -Xshareclasses:name=demo -Xscmx50m -cp shcdemo.jar ClassLoadStress
Address Perm … Size Rss Pss Referenced Anonymous Swap Locked Mapping
…
7fe7d0e00000 rw-s 4 4 2 4 0 0 0 C290M4F1A64P_demo_G35
7fe7d0e01000 r--s 33356 33356 16678 33356 0 0 0 C290M4F1A64P_demo_G35
7fe7d2e94000 rw-s 11096 48 24 48 0 0 0 C290M4F1A64P_demo_G35
7fe7d396a000 r--s 5376 1640 832 1640 0 0 0 C290M4F1A64P_demo_G35
7fe7d3eaa000 rw-s 296 0 0 0 0 0 0 C290M4F1A64P_demo_G35
7fe7d3ef4000 r--s 1072 0 0 0 0 0 0 C290M4F1A64P_demo_G35
… ======= ======= ===== ======== ====== ====== ==== 2732852 120656 90817 97988 62572 0 0 KB
pmap -X 4574
4574: xa6480_openj9/j2sdk-image/jre/bin/java -Xshareclasses:name=demo -Xscmx50m -cp shcdemo.jar ClassLoadStress
Address Perm … Size Rss Pss Referenced Anonymous Swap Locked Mapping …
7f308ce00000 rw-s 4 4 2 4 0 0 0 C290M4F1A64P_demo_G35
7f308ce01000 r--s 33356 33356 16678 33356 0 0 0 C290M4F1A64P_demo_G35
7f308ee94000 rw-s 11080 48 24 48 0 0 0 C290M4F1A64P_demo_G35
7f308f966000 r--s 5392 1632 824 1632 0 0 0 C290M4F1A64P_demo_G35
7f308feaa000 rw-s 296 0 0 0 0 0 0 C290M4F1A64P_demo_G35
7f308fef4000 r--s 1072 0 0 0 0 0 0 C290M4F1A64P_demo_G35
… ======= ======= ===== ======== ====== ====== ==== 2730800 122832 92911 102584 64812 0 0 KB

Conclusion

The Shared Classes feature in the OpenJ9 implementation offers a simple and flexible way to reduce memory footprint and improve JVM startup time. In this article, you have seen how to enable the feature, how to use the cache utilities, and how to get quantifiable measurements of the benefits.

Original Link

Class Sharing in Eclipse OpenJ9: How to Improve Memory, Performance (Part 1)

Memory footprint and startup time are important performance metrics for a Java virtual machine (JVM). The memory footprint becomes especially important in the cloud environment since you pay for the memory that your application uses. In this tutorial, we will show you how to use the shared classes feature in Eclipse OpenJ9 to reduce the memory footprint and improve your JVM startup time.

In 2017, IBM open sourced the J9 JVM and contributed it to the Eclipse foundation, where it became the Eclipse OpenJ9 project. The J9 JVM has supported class sharing from system classes to application classes for over 10 years, beginning in Java 5.

In the OpenJ9 implementation, all systems, application classes, and ahead-of-time (AOT) compiled code can be stored in a dynamic class cache in shared memory. These shared classes feature are implemented on all platforms that OpenJ9 supports. The feature even supports integration with runtime bytecode modification, which we will discuss later in Part 2 of this article.

The shared classes feature is one that you don’t have to think about once it’s started, but it provides a powerful scope for reducing memory footprint and improving JVM startup time. For this reason, it is best suited to environments where more than one JVM is running similar code or where a JVM is regularly restarted.

In addition to the runtime class-sharing support in the JVM and its class loaders, there is also a public Helper API provided for integrating class sharing support into custom class loaders.

You can download the JDK with OpenJ9 from the Adopt OpenJDK project or pull it from the docker image if you’d like to follow along with the example.

How it Works

Let’s start by exploring the technical details of how the shared classes feature operates.

Enabling Class Sharing

To enable class sharing, add -Xshareclasses[:name=] to an existing Java command line. When the JVM starts up, it looks for a shared cache of the name given (if no name is provided, it uses the current username). It either connects to an existing shared cache or creates a new one.

You can specify the shared cache size using the parameter -Xscmx[k|m|g]. This parameter only applies when a new shared cache is created. If this option is omitted, a platform-dependent default value is used. Note that there are operating system settings that limit the amount of shared memory you can allocate. For instance, SHMMAX on Linux is typically set to about 32MB. To learn more about the details of these settings, see the Shared Classes section of this user guide.

Shared Classes Cache

A shared classes cache consists of a shared memory of a fixed size that persists beyond the lifetime of the JVM or a system reboot unless a non-persistent shared cache is used. Any number of shared caches can exist on a system, and all are subject to operating system settings and restrictions.

No JVM owns the shared cache, and there is no master/slave JVM concept. Instead, any number of JVMs can read and write to the shared cache concurrently.

A shared cache cannot grow in size. When it becomes full, JVMs can still load classes from it, but it can no longer store any data into it. You can create a large shared classes cache up front while setting a soft maximum limit on how much shared cache space can be used. You can increase this limit when you want to store more data into the shared cache without shutting down the JVMs that are connected to it. Check out the OpenJ9 documentation for more details about the soft maximum limit.

In addition, there are several JVM utilities to manage actively shared caches. We will discuss these in the Shared Classes Utilities section below.

A shared cache is deleted when it is explicitly destroyed using a JVM command line.

How Are Classes Cached?

When a JVM loads a class, it first looks in the class loader cache to see if the class it needs is already present. If yes, it returns the class from the class loader cache. Otherwise, it loads the class from the filesystem and writes it into the cache as part of the defineClass() call. Therefore, a non-shared JVM has the following class loader lookup order:

  1. Classloader cache
  2. Parent
  3. Filesystem

In contrast, a JVM running with the class sharing feature uses the following order:

  1. Classloader cache
  2. Parent
  3. Shared classes cache
  4. Filesystem

Classes are read from and written to the shared classes cache using the public Helper API. The Helper API is integrated into java.net.URLClassLoader (and jdk.internal.loader.BuiltinClassLoader in Java 9 and up). Therefore, any class loader that extends java.net.URLClassLoader gets class sharing support for free. For custom class loaders, OpenJ9 has provided Helper APIs so that class sharing can be implemented on custom class loaders.

What Is Cached?

A shared classes cache can contain bootstrap and application classes, metadata that describes the classes, and ahead-of-time (AOT) compiled code.

Inside the OpenJ9 implementation, Java classes are divided into two parts:

  • a read-only part called a ROMClass, which contains all of the class’s immutable data
  • a RAMClass that contains mutable data, such as static class variables

A RAMClass points to data in its ROMClass, but these two are completely separated. So, it is quite safe for a ROMClass to be shared between JVMs and also between RAMClasses in the same JVM.

In the non-shared case, when the JVM loads a class, it creates the ROMClass and the RAMClass separately and stores them both in its local process memory. In the shared case, if the JVM finds a ROMClass in the shared classes cache, it only needs to create the RAMClass in its local memory; the RAMClass then references the shared ROMClass.

Because most of the class data is stored in the ROMClass, this is where the memory savings are made (see a more detailed discussion in the “Memory footprint ” sections). JVM startup times are also significantly improved with a populated cache, because some of the work to define each cached class has already been done and the classes are loaded from memory, rather than from the filesystem. Startup time overhead to populate a new shared cache is not significant, as each class simply needs to be relocated into the shared cache as it is defined.

AOT compiled code is also stored into the shared cache. When the shared classes cache is enabled, the AOT compiler is automatically activated. AOT compilation allows the compilation of Java classes into native code for subsequent executions of the same program. The AOT compiler generates native code dynamically while an application runs and caches any generated AOT code in the shared classes cache. Usually, the execution of AOT compiled code is faster than interpreted bytecode but not as fast as JIT’ed code. Subsequent JVMs that execute the method can load and use the AOT code from the shared cache without incurring the performance decrease experienced with generating JIT-compiled code, resulting in a faster startup time. When creating a new shared cache, you can use options -Xscminaot and -Xscmaxaot to set the size of AOT space in the shared cache. If neither -Xscminaot nor – Xscmaxaot is used, the AOT code will be stored to the shared cache as long as there is free space available.

What Happens if a Class Changes on the Filesystem?

Because the share classes cache can persist indefinitely, filesystem updates that invalidate classes and AOT code in the shared cache may occur. If a class loader makes a request for a shared class, then the class returned should always be the same as the one that would have been loaded from the filesystem. This happens transparently when classes are loaded, so users can modify and update as many classes as they like during the lifetime of a shared classes cache, knowing that the correct classes are always loaded.

Pitfalls With Class Changes: Examples

Imagine a class C1 that is stored into the shared cache by a JVM. Then, when the JVM shuts down, C1 is changed and recompiled. When the JVM restarts, it should not load the cached version of C1.

Similarly, imagine a JVM that’s running with a classpath of  /mystuff:/mystuff/myClasses.jar . It loads C2 from  myClasses.jar  into the shared cache. Then a different C2.class is added to  /myStuff  and another JVM starts up running the same application. It would be incorrect for the JVM to load the cached version of C2.

The JVM detects filesystem updates by storing timestamp values into the shared cache and comparing the cached values with actual values on each class load. If it detects that a JAR file has been updated, it has no idea which classes have been changed. Because of this, all classes, as well as AOT code from that JAR in the cache, are immediately marked as stale and cannot be loaded from the cache. When the classes from that JAR are loaded from the filesystem and re-added to the cache, only the ones that have changed are added in their entirety; those that haven’t changed are effectively made not stale.

Classes cannot be purged from the shared classes cache, but the JVM attempts to make the most efficient use of the space it has. For example, the same class is never added twice, even if it is loaded from many different locations. So, if the same class C3 is loaded from  /A.jar ,  /B.jar , and  /C.jar  by three different JVMs, the class data is only added once. But, there are three pieces of metadata to describe the three locations from which it was loaded.

Shared Classes Utilities

There are several utilities that you can use to manage shared classes caches, all of which are sub-options to  -Xshareclasses  ( you can get a complete list of all sub-options via  java -Xshareclasses:help).

To demonstrate the use of these options, let’s walk through some examples.

First, let’s create two shared caches by running a Hello class with different cache names, as Listing 1 shows:

Listing 1. Creating two Shared Caches

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -cp . -Xshareclasses:name=Cache1 Hello
Hello
C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -cp . -Xshareclasses:name=Cache2 Hello
Hello

Running the  listAllCaches  sub-option lists all caches on a system and determines whether they are in use, as you can see in Listing 2:

Listing 2. Listing all Shared Caches

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -Xshareclasses:listAllCaches
Listing all caches in cacheDir C:\Users\Hang Shao\AppData\Local\javasharedresources\
Cache name level cache-type feature last detach time
Compatible shared caches
Cache1 Java8 64-bit persistent cr Mon Apr 23 15:48:12 2018
Cache2 Java8 64-bit persistent cr Mon Apr 23 15:49:46 2018

Running the  printStats  option prints summary statistics on the named cache, as Listing 3 shows. For a detailed description of the  printStats  option, see the user guide.

Listing 3. Summary Statistics for a Shared Cache

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -Xshareclasses:name=Cache1,printStats Current statistics for cache "Cache1": Cache created with: -Xnolinenumbers = false BCI Enabled = true Restrict Classpaths = false Feature = cr Cache contains only classes with line numbers base address = 0x000000001214C000
end address = 0x0000000013130000
allocation pointer = 0x0000000012297DB8 cache size = 16776608
softmx bytes = 16776608
free bytes = 13049592
ROMClass bytes = 1359288
AOT bytes = 72
Reserved space for AOT bytes = -1
Maximum space for AOT bytes = -1
JIT data bytes = 1056
Reserved space for JIT data bytes = -1
Maximum space for JIT data bytes = -1
Zip cache bytes = 902472
Data bytes = 114080
Metadata bytes = 18848
Metadata % used = 0%
Class debug area size = 1331200
Class debug area used bytes = 132152
Class debug area % used = 9% # ROMClasses = 461
# AOT Methods = 0
# Classpaths = 2
# URLs = 0
# Tokens = 0
# Zip caches = 5
# Stale classes = 0
% Stale classes = 0% Cache is 22% full Cache is accessible to current user = true

There are other  printStats  sub-options that can be used to print specific data in the shared cache. They can be found in  printStats=help. For example, you can check the classpath data via  printStats=classpath :

Listing 4. Listing the Classpath Contents of a Shared Cache

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -Xshareclasses:name=Cache1,printStats=classpath Current statistics for cache "Cache1": 1: 0x000000001360E3FC CLASSPATH C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\bin\compressedrefs\jclSC180\vm.jar C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\lib\se-service.jar C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\lib\rt.jar C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\lib\resources.jar C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\lib\jsse.jar C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\lib\charsets.jar C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\lib\jce.jar C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\lib\tools.jar
1: 0x000000001360A144 CLASSPATH C:\OpenJ9
…
…

The shared caches are destroyed using the destroy  option, illustrated in Listing 5. Similarly, option  destroyAll destroys all shared caches that are not in use and that the user has permissions to destroy.

Listing 5. Destroying a Cache

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -Xshareclasses:name=Cache1,destroy
JVMSHRC806I Compressed references persistent shared cache "Cache1" has been destroyed. Use option -Xnocompressedrefs if you want to destroy a non-compressed references cache.

The expire option, illustrated in Listing 6, is a housekeeping option that you can add to the command line to automatically destroy caches to which nothing has been attached for a specified number of minutes. Listing 6 looks for caches that have not been used for a week (10,080 minutes) and destroys them before starting the JVM.

The reset  option always creates a new shared cache. If a cache with the same name exists, it is destroyed and a new one is created.

Listing 6. Destroying Caches That Haven’t Been Used in a Week

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -Xshareclasses:name=Cache1,expire=10080 Hello
Hello

Verbose Options

Verbose options provide useful feedback on what class sharing is doing. They are all sub-options to  -Xshareclasses. This section offers some examples of how to use those verbose options.

The verbose option, illustrated in Listing 7, gives concise status information on JVM startup and shutdown:

Listing 7. Getting JVM Status Information

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -Xshareclasses:name=Cache1,verbose Hello
[-Xshareclasses persistent cache enabled]
[-Xshareclasses verbose output enabled]
JVMSHRC236I Created shared classes persistent cache Cache1
JVMSHRC246I Attached shared classes persistent cache Cache1
JVMSHRC765I Memory page protection on runtime data, string read-write data and partially filled pages is successfully enabled
Hello
JVMSHRC168I Total shared class bytes read=11088. Total bytes stored=2416962
JVMSHRC818I Total unstored bytes due to the setting of shared cache soft max is 0. Unstored AOT bytes due to the setting of -Xscmaxaot is 0. Unstored JIT bytes due to the setting of -Xscmaxjitdata is 0. 

The  verboseIO  option prints a status line for every class load request to the shared cache. To understand  verboseIO  output, you should understand the class loader hierarchy. This can be clearly seen for classes that are loaded by any non-bootstrap class loader. In the output, each class loader is assigned a unique ID, but the bootstrap loader is always 0.

Note that it is normal for  verboseIO  to sometimes show classes being loaded from disk and stored in the cache, even if they are already cached. For example, the first class loaded from each JAR on the application classpath is always loaded from disk and stored, regardless of whether it exists in the cache or not. This is to confirm the JAR in the classpath does exist on the file system.

In Listing 8, the first section demonstrates the population of the cache and the second section shows the reading of the cached classes:

Listing 8. Using VerboseIO

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -Xshareclasses:name=Cache1,verboseIO Hello
[-Xshareclasses verbose I/O output enabled]
Failed to find class java/lang/Object in shared cache for class-loader id 0.
Stored class java/lang/Object in shared cache for class-loader id 0 with URL C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\lib\rt.jar (index 2).
Failed to find class java/lang/J9VMInternals in shared cache for class-loader id 0.
Stored class java/lang/J9VMInternals in shared cache for class-loader id 0 with URL C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\lib\rt.jar (index 2).
Failed to find class com/ibm/oti/vm/VM in shared cache for class-loader id 0.
Stored class com/ibm/oti/vm/VM in shared cache for class-loader id 0 with URL C:\OpenJ9\wa6480_openj9\j2sdk-image\jre\lib\rt.jar (index 2).
Failed to find class java/lang/J9VMInternals$ClassInitializationLock in shared cache for class-loader id 0.
…
…
C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -Xshareclasses:name=Cache1,verboseIO Hello
[-Xshareclasses verbose I/O output enabled]
Found class java/lang/Object in shared cache for class-loader id 0.
Found class java/lang/J9VMInternals in shared cache for class-loader id 0.
Found class com/ibm/oti/vm/VM in shared cache for class-loader id 0.
Found class java/lang/J9VMInternals$ClassInitializationLock in shared cache for class-loader id 0.
…
…

The  verboseHelper  sub-option, illustrated in Listing 9, is an advanced option that gives status output from the Helper API. The  verboseHelper  sub-option helps developers using the Helper API to understand how it is being driven. More details on this output are described in the JVM diagnostics guide.

Listing 9. Status Output From the Helper API

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -Xshareclasses:name=Cache1,verboseHelper Hello
[-Xshareclasses Helper API verbose output enabled]
Info for SharedClassURLClasspathHelper id 1: Verbose output enabled for SharedClassURLClasspathHelper id 1
Info for SharedClassURLClasspathHelper id 1: Created SharedClassURLClasspathHelper with id 1
Info for SharedClassURLClasspathHelper id 2: Verbose output enabled for SharedClassURLClasspathHelper id 2
Info for SharedClassURLClasspathHelper id 2: Created SharedClassURLClasspathHelper with id 2
Info for SharedClassURLClasspathHelper id 1: There are no confirmed elements in the classpath. Returning null.
Info for SharedClassURLClasspathHelper id 2: There are no confirmed elements in the classpath. Returning null.
Info for SharedClassURLClasspathHelper id 2: setClasspath() updated classpath. No invalid URLs found
Info for SharedClassURLClasspathHelper id 2: Number of confirmed entries is now 1
Hello

The  verboseAOT  and  -Xjit:verbose  sub-option, illustrated in Listing 10, give you information on AOT loading and storing activities from/into the shared cache.

Listing 10. Verbose Information on AOT Loading and Storing

C:\OpenJ9>wa6480_openj9\j2sdk-image\bin\java -Xshareclasses:name=demo,verboseAOT -Xjit:verbose -cp shcdemo.jar ClassLoadStress
…
+ (AOT cold) java/nio/Bits.makeChar(BB)C @ 0x00000000540049E0-0x0000000054004ABF OrdinaryMethod - Q_SZ=2 Q_SZI=2 QW=6 j9m=0000000004A4B690 bcsz=12 GCR compThread=1 CpuLoad=298%(37%avg) JvmCpu=175%
Stored AOT code for ROMMethod 0x00000000123C2168 in shared cache.
…
+ (AOT load) java/lang/String.substring(II)Ljava/lang/String; @ 0x0000000054017728-0x00000000540179DD Q_SZ=0 Q_SZI=0 QW=1 j9m=00000000049D9DF0 bcsz=100 compThread=0
Found AOT code for ROMMethod 0x0000000012375700 in shared cache.
…

That’s all for Part 1, be sure to tune in tomorrow when we’ll discuss the next steps for class sharing in Eclipse OpenJ9.

Original Link

2018 Java Ecosystem Executive Insights

This article is featured in the new DZone Guide to Java: Features, Improvements, & Updates. Get your free copy for more insightful articles, industry statistics, and more!

To gather insights on the current and future state of the Java ecosystem, we talked to executives from 14 companies. 

Key Findings

While Java is still the prevalent language in enterprises and there are more Java developers than anything else, this topic does not generate the level of response of the other topics for which we produce research guides. Java content continues to outperform all other content on DZone by a 4:1 margin — but because it’s not new and “sexy,” not a lot of people want to talk about it. I’m most appreciative to the IT professionals that took the time to share their insights on the current and future state of the Java ecosystem.

  1. Java continues to be the platform of choice for a number of enterprise companies, including financial institutions, as a result of its portability and because it allows developers to write once run anywhere. There are plenty of Java developers and it is seeing a resurgence with big data.

  2. The Java Virtual Machine (JVM) was most frequently mentioned as the most important element of the Java ecosystem. The JVM is the most critical element followed by its openness, compatibility, the vastness of the libraries, and the completeness of the toolchains. The JVM enables languages other than Java to flourish. The fact that Java is open-source while championed by a large company was deemed to be important by a couple of respondents, as was the community in which no one participant is more important than the community. The maturity level of Java is high. This results in a lot of frameworks, libraries, and IDEs as well as a high-performance, consistent, compatible language that’s simple and stable.

  3. The most important player in the Java ecosystem is Oracle followed by multiple mentions of IBM, the Apache and Eclipse foundations, Red Hat, and Pivotal. Oracle was seen as the key holder but seems to be pulling back. It was interesting that financial institutions as a whole were mentioned given their use of Java and the number of developers they employ along with other companies like Twitter, Alibaba, Facebook, and Google with GCP and Kubernetes.

    The Eclipse Foundation is likely to become the most significant player with its efforts around MicroProfile and Jakarta EE.

  4. The two most significant changes to the ecosystem in the past year have been the move to semi-annual releases and Java EE moving to the Eclipse Foundation as Jakarta EE, putting the future of enterprise Java into the hands of the community. The move to halfyearly releases will boost interest in, and use of, Java by developers. The “open-sourcing” of Java EE to the Eclipse Foundation, creating EE4J and now the birth of Jakarta EE, is significant. Java has been the dominant player in enterprise applications for two decades. Jakarta EE ensures Java will continue to be the dominant player for enterprise computing for a long time.

  5. The availability of developers helps Java solve many problems in organizations. There is a shortage of security, big data, and AI/ ML/DL/NLP professionals, but there are plenty of Java developers who can get organizations’ work done. Twitter, financial institutions, automobile manufacturers, and AI/ML companies are all heavy users of Java. Java has the ability to support high-speed concurrent processing at scale, which becomes more important as the amount of data continues to grow.

  6. The people I spoke with are all Java devotees and don’t see any problems with Java. They realize that those who are not Java developers see the language as being too verbose. They believe Eclipse is a good steward of open source and, like open source, there needs to be more participation and engagement in the community.

    While Java is considered verbose by some, there are alternatives like Kotlin, Scala, and project Lombok. Java lags behind because it’s heavily used by large enterprises. With slowness comes stability. While it lacks some of the niceties of other languages, it provides quick wins with fast coding.

  7. Serverless was mentioned by a couple of respondents as the future of Java. They believe it will lead to a major reshaping. Java is built for serverless, but it needs work. With Spring Boot, containers can be lighter-weight and great to build serverless upon. There were good changes in Java 8 and 9 for easier execution in container management, memory, and CPU. JVM-based languages and tooling will continue to evolve. The JVM enables many different types of languages to be built. Others see Java thriving in the open-source software ecosystem with innovations continuing to support Java’s ongoing success.

  8. The biggest concerns with the current state of the Java ecosystem were quality deterioration because people were not learning from their mistakes, understanding the value of the ecosystem, or taking security seriously enough. Some question the benefits of the more frequent release cycles because it may lead to release fatigue and meaningless releases that ultimately will not be adopted or supported.

  9. When working with Java, developers need to realize the depth of the ecosystem and not try to reinvent the wheel. Learn the libraries and know your code is going to be attacked — prepare accordingly. Pay attention to the Java roadmap and try new builds before they go live to be seen as an expert. Like any other language, ensure your code is well-designed, extendable, maintainable, and easily understood by others.

    JVM is the top-performing platform. Most languages live on the JVM so start with Java. Be language-agnostic. Learn domain-driven design. Read Design Patterns by the Gang of Four. There’s a good future with Java. Look for tools to help with development. Keep an eye on open source projects through good information outlets.

  10. Additional considerations regarding Java include:

  • Containers are changing how developers deploy applications and that affects Java applications.

  • Pay attention to Kotlin which runs on the JVM. Adopt new languages to help create new applications. There are a lot of really smart people in the ecosystem; pay attention to what they have to say. There’s a lot to continue learning since things will continue to change.

  • Java developers’ participation matters in the continued consistency, stability, and security of the Java ecosystem.

  • One of the strengths of Java that’s undervalued is that it’s a small language that does not offer a huge range of options for how to do things. As a result, two engineers writing the same solution for the same problem tend to end up with the same code. This is a strong advantage of Java that is not true of many other competing languages.

  • We need to recognize the importance of open source — embrace it and contribute to it where we can. If you find problems, work on fixing them. It’s a great community and it’s made better when everyone is involved and contributing.

And here’s who we spoke to:

This article is featured in the new DZone Guide to Java: Features, Improvements, & Updates. Get your free copy for more insightful articles, industry statistics, and more!

Original Link

Free, Fast, Open, Production-Proven, and All Java: OpenJ9

This article is featured in the new DZone Guide to Java: Features, Improvements, & Updates. Get your free copy for more insightful articles, industry statistics, and more!

Eclipse OpenJ9 is a high-performance, scalable, Java virtual machine (JVM) implementation. It is not new. It has been part of the IBM Java Development Kit for many years and it is safe to say that it is an enterprise-grade, production-level component of many large Java-based systems. At the end of last year, it was contributed to the Eclipse Foundation by IBM. OpenJ9 is an alternative to the Hotspot JVM currently mostly used within OpenJDK.

The History

Although the Eclipse OpenJ9 project hasn’t been around for very long, the VM itself has been around for years. It was launched during the fourth JavaOne and parts of the codebase can be traced back to Smalltalk.

It has been powering IBM middleware products for the last decade or more. IBM contributed the VM to the Eclipse Foundation end of 2017 and more than 70 IBM developers are actively involved in the project.

The long-term goal of the Eclipse OpenJ9 project is to foster an open ecosystem of JVM developers that can collaborate and innovate with designers and developers of hardware platforms, operating systems, tools, and frameworks. The Java community has benefited over its history from having multiple implementations of the JVM specification competing to provide the best runtime for your application. Whether adding compressed references, new cloud features, AOT (ahead-of-time) compilation, or straight up faster performance and lower memory use, the ecosystem has improved through that competition. Eclipse OpenJ9 aims to continue to spur innovation in the runtimes space.

As an essential part of this, the teams work closely with the AdoptOpenJDK efforts, which are led by the London Java Community. If you look closely, you realize that a large part of OpenJ9 has been living at the Eclipse Foundation for a couple of years now. Eclipse OpenJ9 embeds Eclipse OMR, which provides cross-platform components for building reliable, high-performance language runtimes.

Eclipse OpenJ9 takes OMR and adds extra code that turns it into a runtime environment for Java applications.

What Exactly Is Exactly Is OpenJ9 and How Does it Relate to OpenJDK?

OpenJ9 replaces HotSpot JVM in the OpenJDK build and becomes a new JVM runtime implementation that can be used with a wide range of system architectures and operating systems. Currently, there are pre-built OpenJDK binaries with OpenJ9 as the JVM available for Java versions 8, 9, and 10 via the AdopOpenJDK project website. A JVM is not the complete Java Development Kit (JDK). It is the execution engine. While the JDK provides tools, class libraries, and more general things that you need to build your application, the JVM is responsible to run the actual bytecode. It is a replacement for the commonly used and known Hotspot JVM which was implemented by Oracle.

In order to create an OpenJDK build with OpenJ9 as the JVM, the OpenJDK project is mirrored into a separate repository that removes Hotspot and provides a place for the needed integration code. The integration code consists mostly of changes to Makefiles and configuration, some patches to the class library code (JCL), and some minor patches for additional OpenJ9 features.

The integration layer mitigates the few places where the interfaces from Hotspot and OpenJ9 are slightly different. All in all, it is roughly 4,000 lines of code, which is really small compared to the size of OpenJ9 or the OpenJDK. The primary goal is to keep this to a necessary minimum and also contribute as much as possible upstream to the OpenJDK.

Image title

With OpenJDK, with the integration code and OpenJ9 coming together via a build process, the end result is a binary that is available for a range of different platforms.

You can find more information about the build process and how to build your own OpenJDK with OpenJ9 on the project website.

Why OpenJ9 Is the Perfect Cloud Runtime

The requirements of cloud runtimes today are different to the characteristics employed by individual, physical machines in the datacenters from just a couple of years ago. A small memory footprint allows a higher application density for providers and reduces the cost per application for users as they can run more instances with less resources.

Another important feature is a fast startup, which lets application instances scale faster and more smoothly.

The OpenJ9 project uses the Daytrader application to monitor performance improvements in comparison with Hotspot. With a 66% smaller memory footprint after startup and a 42% faster startup time, it reaches a comparable throughput in a constraint environment like a Docker container.

Uinque Features and Advantages

The lower memory footprint and faster startup times demonstrate the practical impact of some of the advantages and implementation differences over the classical Hotspot JVM.

Application Class Sharing and AOT Compilation

The first time an application runs, the JVM has to load all the classes that are required to start the application. This process takes time. If the needed classes are stored in cache, the second application run will be much faster. Another implementation difference is that the JVM doesn’t store the Java bytecode classes but an optimized ROMClass version of it. These read-only, optimized versions are carrying symbolic data only. When the JVM is executing a class, it has to be able to do a bunch of lookups, e.g. finding the method it is actually invoking, and it also needs to be able to work on data and save this data. This is where the J9RAMClass comes in. It is a cache for the data at runtime and carries the live data for a particular class.

Further on, the ROMClasses can be shared between multiple JVMs on the same machine. OpenJ9 always shares both the bootstrap and application classes that are loaded by the default system class loader. A detailed view on how this works is available in this article. And there is something else that can be optimized with this shared class cache ahead-of-time compilation (AOT). Unlike just-in-time (JIT) compilation, AOT is dynamically compiling methods into ATO code at runtime and placing these ROMClasses into the shared cache. The VM automatically chooses which methods should be AOT-compiled based on heuristics that identify the start-up phase of large applications. It is enabled via the -Xshareclasses option, which is highly configurable.

As for AOT itself, it works straight out of the box when you enable shared classes and doesn’t require any special tuning.

Tuning for Cloud Environments

The -Xtune:virtualized option is designed to configure OpenJ9 for typical cloud deployments where VM guests are provisioned with a small number of virtual CPUs to maximize the number of applications that can be run. The -Xquickstart option enables an ultra-fast startup of OpenJ9 and works best with short-lived tasks. But be aware that you may trade in the peak throughput capabilities. You can also specify -XX:+IdleTuningGcOnIdle on the command line. When set, OpenJ9 determines whether an application is idle based on CPU utilization and other internal heuristics. When an idle state is recognized, a GC cycle runs if there is significant garbage in the heap and releases unused memory back to the operating system. A more detailed overview is provided in this article.

Garbage Collection in OpenJ9

Eclipse OpenJ9 has a number of GC policies designed around different types of applications and workloads. The Generational Concurrent (-Xgcpolicy:gencon) GC policy is the default policy employed by the JVM. But there are four other alternatives: -Xgcpolicy:balanced, -Xgcpolicy:metronome, -Xgcpolicy:optavgpause, and -Xgcpolicy:optthruput. The Metronome GC (-Xgcpolicy:metronome) policy is especially interesting if your application depends on precise response times and you are running on x86 Linux. The main difference between Metronome and other policies is that garbage collection occurs in small interruptible steps rather than stopping an application completely while garbage is marked and collected.

By default, Metronome pauses for three milliseconds at a time. A full garbage collection cycle requires many pauses, which are spread out to give the application enough time to run. You can limit the amount of CPU that the GC process uses and you can control the pause time. A more complete overview about the individual policies can be found in this article.

Get Started

The AdoptOpenJDK project releases pre-built binaries for Linux x64, Linux s390x, Linux ppc64, and AIX ppc64. The easiest way to get started is to download the one for your operating system and start working with it — although you have the ability to build binaries for other platforms like Linux ARMv7 yourself. Docker images are available via DockerHub.

The JVM documentation is extensive and well-structured to get you started with optimizing for your application.

This article is featured in the new DZone Guide to Java: Features, Improvements, & Updates. Get your free copy for more insightful articles, industry statistics, and more!

Original Link

DZone Research: What Devs Need to Know About Java

To gather insights on the current and future state of the Java ecosystem, we talked to executives from 14 companies. We began by asking, “What do developers need to keep in mind when working with Java?” Here’s what the respondents told us:

Depth

  • Work on applying your Java skills in new environments like Docker and Kubernetes. Pay attention to the Java roadmap. Look into how the platform fits into the cloud with new dynamic APIs. Learn cloud technology and how it applies to Java. Look around at other languages, constantly learning what you can do and bringing the best to Java. 
  • JVM is the top programming platform. Most languages live on the JVM so start with Java. Primary platforms for business apps are Java and .Net. Be language agnostic. Learn design, “Design Patterns” by the Gang of Four.
  • A really big winning point for Java is the high level. Java is on an abstraction level where you can produce and think about the important parts to deliver business value. However, developers who strive to “push” Java a big needs a fundamental knowledge about what’s going on under the hood.

Reuse

  • Don’t reinvent the wheel. Participate in the ecosystem. Be a contributor. 
  • Good IDE can make life easier by reducing verbosity. Since you’re on the JVM you can use many other languages (e.g., Groovy and Kotlin).

Other

Here’s who we spoke to:

Original Link

A Wicked Trick to Make the JVM Forget to Check Exceptions

I’ve been a long-time been critic of the mechanism of compiler-checked exceptions in Java. Whether you love them or hate them, one thing is sure: There are situations where you don’t want to have to deal with them. The solution in Java is to wrap a checked exception in a new RuntimeException, but this gives long stack traces without adding useful information. Sometimes, we just want to tell the compiler to chill.

As it turns out, this is possible through some wicked abuse of the type erasure misfeature of Java generics. Seeing this in action is instructive for understanding the inner workings of Java. Let’s go!

Here is what we want:

public static void main(String[] args) { businessLogic();
} private static void businessLogic() { List < String > configuration = readConfigurationFile(); System.out.println(configuration.get(0));
} private static List < String > readConfigurationFile() { try { return Files.readAllLines(Paths.get("non", "existing", "file")); } catch (IOException e) { throw softenException(e); }
}

Notice that the businessLogic() neither catches IOException nor declares that it throws IOException. Instead, the softenException() method removes the checkedness of the exception. When we run it, we get the following stack trace:

Exception in thread "main" java.nio.file.NoSuchFileException: non\existing\file at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:79) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) at sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:230) at java.nio.file.Files.newByteChannel(Files.java:361) at java.nio.file.Files.newByteChannel(Files.java:407) at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) at java.nio.file.Files.newInputStream(Files.java:152) at java.nio.file.Files.newBufferedReader(Files.java:2784) at java.nio.file.Files.readAllLines(Files.java:3202) at java.nio.file.Files.readAllLines(Files.java:3242) at insanejava.SoftenExceptionsDemo.readConfigurationFile(SoftenExceptionsDemo.java:21) at insanejava.SoftenExceptionsDemo.businessLogic(SoftenExceptionsDemo.java:15) at insanejava.SoftenExceptionsDemo.main(SoftenExceptionsDemo.java:11)

Huh! The exception being thrown in the main method is a NoSuchFileException, which is a subclass of IOException – a checked exception! How can that be? Why didn’t any of the methods in the program have to declare throws IOException?

Here’s the trick:

private static RuntimeException softenException(Exception e) { return checkednessRemover(e);
} private static < T extends Exception > T checkednessRemover(Exception e) throws T { throw (T) e;
}

The checkednessRemover method uses a trick that reveals a few things about the inner workings of Java. First, the generic type argument T is bound to RuntimeException in order to fulfill the contract of softenException. This means that the expression throws T becomes throws RuntimeException, which the compiler interprets as if there were no exceptions thrown.

But the statement throw (T)e; theoretically should evaluate to throw (RuntimeException)e;. Since e is a NoSuchFileException, you would expect this statement to result in a ClassCastException. But the way generics work in Java, the type information is removed by the compiler. So instead, the bytecode reads as throw (Exception)e;, which is fine.

So this strange trick shows that the Java compiler removes generic information from the compiled code and that checked exceptions are purely a feature of the compiler. There are no runtime verifications of checked exceptions.

Would I recommend using this trick in production code? I don’t know. It’s pretty weird and may not be that useful, but I do use it myself when I’m feeling wicked. If nothing else, I hope learning about this has given you some insights into the inner workings of Java.

Disclaimers: (1) I read about this trick somewhere else, but I cannot find the source any longer. I thought it was Heinz Kabutz’ excellent Java Specialist newsletter, but I can’t find the source. (2) This is also implemented in Project Lombok as @SneakyThrows. If you are using Lombok, you should under no circumstance reimplement the trick from this blog. Use @SneakyThrows instead.

Original Link

Behind the Scenes of the Finalize() Method

“What is the purpose of the finalize() method?” is an often asked Java interview question. The typical answer to it is: “The usual purpose of the finalize() method is to perform cleanup actions before the object is discarded.” However, behind the scenes, finalize() methods are handled in a special way. A small mistake in finalize() has the potential to jeopardize entire application’s availability. Let’s study it in detail.

Behind the Scenes

Objects that have “finalize()” methods are treated differently during garbage collection than ones that don’t have it. During garbage collection, objects with “finalize()” methods aren’t immediately evicted from memory. Instead, as the first step, those objects are added to an internal queue of ‘java.lang.ref.Finalizer’. For the entire JVM, there is only one low priority JVM thread, ‘Finalizer’, that executes the “finalize()” methods of each object in the queue. Only after the execution of the “finalize()” method does the object become eligible for garbage collection.

Assume that if the application is producing a lot of objects that have “finalize()” methods and the low priority “Finalizer” thread isn’t able to keep up with executing those finalize() methods, then a significant amount of unfinalized objects will start to build up in the internal queue of ‘java.lang.ref.Finalizer’, which would result in a significant amount of memory wastage.

Sometimes, because of poor programming practices, the “Finalizer” thread may start to WAIT or BLOCK while executing the “finalize()” method. If the “Finalizer” thread starts to wait or block, then the number of unfinalized objects in the internal queue of ‘java.lang.ref.Finalizer’ will start to grow significantly, which would result in an OutOfMemoryError, jeopardizing the entire JVM’s availability.

Example

To illustrate this theory, we wrote a simple sample program.

public class SampleObject { public String data; public SampleObject(String data) { this.data = data; } @Override public void finalize() { try { // Sleep for 1 minute. Thread.currentThread().sleep(1 * 60 * 1000); } catch (Exception e) {} } public static void main(String[] args) { long counter = 0; while (true) { new SampleObject("my-fun-data-" + counter); System.out.println("created: " + counter++); } }
}

Basically, the ‘main()’ method of this class creates ‘SampleObject’ continuously. The interesting part of this program is the “finalize()” method. This method puts the currently executing thread (i.e. the ‘Finalizer’ thread) to sleep for 1 minute. This example illustrates a poor implementation of “finalize()” method.

When we ran the above program with a max heap size of 10 MB (i.e. -Xmx10M), it crashed with ‘java.lang.OutOfMemoryError’ after a few seconds of launch.

This program crashed with a ‘java.lang.OutOfMemoryError’ because only after the execution of the ‘finalize()’ method can SampleObject be evicted from memory. Since the ‘Finalizer’ thread is put to sleep, it can’t execute the “finalize()” method at the rate in which the ‘main()’ method was creating new ‘SampleObjects’. Thus, the memory got filled up and program resulted in a ‘java.lang.OutOfMemoryError’.

On the other hand, when we commented out the “finalize()” method, the program ran continuously without experiencing any ‘java.lang.OutOfMemoryErrors’.

How to Diagnose the Problem?

Your application might contain hundreds, thousands, or millions of classes. It includes classes from 3rd-party libraries and frameworks. Now the question becomes, how will you identify “finalize()” methods that are poorly implemented? This is a tough question to answer. This is where heap dump analysis tools like HeapHero.io might come handy.

When heap dump was captured from the above program and uploaded to HeapHero.io, it generated this beautiful report with several sections. The section that is of interest to us is ‘Objects waiting for finalization’.

objects-waiting-finalize

Fig 1: Excerpt from the report generated by HeapHero.io

This section of the report shows the amount of memory wasted due to objects waiting for finalization of your application. In this hypothetical example, 7.66 MB i.e. 97.2% is the amount of memory that is wasted.

finalization-object-tree

Fig 2: Objects waiting for finalization

When you click on the hyperlink given under ‘What are the objects waiting for finalization?’, you will be able to see the objects waiting to be finalized. Basically, you will be able to see the object tree of “j.l.r.ReferenceQueue” (note that this is the queue of the ‘java.lang.ref.Finalizer’ object that holds the reference of all objects whose finalize() methods need to be executed). If you drill down the tree, it will show the objects that are sitting in the queue waiting to be finalized. Here you can see 2 types of objects that are sitting in the queue:

  1. com.petals.finalize.SampleObject.data occupying 56.8% of memory
  2. com.petals.finalize.SampleObject occupying 11.5% of memory

BINGO!! These are the objects that are created in our sample program!

Original Link

DZone Research: The Future of Java

{{ articles[0].views | formatCount}} Views

Akka from A to Z, An Architects Guide starting off with Actors and Akka Streams, then on to clustering, sharding, event sourcing & CQRS, and more. 

To gather insights on the current and future state of the Java ecosystem, we talked to executives from 14 companies. We began by asking, “What’s the future of Java from your perspective?” Here’s what the respondents told us:

Serverless

  • Serverless, cloud-native, complete disaggregation. Jigsaw is important for disaggregation. Moving to the unikernal where you only get what you need.
  • 1) 2018 will be the year of Eclipse; 2) convergence with containers will accelerate; 3) Kotlin will become the next hot language; 4) the new release model will drive faster innovation; and, 5) serverless will result in a major reshaping of Java. 
  • I expect to see serverless grow over the next two years. In principle, Java is built for serverless but it needs work. With Spring Boot, containers can be lighter weight and great to build serverless upon.

Other

  • Java still has a long future ahead because if its strong presence in the enterprise. The quality has to remain high. If the quality doesn’t improve, it may fall into disfavor.
  • Modules try to get into smaller IoT devices. Simplification of language to get away from boilerplate. Good changes in Java 8. Containerized load for Java 8 and 9 for easier execution in container management, memory, and CPU.
  • It’s very bright. I don’t see any serious challenger to Java on the horizon. New languages come and go, but they don’t have the platform that Java has. Large enterprises don’t move off platforms that they understand, know how to manage at scale, and have built massive processes around. People massively underestimate the power of the JVM and overestimate the importance of “syntactic sugar” when predicting the rise and demise of languages. I’m happy to see a faster release cycle, and hope that Java can continue to innovate.
  • Frequent releases drive innovation, engagement and encourage new developers to add Java to their toolbox. More collaboration between groups like Eclipse and JVM languages on containers, cloud, and machine learning. We have a new JSR for visual recognition.
  • There’s two parts – there’s Java the language which is pretty long-in-the tooth and not likely to change, and there’s another part that is the Java virtual machine (JVM). When you write a Java program and you compile it, you end up with bytecode which is a series of instructions that is actually going to run on a computer and therefore the JVM is the thing that is actually executing the code you wrote. What is interesting about this is other languages can produce the same bytecode and then run on the same JVM, which means there’s a whole slew of languages that are being created that could work seamlessly with Java programs. The assumption is that one of these languages will one day replace Java while retaining access to the Java ecosystem.
  • Faster releases enable developers to have access to more features faster. Faster, lighter, more APIs. More developers are learning Java because they can earn more as a Java developer.
  • This may sound overly optimistic (and if you knew me, you would know that I am often not so positive), continued success in the world’s largest organizations (big businesses, big government, etc.).  The OSS ecosystem continues to thrive, and we are seeing brilliant innovations continuing to support Java’s ongoing success.
  • The future of Java is bright! With other languages such as Scala and Kotlin “competing” on the JVM I believe that that competitiveness will drive the evolution of Java toward an even better language.
  • Letting the language and the compiler be more intelligent. Building larger, more complicated systems with fewer lines of code. Data objects, data types make building AI/ML/NLP on top more easily.
  • Evolution of new JVM-based languages and tooling to support. This becomes more important as different paradigms become more popular. The JVM enables many different types of languages to be built.

Here’s who we spoke to:

Akka from A to Z, An Architects Guide starting off with Actors and Akka Streams, then on to clustering, sharding, event sourcing & CQRS, and more. 

Topics:

java ,serverless ,enterprise development ,jvm

{{ articles[0].views | formatCount}} Views

Opinions expressed by DZone contributors are their own.

Original Link

Java vs Kotlin: Which Programming Language Is Better for Android Developers?

At this year’s Google I/O, Kotlin – the statically-typed language from JetBrains – became an official language for Android. Initially announced in 2011 and launched in early-2016, Kotlin is now included in Android Studio 3.0, which was launched last week. In June, Kotlin ranked a perfectly respectable 43rd on the Tiobe Language Popularity Index, jumping up from the 80th spot in the previous month. Experts feel that, at this rate, the language can soon break into the list of top 20 languages (in June, Apple’s Swift was at 12th).

For coding Android applications, developers need a language that is “understandable” by the Java Virtual Machine (JVM). The arrival of Kotlin has given coders an alternative to the traditionally used Java language. In what follows, we will do an objective, point-by-point Java vs Kotlin comparison and try to find out which one has the better properties.

The Learning Curve

For switching over from an established language like Java to a new one like Kotlin to be a practically viable proposition, the learning curve has to be fairly simple – otherwise, the loss of skills and man-hours can be considerable. For experienced Java developers, the Kotlin language brings with it a high degree of familiarity, like the coding process for making new classes being nearly identical. Java and Kotlin share a number of syntaxes, too, making things easier for developers. In essence, Kotlin can be viewed as an improvement on Java, rather than being a full-blown substitute. If a coder is well-versed in Java, that skill and experience will stand him/her in good stead while starting to work with Kotlin.

The Issue With NullPointerExceptions

NullPointerExceptions, also known as the “Billion Dollar Mistake,” are a constant source of irritation for Android app developers coding with Java. In case a null value is assigned to any of the object references (in Java, any variable can have a null value), this exception is thrown. In Kotlin, this problem is kept out of the equation thanks to its inherent null-safety feature. Unless specially marked as “nullable,” no variable in Kotlin code can hold a null value. Even if you make a mistake and do assign a null value, the code will fail at the time of compilation. In non-trivial pieces of code, looking for the cause(s) of NullPointerExceptions can be problematic and time-consuming. Kotlin offers an easy way out of it.

Kotlin Is the “Modern” Language

A high percentage of Android app makers worldwide work with Java 7, simply because all the features of Java 8 are not yet available on the platform. The Android plugin has to be updated to v.3.0.0, and after that, users can play around with a limited subset of Java 8 features in the toolchain (after disabling Jack). Type annotations, lambda expressions, and method references are some of the important Java 8 features available on Android (some Java 8 language APIs are also supported). On the other hand, Kotlin captures the best points of procedural and functional programming (Java is still mostly procedural), offering the latest tools and resources to the developers. It is easily the more modern, lightweight, and well-rounded language, and is, hence, widely regarded as an enhancement over Java.

Interoperability

The seamless interoperability between Java and Kotlin makes the task of Kotlin beginners that much easier. Within the same project, both Java and Kotlin files can be used without the risk of any error being thrown. Kotlin libraries can be used in primarily Java projects easily (the reverse is also true). The fact that Java and Kotlin classes can reside simultaneously in the project does away with the need for full project conversion (to Kotlin), or having to start a project from scratch. First-time users of Kotlin can try coding with the new language on a specific part of a project – while keeping the legacy Java codes intact. Later, they can proceed to migrating the Java files to Kotlin (the migration should be one file at a time). There can be the occasional code in Kotlin that is difficult to decipher, although the inherent intuitiveness of the language should give developers some idea in such cases too. Interoperability with Java (extending to advanced Java libraries) is a key factor behind the rising adoption rate of Kotlin.

Note: Scala and Clojure are two other languages that are used to build Android apps. While both are JVM-friendly, none of these two have managed to become very popular (although they were launched before Kotlin).

Community Support

By virtue of being one of the oldest open-source programming languages ever, Java has a huge, and highly active, developer community across the globe. Regular updates are available (although they might not be immediately available for Android), and developers can get a lot of codes and references online. The online support for Kotlin cannot come close to matching that for Java, although the newer language is growing at a rapid clip. At last count, there were 2000+ Kotlin-based projects in GitHub – marking the remarkable rate at which Kotlin is growing. Even so, the probability of hitting a roadblock while coding for an app, with no support available, can be slightly higher for Kotlin-users. In the Java community, help is always available at hand.

Verbosity

For carrying out the same function (say, creating a floating action button), the volume of coding required in Java is considerably more than in Kotlin. In general, Kotlin has been designed as a significantly more compact language than Java. With the help of the Kotlin Android Extensions, references can be quickly imported into Activity files (within Views) – and then, that View can be used in the code as part of the Activity. This, in turn, means that developers do not have re-write the ‘findViewByld’ method for each case. Java does not have this feature – and hence, the amount of boilerplate code included in Java programming is a lot higher than that involved in Kotlin. All that the developer has to do before using the Kotlin extensions is add an additional plugin in the build.gradle file (inside the module). Unlike Java, there is no need for third-party dependency injection tools in Kotlin either.

Note: For coding a simple “Hello World” program in Kotlin, no classes or main method(s) are required. Less verbosity also, of course, lowers the probability of manual coding errors as well.

Checked Exceptions

Java has checked exceptions (i.e., Android developers have to catch and declare the exceptions), while Kotlin does away with them. There are many programmers who prefer having these checked exceptions – since they can make the code more robust and effective, by putting a definite emphasis on error recovery. However, in lengthy pieces of code, the many Java try/catch blocks can make things cluttered. From that standpoint, the absence of checked exceptions in this language can be viewed as an advantage. Opinions are divided in this regard – and it all boils down to what individual programmers are more comfortable working with. The overall codebase of Kotlin is more idiomatic than Java.

Increased Size of .apk

Not a major point of concern for small apps – but if the Android application you are working on has a large size, this can be a factor. Professional developers have confirmed that the Kotlin Standard Library increases the average size of .apk files by 800 KB – 1 MB. In Java, there is no such increase in the runtime sizes of .apk files. In select cases, this increased file size can cause issues (end users might consider this while downloading apps that are already ‘heavy’). The relative lack of complex Kotlin examples available online is also holding the language back somewhat. Java has the edge in this context, and Kotlin has plenty of scopes for refinement.

Note: With the plugin available in Android SDK 3.0, Kotlin (Java) files can be converted to Java (Kotlin).

Complete Support for Kotlin in Android Studio

The Android Studio IDE (integrated development environment) offers holistic support for the Kotlin language, ensuring that Android developers can easily make the switch from Java to the newer language. After installing and setting up the Kotlin plugin in Android Studio, configuring a new project is typically a matter of minutes – while there should not be any problems with initial readability and code compilation either. Right from code navigation and code completion, to refactoring, debugging and unit testing – Android Studio offers full support for all important activities in Kotlin. Converting complete source files written in Java to Kotlin is also a breeze. Developers, of course, have to ensure that there are no code losses and/or data corruption during such conversions. Incidentally, both Intellij (the base of Android Studio) and Kotlin have been created by JetBrains.

Implicit Widening Conversions and the Question of Stability

Java offers built-in support for implicit widening conversions with numbers. This keeps things straightforward. While coding in Kotlin though, Android developers have to go through the whole hog of performing similar conversions explicitly (implicit conversions are not supported) – every time a Byte value has to be assigned to an Int variable. In the Java framework, the smaller data types are converted to larger types under the hood – which is both quicker and simpler. Add to that the considerable mindshare that Java still enjoys (there is a certain feeling of stability while using an old, ‘tried-and-tested’ language) – and it becomes pretty much apparent that the popularity of Java is not going to drop off drastically anytime soon.

Higher-Order Functions and Data Classes

With the arrival of Java 8, support was finally extended for lambdas. However, as already mentioned earlier, all its features are not available to Android developers – who generally have to work with Java 6 or Java 7. In Kotlin, things are a lot more developer-friendly – with variables able to store functions for use later on. Lambdas, which are created by passing functions as expressions without declaring it (implicitly or explicitly) are supported in Kotlin, along with other higher-order functions (functions taken as parameters), like callback functions. Since Java does not support these, developers have to create a new interface and implement it separately (in Kotlin, there is only a handful of parameter functions). The conciseness of Kotlin becomes apparent while handling data classes. Unlike Java, where large volumes of boilerplate code have to be written for the different classes which only contain data, Kotlin allows users to perform the same function – simply by adding a keyword (‘data’) inside the class definition. Instead of having to create new fields and constructors for data storage (in Java), all ‘getters/setters’ are auto-generated by the compiler (in Kotlin).

No Extra Garbage Collection

A common bone of concern among many Android app developers working with Java is the extra garbage collection – which often renders the code less than optimally efficient. The problem gets particularly pronounced in instances of lengthy programs. Kotlin comes across as a better alternative – removing such problematic garbage collection (the garbage collector, of course, depends on the runtime, and not on the underlying programming language per se). What’s more – there are several XML-related pain points in Java, which can be removed in Kotlin – thanks to the Kotlin source file named Anko Library.

Note: The extension functions of Kotlin help in creating clean, easy-to-understand APIs.

First-Class Delegation in Kotlin

Also known as implicit delegation, this is a unique feature that gives Kotlin a further boost. Operations can be delegated by received objects to other “delegate” objects (helper objects) – which means that the language itself follows the ‘delegation over inheritance’ rule of thumb. Code duplication (say, on the setters and getters of different properties) can be avoided with the delegated properties of Kotlin – while multiple inheritance is also made possible. In addition, Kotlin also renders the task of type-checking before casting objects (in certain cases) unnecessary. The smart casts of Kotlin ensure that objects that are already checked with the is operator does not have to be cast again inside statements. In other words, the redundancy in casting gets removed.

Speed of Compilation

Java is still the faster language – with tests revealing that it has, on average, ~13% faster compilation speeds (with Gradle) than Kotlin (14.2 seconds vs 16.6 seconds). However, this difference of speeds is only for full builds. Since mobile app developers generally keep compiling codes after making certain changes – the more important parameter over here is the incremental speed of compiling (version 1.0.2 of Kotlin has incremental building). Now, the speed advantage of Java more or less disappears, with Kotlin either being at par, or even just a bit faster, for partial builds with incremental compilation. Summing up, Java performs ‘clean builds’ more quickly – but the two languages are similar (Kotlin might even have a slight advantage) in case of partial builds.

The Advantage of Coroutines

Android is a single-thread platform. In case this main thread is interrupted in any way (say, due to an intensive network I/O operation) – the UI of the app is likely to freeze, till the operation is completed. To avoid this problem, Java makes it necessary for developers to create background thread(s). This has a potentially serious downside – in the form of problems/confusions while managing multiple threads. Kotlin offers the smarter solution through coroutines. Executions can be suspended while performing the intensive operation, and resumed later – and the main thread does not get blocked at any point (the execution is generally restarted on a separate thread). Putting it in another way, Kotlin allows the creation of non-interrupting asynchronous codes, with coroutines being an important utility feature for developers.

Already, many Java libraries have started to provide Kotlin extensions (RxKotlin is a classic example). This, expectedly, is pulling up the usability of Kotlin. The language is further boosted by the built-in support for all important Java libraries and frameworks. It is still advisable for newbie Android developers to learn Java first, and then attempt the switch to Kotlin (type aliases and immutability are two more USPs). The former scores over Kotlin on a couple of factors (the huge community support is perhaps the biggest) – but Kotlin is closing in, and it does offer a fair few advantages over Java.

Original Link

Spice Up Your Java Using Lombok Annotations!

Java is heavily criticized and hated for its verbosity. Even for things such as reading/writing files we have to ceremoniously write a lot of generic stuff. The increasingly popular Java library Lombok has been hugely successful in addressing this issue. It’s an annotation based Java library that plugs itself into editors, IDE, and build tools at compile time, facilitating the generation of boilerplate code and making the code clean and more readable. There are a lot of features and annotations in this library, but to illustrate how useful I have found it in my day to day work I am going to talk about  @Data  annotation.

Let’s say we have a POJO (Plain old Java object) to represent some sort of data in our program. Using vanilla Java this is what we get:

Original Link

The Most Important Element of the Java Ecosystem

{{ articles[0].views | formatCount}} Views

Build vs Buy a Data Quality Solution: Which is Best for You? Gain insights on a hybrid approach. Download white paper now!

To gather insights on the current and future state of the Java ecosystem, we talked to executives from 14 companies. We began by asking, “What do you consider to be the most important elements of the Java ecosystem?” Here’s what the respondents told us:

JVM

  • The JVM is the most critical element followed by the vastness of the libraries and the completeness of the toolchains. The well-roundedness makes it so usable. 
  • The JVM is the foundation of the ecosystem. It provides fundamental VM support as well as for Scala and Kotlin. The interoperability of languages is guaranteed by the JVM. This is the Java ecosystems’ second biggest advantage. The set of libraries can play with each other in a nice, predictable way. 
  • The JVM is a nice platform – fast and runs code everywhere. Open versus others like .Net. Has Maven and Gradle that will work with any operating system.

Maturity

  • 1) The simplicity of the language, which enables the language to be used in so many diverse ways. 2) The stability of the language: because the language evolves slowly, the code you see today when you look at the Internet is very similar to the code in previous years. This enables developers looking for solutions to figure them out easily. 3) The maturity of the ecosystem, which enables a developer to find support to whatever they need, no matter how historical or cutting edge the things they need are. 4)The robustness of the IDEs: Java IDEs are the most advanced IDEs today. 5) The JVM, which enables languages other than Java to flourish. 
  • The maturity level is very high. A lot of frameworks, libraries, and IDEs that makes my life and job easier.

Openness

  • First and foremost, the fact that the language is open source while championed by a large company is critical. I think the fact that Java is moving to an annual release cycle will make it much more agile and responsive to modern software concerns. The JVM is a critical resource. Other languages will come and go (Rust?) but the JVM is what makes Java so important in the enterprise. It takes years to make a platform that can be managed at scale in the enterprise. 
  • Java benefits greatly from an incredibly vibrant and extensive Open Source Software ecosystem.  The OSS community has continued to enrich Java in numerous ways.  There is almost no average enterprise application challenge that has not been met by the Java OSS community.  Plus, even as we see continued evolution in new architecture styles and patterns (e.g. microservices, cloud-native, 12 Factor, FaaS), often driven by the rise of the Cloud, Java’s OSS community continues to innovate at an amazingly rapid clip. 
  • The openness and the will to open source. A lot of the most important parts, such as e.g. the JavaEE-specifications or the Spring Framework, are open sourced. Being bred from Java SE which is also available through the OpenJDK this is in the DNA of Java.

Compatibility

Other

  • The community. No one participant is more important than the community. We need more than one player to participate. We really value developer feedback. We also have corporate contributors and users along with the Apache and Eclipse Foundations and the 400 Java User Groups around the world. 75% of the JCP is made up of individual Java developers. There are 12 million Java developers. 
  • The community, the open JDK, Spring, and Java EE now Jakarta EE serving as the foundation for Spring.
  • The reason we originally picked Java was twofold – it is extremely stable and also very good for massively multithreaded processing. Because we run MMOs, Java lets our servers efficiently run many thousands of concurrent threads which enables thousands of players to interact and chat together in one shared world.

Here’s who we spoke to:

Respondents:

Build vs Buy a Data Quality Solution: Which is Best for You? Maintaining high quality data is essential for operational efficiency, meaningful analytics and good long-term customer relationships. But, when dealing with multiple sources of data, data quality becomes complex, so you need to know when you should build a custom data quality tools effort over canned solutions. Download our whitepaper for more insights into a hybrid approach.

Topics:

java ,jvm ,maturity ,backward compatibility ,openness

{{ articles[0].views | formatCount}} Views

Opinions expressed by DZone contributors are their own.

Original Link

Introduction to Java Bytecode

Reading compiled Java bytecode can be tedious, even for experienced Java developers. Why do we need to know about such low-level stuff in the first place? Here is a simple scenario that happened to me last week: I had made some code changes on my machine a long time ago, compiled a JAR, and deployed it on a server to test a potential fix for a performance issue. Unfortunately, the code was never checked into a version control system, and for whatever reason, the local changes were deleted without a trace. After a couple of months, I needed those changes in source form again (which took quite an effort to come up with), but I could not find them!

Luckily the compiled code still existed on that remote server. So with a sigh of relief, I fetched the JAR again and opened it using a decompiler editor… Only one problem: The decompiler GUI is not a flawless tool, and out of the many classes in that JAR, for some reason, only the specific class I was looking to decompile caused a bug in the UI whenever I opened it, and the decompiler to crash!

Desperate times call for desperate measures. Fortunately, I was familiar with raw bytecode, and I’d rather take some time manually decompiling some pieces of the code rather than work through the changes and testing them again. Since I still remembered at least where to look in the code, reading bytecode helped me pinpoint the exact changes and construct them back in source form. (I made sure to learn from my mistake and preserve them this time!)

The nice thing about bytecode is that you learn its syntax once, then it applies on all Java supported platforms — because it is an intermediate representation of the code, and not the actual executable code for the underlying CPU. Moreover, bytecode is simpler than native machine code because the JVM architecture is rather simple, hence simplifying the instruction set. Yet another nice thing is that all instructions in this set are fully documented by Oracle.

Before learning about the bytecode instruction set though, let’s get familiar with a few things about the JVM that are needed as a prerequisite.

JVM Data Types

Java is statically typed, which affects the design of the bytecode instructions such that an instruction expects itself to operate on values of specific types. For example, there are several add instructions to add two numbers: iadd, ladd, fadd, dadd. They expect operands of type, respectively, int, long, float, and double. The majority of bytecode has this characteristic of having different forms of the same functionality depending on the operand types.

The data types defined by the JVM are:

  1. Primitive types:
    • Numeric types: byte (8-bit 2’s complement), short (16-bit 2’s complement), int (32-bit 2’s complement), long (64-bit 2’s complement), char (16-bit unsigned Unicode), float (32-bit IEEE 754 single precision FP), double (64-bit IEEE 754 double precision FP)
    • boolean type
    • returnAddress: pointer to instruction
  2. Reference types:
    • Class types
    • Array types
    • Interface types

The boolean type has limited support in bytecode. For example, there are no instructions that directly operate on boolean values. Boolean values are instead converted to int by the compiler and the corresponding int instruction is used.

Java developers should be familiar with all of the above types, except returnAddress, which has no equivalent programming language type.

Stack-Based Architecture

The simplicity of the bytecode instruction set is largely due to Sun having designed a stack-based VM architecture, as opposed to a register-based one. There are various memory components used by a JVM process, but only the JVM stacks need to be examined in detail to essentially be able to follow bytecode instructions:

PC register: for each thread running in a Java program, a PC register stores the address of the current instruction.

JVM stack: for each thread, a stack is allocated where local variables, method arguments, and return values are stored. Here is an illustration showing stacks for 3 threads.

jvm_stacks

Heap: memory shared by all threads and storing objects (class instances and arrays). Object deallocation is managed by a garbage collector.

heap.png

Method area: for each loaded class, it stores the code of methods and a table of symbols (e.g. references to fields or methods) and constants known as the constant pool.

method_area.png

A JVM stack is composed of frames, each pushed onto the stack when a method is invoked and popped from the stack when the method completes (either by returning normally or by throwing an exception). Each frame further consists of:

  1. An array of local variables, indexed from 0 to its length minus 1. The length is computed by the compiler. A local variable can hold a value of any type, except long and double values, which occupy two local variables.
  2. An operand stack used to store intermediate values that would act as operands for instructions, or to push arguments to method invocations.

stack_frame_zoom.png

Bytecode Explored

With an idea about the internals of a JVM, we can look at some basic bytecode example generated from sample code. Each method in a Java class file has a code segment that consists of a sequence of instructions, each having the following format:

opcode (1 byte)      operand1 (optional)      operand2 (optional)      ...

That is an instruction that consists of one-byte opcode and zero or more operands that contain the data to operate.

Within the stack frame of the currently executing method, an instruction can push or pop values onto the operand stack, and it can potentially load or store values in the array local variables. Let’s look at a simple example:

public static void main(String[] args) { int a = 1; int b = 2; int c = a + b;
}

In order to print the resulting bytecode in the compiled class (assuming it is in a file Test.class), we can run the javap tool:

javap -v Test.class

And we get:

public static void main(java.lang.String[]);
descriptor: ([Ljava/lang/String;)V
flags: (0x0009) ACC_PUBLIC, ACC_STATIC
Code:
stack=2, locals=4, args_size=1
0: iconst_1
1: istore_1
2: iconst_2
3: istore_2
4: iload_1
5: iload_2
6: iadd
7: istore_3
8: return
...

We can see the method signature for the main method, a descriptor that indicates that the method takes an array of Strings ([Ljava/lang/String; ), and has a void return type (V ). A set of flags follow that describe the method as public (ACC_PUBLIC) and static (ACC_STATIC).

The most important part is the Code attribute, which contains the instructions for the method along with information such as the maximum depth of the operand stack (2 in this case), and the number of local variables allocated in the frame for this method (4 in this case). All local variables are referenced in the above instructions except the first one (at index 0), which holds the reference to the args argument. The other 3 local variables correspond to variables a, b and c in the source code.

The instructions from address 0 to 8 will do the following:

iconst_1: Push the integer constant 1 onto the operand stack.

iconst_1.png

istore_1: Pop the top operand (an int value) and store it in local variable at index 1, which corresponds to variable a.

istore_1.png

iconst_2: Push the integer constant 2 onto the operand stack.

iconst_2.png

istore_2: Pop the top operand int value and store it in local variable at index 2, which corresponds to variable b.

istore_2.png

iload_1: Load the int value from local variable at index 1 and push it onto the operand stack.

iload_1.png

iload_2: Load the int value from the local variable at index 1 and push it onto the operand stack.

iload_2.png

iadd: Pop the top two int values from the operand stack, add them, and push the result back onto the operand stack.

iadd

istore_3: Pop the top operand int value and store it in local variable at index 3, which corresponds to variable c.

istore_3.png

return: Return from the void method.

Each of the above instructions consists of only an opcode, which dictates exactly the operation to be executed by the JVM.

Method Invocations

In the above example, there is only one method, the main method. Let’s assume that we need to a more elaborate computation for the value of variable c, and we decide to place that in a new method called calc:

public static void main(String[] args) { int a = 1; int b = 2; int c = calc(a, b);
} static int calc(int a, int b) { return (int) Math.sqrt(Math.pow(a, 2) + Math.pow(b, 2));
}

Let’s see the resulting bytecode:

public static void main(java.lang.String[]); descriptor: ([Ljava/lang/String;)V flags: (0x0009) ACC_PUBLIC, ACC_STATIC Code: stack=2, locals=4, args_size=1 0: iconst_1 1: istore_1 2: iconst_2 3: istore_2 4: iload_1 5: iload_2 6: invokestatic #2 // Method calc:(II)I 9: istore_3 10: return static int calc(int, int); descriptor: (II)I flags: (0x0008) ACC_STATIC Code: stack=6, locals=2, args_size=2 0: iload_0 1: i2d 2: ldc2_w #3 // double 2.0d 5: invokestatic #5 // Method java/lang/Math.pow:(DD)D 8: iload_1 9: i2d 10: ldc2_w #3 // double 2.0d 13: invokestatic #5 // Method java/lang/Math.pow:(DD)D 16: dadd 17: invokestatic #6 // Method java/lang/Math.sqrt:(D)D 20: d2i 21: ireturn

The only difference in the main method code is that instead of having the iadd instruction, we now an invokestatic instruction, which simply invokes the static method calc. The key thing to note is that the operand stack contained the two arguments that are passed to the method calc. In other words, the calling method prepares all arguments of the to-be-called method by pushing them onto the operand stack in the correct order. invokestatic (or a similar invoke instruction, as will be seen later) will subsequently pop these arguments, and a new frame is created for the invoked method where the arguments are placed in its local variable array.

We also notice that the invokestatic instruction occupies 3 bytes by looking at the address, which jumped from 6 to 9. This is because, unlike all instructions seen so far, invokestatic includes two additional bytes to construct the reference to the method to be invoked (in addition to the opcode). The reference is shown by javap as #2, which is a symbolic reference to the calc method, which is resolved from the constant pool described earlier.

The other new information is obviously the code for the calc method itself. It first loads the first integer argument onto the operand stack (iload_0). The next instruction, i2d, converts it to a double by applying widening conversion. The resulting double replaces the top of the operand stack.

The next instruction pushes a double constant 2.0d  (taken from the constant pool) onto the operand stack. Then the static Math.pow method is invoked with the two operand values prepared so far (the first argument to calc and the constant 2.0d). When the Math.pow method returns, its result will be stored on the operand stack of its invoker. This can be illustrated below.

math_pow.png

The same procedure is applied to compute Math.pow(b, 2):

math_pow2.png

The next instruction, dadd, pops the top two intermediate results, adds them, and pushes the sum back to the top. Finally, invokestatic invokes Math.sqrt on the resulting sum, and the result is cast from double to int using narrowing conversion (d2i). The resulting int is returned to the main method, which stores it back to c (istore_3).

Instance Creations

Let’s modify the example and introduce a class Point to encapsulate XY coordinates.

public class Test { public static void main(String[] args) { Point a = new Point(1, 1); Point b = new Point(5, 3); int c = a.area(b); }
} class Point { int x, y; Point(int x, int y) { this.x = x; this.y = y; } public int area(Point b) { int length = Math.abs(b.y - this.y); int width = Math.abs(b.x - this.x); return length * width; }
}

The compiled bytecode for the main method is shown below:

public static void main(java.lang.String[]); descriptor: ([Ljava/lang/String;)V flags: (0x0009) ACC_PUBLIC, ACC_STATIC Code: stack=4, locals=4, args_size=1 0: new #2 // class test/Point 3: dup 4: iconst_1 5: iconst_1 6: invokespecial #3 // Method test/Point."<init>":(II)V 9: astore_1 10: new #2 // class test/Point 13: dup 14: iconst_5 15: iconst_3 16: invokespecial #3 // Method test/Point."<init>":(II)V 19: astore_2 20: aload_1 21: aload_2 22: invokevirtual #4 // Method test/Point.area:(Ltest/Point;)I 25: istore_3 26: return

The new instructions encountereted here are new , dup, and invokespecial. Similar to the new operator in the programming language, the new instruction creates an object of the type specified in the operand passed to it (which is a symbolic reference to the class Point). Memory for the object is allocated on the heap, and a reference to the object is pushed on the operand stack.

The dup instruction duplicates the top operand stack value, which means that now we have two references the Point object on the top of the stack. The next three instructions push the arguments of the constructor (used to initialize the object) onto the operand stack, and then invoke a special initialization method, which corresponds with the constructor. The next method is where the fields x and y will get initialized. After the method is finished, the top three operand stack values are consumed, and what remains is the original reference to the created object (which is, by now, successfully initialized).

init.png

Next, astore_1 pops that Point reference and assigns it to the local variable at index 1 (the a in astore_1 indicates this is a reference value).

init_store.png

The same procedure is repeated for creating and initializing the second Point instance, which is assigned to variable b.

init2.png

init_store2.png

The last step loads the references to the two Point objects from local variables at indexes 1 and 2 (using aload_1 and aload_2 respectively), and invokes the area method using invokevirtual, which handles dispatching the call to the appropriate method based on the actual type of the object. For example, if the variable a contained an instance of type SpecialPoint that extends Point, and the subtype overrides the area method, then the overriden method is invoked. In this case, there is no subclass, and hence only one area method is available.

area.png

Note that even though the area method accepts one argument, there are two Point references on the top of the stack. The first one (pointA, which comes from variable a) is actually the instance on which the method is invoked (otherwise referred to as this in the programming language), and it will be passed in the first local variable of the new frame for the area method. The other operand value (pointB) is the argument to the area method.

The Other Way Around

You don’t need to master the understanding of each instruction and the exact flow of execution to gain an idea about what the program does based on the bytecode at hand. For example, in my case, I wanted to check if the code employed a Java stream to read a file, and whether the stream was properly closed. Now given the following bytecode, it is relatively easy to determine that indeed a stream is used and most likely it is being closed as part of a try-with-resources statement.

public static void main(java.lang.String[]) throws java.lang.Exception; descriptor: ([Ljava/lang/String;)V flags: (0x0009) ACC_PUBLIC, ACC_STATIC Code: stack=2, locals=8, args_size=1 0: ldc #2 // class test/Test 2: ldc #3 // String input.txt 4: invokevirtual #4 // Method java/lang/Class.getResource:(Ljava/lang/String;)Ljava/net/URL; 7: invokevirtual #5 // Method java/net/URL.toURI:()Ljava/net/URI; 10: invokestatic #6 // Method java/nio/file/Paths.get:(Ljava/net/URI;)Ljava/nio/file/Path; 13: astore_1 14: new #7 // class java/lang/StringBuilder 17: dup 18: invokespecial #8 // Method java/lang/StringBuilder."<init>":()V 21: astore_2 22: aload_1 23: invokestatic #9 // Method java/nio/file/Files.lines:(Ljava/nio/file/Path;)Ljava/util/stream/Stream; 26: astore_3 27: aconst_null 28: astore 4 30: aload_3 31: aload_2 32: invokedynamic #10, 0 // InvokeDynamic #0:accept:(Ljava/lang/StringBuilder;)Ljava/util/function/Consumer; 37: invokeinterface #11, 2 // InterfaceMethod java/util/stream/Stream.forEach:(Ljava/util/function/Consumer;)V 42: aload_3 43: ifnull 131 46: aload 4 48: ifnull 72 51: aload_3 52: invokeinterface #12, 1 // InterfaceMethod java/util/stream/Stream.close:()V 57: goto 131 60: astore 5 62: aload 4 64: aload 5 66: invokevirtual #14 // Method java/lang/Throwable.addSuppressed:(Ljava/lang/Throwable;)V 69: goto 131 72: aload_3 73: invokeinterface #12, 1 // InterfaceMethod java/util/stream/Stream.close:()V 78: goto 131 81: astore 5 83: aload 5 85: astore 4 87: aload 5 89: athrow 90: astore 6 92: aload_3 93: ifnull 128 96: aload 4 98: ifnull 122 101: aload_3 102: invokeinterface #12, 1 // InterfaceMethod java/util/stream/Stream.close:()V 107: goto 128 110: astore 7 112: aload 4 114: aload 7 116: invokevirtual #14 // Method java/lang/Throwable.addSuppressed:(Ljava/lang/Throwable;)V 119: goto 128 122: aload_3 123: invokeinterface #12, 1 // InterfaceMethod java/util/stream/Stream.close:()V 128: aload 6 130: athrow 131: getstatic #15 // Field java/lang/System.out:Ljava/io/PrintStream; 134: aload_2 135: invokevirtual #16 // Method java/lang/StringBuilder.toString:()Ljava/lang/String; 138: invokevirtual #17 // Method java/io/PrintStream.println:(Ljava/lang/String;)V 141: return ...

We see occurrences of java/util/stream/Stream where forEach is called, preceded by a call to InvokeDynamic with a reference to a Consumer. And then we see a chunk of bytecode that calls Stream.close along with branches that call Throwable.addSuppressed. This is the basic code that gets generated by the compiler for a try-with-resources statement.

Here’s the original source for completeness:

public static void main(String[] args) throws Exception { Path path = Paths.get(Test.class.getResource("input.txt").toURI()); StringBuilder data = new StringBuilder(); try(Stream lines = Files.lines(path)) { lines.forEach(line -> data.append(line).append("\n")); } System.out.println(data.toString());
}

Conclusion

Thanks to the simplicity of the bytecode instruction set and the near absence of compiler optimizations when generating its instructions, disassembling class files could be one way to examine changes into your application code without having the source, if that ever becomes a need.

Original Link

The Incredible Shrinking Java Platform

shrinking duke

In an earlier blog, I wrote about the changes for deployment of Java applications that will take effect in JDK 11. Specifically, the removal of support for Applets and Java Web Start. In this post, I thought I’d delve a bit deeper into the suggested way to ship application code when you make the move to JDK 9 or later, so can take advantage of the Java Platform Module System (JPMS).

JDK 9 introduced the jlink command. This is a tool that can generate a custom Java runtime that only contains the modules that are required for a given application. The fundamental idea of jlink is to enable developers to move away from a centralized Java runtime, with all the associated issues of ensuring the correct version is installed, to one runtime per application (or service if you are using a microservices architecture). For ISVs, in particular, this can eliminate a host of potential support issues generated by not having the right version of the JDK that has been used to test and certify their software.

Initially, you might think that a potential drawback is an increase in size for downloads and the disk space that is used. To some extent this is true, but by eliminating unused modules, the runtime size is likely to be a lot smaller than a full JDK image. The jlink command also comes with a number of options that can help to reduce the size even further. Here are some examples:

  • --no-header-files: Excludes header files from the runtime image
  • --no-man-pages: Excludes man pages from the runtime image
  • --strip-debug: Strips debug information from the runtime image
  • --compress={0|1|2}: This allows compression of resources in the runtime image. 0 indicates no compression, 1 indicates constant string sharing and 2 indicates zip compression. It is also possible to add a regular expression to specify a pattern-list of files to include.

There is also an option listed in the documentation, --vm={client, server, minimal, all}, which gives the impression that you can also reduce the size of the JVM to some extent. Sadly, this is something of a red-herring, as the client and minimal VMs are only available on 32-bit platforms. Oracle no longer provide these binaries from JDK 9 onwards, but Azul will happily provide a 32-bit version of Zulu for both Windows and Linux. I tested the Zulu 9 JDK on Windows 7, and this will allow you to select client, server or all but the minimal configuration does not exist.

To build a runtime image with jlink, you need to specify where to find the modules your application or service requires and the modules you want to install in the runtime. For a fully modularised, application all you need to do is specify the module that contains your main() entry point and jlink will work out the dependency graph for you and include those automatically. You also need to tell jlink where to put the resulting runtime.

I constructed a simple example application that uses three modules called com.azul.zapp, com.azul.zoop and com.azul.zeta. The application makes use of the java.sql package so has a dependency on the java.sql module.

Here’s an example of how to use jlink with this application:

jlink --module-path $JAVA_HOME/jmods:zappmods --add-modules com.azul.zapp --output zappruntime

If we run the Java executable in the zappruntime/bin directory with the –list-modules argument we get the following output:

$ zappruntime/bin/java --list-modules com.azul.zapp com.azul.zeta com.azul.zoop java.base@9.0.4 java.logging@9.0.4 java.sql@9.0.4 java.xml@9.0.4

Of the 97 modules available in JDK 9, only the four required are included. The result is that the size of the runtime for this (admittedly rather trivial) application is only 47 Mb, including the application code. The full Java SE 9 JDK is 493Mb. If we use the –no-header-files, –no-man-pages and –strip-debug options the runtime size can be further reduced to only 40 Mb. I also tried this on a 32-bit Windows 7 machine and using the –vm=client option available in Zulu was able to reduce this even further to only 33 Mb.

At this point, you might be saying to yourself, “Hang on, Java has always had the marketing slogan of ‘Write once, run anywhere’. If this runtime only includes these modules, surely it’s not an implementation that conforms to the Java specification?” Prior to JDK 9, you would be right. For a runtime to conform to the Java SE specification, as defined by the relevant JSR, all standard libraries had to be present. This was the cause of the long-running legal dispute between Sun Microsystems and Microsoft at the end of the 1990s. Microsoft had decided to “improve” Java by adding some libraries to the java package namespace and leaving out some that they didn’t feel were necessary.

In the Java SE 9 specification things have changed to allow for tools like jlink to produce conforming runtimes. Section 7 of the specification defines the Modular Platform and includes a section on conformance. Here’s the critical part:

“Every Implementor of this Specification must implement the java.base module. Every Implementor may, further, implement a set of one or more additional standard SE modules so long as that set is closed, i.e., contains every standard module required by any member of the set.”

This defines two rules that need to be obeyed by any tool (since you can write your own jlink alternative) that produces a runtime image.

  1. The java.base module must be included. This is logical because all modules (other than java.base) implicitly have a dependency on java.base. Without it, nothing is going to run.
  2. If any other modules defined by the Java SE specification are included, then all dependencies of those modules must also be included. This equates to any requires elements of the module-info.java file of these modules. Since those modules may, in turn, have dependencies, the whole dependency graph of the included modules must be present.

If these two rules are followed then the module set in the runtime image generated is classed as closed and thus conforms to the specification.

The jlink tool is great if we have an application that has been developed with explicit modules. What about if we want to use jlink for an older application where everything is still on the classpath?

We can still use jlink, but we need to do a little more work.

We’ll use the same example application but this time remove the module-info.class files from the jars and put them all on the classpath. To use jlink, we need to know what JDK modules we require and we can use jdeps to find this information:

$ jdeps --list-deps -cp zapplib/*.jar Zapp.jar java.base java.logging java.sql unnamed module: zapplib/Zeta.jar unnamed module: zapplib/Zoop.jar

Note that if you don’t set the classpath to include the JAR files needed by your application you get this rather confusing output:

$ jdeps --list-deps Zapp.jar java.base java.logging java.sql not found

So, something was not found, but it’s not clear what.

We can now use jlink to build our runtime, thus:

jlink --module-path $JAVA_HOME/jmods --add-modules java.sql,java.logging --output zappruntime

Strictly speaking, we don’t need to specify java.logging as a module to add since it is required by java.sql but, since I used logging in the application I wanted to show the full process.

Checking which modules are in the runtime we get:

$ zappruntime/bin/java --list-modules java.base@9.0.4 java.logging@9.0.4 java.sql@9.0.4 java.xml@9.0.4

Lastly, we can copy our JAR files into the runtime generated by jlink. For our example, we’ll put them all in a directory called dist. Now to run our application we can do this:

$ zappruntime/bin/java -cp ‘zappruntime/dist/*’ com.azul.app.Main

As you can see, whether you want your application to use modules or stick with the classpath for now, it is easy to build a Java runtime tailored to your application. The advantages of doing this are reduced size requirements and eliminating concerns of having the exact version of the JDK to support the application.

With JDK 10 having been released this week, maybe it’s time to make the move to the latest JDK.

Original Link

Java Version Upgrades: GC Overview

I’m guessing there are many companies on the verge of upgrading to a newer Java version because developers love to upgrade, but besides this adventurous attitude, there is also fear. The biggest fear of such a change is that applications can produce unexpected behaviors. In my opinion, the root of this fear is the GC system, the fear of the unknown. 

First, we will do a quick walkthrough of how memory management architecture looks, what kind of GC algorithms are available, and what GC types exist. We will then go over how to come up with a plan that is safe and can persuade management that switching to a newer Java version won’t be the end of worlds. If you are pretty confident on how the GC works, you can skip to the end of the article where the TODOs are listed. For others, let’s start with the architecture for the JVM.

Memory Management Architecture

Many modern MM systems split the available memory into to two key parts based on the type. MetaData (MetaSpace/PermGenSpace) is where you store how data is represented. In Java, this storage is mostly populated by ClassLoaders when loading a class for the first time. This contains, for example, which methods are available in the loaded class and what types of data can be stored in a particular field. Runtime constant pools can also can be found here.

Image titleThe other part is where the actual runtime data resides, known as Java Runtime Memory. This can further be split into two parts, Stack and Heap. The Stack holds data like method local variables, callstack, and Thread-related metadata. They are of relatively small size (~1MB). 

Meanwhile, the Heap holds Object Instance-related data, like actual content of a field defined in your class. Where does Thread-related data get stored? When a thread is started, the JVM allocates a stack separate from the Heap and MetaSpace. Besides size, the Heap also has an important property — how long we actually use certain memory parts.

But why do engineers create this kind of complexity around memory management? Just so they can ask fancy interview questions, or maybe so Oracle can sell support on how to tune your JVM? Behind every complicated solution, there is the desire to building something better. The reason is to have better optimized Garbage Collection systems.

Garbage Collection

We know that memory is finite, so if we are not freeing it up efficiently, we are going to run out of it eventually, leading to us furiously reading the ‘java.lang.OutOfMemoryError’ class’s Javadoc. When dealing with sophisticated languages, the allocation and freeing of memory is taken care of by the environment. Many things can happen in the background, but basically, the pseudo code for GC is the following.

  1. Mark the parts of memory that are potentially still in use (tree traversal found it)

  2. Set the memory pointers free

  3. Defragment memory by moving referenced objects together

This doesn’t sound too complicated, right? Well, for starters, this can be quite expensive in processing power. You can create objects that can’t be reclaimed by the GC, or the GC algorithm simply can’t keep up if your application receives a higher load. When GC runs, Threads in the JVM can be suspended and make your application unresponsive.

Generational Hypothesis

You either die as an Object part of the Young generation, or live long enough to see yourself end up in the Old generation.

Engineers found out that most application data falls into two categories — data that is collected quickly or data that stays around for a long time. The gain here is that you don’t have to run GCs so frequently on the old generation, thus saving a lot of processing power. Of course, objects of the Old Generation can refer to YoungGeneration objects — these are cross-generational links. The JVM has a way of dealing with this — splitting up the Young Generation into three more regions, Eden, Survivor1, and Survivor2.

Based on this information, we can categorize different Garbage Collections strategies.

GC Types

Type Target Trigger Impact

Minor GC

Young generation

Eden getting full

No effect on latency*

Major GC

Old generation

Minor GC fails

Can have latency effect

Full GC

Whole heap + MetaSpace

Minor or Major GC fail

Can have latency effect

* If it fails, it can trigger MajorGC, which can eventually have latency effects
If you are interested in what events are happening in your JVM regarding GC runs, you can start the JVM with -XX:+PrintGCDetails . 

GC Algorithms

Type Threads Algorythm Effect Default in

Serial

YoungGen: single
OldGen: single

YoungGen: mark and copy
OldGen: mark sweep compact

Stop the world every case Java6
if client class*

Parallel

YoungGen: multi
OldGen: single

YoungGen: mark and copy
OldGen: mark sweep compact

Stop the world when OldGen
cleanup is necessary
Java6
if server class**

ParalellOld

YoungGen: multi
OldGen: multi

YoungGen: mark and copy
OldGen: mark summary compact

No stop the world, but
allocation is not so efficient
non default

Concurrent
Mark Sweep

YoungGen: multi
OldGen: multi

YoungGen: mark and copy
OldGen: mark compact

No stop the world, but
allocation is not so efficient

non default
G1 GC

YoungGen: multi
OldGen: multi

memory split into chunks and marked
as usable or “under maintenance”
which is cleaned by the GC currently
Best of both, the promise of
 less GC pauses, more
 predictable GC runs
Java9

*client class: 32-bit architecture or single processor. Today, it’s kind of irrelevant
**server class: Two or more physical processors and 2 or more GB of memory

We can decide which GC algorithms fit best for our needs.
Examples: -XX:+UseSerialGC-XX:+UseG1GC, and -XX:-UseConcMarkSweepGC 

These can be fine-tuned even more with additional parameters, but tuning the GC should be a last resort. 

Monitoring

Memory problems are hard to detect because they are not strictly bound to functional behavior. You can greenlit a project that worked fine for the e2e test and the 5 QA people, but that will break with real load in production in minutes. Know what is happening in your application! If you have already released your software, this should be on point and may only need minor tweaks. If not, you are probably doing blood sacrifices after every retro. Some important measurement values: Memory usage, request response time, threadcount, CPU usage, connections, and GC statistics (-XX:+PrintGCDetails).

These values should be stored as a data series so you can see behavior trendlines. You need to define your system’s current thresholds. 

Examples:

  • CPU percentage should never go over 60%.

  • GC pause never higher than 0.7 sec,

  • Memory usage is always under 70%

  • Avg. Response time is always under 2 sec

  • Active connections less than 300

You will constantly compare to this baseline. Anything worse is not acceptable. If the values start to worsen, it is worth doing an investigation. Try to isolate parts of your system. Threads are also a huge help for such investigations.

The Plan

You should!

So now we know what kind of GCs there are, how are they triggered, what they can cause, and what properties a Minimum Viable Monitoring System (MVMS) should contain. With this knowledge, and after reading many other articles, you can start creating a plan for making a switch. We did this kind of upgrade from Java 7 to Java 8. I tried to create a list of TODOs on what we did for such an upgrade.

  1. Use performance tests and measure the properties listed in the monitoring section. These are crucial to be able to make good decisions later on. Define thresholds for these values. It’s better to run this test regularly as development goes on — bottlenecks could be introduced with new features.

  2. Create easy upgrade and downgrade scripts/configurations for your dev and prod environments. You need to be able to switch easily between the two states of the system. This includes GC configuration, not just the JVM version.

  3. Update your CI environment to enforce compilation with your current JVM version. This also means that you should not start to use the new Java language features for the first release. This is to ensure backward compatibility. Newer versions of the JVM can run code compiled on an earlier version of Java.

  4. Validate your new environment functionality with automated and manual tests. 

  5. Run performance tests using the new Java version environment. Compare the results you got mentioned in step 1. If, somehow, thresholds aren’t met, try to rule out new code changes from development.

  6. Run stability tests. Close to average load should be produced against the system for at least 12 hours. This is the real test of how the GC behaves.

  7. Release the new JVM to the production system. Until this point, we are only doing changes to the JVM and its configuration, no Java 8 code implementation is allowed.

  8. Now you are running your code on a new JVM environment, but the devs see no real benefit from the new functionalities. It’s time to switch your CI environment to compile with the new version of Java.

  9. Repeat step 4-5-6-7. You might think you are done, but there is one last additional step.

  10. Educate your fellow developers of the capabilities of your new Java version. Do some knowledge sharing. Explain how the new GC system you are using works. This is the real fruit of the upgrade, and hopefully sees some performance gain.

Original Link

Hello OpenJ9 on Windows, I Didn’t Expect You so Soon!

Faster startup time, lower memory footprint, and higher application throughput only by replacing the Java Virtual Machine? That sounds too good to be true. So far, there has been no real alternative to Oracle’s Java HotSpot VM on Windows. With Eclipse OpenJ9, which emerged from open-sourcing IBM’s J9 VM, there is now the alternative that promises exactly this.

At the end of January, the first OpenJDK 9 with Eclipse OpenJ9 nightly builds for Windows were published, but they were not very stable at that time. This week, I tested the nightly builds again to run the Eclipse IDE and I was pleasantly surprised: OpenJ9 ran without crashing. Here are my results: the start time of the Eclipse Oxygen.2 Java IDE improves with OpenJ9 from 20 to 17 seconds, with some tuning (see below) even to 12 seconds compared to the Java 9 JDK with Oracle’s HotSpot VM on my more than six-year-old laptop. Also, the Windows Task Manager shows less memory used by the Eclipse IDE and tasks like compiling a large project are a bit faster with OpenJ9.

To start the Eclipse IDE with OpenJ9, in eclipse.ini add the following two lines above -vmargs:

-vm
C:\path\to\jdk-9+181\bin\javaw.exe

Embedding the JDK into an Eclipse installation directory as a jre subdirectory does not yet work, but as long as you do not start the Eclipse IDE from the command line from another directory, you can use -vm with jre\bin\javaw.exe. To further improve the startup time, add the following two lines below -vmargs:

-Xtune:virtualized
-Xshareclasses:cacheDir=C:\path\to\shareclasses

The cloning of a GitHub repository fails due to missing certificate authority (CA) certificates. You can fix this OpenJDK 9 issue by replacing the lib\security directory (which contains the cacerts file) with the same directory of an OpenJDK 10 early access build.

In the Eclipse IDE that is running on OpenJDK the standard basic text font defaults (for reasons I don’t know) to Courier New 10 instead of Consolas 10. You can change this in Window > Preferences: General > Appearance > Colors and Fonts by selecting Basic > Text Font and pressing Edit… (if you like, you can also use Source Code Pro 9 like the Clean Sheet theme does).

I have not noticed so far any further differences between the Eclipse IDE running on Oracle’s JDK 9 and the Eclipse IDE running on the current OpenJDK 9 with OpenJ9 nightly build. Debugging and hot code replace works as expected.

Many thanks to the OpenJ9 team! I look forward to the final release. It’s great to have two good open source Java virtual machines for Windows. Who knows, but with only one of the two, neither of the two might be open source today.

PS: If the Eclipse IDE still starts too slowly for you, have a look at a developer build of the upcoming Eclipse Photon IDE.

Original Link

IntelliJ IDEA 2018.1 Public Preview

Good news everyone! IntelliJ IDEA 2018.1 is now ready for Public Preview! The upcoming IntelliJ IDEA 2018.1 brings a lot of important improvements: support for Git partial commits, inline external annotations, merged features from Android Studio 3.0 and many more. We are excited about all these new features, and we encourage you to take the preview build for a ride right away!

Enhancements in Code Completion

In the upcoming IntelliJ IDEA 2018.1, there are enhancements to the code completion. Now completion in the Stream API chains is aware of the type casts. The code completion not only suggests a completion item according to the existing call filter(String.class::isInstance), but also an automatically typecasted completion item.

image37

We have also improved the postfix code completion in the upcoming release. Now the IDE allows you to create your own Java templates or edit and rename some of the predefined Java templates at Preferences | Editor | General | Postfix Completion.

image4

image3

Improvements in Data Flow Analysis

With the upcoming IntelliJ IDEA 2018.1 release we’ve improved our data flow analysis so now it detects a wider variety of potential problems in your code.
First of all, the data flow analysis now tracks relations between variables like “greater than” and “less than”. The IDE detects when a condition is always true (or false) in all the possible code paths when the variables are compared.

image17

Another enhancement for the data flow analysis in the upcoming release is that it now works for non-terminated stream API chains.

image19

The IDE will now warn you when you try to assign a variable to the same value it already contains, this may help you to detect and then remove some redundant code.

image32

The IDE also warns you about modifications of immutable collections.

image6

If you are interested, you can read this blog post with more detailed information about the enhancements in data flow analysis.

As always, the upcoming IntelliJ IDEA 2018.1 has a bagful of new inspections and quick fixes. Now, the IDE detects and warns you about the while-loops with an idempotent body, as in most cases this indicates a programming error and can lead to a program hang.

image33

Also, the IDE detects while-loops with a conditional break at the end or beginning of an infinite loop. The IDE now offers a quick-fix to replace a break condition with a loop condition, because in most cases it’ll make your code look clearer.

image29

The upcoming IntelliJ IDEA now warns you about any infinite streams that weren’t short-circuited, as such operations can be completed only by throwing an exception. Such code may result in an infinite loop or a running out of memory issue.

image22

The IDE now gives you the ability to sort the array content alphabetically.

image18

If there is a copy constructor that doesn’t copy all the fields in a class, you’ll get a notification about it. The IDE considers fields with a transient modifier unnecessary to copy.

image12

The upcoming IntelliJ IDEA now warns you about an explicitly redundant close() call and provides a handy quick-fix to remove it.

image39

The upcoming IntelliJ IDEA features Java 9 specific inspections and quick-fixes. The IDE checks that a service loaded by ServiceLoader is declared in the module-info.java file and provides the quick-fix to add a missing statement to the module-info.java file.

image24

For an unresolved class mentioned in module-info.java, the IDE now suggests creating that missing class. The IDE suggests creating missing exported packages as well. The IDE creates the package with the class in the required directory, as you can’t export an empty package in Java 9.

image2

Now when there are several different approaches on how to fix possible problems in the chosen scope IntelliJ IDEA gives you the ability to group all the suggested quick-fixes by their quick-fix type under the Fix partially button in the Inspection Results Tool Window.

image28

JUnit 5 @Tag Annotation Support

The upcoming IntelliJ IDEA 2018.1 now supports the JUnit5 @Tag annotation to give you the ability to include in the testing scope, tagged classes and tagged methods. Select the Tags (JUnit 5) option in the test kind field in the Run/Debug Configuration dialog. Use the Uniqueld field to filter tests according to their id.

image13

Code Generation

With IntelliJ IDEA you can generate a test class, and by default, the IDE will add the Test suffix to the test class name. Now you have the ability to customize a test class template to make it possible to create a test class with a Test prefix in the test class name: Preferences | Editor | Code Style | Java | Code Generation.

JVM Debugger

The new Throw Exception action now allows you to throw an exception from a certain location in your program without changing the code. It is available from the Run | Throw Exception menu, or the frame context menu during a debugging session.

image43

Print breakpoint Stacktraces

The upcoming IntelliJ IDEA 2018.1 allows you to print breakpoints Stacktraces to the console. You can enable the Stacktrace option in the Breakpoints dialog box. The IDE also provides you with the ability to observe multiple breakpoints Stacktraces at the same time in the Console log.

image9

Also, you can now copy the current thread stack trace via a new Copy Stack action which is available from the frame context menu.

image27

Java Compiler

There is a new Use --release option for cross-compilation (Java 9 and later) check-box on the Java Compiler page at Preferences | Build, Execution, Deployment | Compiler | Java Compiler that is enabled by default. When you need to use the --source and --target options with Java 9 and link against Java 9 classes at the same time you now can disable the new checkbox.

Also now there is a new ability to use a specific version of the ECJ compiler. Select Eclipse from the Use Compiler drop-down menu, and in the field specify the path to jar with the needed compiler.

Editor

In IntelliJ IDEA you can annotate your code with external annotations even when direct annotation of the source code is not possible (library classes). You can configure your annotations in the annotations.xml files, which are stored outside of your source code. Previously the IDE only showed the @ icon in the gutter near the externally annotated code; now the IDE shows these external annotations inline in your code.

image30

IntelliJ IDEA now lets you view the automatic inferences of @NotNull or @Nullable annotations right in your source code (not only in the gutter icon near the inferred annotation as it was before). You can enable the Show inferred annotations inline check-box in the Preferences | Editor | General | Appearance.

image26

The upcoming IntelliJ IDEA 2018.1 gives you the ability to quickly find a problem in your code. The IDE now highlights the folded code regions that contain errors or warnings, and colors such blocks according to their validation status.

image11

The IDE also highlights the folded code regions if they contain any matching occurrences when you search through the current file.

image14

When you place the caret on an identifier, and the IDE highlights its usages you can now use the new Alt + Wheel down or Alt + Wheel up shortcuts to jump to the next or previous identifier occurrence.

image41

Project Configuration

There is a new Include dependencies with “Provided” scope option for the Application and Spring Boot configurations in the Run/Debug Configurations dialog. This new feature allows you to add “provided” dependencies to the classpath when needed. Please note that the Include dependencies with “Provided” scope option is enabled by default for the Spring Boot applications.

The upcoming IntelliJ IDEA now gives you the ability to change qualified names for several modules at once by using the new Change Module Names… action from the context menu of the Project Structure dialog.

image5

Replace Improvements

The upcoming IntelliJ IDEA 2018.1 has the ability to preview a regex replacement in the Replace in Path window.

image1

Structural Search Enhancements

We have improved our Structural Search feature to give you the ability to find method calls to annotated methods. In the Structural Search dialog, you can create your own search template or choose from the existing search templates. In the following example, the Structural Search finds all method calls to methods marked as @Deprecated.

image40

StructuralSearch

Groovy

For Groovy files and modules a new refactoring action is available from the context menu in the Refactor | Convert to @CompileStatic.
The Convert to @CompileStatic action annotates every groovy class in the scope of the @CompileStatic annotation.

2018-02-27 18_05_36

Android

Here is some long-awaited news for Android Developers! The upcoming IntelliJ IDEA 2018.1 merges the changes from Android Studio 3.0 and brings in dozens of new features, with the following major new ones:

First of all, IntelliJ IDEA now supports the latest Android Oreo APIs, and lets you build Java 8 Android applications, as well as Kotlin Android applications.

Furthermore, now the IDE supports Gradle 3.0.0 for Android applications.

The Layout Editor has been improved: a new toolbar layout and icons, updated layouts in the component tree, a new error panel and more.

image36

Now you can create App Icons with the updated Image Asset Studio. Right-click the res folder in your Android project, and select New | Image Asset from the context menu. In the Asset Studio window, select Launcher Icons (Adaptive and Legacy) as the icon type.

image25

The IDE now supports the ability to build Instant Apps – lightweight Android apps that can be run without installation. To start building Instant Apps, make sure that the Instant Apps Development SDK is installed. You can check which SDK tools are installed if you need to in Preferences | Appearance & Behaviour | System Settings | Android SDK in the SDK tab.

image35

image10

The new Device File Explorer Tool Window displays the file and directory structure of your Android device or emulator. Use the Device File Explorer Tool Window to view, copy or delete files on an Android device. You can access it through View | Tool Windows | Device File Explorer.

image31

The upcoming IntelliJ IDEA 2018.1 has also merged the Android Profiler, a brand new suite of profiling tools that provide real-time data for your app’s CPU, memory, and network activity.

image15

To learn more, please refer to the Android Studio Release Notes.

Version Control System

One of the highlights of the upcoming release is the support for partial Git commits (git add -p). IntelliJ IDEA now provides you with the ability to associate the code chunks with a changelist. Create a changelist, put all the needed code chunks there and then commit it. The IDE now commits only the selected changes from the file and skips all other changes.

To add the needed code chunks to a commit – use the checkboxes in the gutter in the Diff pane of the Commit Changes dialog.

image34

To move the code chunks between changelists use the Move to another changelist action from the context menu of the Diff pane in the Commit Changes dialog.

image42

As another option, the IDE gives you the ability to add code chunks to a changelist from the editor by simply clicking on a change marker in the gutter.

Furthermore, the upcoming IntelliJ IDEA 2018.1 includes the ability to toggle the grouping of your local changes in the Local Changes tab of the Version Control Tool Window via the new Group by icon. Use to group local changes according to their: directory, module or repository. The IDE now gives you the ability to select one of the grouping options or all three at once.

image8

There are several improvements in the Log tab – the tab which is available for Git and Mercurial VCS. The Commit Details pane of the Log tab has been redesigned. Now you can quickly navigate to a commit in the Log by clicking the corresponding commit hash in the Commit Details pane.

image21

For Git integration, we’ve improved performance of the History for revision tab. Also this tab now has a new UI.

image16

Also, the Abort Rebase, Continue Rebase, and Skip Commit actions are now available from the Git Branches pop-up if there is an ongoing rebase process.

image23

The IDE has the new default Alt + Cmd + K (macOS) or Alt + Ctrl + K (Windows and Linux) shortcut to perform the Commit and Push…action from the Commit Changes dialog.

Moreover, the Clone Repository dialogs for Git and GitHub have been merged into one. Also, the autocompletion for the GitHub repositories is working in the new Clone Repository dialog. You just need to log in to your GitHub account using the Log in to GitHub… button.

image38

Also, the SVNKit library has been removed. You can find more information in the dedicated blog post.

Enhancement in Docker Compose

The Docker Compose workflow has been improved. The Run/Debug Configurations dialog for the Docker Compose run configuration has been improved to make it possible to use important Docker Compose features such as support of multiple compose files, and the ability to choose a service to run.

The Docker plugin now supports Multiple Docker Compose files and respects not only a docker-compose.yml but also an optional docker-compose.override.yml file. You can add the docker-compose.override.yml as any other override file right after the base configuration file.

Screen Shot 2018-02-27 at 11.07.19

For Docker Compose files you can now define an environment variable in the updated Run/Debug Configurations dialog. Also, if you want to use the --build command line options, enable the Force build checkbox.

The Docker plugin allows you to choose the set of services to run after you choose configuration files in the Docker Compose Run configuration.

The Spring Boot Enhancement

  • Support for Spring Boot Devtools.
  • A new gutter icon provides you with the ability to open methods with @RequestMapping annotations via the new REST client.

Other

  • The IDE automatically resizes the graphics to fit the window.
  • During the import of an Eclipse project, IntelliJ IDEA is now able to import your code style configuration.
  • There is a new Open in terminal action that launches the integrated terminal on the folder’s path.
  • Better HiDPI support on multiple displays for Windows

Also, the JDK has been updated to 1.8.0_152-release-1136-b16, with the following fixes integrated:

  • Follow up fix for the issue with running IDE on 32-bits Windows  JRE-590.
  • Position of IME Composition windows was fixed  JRE-668,  JRE-669.
  • The issue with displaying UI after changing DPI was fixed  JRE-660.

With the upcoming IntelliJ IDEA 2018.1 we have completely reworked our focus subsystem. Many focus-related issues have been already fixed, such as: the Search Everywhere pop-up that now receives focus, and the Project tool window that now receives focus when invoking from Select In pop-up and many others, check this link. We expect that with the updated focus subsystem we can fix focus related issues much faster.

A big thanks for all the bug reports, please keep them coming!

Last but not least, the built-in SSH executable is now compatible with the new GitHub cryptographic standards. Learn more here.

You can look through the full list of v2018.1 changes in the Public Preview release notes.
Here’re the release notes for the 181.3986.9 build.

We have tons of improvements coming with the upcoming release! Download the IntelliJ IDEA 2018.1 Public Preview build and see for yourself! We welcome your feedback, so please reach out to us in the EAP discussion forum, issue tracker, Twitter or in the comments below.

Happy developing!

Original Link

CodeTalk 2.0: Containers, the Java EE Way [Podcast]

Introduction

Intro by DZone Content and Community Manager, Michael Tharrington

Welcome to the first episode in DZone’s relaunch of our official CodeTalk podcast!

Image title

Picking up the podcast where our previous host, John Esposito, left off, our new joint hosts, Travis Van and Travis Carlson, are adapting the show slightly to fit the theme of early conversations with the creators of new developer technologies.

While I’m sure you’ll get to know them better through the show, I’d like to take a brief moment to introduce our hosts before we dive into the first episode.

Image title

Travis Van comes from a decade-long background in tech PR and marketing, having worked at a variety of places including MuleSoft where he was “Employee #3 (first marketing hire).” In 2007, underwhelmed by the range of PR and marketing research tools available to tech companies, Travis decided to build something better and thus founded TechNews.io, a platform for promoting tech companies.

Image title

Travis Carlson is a veteran software dev focused on back-end enterprise systems. He’s been working with the JVM since Java 1.2 and distributed systems since the advent of the Internet. As a systems architect, he has managed the entire stack—from DevOps in AWS all the way to front-end development with AngularJS. His passion is creating systems which are agile, maintainable, scalable, and robust.

So, now that you know a little bit about our hosts, let’s get to the meat—our first new episode in the CodeTalk relaunch. Take it away Travis and Travis!

CodeTalk: Containers, the Java EE Way

Sebastian Daschner joins CodeTalk this week to preview a talk he’ll be giving at the Index Developer Conference organized by IBM (Feb. 20-22 in San Francisco): “Containers and Container Orchestration — The Java EE Way.”

DZone readers who have been following containers may have noticed how conspicuously absent the Java stack is from the whole Docker/Kubernetes/Container discussion. We were really interested to jump in and understand how Java EE fits the world of Docker.

If you’re at an enterprise Java shop, you’re going to find Daschner’s talk an interesting reference point in understanding:

  • When it makes sense to containerize Java applications
  • How Kubernetes and Istio make it possible to orchestrate Java EE microservices in a modern enterprise system
  • How this convergence fits into the overall Java developer priority to build and scale faster

Want More CodeTalk?

We’re still in the early stages here with the relaunch. But soon we hope to give our CodeTalk landing page a face lift and are going to house all future and past episodes in this one place. 

For now, stay tuned to DZone for weekly episodes released each Friday. And, if you’d like to contact the showrunners to get involved as an interviewee or just simply share your feedback, feel free to message Travis and Travis here: codetalkpodcast@gmail.com.

Original Link

Creating Annotations in Java

Annotations are a powerful part of Java, but most times we tend to be the users rather than the creators of annotations. For example, it is not difficult to find Java source code that includes the @Override annotation processed by the Java compiler, the @Autowired annotation used by the Spring framework, or the @Entity annotation used by the Hibernate framework; but rarely do we see custom annotations. While custom annotations are an often-overlooked aspect of the Java language, they can be a very useful asset in developing readable code and just as importantly, useful in understanding how many common frameworks, such as Spring or Hibernate, succinctly accomplish their goals.

In this article, we will cover the basics of annotations, including what annotations are, how they are useful in large-than-academic examples, and how to process them. In order to demonstrate how annotations work in practice, we will create a Javascript Object Notation (JSON) serializer that processes annotated objects and produces a JSON string representing each object. Along the way, we will cover many of the common stumbling blocks of annotations, including the quirks of the Java reflection framework and visibility concerns for annotation consumers. The interested reader can find the source code for the completed JSON serializer on GitHub.

What Are Annotations?

Annotations are decorators that are applied to Java constructs, such as classes, methods, or fields, that associate metadata with the construct. These decorators are benign and do not execute any code in-and-of-themselves, but can be used by runtime frameworks or the compiler to perform certain actions. Stated more formally, the Java Language Specification (JLS), Section 9.7, provides the following definition:

An annotation is a marker which associates information with a program construct, but has no effect at run time.

It is important to note the last clause in this definition: Annotations have no effect on a program at runtime. This is not to say that a framework may not change its behavior based on the presence of an annotation at runtime, but that the inclusion of an annotation does not itself change the runtime behavior of a program. While this may appear to be a nuanced distinction, it is a very important one that must be understood in order to grasp the usefulness of annotations.

For example, adding the @Autowired annotation to an instance field does not in-and-of-itself change the runtime behavior of a program: The compiler simply includes the annotation at runtime, but the annotation does not execute any code or inject any logic that alters the normal behavior of the program (the behavior expected when the annotation is omitted). Once we introduce the Spring framework at runtime, we are able to gain powerful Dependency Injection (DI) functionality when our program is parsed. By including the annotation, we have instructed the Spring framework to inject an appropriate dependency into our field. We will see shortly (when we create our JSON serializer) that the annotation itself does not accomplish this, but rather, the annotation acts as a marker, informing the Spring framework that we desire a dependency to be injected into the annotated field.

Retention and Target

Creating an annotation requires two pieces of information: (1) a retention policy and (2) a target. A retention policy specifies how long, in terms of the program lifecycle, the annotation should be retained for. For example, annotations may be retained during compile-time or runtime, depending on the retention policy associated with the annotation. As of Java 9, there are three standard retention policies, as summarized below:

Policy Description
Source Annotations are discarded by the compiler
Class Annotations are recorded in the class file generated by the compiler but are not required to be retained by the Java Virtual Machine (JVM) that processes the class file at runtime
Runtime Annotations are recorded in the class file by the compiler and retained at runtime by the JVM

As we will see shortly, the runtime option for annotation retention is one of the most common, as it allows for Java programs to reflectively access the annotation and execute code based on the presence of an annotation, as well as access the data associated with an annotation. Note that an annotation has exactly one associated retention policy.

The target of an annotation specifies which Java constructs an annotation can be applied to. For example, some annotations may be valid for methods only, while others may be valid for both classes and fields. As of Java 9, there are eleven standard annotation targets, as summarized in the following table:

Target Description
Annotation Type Annotates another annotation
Constructor Annotates a constructor
Field Annotates a field, such as an instance variable of a class or an enum constant
Local variable Annotates a local variable
Method Annotates a method of a class
Module Annotates a module (new in Java 9)
Package Annotates a package
Parameter Annotates a parameter to a method or constructor
Type Annotates a type, such as a class, interfaces, annotation types, or enum declarations
Type Parameter Annotates a type parameter, such as those used as formal generic parameters
Type Use Annotates the use of a type, such as when an object of a type is created using the newkeyword, when an object is cast to a specified type, when a class implements an interface, or when the type of a throwable object is declared using the throws keyword (for more information, see the Type Annotations and Pluggable Type Systems Oracle tutorial)

For more information on these targets, see Section 9.7.4 of the JLS. It is important to note that one or more targets may be associated with an annotation. For example, if the field and constructor targets are associated with an annotation, then the annotation may be used on either fields or constructors. If on the other hand, an annotation only has an associated target of method, then applying the annotation to any construct other than a method results in an error during compilation.

Annotation Parameters

Annotations may also have associated parameters. These parameters may be a primitive (such as int or double), String, class, enum, annotation, or an array of any of the five preceding types (see Section 9.6.1 of the JLS). Associating parameters with an annotation allows for an annotation to provide contextual information or can parameterize a processor of an annotation. For example, in our JSON serializer implementation, we will allow for an optional annotation parameter that specifies the name of a field when it is serialized (or use the variable name of the field by default if no name is specified).

How Are Annotations Created?

For our JSON serializer, we will create a field annotation that allows a developer to mark a field to be included when serializing an object. For example, if we create a car class, we can annotate the fields of the car (such as make and model) with our annotation. When we serialize a car object, the resulting JSON will include make and model keys, where the values represent the value of the make and model fields, respectively. For the sake of simplicity, we will assume that this annotation will be used only for fields of type String, ensuring that the value of the field can be directly serialized as a string.

To create such a field annotation, we declare a new annotation using the @interface keyword:

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.FIELD)
public @interface JsonField { public String value() default "";
}

The core of our declaration is the public @interface JsonField, which declares an annotation type with a publicmodifier, allowing our annotation to be used in any package (assuming the package is properly imported if in another module). The body of the annotation declares a single String parameter, named value, that has a type of String and a default value of an empty string.

Note that the variable name value has a special meaning: It defines a Single-Element Annotation (Section 9.7.3. of the JLS) and allows users of our annotation to supply a single parameter to the annotation without specifying the name of the parameter. For example, a user can annotate a field using @JsonField("someFieldName") and is not required to declare the annotation as @JsonField(value = "someFieldName"), although the latter may still be used (but it is not required). The inclusion of a default value of empty string allows for the value to be omitted, resulting in value holding an empty string if no value is explicitly specified. For example, if a user declares the above annotation using the form @JsonField, then the value parameter is set to an empty string.

The retention policy and target of the annotation declaration are specified using the @Retention and @Target annotations, respectively. The retention policy is specified using the java.lang.annotation.RetentionPolicy enum and includes constants for each of the three standard retention policies. Likewise, the target is specified using the java.lang.annotation.ElementTypeenum, which includes constants for each of the eleven standard target types.

In summary, we created a public, single-element annotation named JsonField, which is retained by the JVM during runtime and may only be applied to fields. This annotation has a single parameter, value, of type String with a default value of an empty string. With our annotation created, we can now annotate fields to be serialized.

How Are Annotations Used?

Using an annotation requires only that the annotation is placed before an appropriate construct (any valid target for the annotation). For example, we can create a Carclass using the following class declaration:

public class Car { @JsonField("manufacturer") private final String make; @JsonField private final String model; private final String year; public Car(String make, String model, String year) { this.make = make; this.model = model; this.year = year; } public String getMake() { return make; } public String getModel() { return model; } public String getYear() { return year; } @Override public String toString() { return year + " " + make + " " + model; }
}

This class exercises the two major uses of the @JsonField annotation: (1) with an explicit value and (2) with a default value. We could have also annotated a field using the form @JsonField(value = "someName"), but this style is overly verbose and does not aid in the readability of our code. Therefore, unless the inclusion of an annotation parameter name in a single-element annotation adds to the readability of code, it should be omitted. For annotations with more than one parameter, the name of each parameter is required to differentiate between parameters (unless only one argument is provided, in which case, the argument is mapped to the value parameter if no name is explicitly provided).

Given the above uses of the @JsonField annotation, we would expect that a Car ject is serialized into a JSON string of the form {"manufacturer":"someMake", "model":"someModel"} (note, as we will see later, we will disregard the order of the keys–manufacturer and model–in this JSON string). Before we proceed, it is important to note that adding the @JsonField annotations does not change the runtime behavior of the Car class. If we compile this class, the inclusion of @JsonField annotations does not enhance the behavior of the Car class anymore than had we omitted the annotations. These annotations are simply recorded, along with the value of the value parameter, in the class file for the Car class. Altering the runtime behavior of our system requires that we process these annotations.

How are Annotations Processed?

Processing annotations is accomplished through the Java Reflection Application Programming Interface (API). Sidelining the technical nature of the reflection API for a moment, the reflection API allows us to write code that will inspect the class, methods, fields, etc. of an object. For example, if we create a method that accepts a Car object, we can inspect the class of this object (namely, Car) and discover that this class has three fields: (1) make, (2) model, and (3) year. Furthermore, we can inspect these fields to discover if each is annotated with a specific annotation.

Using this capability, we can iterate through each field of the class associated with the object passed to our method and discover which of these fields are annotated with the @JsonField annotation. If the field is annotated with the @JsonField annotation, we record the name of the field and its value. Once all the fields have been processed, then we can create the JSON string using these field names and values.

Determining the name of the field requires more complex logic than determining the value. If the @JsonField includes a provided value for the value parameter (such as "manufacturer" in the previous @JsonField("manufacturer") use), we will use this provided field name. If the value of the value parameter is an empty string, we know that no field name was explicitly provided (since this is the default value for the value parameter), or else, an empty string was explicitly provided. In either case, we will use the variable name of the field as the field name (for example, model in the private final String model declaration).

Combining this logic into a JsonSerializer class, we can create the following class declaration:

public class JsonSerializer { public String serialize(Object object) throws JsonSerializeException { try { Class<?> objectClass = requireNonNull(object).getClass(); Map<String, String> jsonElements = new HashMap<>(); for (Field field: objectClass.getDeclaredFields()) { field.setAccessible(true); if (field.isAnnotationPresent(JsonField.class)) { jsonElements.put(getSerializedKey(field), (String) field.get(object)); } } System.out.println(toJsonString(jsonElements)); return toJsonString(jsonElements); } catch (IllegalAccessException e) { throw new JsonSerializeException(e.getMessage()); } } private String toJsonString(Map<String, String> jsonMap) { String elementsString = jsonMap.entrySet() .stream() .map(entry -> "\"" + entry.getKey() + "\":\"" + entry.getValue() + "\"") .collect(joining(",")); return "{" + elementsString + "}"; } private static String getSerializedKey(Field field) { String annotationValue = field.getAnnotation(JsonField.class).value(); if (annotationValue.isEmpty()) { return field.getName(); } else { return annotationValue; } }
}

We also create an exception that will be used to denote if an error has occurred while processing the object supplied to our serialize method:

public class JsonSerializeException extends Exception { private static final long serialVersionUID = -8845242379503538623L; public JsonSerializeException(String message) { super(message); }
}

Although the JsonSerializer class appears complex, it consists of three main tasks: (1) finding all fields of the supplied class annotated with the @JsonField annotation, (2) recording the field name (or the explicitly provided field name) and value for all fields that include the @JsonField annotation, and (3) converting the recorded field name and value pairs into a JSON string.

The line requireNonNull(object).getClass() simply checks that the supplied object is not null (and throws a NullPointerException if it is) and obtains the Class object associated with the supplied object. We will use this Class object shortly to obtain the fields associated with the class. Next, we create a Map of Strings to Strings, which will be used store the field name and value pairs.

With our data structures established, we next iterate through each field declared in the class of the supplied object. For each field, we configure the field to suppress Java language access checking when accessing the field. This is a very important step since the fields we annotated are private. In the standard case, we would be unable to access these fields, and attempting to obtain the value of the private field would result in an IllegalAccessException being thrown. In order to access these private fields, we must instruct the reflection API to suppress the standard Java access checking for this field using the setAccessible method. The setAccessible(boolean) documentation defines the meaning of the supplied boolean flag as follows:

A value of true indicates that the reflected object should suppress Java language access checking when it is used. A value of false indicates that the reflected object should enforce Java language access checks.

Note that with the introduction of modules in Java 9, using the setAccessible method requires that the package containing the class whose private fields will be accessed should be declared open in its module definition. For more information, see this explanation by Michał Szewczyk and Accessing Private State of Java 9 Modules by Gunnar Morling.

After gaining access to the field, we check if the field is annotated with the @JsonField. If it is, we determine the name of the field (either through an explicit name provided in the @JsonField annotation or the default name, which equals the variable name of the field) and record the name and field value in our previously constructed map. Once all fields have been processed, we then convert the map of field names to field values (jsonElements) into a JSON string.

We accomplish by converting the map into a stream of entries (key-value pairs for each entry in the map), mapping each entry to a string of the form "<fieldName>":"<fieldValue>", where <fieldName> is the key for the entry and <fieldValue> is the value for the entry. Once all entries have been processed, we combine all of these entry strings with a comma. This results in a string of the form "<fieldName1>":"<fieldValue1>","<fieldName2>":"<fieldValue2>",.... Once this terminal string has been joined, we surround it with curly braces, creating a valid JSON string.

In order to test this serializer, we can execute the following code:

Car car = new Car("Ford", "F150", "2018");
JsonSerializer serializer = new JsonSerializer();
serializer.serialize(car);

This results in the following output:

{"model":"F150","manufacturer":"Ford"}

As expected, the maker and model fields of the Car object have been serialized, using the name of the field (or the explicitly supplied name in the case of the maker field) as the key and the value of the field as the value. Note that the order of JSON elements may be reversed from the output seen above. This occurs because there is no definite ordering for the array of declared fields for a class, as stated in the getDeclaredFields documentation:

The elements in the returned array are not sorted and are not in any particular order.

Due to this limitation, the order of the elements in the JSON string may vary. To make the order of the elements deterministic, we would have to impose ordering ourselves (such as by sorting the map of field names to field values). Since a JSON object is defined as an unordered set of name-value pairs, as per the JSON standard, imposing ordering is unneeded. Note, however, a test case for the serialize method should pass for either {"model":"F150","manufacturer":"Ford"} or {"manufacturer":"Ford","model":"F150"}.

Conclusion

Java annotations are a very powerful feature in the Java language, but most often, we are the users of standard annotations (such as @Override) or common framework annotations (such as @Autowired), rather than their developers. While annotations should not be used in place of interfaces or other language constructs that properly accomplish a task in an object-oriented manner, they can greatly simplify repetitive logic. For example, rather than creating a toJsonStringmethod within an interface and having all classes that can be serialized implement this interface, we can annotate each serializable field. This takes the repetitive logic of the serialization process (mapping field names to fields values) and places it into a single serializer class. It also decouples the serialization logic from the domain logic, removing the clutter of manual serialization from the conciseness of the domain logic.

While custom annotations are not frequently used in most Java applications, knowledge of this feature is a requirement for any intermediate or advanced user of the Java language. Not only will knowledge of this feature enhance the toolbox of a developer, just as importantly, it will aid in the understanding of the common annotations in the most popular Java frameworks.

Original Link

Problems With Kafka Streams: The Saga Continues

This blog continues where the previous left, so be sure to check it before proceeding with the current.

After many long years, the first stable Kafka version has been released: 1.0.0. In the previous version of Kafka Streams (0.11.0.0), we saw some suboptimal behavior when using the Processor API. With the new version came updates that are supposed to alleviate some pressure from the developer when using Kafka Streams.

Kafka Streams Overview

Before diving directly into the problem, let’s see how Kafka Streams are implemented. Warning: There are traces of some actual code!

Firstly, we need a main class that will contain topology, so let’s implement a dummy one:

object DummyMainClass { def main(args: Array[String]): Unit = { val properties = CreateKafkaStreamProperties() val topology: TopologyBuilder = new TopologyBuilder topology.addSource("Input", properties.inputTopic) topology.addProcessor("Input_processor", new DummyProcessor(properties.punctuateTime), "Input") topology.addSing("Output", properties.outputTopic, "Input_processor") val streams: KafkaStreams = new KafkaStreams(topology, properties) streams.start() } def CreateKafkaStreamProperties(): Properties = { //some code }
}

In several lines of code, we have created a Topology for the Kafka Streams application that reads messages from input topic, processes them with DummyProcessor, and outputs them to output topic. For more details, consult the documentation. Next in line is DummyProcessor:

class DummyProcessor(punctuateTime: Long) extends ProcessorSupplier[String, String] with Logging { override def get(): Processor[String, String] = new Processor[String, String] { var context: ProcessorContext = _ var arrived: Int = 0 override def init(context: ProcessorContext): Unit = { this.context = context this.context.schedule(punctuateTime) } override def process(key: String, value: String): Unit = { this.arrived += 1 this.logger.debug(s"Arrived: $value") } override def punctuate(timestamp: Long): Unit = { this.logger.debug("Punctuate call") this.context.forward(null, s"Processed: ${this.arrived}") this.arrived = 0 this.context.commit() } override def close(): Unit = { // does nothing } }
}

This DummyProcessor is only used for demonstration purposes. Its job is to output to the topic the number of arrived messages per punctuate period; for debugging, we have added console logging. Now, we have the entire pipeline needed for testing how Kafka Streams operate.

The input topic has three partitions and a replication factor of 1. The output topic has only one partition and a replication factor of 1.

Testing Kafka Streams

We are going to repeat the testing mentioned in the previous article. Input messages for all tests are going to be same: message_<offset>. The punctuate period is set to occur every ten seconds. There are two scenarios that we need to check for both Kafka versions:

  • For the first test, we have a constant rate of messages on input topic — one message per second.

  • For the second test, we have nine messages (one message per second), followed by a pause that lasts 21 seconds, followed by one more message.

Older Kafka Streams Version

In the first test scenario, we see ten messages on the console in the format “Arrived: message_<offset>” accompanied by one “Punctuate call”, followed by one message in the output topic. Messages in the output topic are “Processed: 10”. This behavior repeats as long as we have messages on the input stream.

In the second test scenario, we see nine messages on the console in the format “Arrived: message_<offset>”. After the 21-second long pause has passed on the console, three messages appear: “Punctuate call”. On output topic, we can see three messages: “Processed: 10”, “Processed: 0”, and “Processed 0”. The late message ended up in same time window as the first nine messages and we have two unwanted punctuate calls.

New Kafka Streams Version

After switching to a new version, we have some modification in the code:

  • TopologyBuilder in main class has to be changed to Topology class.

  • Schedule method in DummyProcessor class is now deprecated.

One would expect that by changing the version, the previous behavior would remain the same. Well, it hasn’t. What has changed?

After each process method, a punctuate method is called. After punctuateInterval is scheduled, punctuate also occurs. This means the following:  

  • In the first test scenario, each “Arrived: message_<offset>” message in the console is accompanied with “Punctuate call”. Unsurprisingly, we have one: “Processed: 1” message in output topic. After ten messages, we have another: “Punctuate call” and “Processed: 0” pair. 

  • In the second scenario, we have nine: “Arrived: message_<offset>” and “Punctuate call” pairs on the console, followed with 9: “Processed: 1” in the output topic. After the pause and tenth message we have: “Arrived: message_<offset>” and 3 “Punctuate call”. In the output topic, we see “Processed: 1”, “Processed: 0”, and “Processed 0”.

Now, if we want the same behavior as with the previous version, we need to manually check if the punctuation period has passed and then output the message to output topic and reset the counter. From one problem (multiple punctuate calls for the late messages), we now have two problems (multiple punctuate calls for the late messages and punctuate call after each processed message).

Putting that aside, we have noticed that the schedule method is deprecated. Along with the new version came a new API for scheduling. The ald format has been preserved for backward compatibility — but it works differently.

The new schedule method has the following format:

context.schedule(punctuateInterval, PuntuationType, Punctuator)

The first parameter remains the same — the interval on which scheduled punctuation should occur. The second parameter is something new: PunctuationType. It’s enum and it has two values STREAM_TIME and WALL_CLOCK_TIME (more on them later). The third and last parameter is Punctuator — interface with only one method punctuate (Long). With these changes, our DummyProcessor has new look:

class DummyProcessor(punctuateTime: Long) extends ProcessorSupplier[String, String] with Logging { override def get(): Processor[String, String] = new Processor[String, String] { var context: ProcessorContext = _ var arrived: Int = 0 override def init(context: ProcessorContext): Unit = { this.context = context this.context.schedule(punctuateTime, PunctuationType.STREAM_TIME, new Punctuator { override def punctuate(timestamp: Long): Unit = { this.logger.debug("Inner punctuate call") this.context.forward(null, s"Inner processed: ${this.arrived}") this.arrived = 0 this.context.commit() } }) } override def process(key: String, value: String): Unit = { this.arrived += 1 this.logger.debug(s"Arrived: $value") } override def punctuate(timestamp: Long): Unit = { this.logger.debug("Punctuate call") this.context.forward(null, s"Processed: ${this.arrived}") this.arrived = 0 this.context.commit() } override def close(): Unit = { // does nothing } }
}

This is not a typo — we have two punctuate methods. The only difference is that the inner punctuation method has the “Inner” prefix in its messages. Running both test scenarios yields almost identical results, with the only difference being outputted messages. On the console, “Inner punctuate call” will replace “Punctuate call” messages and in the output topic, “Inner processed: *” will replace “Processed: *” messages. We have identical behavior to when using the schedule (punctuateInterval) method, but the outer punctuate method is never called even though we are obligated to implement it.

Using WALL_CLOCK_TIME and running test scenarios, we get following results:

  • In the first scenario, after starting the application, three punctuate methods occur (one for each partition in input topic). After each processing message, one inner punctuate method is called. After the tenth message, we see three punctuate calls.

  • In the second scenario, the init method is followed by three punctuate methods. The first nine messages are followed by inner punctuate call. Then, after one second, we see three scheduled inner punctuate calls. As no messages arrive for another ten seconds, we see again three scheduled inner punctuate calls. Ten more seconds pass and we see three scheduled inner punctuate calls followed by processing of the tenth message and one punctuate call (for the corresponding partition to which message entered).

WALL_CLOCK_TIME solves the problem of multiple consecutive punctuate calls but introduces the problem of the punctuate call after each processed message. We are just shifting the problem from one place to another.

A few notes:

  • After starting Kafka Streams with WALL_CLOCK_TIME, before consuming messages, the inner punctuate for each partition is called.

  • The duration of the process method should be shorter than the punctuate interval. If the process method is longer than the punctuate interval, multiple consecutive calls will occur.

  • Inside the process method, we can call context.partition() to get partition id for a partition to which message arrived. Even though Kafka Streams creates one thread (worker) per partition, calling context.partition() inside punctuate method (both inner and outer) will return -1. This means that in punctuate, we can’t know which partition the code refers to.

Conclusion

Upgrading to the first stable Kafka version requires testing and checking existing applications. Some classes have been replaced or renamed so that users will pay attention to those changes to ensure that code compiles. The problem lies in changes that don’t directly require interventions. There have been some changes under the hood, and now, Kafka Streams applications that compile with the newer version will produce different results. 

The problems mentioned in the previous article haven’t been solved completely. Some shortcomings have been fixed while some new problems have been introduced — the cycle continues.

Original Link

Setting Up JRebel for WebSphere AS in a Docker Environment

Getting any Java application server up and running in the development environment is usually a fairly simple task. You can just download the ZIP archive and start the container either from the command line or via IDE integration. Configuring a JRebel agent for the server is also quite straightforward. However, there are some exceptions to that. For instance, if you’d like to try JRebel on WebSphere AS and you are using MacOS, then you will have to take another route.

WebSphere Application Server is available for Linux and Windows platforms, but not for MacOS. The good news is that there is a WebSphere Docker image that you can use for development.

This developerWorks’ article demonstrates pretty clearly what needs to be done in order to get WebSphere running in Docker and deploy a web application. With the help of the article, I have assembled a demo project that deploys a Petclinic application on WebSphere running with JRebel in Docker container.

Let me explain some interesting bits of the outcome.

First of all, we need to derive from the base image, package the application archive into the new image, and make sure that WebSphere will deploy the application when it starts:

#Dockerfile
FROM ibmcom/websphere-traditional:profile
COPY target/petclinic.war /tmp/petclinic.war
RUN wsadmin.sh -lang jython -conntype NONE -c "AdminApp.install('/tmp/petclinic.war', \ '[ -appname petclinic -contextroot /petclinic -MapWebModToVH \
[[ petclinic petclinic.war,WEB-INF/web.xml default_host]]]')"

As you’ve noticed, it is not enough just to copy the application archive to some folder. You also need to invoke a script to actually deploy the application: call wsadmin.sh by providing it a snippet of Jython code.

Next, as we want to enable JRebel, we also need to package the agent binary into the image and we need to modify JVM arguments of the application server. Hence, the corresponding Dockerfile will get a little more complicated:

#Dockerfile
FROM ibmcom/websphere-traditional:profile
COPY ["jrebel-7.1.2","/tmp/jrebel"]
RUN wsadmin.sh -lang jython -conntype NONE -c "AdminConfig.modify(AdminConfig.list('JavaVirtualMachine', \
AdminConfig.list('Server')), [['genericJvmArguments', \ '-Xshareclasses:none -agentpath:/tmp/jrebel/lib/libjrebel64.so']])"
COPY target/petclinic.war /tmp/petclinic.war
RUN wsadmin.sh -lang jython -conntype NONE -c "AdminApp.install('/tmp/petclinic.war', \ '[ -appname petclinic -contextroot /petclinic -MapWebModToVH \
[[ petclinic petclinic.war,WEB-INF/web.xml default_host]]]')"

The Dockerfile above packages JRebel distribution into the image. That’s an easy part. The hard part was to figure out how to configure JVM arguments. In WebSphere, JVM arguments are set via server.xml configuration which is quite unusual. Normally, a developer would use an administrative user interface to modify the parameters, but in our case, we need the arguments to be in the right place right at the start. Hence, we need to do some Jython scripting via wsadmin again.

Now that the Dockerfile is ready, we can build and run the new image. In the terminal:

$ docker build -t waspet .
$ docker run -d -p 9043:9043 -p 9443:9443 -v `pwd`:/tmp/petclinic -v ~/.jrebel:/home/was/.jrebel waspet

The docker run command above also maps a few directories: a project folder and JRebel’s home folder. We map the project folder because JRebel agent could then see if any resource is updated. JRebel’s home folder (~/.jrebel) includes cached resources, so that if we would have to restart the Docker image then the application will start faster the next time.

Now, it is possible to use JRebel to update the application instantly without restarting the application server or redeploying the application. For the full list of instructions, see the README file in GitHub repository.

Original Link

How to Monitor Your Java Application

Java was presented as a no-cost solution with the benefit of being a “Write Once, Run Anywhere” language when it was released publicly in 1995 by Sun Microsystems. Many years and almost as many versions later, Java is now one of the most prominent computer languages in the industry. Java-based applications power banking, commerce, and a host of other sectors.

As with any computer system, Java applications and services are invaluable when running as they should, and as engineers, it is our job to ensure that they are robust and keep running as efficiently and effectively as possible. In this article, we’re going to look at one key element of keeping your Java applications operating properly. Monitoring is the process whereby a computer system is configured to report on key metrics, which are analyzed to display the near real-time health of the system, and integrated with logic constraints to alert operations personnel when the system begins trending towards less-than-optimal performance.

Developing a Solid Monitoring Plan

Whether you are implementing a new system or maintaining an existing Java-based ecosystem, having a comprehensive and well thought-out monitoring plan is vital to the success of your project. There are two types of metrics that we need to be concerned with-  those related to the Java Virtual Machine (JVM), and those related to the underlying infrastructure.

Specific Considerations for Java

Let’s look at specific metrics that should be monitored on a Java-based system, as well as the reasons why they are important.

Java Heap Usage

The Heap is the allocated portion of the computer’s memory for the creation and storage of objects. The JVM will be unable to create additional objects if this space has already been consumed. At the very least, the JVM may pause and experience a degradation in performance, or the JVM may abruptly terminate.

Potential problems in the application (which can result in problems for Heap usage) are the inadequate allocation of Heap memory when the application is started, or a memory leak. In the case of the latter, a constant increase in Heap usage over time is often observed.

Garbage Collection

Closely related to the Heap is the Garbage Collector (GC). This process is responsible for identifying objects that are no longer in use, and can be removed to free up space for the application. This necessary process is very resource-intensive, so if the JVM is initiating it too often, or if it is being run for extended periods, it can be indicative of a memory leak, or simply necessary additional tuning.

GC metrics are different for each application, and monitoring them over time will help to identify trends and indicate when improvements may be needed.

Active Threads

Threads are how applications process their work. Highly performant applications can complete their work quickly and free up their threads. A high count of active threads may indicate that the system is processing a higher-than-expected load, and may lead to decreased performance over time.

Response Time

Computer applications are written to perform tasks quickly and efficiently. Response time is a direct measurement of how quickly an application is responding to requests. Analyzing this metric over time allows you to create a performance baseline, and deviations from this baseline may indicate that something is wrong- either with the load being processed by the server, or there could be an internal issue.

Key Infrastructure Metrics

In addition to monitoring the health of the JVM, it is also important to monitor the supporting infrastructure and any supporting systems. For a Java application, this would include the application server as well. Some examples would be Jetty, Netty, or Tomcat.

Infrastructure Specific Metrics:

  • CPU Utilization
  • Memory Usage/RAM
  • Available Threads
  • Open File Descriptors
  • Database Connections
  • SLA
  • Latency
  • Success of API Endpoints
  • Average Response Times
  • Errors and Exceptions
  • Health and Status of Dependencies

Original Link

Elastic Stack 6: What You Need to Know

Elastic Stack 6 was released last November, and now’s a good time as any to evaluate whether to upgrade. To help you guys make that call, we are going to take a look at some of the major changes included in the different components in the stack and review the main breaking changes.

The main changes to Elasticsearch are designed to boost performance, improve reliability, and make recovery easier. The most significant changes in Logstash are focused on allowing users to run multiple self-contained pipelines on the same JVM. There are no major changes to Kibana, but a relatively large amount of minor usability improvements were added. With the exception of the addition of Auditbeat, the main changes made to Beats are focused on improving the performance of and enhancing existing log shippers.

Let’s take a closer look.

Elasticsearch 6

Changes to Elasticsearch are mostly internal and shouldn’t require most organizations to alter how they configure or administer the Elasticsearch, with the big exception being the change to mapping types.

Sparse Doc Values

A sparse values situation (when documents do not have values for each of the fields in our indices) results in the use of a large amount of disk space and file-system cache. The change to Lucene 7 allows Elasticsearch to now support sparse doc values, a new encoding format that reduces disk space and improves query throughput.

Index Sorting

Lucene 7 also brings the ability to specify a sort order to indices. This boosts performance by allowing for the sorting of indices during re-indexing (while documents are written) instead of when documents are read. The indices are written to disk in the order of the specified sort.

Specifying a sort of indices means that a search can terminate when it has found the documents requested by the query. For example, if the sort is alphabetical, then a search for entries beginning with “E” can quit when it reaches “F” because all items beginning with E have been read.

No More Mapping Types

This is one of the most talked-about changes in Elasticsearch 6, and a somewhat controversial one at that.

In this version, indices can only have one mapping type. Indices created in 5.x with multiple mapping types will continue to function as before. The plan is to remove mapping types completely in version 7.

The main reason for this move is to simplify the understanding and usage of the underlying data structure in Elasticsearch. Comparisons to RDBMS databases have led to a faulty understanding that types can be compared to tables. This has led in turn, to an expectation for fields to be independent across types whereas they must be of the same field type.

While this change fundamentally changes the way we index data, and as such has received a decent amount of criticism in the community, it also promises to speed searches by forcing users to use indices in a fashion tailored to the underlying database structure. The common existing practice of treating indices like tables and types like tables is suboptimal. Enforcing a single type per index should provide for significant performance increases.

Better Shard Recovery

A new feature called Sequence IDs promises to guarantee more successful and efficient shard recovery.

Every index, update, and delete operation receives an ID that is logged in the primary shard’s transaction log. A replica can now refer to the operations recorded in this log and use them to update itself without needing to copy all the files, thus making recovery much faster. You’ll be able to configure how long to keep these transaction logs.

Replicas can run unacknowledged and different operations — meaning that in case of a primary shard failing, the replicas will be able to sync with the new primary shard without waiting for the next recovery.

Upgrades

Updating to the new Elasticsearch version is made easier with a series of upgrade improvements that aim at tackling some of the traditional hurdles facing upgrade procedures.

A new rolling restart feature negates the need for a full cluster restart and thus minimizes downtime. Elasticsearch 5.x indices will be able to be searched using cross-cluster searching: a new approach to cross-cluster operations that replaces the traditional tribe-node based approach. Deprecation logs have been reinforced with important info on breaking changes. And if you’re an X-Pack user, you will be able to manage the upgrade via the UI.

Logstash 6

Logstash was originally intended to handle one type of event per instance, and prior to this version, each Logstash instance supported only a single event pipeline. Users can circumvent this restriction using conditional statements in the configuration, which often leads to a new set of problems.

Logstash now supports native support for multiple pipelines. These pipelines are defined in a pipelines.yml file, which is loaded by default. If Logstash is started with the -r flag, it will periodically reread the pipelines.yml file looking for changes, making dynamic multiple pipelines possible for a single Logstash instance.

Each pipeline’s independent configuration means that each pipeline can be assigned different levels of resources. A single Logstash instance could, for example, have multiple pipelines with a single worker thread and a high-volume pipeline with ten worker threads. Combining this with Logstash’s ability to periodically reload configurations means dynamic reconfiguration without having to tear the entire application down to meet changing needs.

X-Pack users will be able to manage multiple pipelines within Kibana. This solution uses Elasticsearch to store pipeline configurations and allows for on-the-fly reconfiguration of Logstash pipelines.

Another noteworthy addition to Logstash is a new conversion tool to help those using Elasticsearch Ingest Nodes to begin using Logstash. The tool allows users to enter their Ingest Node configuration and outputs a Logstash pipeline configuration.

Last but not least, a new pipeline viewer now allows users to monitor and analyze Logstash pipelines in Kibana. Pipelines are displayed on a graph where each component is accompanied by relevant metrics. I explored the pipeline viewer in this article.

Kibana 6

As mentioned in the introduction to this article, there are no major changes to the stack’s UI, but there is a nice list of usability improvements that users will most likely enjoy.

The main changes to the UI include a new CSV export option, new colors for better contrast, and improved screen reading and keyboard navigation.

Dashboarding in Kibana has two new features. A new full-screen mode was added when viewing dashboards to allow users to enjoy all the Kibana goodness. In addition, the new dashboard only mode enables administrators to share dashboards safely. Using a new kibana_dashboard_only_user role, specific users can be given limited access to Kibana with read-only data visibility and with no ability to modify or delete any of the dashboards.

Also of note is a new querying language called Kuery that seeks to address the issues of previous search mechanisms and that is meant to ultimately simplify querying and filtering in Kibana. Disabled by default, the new language supports all the query types that are supported by the Elasticsearch DSL, including those not supported by Lucene query syntax (such as aggregations and visualizations) — directly in the query language.

Beats 6

Auditbeat is the only new shipper released in this version and is basically a native and easy-to-use auditing tool for the Elastic Stack. I tried the beta back in September and am pretty sure this will become one of the more popular beats.

Most of the other changes center around improving support and performance of existing beats, mostly Filebeat and Metricbeat.

Filebeat and Metricbeat have a tighter integration with Docker and Kubernetes now with new processors that add metadata to the logs collected from these environments, such as details on the container name, image, labels, pod name, and so forth. A new Kubernetes module was also added to Metricbeat that gives insights into your Kubernetes deployment.

Additional changes to Filebeat and Metricbeat include deployment manifests for easier setup in Kubernetes and a new modules.d directory that contains configuration files for the different supported modules.

A new set of commands for Beats allows users to list, enable, and disable modules, export configurations and the Elasticsearch mapping template, and test Logstash or Elasticsearch connectivity.

Last but not least, internal changes were introduced in the Beats pipeline architecture to improve performance, including a transition to an asynchronous pipeline that allows other processes to continue even as one hangs in a wait state. This also made the Filebeat internal spooler obsolete. For Metricbeat specifically, polling frequency was reduced and sharding behavior changed; that, coupled with the changes made to Elasticsearch, results in less storage used.

What Will Break?

We’re going to end this piece with a summary of the main gotchas that need to be considered before upgrading. Keep in mind this is a partial list of breaking changes only. As always, we recommend you read the official documentation carefully and test the upgrade process.

Elasticsearch

  • As mentioned above, the main breaking change in this version is the gradual removal of mapping types from indices, so Elasticsearch 6.0 will be able to read indices created in version 5.0 or above only.
  • Elasticsearch 6.0 requires a reindexing before full functionality can be achieved; this is because of a number of changes to the Elasticsearch indices. These are accompanied by changes to the CAT, Document, and Java APIs and the Search and Query DSL, as well as REST changes. The result is a number of key changes that affect Elasticsearch deeply, altering everything from the structure of queries and scripts to how internal components communicate.

A full list of the breaking changes is available here.

Logstash

Changes in Logstash 6 involve several breaking changes, including a number of changes to the input and output options. There are also numerous plugin changes and a change to the config.reload.interval configuration option, which now uses time value strings (5m, 10s, etc.) instead of millisecond time values.

A full list of the breaking changes is available here.

Kibana

To migrate existing Kibana installations to Kibana 6.0, the Kibana index needs to be re-indexed. A full list of the breaking changes is available here.

Beats

  • The main breaking change in Beats 6 is the removal of the Filebeat internal spooler, as described above. This results in the removal of some of the configuration options and the addition of others.
  • Following the changes made to the pipeline architecture, there is no longer an option to enable two outputs at the same time (you can use multiple outputs in Logstash instead or run multiple instances of your beat shipper).
  • If you’re outputting to Logstash from your beat, you now need to include a version in the Logstash index setting.

A full list of the breaking changes is available here.

Summing It Up

As with any major release of the stack, there are quite a large number of changes in Elastic Stack 6.0. Many of these changes stem from the upgrade to the Lucene 7 database engine, but just as many are part of a general push towards increased efficiency and performance.

Elastic Stack 6.0 also brings a number of important security changes and enhancements and sets the stage for many more. This follows the needs of users deploying Elastic Stack in production environments, where complex security requirements are increasingly the standard.

In many ways, version 6.0 is a transitionary release. It is a midpoint between traditional deployments of the stack that clung to concepts of data organization and production deployments belonging to relational database designs and more modern approaches.

That a major version of the stack comes with a need to re-index, changes to the index structure, and a number of configuration changes to various plugins should come as no surprise. Migration from previous versions will need to be planned carefully and, above all, tested.

Original Link

Microservices on the JVM With Actors

This article is featured in the new DZone Guide to Microservices. Get your free copy for more insightful articles, industry statistics, and more! 

As mobile and data-driven applications increasingly dominate, users are demanding real-time access to everything everywhere. System resilience and responsiveness are no longer “nice to have;” they’re essential business requirements. Businesses increasingly need to trade up from static, centralized architectures in favor of flexible, distributed, and elastic systems.

But where to start and which architecture approach to use is still a little blurry, and the microservices hype is only slowly settling while the software industry explores various architectures and implementation styles.

For a decade or more, enterprise development teams have built their Java EE projects inside large, monolithic application server containers without much regard to the individual lifecycle of their module or component. Hooking into startup and shutdown events was simple, as accessing other components was just an injected instance away. It was comparably easy to map objects into single relational databases or connect to other systems via messaging. One of the greatest advantages of this architecture was transactionality, which was synchronous, easy to implement, and
simple to visualize and monitor.

By keeping strong modularity and component separation a first-class priority, it was manageable to implement the largest systems that still power our world. Working with compartmentalization and introducing modules belongs to the core skills of architects. Our industry has learned how to couple services and build them around organizational capabilities.

The new part in microservices-based architectures is the way truly independent services are distributed and connected back together. Building an individual service is easy. Building a system out of many is the real challenge, because it introduces us to the problem space of distributed systems. This is the major difference from classical, centralized infrastructures.

There Isn’t Just One Way of Doing Microservices

There are many ways to implement a microservices-based architecture on or around the Java Virtual Machine (JVM). The pyramid in Figure 1 was introduced in my first book. It categorizes some technologies into layers, which can help identify the level of isolation that is needed for a microservices-based system.

Starting at the virtualization infrastructure with virtual machines and containers, as they are means of isolating applications from hardware, we go all the way up the stack to something that I summarize under the name “application services.” This category contains specific microservices frameworks aimed at providing microservices support across the complete software development lifecycle.

Image title

Figure 1: Pyramid of modern enterprise Java development (Source: Modern
Java EE Design Patterns
, Eisele)

The three frameworks in the application services and infrastructure categories are all based on the principles of the Reactive Manifesto. It defines traits that lead to large systems that are composed of smaller ones, which are more flexible, loosely-coupled, and scalable. As they are essentially message-driven and distributed, these frameworks fit the requirements of today’s microservices architectures.

While Lagom offers an opinionated approach on close guardrails that only support microservices architectures, Play and Akka allow you to take advantage of the reactive traits to build a microservices-style system but doesn’t limit you to this approach.

Microservices With Akka

Akka is a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications on the JVM. Akka “actors” are one of the tools in the Akka toolkit that allow you to write concurrent code without having to think about low-level threads and locks. Other tools include Akka Streams and Akka HTTP. Although Akka is written in Scala, there is a Java API, too.

Actors were invented decades ago by Carl Hewitt. But, relatively recently, their applicability to the challenges of modern computing systems has been recognized and proven to be effective. The actor model provides an abstraction that allows you to think about your code in terms of communication, not unlike people in a large organization.

Systems based on the actor model using Akka can be designed with incredible resilience. Using supervisor hierarchies means that the parental chain of components is responsible for detecting and correcting failures, leaving clients to be concerned only about what service they require.

Unlike code written in Java that throws exceptions, clients of actor-based services never concern themselves with dealing with failures from the actor from which they are requesting a service. Instead, clients only must understand the request-response contract that they have with a given service, and possibly retry requests if no response is given in some time frame. When people talk about microservices, they focus on the “micro” part, saying that a service should be small.

I want to emphasize that the important thing to consider when splitting a system into services is to find the right boundaries between services, aligning them with bounded contexts, business capabilities, and isolation requirements. As a result, a microservices-based system can achieve its scalability and resilience requirements, making it easy to deploy and manage. The best way to understand something is to look at an example. The Akka documentation contains an extensive walkthrough of a simplistic IoT management application that allows users to query sensor data. It does not expose any external API to keep things simpler, only focuses on the design of the application, and uses an actor-based API for devices to report their data back to the management part. You can find a high-level architecture diagram in Figure 2.

Image title

Figure 2: IoT sample application architecture (Source: Akka documentation)

Actors are organized into a strict tree, where the lifecycle of every child is tied to the parent, and where parents are responsible for deciding the fate of failed children. All you need to do is to rewrite your architecture diagram so that it contains nested boxes into a tree, as shown in Figure 3.

In simple terms, every component manages the lifecycle of the subcomponents. No subcomponent can outlive the parent component. This is exactly how the actor hierarchy works. Furthermore, it is desirable that a component handles the failure of its subcomponents. A “contained-in” relationship of components is mapped to the “children-of” relationship of actors.

If you look at microservice architectures, you would have expected that the top-level components are also the top-level actors. That is indeed possible, but not recommended. As we don’t have to wire the individual services back together via external protocols and the Akka framework also manages the actor lifecycle, we can create a single top-level actor in the actor system and model the main services as children of this actor. The actor architecture is built on the same traits that a microservice architecture should rely on, which are isolation, autonomy, single responsibility, exclusive state, asynchronous communication, explicit communication protocols, and distribution and location transparency.

Image title

Figure 3: An Actor representation of the IoT architecture.

You find the details about how to implement the IoTSupervisor and DeviceManager classes in the official Akka tutorial. Until now, I only looked at the complete system at large. But there is also the individual actor that represents a device. His simple task will be to collect temperature measurements and report the last measured data back on request. When working with objects, you usually design APIs as interfaces, which are basically collections of abstract methods to be filled out by the actual implementation. In the world of actors, the counterparts of interfaces are protocols. The protocol in an actor-based application is the message for the devices.

function counter(state: AppState = 0, action: AppAction): public
static final class ReadTemperature { long requestId; public ReadTemperature(long requestId) { this.requestId = requestId; }
}
public static final class RespondTemperature { long requestId; Optional < Double > value; public RespondTemperature(long requestId, Optional < Double > value) { this.requestId = requestId; this.value = value; }
}

Code 1: message protocol for the device actor

I am skipping a lot of background on message ordering and delivery guarantees. Designing a system with the assumption that messages can be lost in the network is the safest way to build a microservices-based architecture. This can be done, for example, by implementing a “re-send” functionality if a message gets lost. And this is the reason why the message also contains a requestId. It will now be the responsibility of the querying actor to match requests to actors. A first rough sketch of the Device Actor is below.

class Device extends AbstractActor { //… Optional < Double > lastTemperatureReading = Optional.empty(); @Override public void preStart() { log.info(“Device actor {} - {} started”, groupId, deviceId); } @Override public void postStop() { log.info(“Device actor {} - {} stopped”, groupId, deviceId); } @Override // react to received messages of ReadTemperature public Receive createReceive() { return receiveBuilder() .match(ReadTemperature.class, r - > { getSender().tell(new RespondTemperature(r.requestId, lastTemperatureReading), getSelf()); }) .build(); }
}

Code 2: The device actor

The current temperature is initially set to Optional.empty(), and simply reported back when queried. A simple test for the device is shown below.

@Test
public void testReplyWithEmptyReadingIfNoTemperatureIsKnown() { TestKit probe = new TestKit(system); ActorRef deviceActor = system.actorOf(Device.props(“group”, “device”)); deviceActor.tell(new Device.ReadTemperature(42 L), probe.getRef()); Device.RespondTemperature response = probe. expectMsgClass(Device.RespondTemperature.class); assertEquals(42 L, response.requestId); assertEquals(Optional.empty(), response.value);
}

 Code 3: Testing the device actor

The complete example of the IoT System is contained in the Akka documentation.

Where to Get Started

Most of today’s enterprise software was built years ago and still undergoes regular maintenance to adopt the latest regulations or new business requirements. Unless there is a completely new business case or significant internal restructuring, the need to reconstruct a piece of software from scratch is rarely given.

If this is the case, it is commonly referred to as “greenfield” development, and you are free to select the base framework of your choice. In a “brownfield” scenario, you only want to apply the new architecture to a certain area of an existing application. Both approaches offer risks and challenges and there are advocates for both. The common ground for both scenarios is your knowledge of the business domain.

Especially in long-running and existing enterprise projects, this might be the critical path. They tend to be sparse on documentation, and it is even more important to have access to developers who are working in this domain and have firsthand knowledge.

The first step is an initial assessment to identify which parts of an existing application can take advantage of a microservices architecture. There are various ways to do this initial assessment. I suggest thinking about service characteristics. You want to identify either core or process services first.

While core services are components modeled after nouns or entities, the process services already contain complex business or flow logic.

Selective Improvements

The most risk-free migration approach is to only add selective improvements. By scraping out the identified parts into one or more services and adding the necessary glue to the original application, you’re able to scale out specific areas of your application in multiple steps.

The Strangler Pattern

First coined by Martin Fowler as the Strangler Application, the extraction candidates are move into a separate system which adheres to a microservices architecture, and the existing parts of the applications remain untouched. A load balancer or proxy decides which requests need to reach the original application and which go to the new parts. There are some synchronization issues between the two stacks. Most importantly, the existing application can’t be allowed to change the microservices’ databases.

Big Bang: Refactor an Existing System

In very rare cases, complete refactoring of the original application might be the right way to go. It’s rare because enterprise applications will need ongoing maintenance during the complete refactoring.

What’s more, there won’t be enough time to make a complete stop for a couple of weeks — or even months, depending on the size of the application — to rebuild it on a new stack. This is the least recommended approach because it carries a comparably high risk of failure.

When Not to Use Microservices

Microservices are the right choice if you have a system that is too complex to be handled as a monolith. And this is exactly what makes this architectural style a valid choice for enterprise applications.

As Martin Fowler states in his article about “Microservice Premium,” the main point is to not even consider using a microservices architecture unless you have a system that’s too large and complex to be built as a simple monolith. But it is also true that today, multicore processors, cloud computing, and mobile devices are the norm, which means that all-new systems are distributed systems right from the start.

And this also results in a completely different and more challenging world to operate in. The logical step now is to switch thinking from collaboration between objects in one system to a collaboration of individually scaling systems of microservices.

Summary

The actor model provides a higher level of abstraction for writing
concurrent and distributed systems, which shields the developer
from explicit locking and thread management. It provides the core
functionality of reactive systems, defined in the Reactive Manifesto
as responsive, resilient, elastic, and message-driven. Akka is an
actor-based framework that is easy to implement with full Java 8
Lambda support. Actors enable developers to design and implement
systems in ways that help focus more on the core functionality
and less on the plumbing. Actor-based systems are the perfect
foundation for quickly evolving microservices architectures.

This article is featured in the new DZone Guide to Microservices. Get your free copy for more insightful articles, industry statistics, and more! 

Original Link

Problems With Kafka Streams

Before diving straight into the main topic, let me introduce you to Kafka Streams first.

We have all heard about Apache Kafka, as it has been used extensively in the big data and stream processing world. For those who are new in the Kafka world, this blog written by Nikola Ivancevic is a good intro. Kafka Streams is a library that comes with Apache Kafka. It enables easy and powerful stream processing of Kafka events. This way, existing applications can use a Kafka Streams API by simply importing the library. This means that all applications that use Kafka Streams can be run in virtually any environment.

The Kafka Streams API is implemented in Java. For now, only applications written in JVM can utilize this library.

How It Works

Kafka Streams library can be used in two distinct ways:

  1. High-level stream DSL

  2. Processor API

1. High-Level Stream DSL

Using the Stream DSL, a user can express transformation, aggregations, grouping, etc. by using several simple provided methods. In several lines of code and within a couple of minutes, it’s possible to create and deploy a working application, at least in theory. When transforming data from one format to another, internal Kafka Topics are used for storing intermediate results. This means that for each transformation in the chain, Kafka Stream will create one internal topic. With this approach, users enjoy all the benefits and security of Kafka without having to invest time in development. If anything unexpected occurs during runtime, the application is able to continue from where it was before the breakdown.

All this sounds like a neat solution. But it has some serious drawbacks:

  • With each transformation, data has to be serialized and written into the topic. Then, for the next operation in the chain, it has to be read from the topic, meaning that all side operations happen for every entity (like partition key calculation, persisting to disk, etc.).

  • Kafka Streams assumes that the Serde class used for serialization or deserialization is the one provided in the config. With changing the format of data in the operation chain, the user has to provide the appropriate Serde. If existing Serdes can’t handle the used format, the user has to create a custom Serde. No big deal — just extend the Serde class and implement a custom serializer and deserializer. From one class, we ended up with 4 — not so optimal. So, for each custom format of data in the operation chain, we create three additional classes. An alternative approach is to use generic JSON or AVRO Serdes. One more thing: the user has to specify a Serde for both key and value parts of the message.

  • Restarting of the application. After the breakdown, the application will go through each internal topic to the last valid offset, and this can take some time — especially if log compaction is not used and/or retention period is not set up.

2. Processor API

Another way to use Kafka Streams is to use the Processor API. The Processor API provides complete flexibility; the user can implement anything not supported by Stream DSL. It’s intended for users who haven’t completely grasped Stream DSL or for when some exotic functionality is required. To use the Processor API, the user has to provide a concrete implementation of ProcessorSuplier that returns the user’s implementation of the Processor class.

The user’s implementation of the Processor class needs to override four methods:

  • init: A method that will be called upon creating an instance of the Processor class. It provides a ProcessorContext object for the user. ProcessorContext is commonly used to schedule punctuation periods, forward transformed messages to output topic, or commit Processor state.

  • process: A method that is called for each arriving record; this is where the transformations usually happen.

  • punctuate: A scheduled method that is called on the configured period of times, often used to send messages in batches, export metrics, populate a database, etc.

  • close: A method called upon terminating the app to close connections.

Where Does the Problem Lie?

To completely understand the problem, we will first go into detail how ingestion and processing occur by default in Kafka Streams. For example purposes, the punctuate method is configured to occur every ten seconds, and in the input stream, we have exactly one message per second. The purpose of the job is to parse input messages, collect them, and, in the punctuate method, do a batch insert in the database, then to send metrics.

After running the Kafka Stream application, the Processor will be created, followed by the init method. Here is where all the connections are established. Upon successful start, the application will listen to input topic for incoming messages. It will remain idle until the first message arrives. When the first message arrives, the process method is called — this is where transformations occur and where the result is stored for later use. If no messages are in the input topic, the application will go idle again, waiting for the next message. After each successful process, the application checks if punctuate should be called. In our case, we will have ten process calls followed by one punctuate call, with this cycle repeating indefinitely as long as there are messages.

A pretty obvious behavior, isn’t it? Then why is one bolded?

This is where the things get tricky. An application can call more than one punctuate method subsequently. To understand why, read the following sentence again: After each successful process, the application checks if punctuate should be called.

What would happen if we have processed nine messages and there are no more messages in input topic?

The application will wait until new messages arrive in the topic. But we have started the punctuate period, and now more than ten seconds have passed. The application is still idle and will remain like that without regard to the started punctuate period. For the purpose of this example, the next message arrives one hour later. That message will be processed and put in the same punctuate period before calling insert for the entire batch. After the first expected punctuate, the application will call 359 more punctuate methods (60 minutes * 6 punctuate per minute). Only after finishing these 360 punctuate calls will the application resume consuming input messages.

There are two problems with this approach:

  1. Messages that arrive long after the last processed can end up in the wrong batch.

  2. Multiple punctuate methods can be called in succession.

If we want to group messages in timed windows so that each window contains only records that arrived in Kafka in the exact time bucket before adding a message to bulk, we should check the timestamp of the arriving message. After a bulk insert (to a database, to Kafka, etc.), save the current time and use that time to manually check if arriving messages belong to the next batch. The first message that doesn’t belong to the batch will trigger a bulk insert and start a new batch.

What if we are outputting metrics during the punctuate method? One solution is to remember the time of the last punctuate. A new punctuate will do output only if the correct amount of time has passed. Again, for the purpose of this example, let’s say that the first punctuate takes one second to complete and the remaining 359 punctuate methods in succession take one second to check (without metrics output). The application will process messages for eight seconds before bulk insert. The batch collect period has been shortened by 20% because of punctuate methods.

Why does it last only eight seconds?

Kafka Streams maps periods on regular and “correct” intervals by default. It will always start counting from zero. For our configuration intervals, this will look like:

[0, 10000),[10000, 20000), …

Note that Kafka Streams counts the duration of intervals in milliseconds. This means that the scheduled time for the first punctuate method is 10,000 ms from starting time. No matter how many input messages arrive, the first process method that finishes after 10,000 ms will trigger punctuate. The second punctuate is scheduled to occur 20,000 ms from starting time, it will not be affected by the duration of punctuate or process methods.

This will cause wrong metrics data (number of processed messages, latency time, etc.).

Multiple punctuate calls in succession can occur if the punctuate method lasts longer than intended. For example, network slowdown occurs when a database large population and high database usage are present (i.e. large inserts in the same tables, thus locking the table and preventing access) results in punctuates last longer that ten seconds (our interval period); for example, it lasted 30 seconds. This means that the next process will trigger punctuate again, followed by two unwanted punctuate calls.

One can argue that if we have such “slow” input, then stream processing is perhaps not the correct solution. This scenario can occur in systems with “fast” input; it all depends on the rate of messages and the punctuate period. Many legacy systems still have regular downtimes, when batch jobs are run, the database is updated, a new deploy is in progress, etc. Scheduled downtimes in the legacy part of a system can affect the correctness of the stream processing part of the system — and developers have to keep that in mind.

Another way to avoid the mentioned problems is to apply stateless processing. After each processed message sends the result to output topic, leave the batching to Kafka Producer. This will only shift problems down the line, as this introduces more database calls, more tcp/ip packets over the network, and more IO calls to replicate and permanently store Kafka messages on disks.

Conclusion

The Kafka Streams library provides a performant and scalable solution that can be incorporated into virtually any existing application. It’s developed with ease of use in mind. It provides the most commonly used functions while allowing users to implement their own.

Kafka Streams is by no means a silver bullet, and the same solution might not work in different use cases. The Processor API is intended to be flexible, but that only places the implementation of corner cases on the user. The high-level stream DSL provides several solutions for windowing and aggregations, while not covering all.

Some frameworks like Apache Flink put a strong emphasis on windowing and correctness, at some cost in performance or usability. No matter the framework, corner cases always require special care.

Original Link

Choosing the Right GC

Size matters when it comes to software. It has become clear that using small pieces within a microservices architecture delivers more advantages compared to the big monolith approach. The recent Java release of Jigsaw helps decompose legacy applications or build new cloud-native apps from scratch.

This approach reduces disk space, build time, and startup time. However, it doesn’t help enough with RAM usage management. It is well-known that Java consumes a large amount of memory in many cases. At the same time, many have not noticed that Java has become much more flexible in terms of memory usage and provided features to meet the requirements of microservices.

In this article, we will share our experiences in tuning RAM usage in a Java process to make it more elastic and gain the benefits of faster scaling and lower total cost of ownership (TCO).

There are three options for scaling: vertical, horizontal, and a combination of both. In order to maximize the outcome, first, you need to set up vertical scaling in an optimal way. Then, when your project grows horizontally, the preconfigured resource consumption within a single container will be replicated to each instance, and efficiency will grow proportionally.

If configured properly, vertical scaling works perfectly for both microservices and monoliths, optimizing memory and CPU usage according to the current load inside containers. The selected garbage collector is one of the main foundational bricks, and its settings can influence the whole project.

There are five widely used garbage collector solutions for OpenJDK:

  • G1
  • Parallel
  • ConcMarkSweep (CMS)
  • Serial
  • Shenandoah

Let’s see how each of these performs in terms of scaling and what settings can be applied to improve results.

For testing, we’ll use a sample Java application that helps to track JVM vertical scaling results.

The following JVM start options will be initiated for each GC test:

java -XX:+Use[gc_name]GC -Xmx2g -Xms32m -jar app.jar [sleep]

Where:

  • [gc_name] will be substituted with the specific garbage collector type
  • Xms is the scaling step (32 MB in our case).
  • Xmx is the maximum scaling limit (2 GB in our case).
  • [sleep] is the interval between memory load cycles in milliseconds. The default is 10.

At the moment, invoking Full GC is required for a proper release of unused resources. It can be easily initiated with various options:

  • jcmd <pid> GC.run — executing external calls.
  • System.gc() — inside the source code.
  • jvisualvm — manually via the great VisualVM troubleshooting tool.
  • -javaagent:agent.jar — pluggable commonly used approach. This open source automation add-on is available at the GitHub repo Java Memory Agent.

Memory usage can be tracked in output logs or using VisualVM for a deeper review.

G1 Garbage Collector

The good news for the Java ecosystem is that, starting with JDK 9, the modern shrinking G1 garbage collector is enabled by default. If you use a JDK of lower release, G1 can be enabled with the -XX:+UseG1GC parameter.

One of G1’s main advantages is the ability to compact free memory space without lengthy pause times. It can also uncommit unused heaps. We found this GC to be the best option for vertical scaling of Java applications running on OpenJDK or HotSpot JDK.

To gain a better understanding of how the JVM behaves at different memory pressure levels, we’ll run three cases:

  1. Fast memory usage growth.

  2. Medium memory usage growth.

  3. Slow memory usage growth.

In this way, we can check how smart G1’s ergonomics are and how GC handles different memory usage dynamics.

Fast Memory Usage Growth

java -XX:+UseG1GC-Xmx2g -Xms32m -jar app.jar 0

Memory grew from 32 MiB to 1 GiB in 25 seconds.

G1 fast memory usage growth

If memory usage growth is very fast, the JVM’s ergonomics ignore Xms scaling steps and reserves RAM faster according to its internal adaptive optimization algorithm. As a result, we see much faster RAM allocation for JVM (orange) relative to the fast real usage (blue) growth. So with G1, we are safe, even in the event of load spikes.

Medium Memory Usage Growth

java -XX:+UseG1GC-Xmx2g -Xms32m -jar app.jar 10

Memory grew from 32 MiB to 1 GiB in 90 seconds during four cycles.

Sometimes it requires several cycles for JVM ergonomic to find an optimal RAM allocation algorithm.

G1 medium memory usage growthAs we can see, the JVM tuned the RAM allocation ergonomics at the fourth cycle to make vertical scaling very efficient for repeatable cycles.

Slow Memory Usage Growth

java -XX:+UseG1GC-Xmx2g -Xms32m -jar app.jar 100

Memory grew from 32 MiB to 1 GiB with a delta time growth of about 300 seconds. That’s very elastic and efficient resource scaling that meets our expectations — impressive.

G1 slow memory usage growth

As you can see, the orange area (reserved RAM) increases slowly, corresponding to the blue area (real usage) growth. That means no overcommitting or unnecessarily reserved memory.

Important: Aggressive Heap or Vertical Scaling

Popular JVM configurations for improving Java application performance can often impede the ability to efficiently scale vertically. So, you need to choose between the priorities of what attributes are most essential for your application.

One of many widely-used settings is the activation of Aggressive Heap in an attempt to make maximum use of physical memory for the heap. Let’s analyze what happens while using this configuration.

java -XX:+UseG1GC -Xmx2g -Xmx2g

or

java -XX:+UseG1GC -Xmx2g -XX:+AggressiveHeap

G1 aggressive heap

As we can see, Reserved Heap (orange) is constant and doesn’t change throughout time, so there is no vertical scaling of the JVM in the container. Even if your application uses only a little part of available RAM (blue), the rest cannot be shared with other processes or other containers, as it’s fully allocated for the JVM.

So if you want to scale your application vertically, make sure that Aggressive Heap is not enabled (the parameter should be -XX:-AggressiveHeap) and do not define -Xms as high as -Xmx (for example, do not state -Xmx2g -Xms2g).

Parallel Garbage Collector

Parallel is a high-throughput GC and is used by default in JDK8. At the same time, it does not suit memory shrinking, which makes it inappropriate for flexible vertical scaling. To confirm this, let’s run a test with our sample application:

java -XX:+UseParallelGC-Xmx2g -Xms32m -jar app.jar 10

Parallel garbage collector

As we can see, the unused RAM is not released back to the OS. The JVM with Parallel GC keeps it forever, even disregarding the explicit Full GC calls.

So if you want to benefit from vertical scaling according to the application load, change the Parallel to the shrinking GC available in your JDK. It will package all the live objects together, remove garbage objects, and uncommit and release unused memory back to the operating system.

Serial and ConcMarkSweep Garbage Collector

Serial and ConcMarkSweep are also shrinking garbage collectors and can scale memory usage in the JVM vertically. But in comparison to G1, they require four Full GC cycles to release all unused resources.

Let’s see the results of the test for the both of these garbage collectors:

java-XX:+UseSerialGC-Xmx2g -Xms32m -jar app.jar 10

Serial garbage collector

java -XX:+UseConcMarkSweepGC-Xmx2g -Xms32m -jar app.jar 10

ConcMarkSweep garbage collector

Starting from JDK9, the releasing of memory can be sped up with the new JVM option -XX:-ShrinkHeapInSteps, which brings down committed RAM right after the first Full GC cycle.

Shenandoah Garbage Collector

Shenandoah is a rising star among garbage collectors that can already be considered as the best upcoming solution for vertical JVM scaling.

The main difference, compared to the others, is the ability to shrink (uncommit and release unused RAM to OS) asynchronously without needing to call a Full GC. Shenandoah can compact live objects, clean garbage, and release RAM back to the OS almost immediately after detecting free memory. And the possibility of omitting Full GC leads to eliminating related performance degradation.

Let’s see how it works in practice:

java -XX:+UseShenandoahGC -Xmx2g -Xms32m -XX:+UnlockExperimentalVMOptions -XX:ShenandoahUncommitDelay=1000 -XX:ShenandoahGuaranteedGCInterval=10000 -jar app.jar 10

Here we added some extra parameters available in Shenandoah:

  • -XX:+UnlockExperimentalVMOptions — needed to enable the uncommit option listed below.
  • -XX:ShenandoahUncommitDelay=1000 — GC will start uncommitting memory that was not used for more than a designated time (in milliseconds). Note that making the delay too low may introduce allocation stalls when the application would like to get the memory back. In real-world deployments, making the delay larger than 1 second is recommended.
  • -XX:ShenandoahGuaranteedGCInterval=10000 — guarantees a GC cycle within the stated interval (in milliseconds).

Shenandoah garbage collector

Shenandoah is very elastic and allocates only necessary resources. It also compacts used RAM (blue) and releases unconsumed reserved RAM (orange) back to the OS on the fly and without costly Full GC calls. Please note this GC is experimental, so your feedback about stability will be helpful for its creators.

Conclusion

Java keeps perfecting and adapting to always-changing demands. Currently, its RAM appetite is no longer a problem for microservices and cloud hosting with traditional applications, as there are already the right tools and ways to scale it properly, clean the garbage, and release resources for required processes. Being configured smartly, Java can be cost-effective for all ranges of projects — from cloud-native startups to legacy enterprise applications.

Original Link

Inside the JVM (Part 4): ClassLoader [Video]

This video discusses the JVM ClassLoader. This is a detailed and advance discussion of what each classloading phase does and the internal process of loading, linking, and initialization. Also, we will discuss how and why Java initializes parent classes while we use the child class and the six points that Java needs to initialize objects at. Finally, we will discuss four ways to create new instances and how the initial value assignment works each way.

Original Link

IntelliJ IDEA 2017.3 Public Preview

IntelliJ IDEA 2017.3 is now available for public preview! Everyone is welcome to download the public preview build right away. We are committed to creating a better product, we really appreciate your input when you share your feedback and send us bug reports. We are constantly working to improve user experience and productivity, and, of course, on bug fixes. In this post, we’d like to give you a glimpse of the highlights of the upcoming release.

Java Inspections and Quick-Fixes

IntelliJ IDEA 2017.3 has loads of new improvements made to the data flow analysis.
The IDE now detects possible nullability issues in Stream API call chains:

Now you can replace StringBuilder usages with Stream.collect and Collectors.joining:

You can replace code that iterates to find a maximum or a minimum with Stream.max and Stream.min:

We’ve made the data flow analysis for the Optional type even smarter:

It now even detects issues that were not obvious:

The updated inspection infers nullability annotations for parameters of sealed and private methods:

It offers a quick-fix to explicitly declare the inferred annotation:

Redundant throws declarations can now be detected and fixed on the fly.

For deprecated code, the IDE now suggests a quick-fix if there is a replacement method in the JavaDoc.

The inspection for Inverting a boolean method is now available on-the-fly.

The IDE now detects duplicating Map keys and duplicating Set elements.

You will also find new inspections for the Fuse Stream API call chain:

There’s a new intention action called Unroll loop:

Note that there’s an easy way to find all the new inspections. Open the Settings/Preferences dialog select Editor | Inspections, and click on the Filter icon.

Smart Completion in IntelliJ IDEA 2017.3 is aware of type casts and uses them to suggest chains of method calls.

JVM Debugger

There’s a new Overhead tab in the Debugger tool window. It displays the overhead added either when stepping over the code or when Data Renderers evaluate values.

One more enhancement to the debugger is the new On-demand Data Renderers feature that helps reduce overhead. Now, the evaluation of the values in Variables, Watches, and other places can be done on demand. Simply click on them when needed, instead of having them evaluated automatically:

To enable On-demand Data Renderers for each renderer, go to the Data Renderers settings page.

In addition, a renderer can be “muted” from the context menu.

Async Stacktraces

The Async Stacktraces feature shows the combined stack traces of all threads. IntelliJ IDEA 2017.3 extends the Async Stacktraces feature with a new Instrumenting agent, which provides almost the same functionality and works out of the box as common capture points are built-in. The new Instrumenting agent causes very low overhead. The Instrumenting agent option is enabled by default.

For this code example, we’ll have the following stack trace:

Java Stream Debugger

The Java Stream Debugger plugin, which visualizes Java Stream operations, is built into IntelliJ IDEA 2017.3. The new functionality is available inside the Debugger tool window (click the Trace Current Stream Chain button). This plugin evaluates the current data stream and presents a visual representation of what exactly happens to each element, from the first call to the last.

Use the Split Mode/Flat Mode button to toggle the way operations are displayed: all at once or individually.

Gradle

IntelliJ IDEA 2017.3 lets you run tests with coverage with Gradle Test Runner. You can run tests with coverage even if you select the option Delegate IDE build/run action to Gradle.

In the Preferences window, you can enable the option Let me choose per test to choose how to run your tests with coverage per test: with the platform test runner, or with the Gradle test runner. You can select the test runner type from the editor.

You can also run tests with coverage from the editor, even if you delegate a test to Gradle.

Build Tool Window

IntelliJ IDEA 2017.3 introduces a new Build tool window as a single place to view all of the output related to a Gradle build. Previously, the output of a Gradle build was displayed in different places depending on the context, including the Run or Messages windows. In some cases, it was even hidden as a background process.

If a project is set to auto-import, it will be automatically rebuilt when there are changes that affect a Gradle build. The new Build tool window shows the progress of this process as well.

Even when you enable Delegate IDE build/run actions to gradle, the build output will still be displayed in the Build tool window.

Run Dashboard

In IntelliJ IDEA 2017.2, we introduced the Run Dashboard for Spring Boot applications. We are extending this functionality in v2017.3, so that the IDE now lets you add different types of run configurations to the Run Dashboard. In the Run/Debug Configurations window, you can add the required run configuration to the Run Dashboard.

You can manage multiple run configurations at once by using the Run Dashboard.

The Run Dashboard toolbar lets you rerun, stop, pause, or terminate an application. The right-hand side shows application-specific information.

Command Line Shortener

IntelliJ IDEA 2017.3 introduces a configurable command line shortener — a convenient new way to specify a method used to shorten the command line for each configuration.

Look in the Run/Debug Configuration dialog for a new field called Shorten command line.


You can set the default way to shorten the command line and use it as a template for future configurations. IntelliJ IDEA 2017.3 also enables you to share your configuration with your colleagues.

You can also preview the full command line if a long classpath was shortened via a temporary classpath.jar (the JAR Manifest option in the Run/Debug Configuration dialog).

Project Configuration

Now you can choose how to organize modules: by using module groups, or by grouping modules based on their qualified names.

There are also new improvements for unloaded modules. When you update a project that has unloaded modules via a VCS, the IDE will analyze all the dependencies between the modules. If newly added modules depend on the existing ones, they will be marked as loaded; otherwise, as unloaded.

Also, the IDE now checks that unloaded modules compile successfully before a commit.

Java EE 8

IntelliJ IDEA 2017.3 expands its support for the key features in Java EE 8. The IDE now supports Asynchronous CDI Events, and allows you to easily navigate between the place where the event was fired and where this event was received. This is done by using icons in the gutter.

Spring and Spring Boot

The IDE now automatically detects the Spring configuration in your code. It will automatically create a Spring facet for your project, or will alert you if any configuration is missing.

Also, the Spring Beans Dependencies diagram has been extended with a new Neighborhood Mode feature, which lets you choose only the necessary beans and only view their dependencies.

For better readability, you can easily switch to the Borderless View.

For Spring Boot MVC web applications, the IDE now automatically detects an MVC context. A Web Facet and a Spring Boot MVC Context will be set up automatically.

For Spring Boot MVC web applications, the IDE now fully supports all the major features including auto-completion, syntax highlighting, and navigation to related views.

Support for Spring Boot 2 is now available.

REST Client

IntelliJ IDEA 2017.3 introduces a brand new editor-based REST client. To start using the new REST client, create a scratch file with the .http extension.

Use the icon on the left-hand panel of the editor to run a request:

For easier navigation, the IDE will add a link to the request results.

In the new editor-based REST client, you can define the context for executing a request. Create a rest-client.env.json file inside your project and define an environment with variables. Once the environment variables are defined, you can easily switch between different environments such as production, testing, and development.

VCS

For Git integration, IntelliJ IDEA 2017.3 provides faster branch operations, especially for big projects.

The Log viewer now offers a new action called Interactively Rebase from Here.

To learn more about all the exciting new features that v2017.3 offers for version control tools, click here.

Settings Synchronization

IntelliJ IDEA 2017.3 ensures better synchronization of your settings across different installations. It provides a more convenient method for storing your settings, utilizing a JetBrains repository to store items such as the UI theme, colors, and fonts settings, and so on. You can apply these settings to all of your IDE instances using your JetBrains Account (JBA).

You can also synchronize the list of installed and enabled and disabled plugins.

The settings you can sync include:

  1. Look And Feel (Preferences | Appearance & Behavior | Appearance | Theme)
  2. Keymap (Preferences | Keymap)
  3. Color Scheme (Preferences | Editor | Color Scheme)
  4. General Settings (Preferences | Appearance & Behavior | System Settings)
  5. UI Settings (Preferences | Appearance & Behavior | Appearance)
  6. Menus and Toolbars (Preferences | Appearance & Behavior | Menus and Toolbars)
  7. ProjectView Settings (Project Tool Window – syncs only IDE settings, all other settings are automatically saved in a project)
  8. Editor Settings (Preferences | Editor | General)
  9. Code Completion (Preferences | Editor | General | Code Completion)
  10. Parameter Name Hints (Preferences | Editor | General)
  11. Live Templates (Preferences | Editor | Live Templates)
  12. Code Style Schemes (Preferences | Editor | Code Style)Plugins (Preferences | Plugins)

In order to test the new settings synchronization functionality before its release, you need to activate the IDE Settings Sync plugin. You can learn how to activate it in this blog post.

To learn about the enhancements for JavaScript and TypeScript, please take a look at the Webstorm blog. Make sure to check the improvements affecting database tools and SQL support in the DataGrip blog.

Last but not least, here’s a list of some notable bugfixes in IntelliJ IDEA 2017.3:

  • JRE-509 JDK was updated to 1.8.0_152-release-1024-b06 x86_64: the HiDPI mode fallback logic was removed

  • IDEA-26988 the IDEA Coverage ignores empty private Constructors of Utility Classes
    Two new options added in the Coverage settings:

    • to exclude implicit constructors from the coverage

    • to exclude empty private constructors coverage data in classes when all of the other methods are static

  • IDEA-101502 Dir diff: Files with different line separators are shown as different, but they are actually the same.

  • IDEA-178764 The pull request fails after the pull request creation

  • IDEA-170103 The Find in Path dialog don’t close (2017.1 version, Ubuntu 16.04)

  • IDEA-121059 JPA2.1: Error “Element entity-mapping must be declared” in the mapping file in case of standard schema location.

  • IDEA-138029 Cannot index custom maven repositories provided by Bintray

  • IDEA-134885 Gradle plugin incorrectly adds Maven artifact dependencies when they should resolve to module dependencies.

  • And many more, just check this link

Whew, that was quite a list! To get started and discover these new features, download the preview build! Share your feedback with us via the comments below, our issue tracker or Twitter.

Happy developing!

Original Link

Kotlin Features I Miss in Java

Although I’m a big supporter of the Kotlin programming language, I still do a lot of Java programming on a daily basis for my employer. Since I’m aware of the great functionalities of Kotlin, I’m often struggling with Java, as it has some “pitfalls,” requires additional boilerplate, and misses many features.

In this post, I’d like to describe which Kotlin features I miss most when coding in Java.

new and Semicolon

Ever since I’ve been doing Kotlin, there are two things I always forget when coding in Java: the new keyword for constructor invocations and the annoying ; to complete statements. Kotlin doesn’t have new, and even semicolons are optional. I really appreciate this decision because it reduces the “syntactic noise.

Data Classes

In Kotlin, data classes are used for simple data containers, representing JSON objects or returning compound objects from functions, amongst other use cases. Of course, Java doesn’t support this special type of class yet. As a result, I often have to implement my own data class, which means a lot of boilerplate in Java.

One special use case is compound objects returned from functions. For example, let’s imagine a function that needs to return two objects. In Kotlin, we could use a data class, or simpler, a Pair directly.

In Java, I tend to create a value object, which is a class with several final fields, each of which instantiated through the constructor. Similar to Kotlin, I don’t implement getters and setters, but use the class’s fields directly as public properties. Unfortunately, this is not what we learned as a best practice, and most Java code style-checkers will complain about it. I do not see any encapsulation issues here, and it’s the least verbose approach in Java. The following shows such a compound object, the inner class Multi. In Kotlin, this would be a one-liner.

public class MultiReturn { public static void main(String[] args) { new MultiReturn().useMulti(); } public void useMulti() { Multi multi = helper(); System.out.println("Multi with " + multi.count + " and " + multi.name); } private Multi helper() { return new Multi(2, "test"); } private static class Multi { private final int count; private final String name; public Multi(int count, String name) { this.count = count; this.name = name; } }
}

Local Functions

In many situations, we tend to create private methods that are only used inside another single method in order to make this one more readable. In Kotlin, we can use local functions, i.e. functions inside functions (inside functions…), which enables some kind of scope. For me, this is a much cleaner approach because the function is only accessible inside the function that actually uses the local one. Let’s look at an example.

fun deployVerticles() { fun deploy(verticleClassName: String) { vertx.deployVerticle(verticleClassName, opt, { deploy -> LOG.info("$verticleClassName has been deployed? ${deploy.succeeded()}") }) } deploy("ServiceVerticle") deploy("WebVerticle")
}

It’s taken from a sample vert.x application and defines a local function that is reused twice afterward. A great way to simplify your code.

Single Expression Functions

We can create single expression functions in Kotlin, i.e. functions without an actual body. Whenever a function contains only a single expression, it can be placed after an = sign following the function declaration:

fun trueOrFalse() = Random().nextBoolean()

In Java, on the other hand, we always have to use a function body enclosed in {}, which ranges over at least three lines. This is also “syntactic noise” I don’t want to see anymore. To be fair, Java 1.8 makes it possible to define lambdas, which can also solve this —but they’re less readable though (can also be applied to local functions):

public class SingleExpFun { private BooleanSupplier trueOrFalse = new Random()::nextBoolean; private boolean getNext(){ return trueOrFalse.getAsBoolean(); }
}

Default Parameters

One very annoying part of Java is the way methods have to be overloaded. Let’s see an example:

public class Overloade public static void main(String[] args) { Overloader o = new Overloader(); o.testWithoutPrint(2); o.test(2); } public void test(int a, boolean printToConsole) { if (printToConsole) System.out.println("int a: " + a); } public void testWithoutPrint(int a) { test(a, false); } public void test(int a) { test(a, true); } }

We can see a class with a method test(int, boolean) that is overloaded for the default case, and a convenience method is also available. For more complex examples, it can lead to a lot of redundant code, which is simpler in Kotlin by using default parameters.

fun test(a: Int, printToConsole: Boolean = true) { if (printToConsole) println("int a: " + a)
} fun testWithoutPrint(a: Int) = test(a, false) fun main(args: Array) { testWithoutPrint(2) test(2)
}

Calling Multiple Methods on an Object Instance (with)

Obviously, Kotlin is more functional than Java. It makes use of higher-order functions in many situations and provides many standard library functions that can be used as such. One of my favorites is with, which I miss a lot whenever I can’t use Kotlin. The with function can be used to create scopes that actually increase the readability of code. It’s always useful when you sequentially call multiple functions on a single object.

class Turtle { fun penDown() fun penUp() fun turn(degrees: Double) fun forward(pixels: Double)
} with(Turtle()) { penDown() for(i in 1..4) { forward(100.0) turn(90.0) } penUp()
}

The great thing is the usage of lambdas with receiver, which you can read about in one of my other posts.

Null-Safety

Whenever I work with nullable types, since the time I started with Kotlin, I actually miss the type system’s tools to prevent null-related errors. Kotlin did a very good job by distinguishing nullable types from non-nullable ones. If you strictly make use of these tools, there is no chance you’ll ever see a NullpointerException at runtime.

Lambdas and Collection Processing

Kotlin places a lot of value on its lambdas. As shown in the with example earlier, there’s special syntax available for lambdas that makes its usage even more powerful. I want to underline that the way functions, and especially lambdas, are treated in the language makes it dramatically superior to Java. Let’s see a simple example of Java’s Streams, which were introduced along with lambdas in Java 1.8:

List list = people.stream().map(Person::getName).collect(Collectors.toList());

It’s a rather simple example of a Stream that is used to get a list of names from a list of persons. Compared to what we did before 1.8, this is awesome. Still, it’s too noisy compared to a real functional approach as pursued by Kotlin:

val list = people.map { it.name }

Or yet another example, in which salaries of employees are summed up to a total amount:

int total = employees.stream() .collect(Collectors.summingInt(Employee::getSalary)));

So much simpler in Kotlin:

val total = employees.sumBy { it.salary }

The Kotlin examples show how simple it can be. Java isn’t a functional language and has a hard time trying to adopt functional features like lambdas and streams, as we can easily observe in the snippets. It really sucks to go back to Java if you’ve ever experienced the beauty of Kotlin. Have you ever tried to use Eclipse after being familiar with IntelliJ? You know what I mean then.

Wrap Up

In this short post, I presented you my top Kotlin features I always miss when coding in Java. It’s just a selection of things, which will hopefully find their way into the Java language soon. But to be honest, there’s no reason to always wait for Java when there already is a much sweeter language available… I want to point out, that starting with Kotlin really made me a much better programmer because I began wondering about certain features in both languages and also tried to find ways to use Kotlin-dedicated things in Java by finding workarounds, like arranging my code differently.

I’d be interested in the features you like most — feel free to comment.

Also, if you like, have a look at my Twitter account and follow if you’re interested in more Kotlin stuff! Thanks a lot.

If you want to read more about Kotlin’s beautiful features I recommend the book Kotlin in Action and my other articles to you.

Original Link

Inside the Java Virtual Machine (Part 3): Data Types [Video]

This video discusses data types inside the JVM, including breakdowns of primitive and reference types and how data resides in the JVM’s memory. This is next post of my series.

Original Link

Inside the Java Virtual Machine (Part 2)

In my last article, I explained virtual machines, the Java Virtual Machine, and the JVM’s more basic concepts. This video will go a little deeper into the JVM.

Original Link

Looking Into the Java Virtual Machine [Video]

A lot of Java developers, even experienced ones, struggle to understand the JVM. When I  interview people, I realize there are a lot of misconceptions around the JVM and virtual machines in general. So, with that in mind, I created a video to explain the important concepts of the JVM.

Original Link

jOOQ Tuesdays: Nicolai Parlog Talks About Java 9

Welcome to the jOOQ Tuesdays series. In this series, we’ll publish an article on the third Tuesday every other month where we interview someone we find exciting in our industry from a jOOQ perspective. This includes people who work with SQL, Java, Open Source, and a variety of other related topics.

I’m very excited to feature today Nicolai Parlog, author of The Java Module System

Nicolai, your blog is an “archeological” treasure trove for everyone who wants to learn about why Java expert group decisions were made. What made you dig out all these interesting discussions on the mailing lists?

Ha, thank you, didn’t know I was sitting on a treasure.

It all started with everyone’s favorite bikeshed: Optional. After using it for a few months, I was curious to learn more about the reason behind its introduction to Java and why it was designed the way it was, so I started digging and learned a few things:

  • Piperman, the JDK mailing list archive, is a horrible place to peruse and search.
  • Mailing list discussions are often lengthy, fragmented, and thus hard to revisit.
  • Brian Goetz was absolutely right: Everything related to Optional seems to take 300 messages.

Consequently, researching that post about Optional’s design took a week or so. But as you say, it’s interesting to peek behind the curtain and once a discussion is condensed to its most relevant positions and peppered with some context it really appeals to the wider Java community.

I actually think there’s a niche to be filled, here. Imagine there was a site that did regularly (at least once a week) what I did with a few selected topics: Follow the JDK mailing list, summarize ongoing discussions, and make them accessible to a wide audience. That would be a great service to the Java community as it would make it much easier to follow what is going on and to chime in with an informed opinion when you feel you have something to contribute. Now we just need to find someone with a lot of free time on their hands.

By the way, I think it’s awesome that the comparatively open development of the JDK makes that possible.

I had followed your blog after Java 8 came out, where you explained expert group decisions in retrospect. Now, you’re mostly covering what’s new in Java 9. What are your favorite “hidden” (i.e. non-Jigsaw) Java 9 features and why?

From the few language changes, it’s easy pickings: definitely private interface methods. I’ve been in the situation more than once that I wanted to share code between default methods but found no good place to put it without making it part of the public API. With private methods in interfaces, that’s a thing of the past.

When it comes to API changes, the decision is much harder as there is more to choose from. People definitely like collection factory methods and I do, too, but I think I’ll go with the changes to Stream and Optional. I really enjoy using those Java 8 features and think it’s great that they’ve been improved in 9.

A JVM feature I really like is multi-release JARs. The ability to ship a JAR that uses the newest APIs, but degrades gracefully on older JVMs will come in very handy. Some projects, Spring for example, already do this, but without JVM support it’s not exactly pleasant.

Can I go on? Because there’s so much more! Just two: Unified logging makes it much easier to tease out JVM log messages without having to configure logging for different subsystems and compact strings and indified string concatenation make working with strings faster, reduce garbage and conserve heap space (on average, 10% to 15% less memory!). Ok, that was three, but there you go.

You’re writing a book on the Java 9 module system that can already be pre-ordered on Manning. What will readers get out of your book?

All they need to become module system experts. Of course, it explains all the basics (declaring, compiling, packaging, and running modular applications) and advanced features (services, implied readability, optional dependencies, etc), but it goes far beyond that. More than how to use a feature it also explains when and why to use it, which nuances to consider, and what are good defaults if you’re not sure which way to go.

It’s also full of practical advice. I migrated two large applications to Java 9 (compiling and running on the new release, not turning them into modules) and that experience, as well as the many discussions on the mailing list, informed a big chapter on migration. If readers are interested in a preview, I condensed it into a post on the most common Java 9 migration challenges. I also show how to debug modules and the module system with various tools (JDeps for example) and logging (that’s when I started using uniform logging), Last but not least, I plan to include a chapter that simply lists error messages and what to do about them.

In your opinion, what are the good parts and the bad parts about  Jigsaw? Do you think Jigsaw will be adopted quickly?

The good, the bad, and the ugly, eh? My favorite feature (of all of Java 9 actually) is strong encapsulation. The ability to have types that are public only within a module is incredibly valuable! This adds another option to the private-to-public-axis and once people internalize that feature we will wonder how we ever lived without it. Can you imagine giving up private? We will think the same about exported.

I hope the worst aspect of the module system will be the compatibility challenges. That’s a weird way to phrase it, but let me explain. These challenges definitely exist and they will require a non-neglectable investment from the Java community as a whole to get everything working on Java 9, in the long run as modules. (As an aside: This is well invested time – much of it pays back technical debt.)

My hope is that no other aspect of the module system turns out to be worse. One thing I’m a little concerned about is the strictness of reliable configuration. I like the general principle and I’m definitely one for enforcing good practices, but just think about all those POMs that busily exclude transitive dependencies. Once all those JARs are modules, that won’t work – the module system will not let you launch without all dependencies present.

Generally speaking, the module system makes it harder to go against the maintainers’ decisions. Making internal APIs available via reflection or altering dependencies now goes against the grain of a mechanism that is built deeply into the compiler and JVM. There are of course a number of command line flags to affect the module system but they don’t cover everything. To come back to excluding dependencies, maybe –ignore-missing-modules ${modules} would be a good idea…

Regarding adoption rate, I expect it to be slower than Java 8. But leaving those projects aside that see every new version as insurmountable and are still on Java 6, I’m sure the vast majority will migrate eventually. If not for Java 9’s features than surely for future ones. As a friend and colleague once said: “I’ll do everything to get to value types.”

Now that Java 9 is out and “legacy”, what Java projects will you cover next in your blog and your work?

Oh boy, I’m still busy with Java 9. First I have to finish the book (November hopefully) and then I want to do a few more migrations because I actually like doing that for some weird and maybe not entirely healthy reason (the things you see…). FYI, I’m for hire, so if readers are stuck with their migration they should reach out.

Beyond that, I’m already looking forward to primitive specialization, e.g. ArrayList<int>, and value types (both from Project Valhalla) as well as the changes Project Amber will bring to Java. I’m sure I’ll start discussing those in 2018.

Another thing I’ll keep myself busy with and which I would love your readers to check out is my YouTube channel. It’s still very young and until the book’s done I won’t do a lot of videos (hope to record one next week), but I’m really thrilled about the whole endeavour!

Original Link