Serial Programming/Serial Java

Introduction
Because of Java's platform-independence, serial interfacing is difficult. Serial interfacing requires a standardized API with platform-specific implementations, which is difficult for Java.

Unfortunately, Sun didn't pay much attention to serial communication in Java. Sun has defined a serial communication API, called JavaComm, but an implementation of the API was not part of the Java standard edition. Sun provided a reference implementation for a few, but not all Java platforms. Particularly, at the end of 2005 Sun silently withdrew JavaComm support for Windows. Third party implementations for some of the omitted platforms are available. JavaComm hasn't seen much in the way of maintenance activities, only the bare minimum maintenance is performed by Sun, except that Sun apparently responded to pressure from buyers of their own Sun Ray thin clients and adapted JavaComm to this platform while dropping Windows support.

This situation, and the fact that Sun originally did not provide a JavaComm implementation for Linux (starting in 2006, they now do) led to the development of the free-software RxTx library. RxTx is available for a number of platforms, not only Linux. It can be used in conjunction with JavaComm (RxTx providing the hardware-specific drivers), or it can be used stand-alone. When used as a JavaComm driver the bridging between the JavaComm API and RxTx is done by JCL (JavaComm for Linux). JCL is part of the RxTx distribution.

Sun's negligence of JavaComm and JavaComm's particular programming model gained JavaComm the reputation of being unusable. RxTx - if not used as a JavaComm driver - provides a richer interface, but one which is not standardized. RxTx supports more platforms than the existing JavaComm implementations. Recently, RxTx has been adopted to provide the same interface as JavaComm, only that the package names don't match Sun's package names.

So, which of the libraries should one use in an application? If maximum portability (for some value of "maximum") is desired, then JavaComm is a good choice. If there is no JavaComm implementation for a particular platform available, but an RxTx implementation is, then RxTx could be used as a driver on that platform for JavaComm. So, by using JavaComm one can support all platforms which are either directly supported by Sun's reference implementation or by RxTx with JCL. This way the application doesn't need to be changed, and can work against just one interface, the standardized JavaComm interface.

This module discusses both JavaComm and RxTx. It mainly focuses on demonstrating concepts, not ready-to-run code. Those who want to blindly copy code are referred to the sample code that comes with the packages. Those who want to know what they are doing might find some useful information in this module.

jSSC (Java Simple Serial Connector) should also be considered

There is also another library called jSerialComm that includes all its platform specific files in its jar, this makes it truly portable as there is no installation necessary.

Getting started

 * Learn the basics of serial communication and programming.
 * Have the documentation of the device you want to communicate with (e.g. the modem) ready.
 * Set up all hardware and a test environment
 * Use, for example, a terminal program to manually communicate with the device. This is to be sure the test environment is set up correctly and you have understood the commands and responses from the device.
 * Download the API implementation you want to use for your particular operating system


 * Read
 * the JavaComm and/or RxTx installation instruction (and follow it)
 * the API documentation
 * the example source code shipped

General Issues
Both JavaComm and RxTX show some installation quirks. It is highly recommended to follow the installation instructions word-for-word. If they say that a jar file or a shared library has to go into a particular directory, then this is meant seriously! If the instructions say that a particular file or device needs to have a specific ownership or access rights, this is also meant seriously. Many installation troubles simply come from not following the instructions precisely.

It should especially be noted that some versions of JavaComm come with two installation instructions. One for Java 1.2 and newer, one for Java 1.1. Using the wrong one will result in a non-working installation. On the other hand, some versions/builds/packages of RxTx come with incomplete instructions. In such a case the corresponding source code distribution of RxTx needs to be obtained, which should contain complete instructions.

Many linux distributions offer a RxTx package in their repositories (ArchLinux - 'java-rxtx', Debian/Ubuntu - 'librxtx-java'), these packages include only the platform specific parts of the library, but are generally ready to use.

It should be further noticed that it is also typical for Windows JDK installations to come with up to three VMs, and thus three extension directories. Some even claim to have a fourth JRE somewhere in the \Windows directory hierarchy.
 * One as part of the JDK,
 * one as part of the private JRE which comes with the JDK to run JDK tools, and
 * one as part of the public JRE which comes with the JDK to run applications

JavaComm should at least be installed as extension in the JDK and in all public JREs.

JavaComm
A general problem, both for JavaComm and RxTx is, that they resist installation via Java WebStart:

JavaComm is notorious, because it requires a file called javax.comm.properties to be placed in the JDK lib directory, something which can't be done with Java WebStart. This is particularly sad, because the need for that file is the result of some unnecessary design/decision in JavaComm and could have easily been avoided by the JavaComm designers. Sun constantly refuses to correct this error, citing the mechanism is essential. Which is, they are lying through their teeth when it comes to JavaComm, particular, because Java for a long time has a service provider architecture exactly intended for such purposes.

The contents of the properties file is typically just one line, the name of the java class with the native driver, e.g.:

The following is a hack which allows to deploy JavaComm via Web Start ignoring that brain-dead properties file. It has serious drawbacks, and might fail with newer JavaComm releases - should Sun ever come around and make a new version.

First, turn off the security manager. Some doofus programmer at Sun decided that it would be cool to again and again check for the existence of the dreaded javax.comm.properties file, even after it has been loaded initially, for no other apparent reason than checking for the file.

Then, when initializing the JavaComm API, initialize the driver manually:

RxTx
RxTx on some platforms requires changing ownership and access rights of serial devices. This is also something which can't be done via WebStart.

At startup of your program you could ask the user to perform the necessary setup as super user.

Further, RxTx has a pattern matching algorithm for identifying "valid" serial device names. This often breaks things when one wants to use non-standard devices, like USB-to-serial converters. This mechanism can be overridden by system properties. See the RxTx installation instruction for details.

jSerialComm
jSerialComm in contrast to both RxTx and JavaComm is ready to use on many operating Systems and Platforms (e.g. Windows x86/x86_64, Linux x86/x86_64, ARM, and even android - full list in the specific library jar) without any changes. However it still needs permissions to access the device (more information on the jSerialComm Homepage).

SerialPundit

SerialPundit is another feature rich library for accessing serial port in Java. It includes features like detecting when a USB-UART device like FTDI232 has been plugged into system, automatically identifies operating system and CPU architecture, does not require any installation, comprehensively documented, well tested and have a support/discussion group as well.

Introduction
The official API for serial communication in Java is the JavaComm API. This API is not part of the standard Java 2 version. Instead, an implementation of the API has to be downloaded separately. Unfortunately, JavaComm has not received much attention from Sun, and hasn't been really maintained for a long time. From time to time Sun does trivial bug-fixes, but doesn't do the long overdue main overhaul.

This section explains the basic operation of the JavaComm API. The provided source code is kept simple to demonstrate important point. It needs to be enhanced when used in a real application.

The source code in this chapter is not the only available example code. The JavaComm download comes with several examples. These examples almost contain more information about using the API than the API documentation. Unfortunately, Sun does not provide any real tutorial or some introductory text. Therefore, it is worth studying the example code to understand the mechanisms of the API. Still, the API documentation should be studied, too. But the best way is to study the examples and play with them. Due to the lack of easy-to-use application and people's difficulty in understanding the APIs programming model, the API is often bad-mouthed. The API is better than its reputation, and functional. But no more.

The API uses a callback mechanism to inform the programmer about newly arriving data. It is also a good idea to study this mechanism instead of relying on polling the port. Unlike other callback interfaces in Java (e.g. in the GUI), this one only allows one listener listening to events. If multiple listeners require to listen to serial events, the one primary listener has to be implemented in a way that it dispatches the information to other secondary listeners.

Download
Sun's JavaComm web page points to a download location. Under this location Sun currently (2007) provides JavaComm 3.0 implementations for Solaris/SPARC, Solaris/x86, and Linux x86. Downloading requires to have registered for a Sun Online Account. The download page provides a link to the registration page. The purpose of this registration is unclear. One can download JDKs and JREs without registration, but for the almost trivial JavaComm Sun cites legal and governmental restrictions on the distribution and exportation of software.

The Windows version of JavaComm is no longer officially available, and Sun has - against their own product end-of-live policy - not made it available in the Java products archive. However, the 2.0 Windows version (javacom 2.0) is still downloadable from here.

Installation
Follow the installation instructions that come with the download. Some versions of JavaComm 2.0 come with two installation instructions. The most obvious of the two instructions is unfortunately the wrong one, intended for ancient Java 1.1 environments. The information referring to the also ancient Java 1.2 (jdk1.2.html) is the right one.

Particularly Windows users are typically not aware that they have copies of the same VM installed in several locations (typically three to four). Some IDEs also like to come with own, private JRE/JDK installations, as do some Java applications. The installation needs to be repeated for every VM installation (JDKs and JREs) which should be used in conjunction with the development and execution of a serial application.

IDEs typically have IDE-specific ways of how a new library (classes and documentation) is made known to the IDE. Often a library like JavaComm not only needs to be made known to the IDE as such, but also to each project that is supposed to use the library. Read the IDE's documentation. It should be noted that the old JavaComm 2.0 version comes with JavaDoc API documentation that is structured in the historic Java 1.0 JavaDoc layout. Some modern IDEs are no longer aware of this structure and can't integrate the JavaComm 2.0 documentation into their help system. In such a case an external browser is needed to read the documentation (a recommended activity ...).

Once the software is installed it is recommended to examine the samples and JavaDoc directories. It makes sense to build and run one of the sample applications to verify that the installation is correct. The sample applications typically need some minor adaptations in order to run on a particular platform (e.g. changes to the hard-coded com port identifiers). It is a good idea to have some serial hardware, like cabling, a null modem, a breakout box, a real modem, PABX and others available when trying out a sample application. Serial_Programming:RS-232 Connections and Serial_Programming:Modems and AT Commands provide some information on how to set up the hardware part of a serial application development environment.

Finding the desired serial Port
The first three things to do when programming serial lines with JavaComm are typically


 * 1) to enumerate all serial ports (port identifiers) available to JavaComm,
 * 2) to select the desired port identifier from the available ones, and
 * 3) to acquire the port via the port identifier.

Enumerating and selecting the desired port identifier is typically done in one loop:

Once a port identifier has been found, it can be used to acquire the desired port:

Initialize a Serial Port
The initialization of a serial port is straight forward. Either individually set the communication preferences (baud rate, data bits, stop bits, parity) or set them all at once using the setSerialPortParams(...) convenience method.

As part of the initialization process the Input and Output streams for communication will be configured in the example.

Simple Writing of Data
Writing to a serial port is as simple as basic Java IO. However there are a couple of caveats to look out for if you are using the AT Hayes protocol:


 * 1) Don't use println (or other methods that automatically append "\n") on the OutputStream. The AT Hayes protocol for modems expects a "\r\n" as the delimiter (regardless of underlying operating system).
 * 2) After writing to the OutputStream, the InputStream buffer will contain a repeat of the command that was sent to it (with line feed), if the modem is set to echoing the command line, and another line feed (the answer to the "AT" command). So as part of the write operation make sure to clean the InputStream of this information (which can actually be used for error detection).
 * 3) When using a Reader/Writer (not a really good idea), at least set the character encoding to US-ASCII instead of using the platform's default encoding, which might or might not work.
 * 4) Since the main operation when using a modem is to transfer data unaltered, the communication with the modem should be handled via InputStream/OutputStream, and not a Reader/Writer.

Simple Reading of Data (Polling)
If you correctly carried out the write operation (see above) then the read operation is as simple as one command:

Problems with the simple Reading / Writing
The simple way of reading and/or writing from/to a serial port as demonstrated in the previous sections has serious drawbacks. Both activities are done with blocking I/O. That means, when there is the read or write method ( or   in the previous example) do not return, and the application comes to a halt. More precisely, the thread from which the read or write is done gets blocked. If that thread is the main application thread, the application freezes until the blocking condition is resolved (data becomes available for reading or device accepts data again).
 * no data available for reading, or
 * the output buffer for writing is full (the device does not accept (any more) data),

Unless the application is a very primitive one, freezing of the application is not acceptable. For example, as a minimum some user interaction to cancel the communication should still be possible. What is needed is non-blocking I/O or asynchronous I/O. However, JavaComm is based on Java's standard blocking I/O system, but with a twist, as shown later.

The mentioned "twist" is that JavaComm provides some limited support for asynchronous I/O via an event notification mechanism. But the general solution in Java to achieve non-blocking I/O on top of the blocking I/O system is to use threads. Indeed, this is a viable solution for serial writing, and it is strongly recommended to use a separate thread to write to the serial port - even if the event notification mechanism is used, as explained later.

Reading could also be handled in a separate thread. However, this is not strictly necessary if the JavaComm event notification mechanism is used. So summarize:

The following sections provide some details.

Introduction
The JavaComm API provides an event notification mechanism to overcome the problems with blocking I/O. However, in the typical Sun manner this mechanism is not without problems.

In principle an application can register event listeners with a particular  to be kept informed about important events happening on that port. The two most interesting event types for reading and writing data are
 * and

But there are also two problems:


 * 1) Only one single event listener per  can be registered. This forces the programmer to write "monster" listeners, discriminating according to the event type.
 * 2)  is an optional event type. Well hidden in the documentation Sun states that not all JavaComm implementations support generating events of this type.

Before going into details, the next section will present the principal way of implementing and registering a serial event handler. Remember, there can only be one handler at all, and it will have to handle all possible events.

Setting up a serial Event Handler
Once the listener is implemented, it can be used to listen to particular serial port events. To do so, an instance of the listener needs to be added to the serial port. Further, the reception of each event type needs to be requested individually.

Setting up a separate Thread for Writing
Using a separate thread for writing has one purpose: Avoiding that the whole application blocks in case the serial port is not ready for writing.

A simple, thread-safe Ring Buffer Implementation
Using a separate thread for writing, separate from some main application thread, implies that there is some way to hand off the data which needs to be written from the application thread to the writing thread. A shared, synchronized data buffer, for example a  should do. Further, there needs to be a way for the main application to determine if it can write to the data buffer, or if the data buffer is currently full. In case the data buffer is full it could indicate that the serial port is not ready, and output data has queued up. The main application will have to poll the availability of new space in the shared data buffer. However, between the polling the main application can do other things, for example updating a GUI, providing a command prompt with the ability to abort the sending, etc.

At first glance a  pair seems like a good idea for this kind of communication. But Sun wouldn't be Sun if the a piped stream would actually be useful. blocks if the corresponding  is not cleared fast enough. So the application thread would block. Exactly what one wants to avoid by using the separate thread. A  suffers from the same problem. Its blocking behavior is platform dependent. And adapting the classic I/O used by JavaComm to NIO is anyhow not a nice task.

In this article a very simple synchronized ring buffer is used to hand over the data from one thread to another. In a real world application it is likely that the implementation should be more sophisticated. E.g. in a real world implementation it would make sense to implement OutputStream and InputStream views on the buffer.

A ring buffer as such is nothing special, and has no special properties regarding threading. It is just that this simple data structure is used here to provide data buffering. The implementation is done so that access to this data structure has been made thread safe.

.     * @return       Amount of data actually written */    int write(byte data[], int off, int n) { if(n <= 0) return 0; int remain = n;        // @todo check if off is valid: 0= <= off

With this ring buffer one can now hand over data from one thread to another in a controlled way. Any other thread-safe, non-blocking mechanism would also do. The key point here is that the write does not block when the buffer is full and also does not block when there is nothing to read.

Usage of OUTPUT_BUFFER_EMPTY Event in Writing
Referring to the skeleton event handler presented in the section Setting up a serial Event Handler, one can now use a shared ring buffer from section A simple, thread-safe Ring Buffer Implementation to support the   event. The event is not supported by all JavaComm implementations, therefore the code might never be called. However, in case the event is available it is one building block for ensuring best data throughput, because the serial interface is not left idle for too long.

The skeleton event listener proposed a method, which could be implemented as it follows.

Reading of Data
The following example assumes that the data's destination is some file. Whenever data becomes available it is fetched from the serial port and written to the file. This is an extremely simplified view, because in reality one would need to check the data for an end-of-file indication to, for example, return to the modem command mode.

Modem Control
JavaComm is strictly concerned with the handling of a serial interface and the transmission of data over that interface. It does not know, or provide, any support for higher-layer protocols, e.g. for Hayes modem commands typically used to control consumer-grade modems. This is simply not the job of JavaComm, and not a bug.

Like with any other particular serial device, if the control of a modem is desired via JavaComm the necessary code has to be written on top of JavaComm. The page "Hayes-compatible Modems and AT Commands" provides the necessary basic generic information to deal with Hayes modems.

Some operating systems, e.g. Windows or certain Linux distributions provide a more or less standardized way how modem control commands for a particular modem type or brand are configured for the operating system. Windows modem "drivers", for example, are typically just registry entries, describing a particular modem (the actual driver is a generic serial modem driver). JavaComm as such has no provisions to access such operating-system specific data. Therefor, one either has to provide a separate Java-only facility to allow a user to configure an application for the usage of a particular modem, or some platform-specific (native) code needs to be added.

Overview and Versions
Due to the fact that Sun didn't provide a reference implementation of the JavaComm API for Linux, people developed RxTx for Java and Linux. RxTx was then further ported to other platforms. The latest version of RxTx is known to work on 100+ platforms, including Linux, Windows, Mac OS, Solaris and other operating systems.

RxTx can be used independent of the JavaComm API, or can be used as a so called provider for the JavaComm API. In order to do the latter, a wrapper called JCL is also needed. JCL and RxTx are usually packaged together with Linux/Java distributions, or JCL is completely integrated into the code. So, before trying to get them separately, it is worth having a look at the Linux distribution CD.

There seems to be a trend to abandon the JavaComm API, and using RxTx directly instead of via the JCL wrapper, due to Sun's limited support and improper documentation for the JavaComm API. However, RxTx's documentation is extremely sparse. Particularly, the RxTX people like to make a mess of their versions and package contents (e.g. with or without integrated JCL). Starting with RxTx version 1.5 RxTx contains replacement classes for the public JavaComm classes. For legal reasons they are not in the  package, but in the   package. However, the two currently available RxTx versions are packaged differently:


 * RxTx 2.0:RxTx version supposed to be used as a JavaComm provider. This one is supposed to have its roots in RxRx 1.4, which is the RxTx version before the  package was added.
 * RxTx 2.1:RxTx version with a full  package replacement for  . This version is supposed to have its roots in RxTx 1.5, where   support started.

So, if one wants to program against the original JavaComm API one needs


 * 1) Sun's generic JavaComm version. As of this writing this is in fact the Unix package (which contains support for various Unix versions like Linux or Solaris). Even when used on Windows, the Unix package is needed to provide the generic  implementations. Only the part implemented in Java is used, while the Unix native libraries are just ignored.
 * 2) RxTx 2.0 in order to have a different provider below the generic generic JavaComm version than the ones coming with the JavaComm package

However, if one just wants to program against the  replacement package, then
 * only RxTx 2.1 is needed.

Converting a JavaComm Application to RxTx
So, if you belong to the large group of people who have been let down by Sun when they dropped Windows support for JavaComm, you are in need to convert a JavaComm application to RxTx. As you can see from the above, there are two ways to do it. Both assume that you manage to install a version of RxTx first. Then the options are either


 * 1) Using RxTx 2.0 as a JavaComm provider
 * 2) Porting the application to RxTx 2.1

The first option has already been explained. The second option is surprisingly simple. All one has to do to port some application from using JavaComm to using RxTx 2.1 is to replace all references to  in the application source code with references to. If the original JavaComm application was properly written there is nothing more to do.

RxTx 2.1 even provides the tool  to perform the global replacement on a source tree under Unix. On other platforms such a global replacement is easy to do with IDEs supporting a decent set of refactoring features.