Joshua Nichols

As a recommendation to people who use /etc/portage/package.unmask regularly: You really ought to keep tabs on the thing that you’re unmasking… and once it has been unmasked, you should remove the entries from package.unmask.

Here’s a scenario: You heard about the new Java system being in package.mask, so you decided to try it. Cool. It has since come out of package.mask… but you still have it unmasked via /etc/portage/package.unmask. Now, we release a new revision of java-config that we put in package.mask for testing… and now you get the version that we are heavily tested, and it doesn’t quite work 100%…

So, please, be mindful of your package.unmasks.

So, one of the issues that has come up on occaision is that JAVA_HOME ends up pointing at, for instance, /opt/blackdown-jdk-1.4.2.03, instead of the shiny, recently unmasked Java 1.5 you installed and set as your VM of choice.

The reason for this has is legacy support. java-config-1 isn’t so smart. It determines what the current VM is based on JAVA_HOME in the environment. It expects it to be living under /opt/${P}, where ${P} is the name and version of the VM.

The workaround we found is described in the upgrade guide. However, the problem is, that java-config-1 would then no longer be able to know what VM is currently being used, which is a bit problematic.

Thanks to the hax0ring skills, we now have a patched java-config-1 that uses VMHANDLE instead of JAVA_HOME. VMHANDLE is the token by which a VM is identified with the new Java system, and happens to be in the environment.

We have added a new entry for /etc/profile.d, which uses the prescribed way of setting JAVA_HOME from the upgrade guide. This means that JAVA_HOME would be set correctly most of the time. The only time it wouldn’t be accurate would be if you did not have a user vm set (JAVA_HOME points at /etc/java-config-2/current-system-vm), and then you set a user vm, JAVA_HOME would point at current-system-vm, instead of ~/.gentoo/java-config-2/current-user-vm. Running ‘source /etc/profile’ though, remedies the problem.

Perhaps java-config-2 can be updated to give notice that you need to source /etc/profile ? Or perhaps it can directly update JAVA_HOME… The only problem with the the latter point ist hat it would probably only fix JAVA_HOME in the current terminal, and not in all the other open terminals.

I’ve added revision bumps of java-config to the tree that work as described here. Because of the changes though, I’ve put them in package.mask while we finish testing the new way of handling JAVA_HOME. The package.mask entry is below, for those interested in testing and providing feedback:

# Joshua Nichols <nichoj@gentoo.org> (02 Aug 2006)
# Testing new versions of java-config, which will let us remove
# JAVA_HOME from env, which points to generation-1 system vm
=dev-java/java-config-1.3.0-r3
=dev-java/java-config-2.0.26-r6

Assuming there aren’t any issues while testing, I expect to unmask it in a day or two. At that point, I’ll also update all the env.d files for the VMs, so they no longer export JAVA_HOME (note: this is different from the profile.d, which is actually overwriting the JAVA_HOME originally being set by the env.d file).

The answer, as of this moment, is most definitely. There are two reasons for this at the moment.

1. there are still packages which don’t compile when built with Java 1.5. In these cases, the new Java build system switches to a 1.4 JDK for building.
2. it is needed for packages that haven’t been migrated to use the new Java system. This is necessary, otherwise we encounter the same exact problems that prompted the initial package.mask of Java 1.5.

Just to recap why Java 1.5 was package was package.mask’d…. It really comes down to backwards compatibility, and that Java 1.5 isn’t 100% backwards compatible with Java 1.4. Things that ran with Java 1.4 should still run with Java 1.5… but some things that once built with Java 1.4 no longer compile with Java 1.5. Specifically:

1. There’s a new reserved keyword with Java 1.5, enum. So where you used to be able to use enum as a variable name, you no longer can in 1.5.
2. A number of new methods have been introduced to some abstract classes and interfaces. If a particular implementation doesn’t implement the new methods, then it won’t compile against Java 1.5

Now, say you’re using Java 1.5 to build all your packages, and then you come up to a package that can’t compile for one of these reasons? It’d fail for one. But then you may try with Java 1.4… and all is well until you see a compile error: UnsupportedClassVersionError. What does this mean though?

Well, each major release of Java has it’s own version of bytecode. Bytecode is forward compatible, ie run 1.4 bytecode in 1.5, but not backwards compatible. If you try to use 1.5 bytecode in 1.4, you will see the infernal UnsupportedClassVersionError. This happens because, since you were using Java 1.5, the dependencies of the package you were just trying to compile with 1.4 were built with 1.5, and the default behavior is to build the highest possible bytecode.

So, these are the reasons Java 1.5 was package.mask’d, and why we need to use Java 1.4 for building unmigrated packages.

But how does the new Java system help?

• We get around the problem of enum by specifically telling the compiler to use source=”1.4″, where enum is a valid variable name.
• We get around the changes in API by switching to a version where it can be compiled, until such time as the package can be properly patched.
• Additionally, we make javac compile the lowest possible bytecode, so it can be used by as many other packages as possible.

I’m pleased to announce that www-servers/tomcat-5.5 and dev-util/eclipse-sdk-3.2_rc7 have been added to portage over the long weekend.

We strongly encourage users to update to tomcat 5.5, and refrain from using earlier versions, as recommended by upstream. 5.0 and earlier will likely be removed in the future, and have put it in package.mask. Don’t worry though, it won’t go away right away, so users should have time to migrate.

As for eclipse, there are few few changes between 3.2_rc7 and the official 3.2 release. Once we verify our patchset works against 3.2, you can expect to see the official release added to portage. It is in package.mask for testing, and until we get the final release out.

The Gentoo/Java Team is proud to announce the release of the updated Java system. This means that Java 1.5 is finally unmasked.

It is currently in ~arch. To begin using it, follow the upgrade guide:

http://www.gentoo.org/proj/en/java/java-upgrade.xml

Here are some highlights of the new system

* Ability to switch the current VM on the fly
* Changes to the user and system VM take effect immediately, and no longer are tied to the shell environment, which means you no longer have to run env-update followed by source /etc/profile when you switch the system VM
* Now has the concept of a “build VM”, which is used to emerge packages, and is configured independently of the system VM
* For each version of Java, ie 1.3, 1.4, 1.5, etc, the build VM can be configured as to which vendor and version of a VM to use
* The VM at emerge time will be switched on the fly according to its configuration, as well as the dependency of the package. For example, some packages won’t compile with 1.5. In these cases, a 1.4 VM will be used at build time
* Java packages which build with ant will have their build.xml rewritten at build time in order to ensure that the correct version of Java bytecode is compiled
* Java 1.5 has been unmasked, and we will be ready for Java 1.6 when it is released this fall.

Just to keep people updated about how the new Java stuff is going… it’s going pretty good 😀

A lot of smaller kinks have been worked out, better documentation, more useful error / warnings, and such forth. All in all, there has been a lot of good input and feedback.Most of the issues users have been running into is not having everything unmasked and keyworded properly. With unmasking,the former will no longer be an issue.

So, I hope to go forward with unmasking over the long (read: 4 1/2 days) weekend. At that point, we’ll begin switching packages over to using the new eclasses, doing version bumps, bug squashing, and all that jive.

I’m pleased to announce that the new Java hotness has finally hit the tree 🙂 It’s currently in package.mask, but I expect to unmask it in the next few days.

To begin using it, you will need to add the appropriate entries to /etc/portage/package.unmask, and then follow the upgrade guide.

For those not familar with that this means, here are some highlights:

• Ability to switch the current VM on the fly
• Changes to the user and system VM take effect immediately, and no
  longer are tied to the shell environment (ie no more running env-update &&
  source /etc/profile after switching the sytem VM)
• Now has the concept of a
  

The road to unmasking Java 1.5 has been long and arduous. There have been enough issues to warrant it’s own FAQ and an entire redesign of the way we handle Java packages.

I won’t go into detail about the specific problems incurred, as the FAQ should sufficiently cover them.

Nearly a year ago, work was begun on a new build system (ie new eclasses and a new version of java-config). About six months later, we had a new fully featured, flexible build system. Highlights include build-time VM switching, build-time build.xml rewriting, and a more flexible means of storing/setting the system and user VM.

Unfortunately, there were enough incompatibilities between this new system and the old one that we couldn’t just drop it in the tree as is. A few months were needed to determine how to best migrate to the new system, in such a way that it could be done incrementally, and it could be properly tested (ie be in ~arch). Notes about the method we choose can be found on our wiki.

Since then, a lot of work has been done to implement the migration system, as well as code and documentation cleanup.

So now, we (kartlk and I), have deemed the new system ready for prime time. This coming week, we will be finishing any last minute cleanups, and bringing in any packages that require 1.5. Once that is complete, we will be announcing the changes on the developer mailing for public scrutiny/input. Assuming that there are no issues that crop up, the new system would be brought into the main tree a week or so later.

You may begin rejoicing now.

As has been previously announced, we are officially part of Google’s Summer of Code. I’ve decided to step up to be a mentor, and in particular, to mentor projects related to Java on Linux.

A few ideas follow. The 1a and 1b are listed already on our SoC page to an extent, but the idea is elaborated further here. If you plan on proposing one of these ideas, I really would expect you to even further elaborate.

1a)
Summary:

Build most of our Java packages with free (libre) virtual machines and free implementations of public APIs.

Background:

Currently, we really only support using a proprietary virtual machine (ie sun, blackdown, ibm, etc), because packages are likely to fail for various reason with the open ones.

For many open apis, such as javamail, java activation framework, etc, we have binary packages of Sun’s proprietary implementations. In a number of cases, there are open implementations. However, our packages compile against and run using the proprietary implementations.

For reasons why one would want to be using Free Java, see the article on the Java trap.

As for a practical reason, use of proprietary packages from Sun and IBM can annoying for the end user, because in both cases, it requires placing a fetch restriction on the distfiles. To the end user, this means that an emerge gets halted until they agree to a license, and download the files.

Goals:

• Build/run all/most packages using free virtual machines
• Build/run all/most packages against free implementations of public APIs
• Might want to target specific big name packages, like eclipse, azureus, tomcat.
• Be able to select between different implementations of the same apis

Tasks:

• Work with upstream of packages that use propertary classes from the virtual machine (ie com.sun.*, sun.*)
• Work with upstream of virtual machines and packages when packages don’t compile or run with using free java
• Find and package open implementations of public APIs

Hurdles:

• Lots of Java packages (300+). It is unknown how many will need to be patched.
• Upstream might not care about free java
• Might not be open implementations of all APIs

1b)

Summary:
Native compiling with gcj

Background:

GNU Compiler for Java (gcj) allows for compiling Java code to native machine code, as opposed to Java bytecode. Fedora/Redhat has done this to an extent for some time. It’d be great to be able to do this under Gentoo as well.

Tasks:

• Create a ‘JDK’ which uses gcj as a backend. In this case, JDK really means providing a javac, javadoc, jar, etc
• Have a way of keeping track of native bytecode
• Integrate methods for native compiling into the current build system for building Java packages on Gentoo (ie eclasses)
• Target native compilation of high profile applications, ie azreus, eclipse, freemind

2a)

Summary:

Build maven entirely from source.

Background:

We currently only provide binary packages of maven, due to the fact that it is a bootstrapping nightmare because of the monolithic nature of the build process. The build system, at its basic level, does the following:

• Build the ‘core’ maven functionailty using ant
• Build the ‘core’ maven functionality using maven
• Build all the plugins using maven

The problem with building all the plugins at once is that there are a ton of plugins, with many, many dependencies. Because of these dependencies, it would make it impractical to have a monolithic package, which would be used to build other packages.

Goals:

• Have modular packages for maven
• Have one package per plugin
• Have a ‘core’ maven package
• Have a ‘minimal’ maven package, which is core + minimal packages for compiling, javadocs, and jarring
• Have a ‘full’ maven package which is all the plugins
• Would need to work out a way to make the build much more modular

2b)

Summary
Build packages using maven

Background
Many packages, particular Apache projects, have been switching over

The main issue is getting jars from a maven repository. The normal behavior of maven is to download dependencies from a maven mirror. First
of all, things at build time should not be using the network. Secondly, dependencies should be provided by jars from packages that have previously been emerged.

Another issue is that maven 2 repositories contain metadata (specifically, pom.xml files) about the packages contained within. This is normally fetched from the repository, but since we don’t want to be pulling from there, the question remains open as to where the metadata is gotten from and how its stored on the system.

Goals:

• Have a system to have maven use jars that we have built through portage
• Update packages to use maven when possible. These should have fine grain control to the plugins that are available at build time. Packages should depend on the minimal install, and whatever extra plugins they might need, and only those should be available at build time.

Possible solutions for dealing with maven repositories:
A) Create a maven-repository like structure, and have it populated with jars from the system
B) maven (2.0 at least) has a mechanism for providing alternative ways of fetching dependencies. We could then create an alternative way that’s backed by portage.
C) Find a way to tell maven not to fetch dependencies, and hope that there are the appropriate classes available on the classpath

So, I’ve recently joined the dev-tools team in order to help maintain dev-util/eclipse-sdk. The ebuilds themselves haven’t been touched much lately, aside from the occasional version bumps, and so have been in need of a little love.

I’ve just added 3.1.2 to the tree recently. In terms of upstream changes since 3.1.1, there isn’t too much to write home about. In terms of the ebuilds, I’ve manage clean up a number of things that would make mere mortals tremble. This isn’t to say the ebuilds are quite ideal yet, but it is certainly a step in the right direction.

In the coming weeks, I hope to continue sanitizing the Eclipse ebuilds, in addition to getting ready for the 3.2 release, which just entered release candidate last week.