Author: agriffis

requoting in bash

While working with Uberlord on the Gentoo netscripts, I had a chance to review our requoting function. Here it is:

function requote {
        local q=\'
        set -- "${@//\'/$q\'$q}"        # quote inner instances of '
        set -- "${@/#/$q}"              # add ' to start of each param
        set -- "${@/%/$q}"              # add ' to end of each param
        echo "$*"
}

The purpose of this function is to make the arguments suitable for evaluation by the shell. This happens whenever you need a single variable that will correctly evaluate to multiple arguments without distorting the original content. In the case of passing the variable to an external program, you can’t even use bash arrays, so requoting is the only option. Here’s a simple example:

connect=$(requote chat -v '' ATZ OK ATDT318714 CONNECT '' ogin: ppp word: '<pa$$w0rd!>')
pppd connect "$connect"

In this case it’s important that quoting is preserved so that the special characters in the password, including the angle brackets that could be misinterpreted as I/O redirection, are passed to the chat program safely.

Some time after writing this function, I learned about bash printf’s %q, which “means to quote the argument in a way that can be reused as shell input” (from bash built-in help). It turns out it isn’t very easy to update our requote function to use it because, right or wrong, it drops empty arguments…

$ printf "%q " one '' '$<$'; echo
one  \$\<\$

This is the best I could come up with for now, which unfortunately needs a bash loop. If somebody comes up with a better implementation using printf %q, I’d be interested in knowing it!

function requote {
    declare arg
    for arg; do 
        arg=$(printf '%q' "$arg")
        printf '%s ' "${arg:-''}"
    done
}

comment contest

This contest stems from David’s post regarding a script to comment out words in a C file. The general applicability of this script is questionable, but it’s still a fun exercise. Here are the instructions:

  1. Write a command-line filter which will comment out all matches of a pattern given on the command-line, reading a C program from stdin and filtering to stdout. Whether the pattern is a literal, glob, regular expression is up to the contestant.
  2. The filter should ignore matches that are already contained within a comment (including C++ style comments) or double-quoted string.

The existing gawk implementation on David’s blog is an interesting starting point, though it doesn’t fulfill all the requirements above.

Samples:

  1. simple: input / output (“add”)
  2. wacky: input / output (“wacky”)
  3. deranged: input / output (“int”)

bash default/alternate values

Many people are familiar with the bourne shell constructs for default or alternate values, but there’s a detail that I suspect most people don’t know: the semantic difference between the bash and bourne shell syntax. Here is the summary:

${var-word}    # insert $var, or default value if var is unset
${var+word}    # insert alternate value if var is set
${var:-word}   # insert $var, or default value if var is unset or null
${var:+word}   # insert alternate value if var is set and non-null

Do you see the difference between these? The first two are classic bourne shell syntax, the second two are bash additions (they’re also present in most bourne shell derivatives, including posix shell). The semantic difference is that the first two only check for the existence of the variable, but the second two additionally check that it has non-zero length.

Here is an example of when the older bourne shell syntax is useful. Imagine you have a configuration file in shell syntax, so you’re going to source it into the current shell to pick up the variable settings. Something like this, which happens to work because Gentoo’s /etc/make.conf is valid shell syntax:

source /etc/make.conf

You might like to know if GENTOO_MIRRORS is set by the environment or by the configuration file, even if it was set to a null string. If it is unset after consulting both sources, we’ll call that an error. Here’s one way you could do it:

if [[ -z ${GENTOO_MIRRORS+set} ]]; then
    source /etc/make.conf
fi
if [[ -z ${GENTOO_MIRRORS+set} ]]; then
    echo "Error: GENTOO_MIRRORS not found in environment or /etc/make.conf" >&2
    exit 1
fi

mozilla braindump

On Friday, for better or worse, I marked stable the current versions of the mozilla-* ebuilds. These were pushed to stable quickly because of security issues (98855 98846) in the older versions, plus extension compatibility problems in mozilla.org’s first attempt at releasing a security update. Here are the versions that went stable:

mozilla-launcher-1.41.ebuild
mozilla-1.7.10-r1.ebuild
mozilla-firefox-1.0.6-r2.ebuild
mozilla-thunderbird-1.0.6-r2.ebuild
mozilla-bin-1.7.10.ebuild
mozilla-firefox-bin-1.0.6-r1.ebuild
mozilla-thunderbird-bin-1.0.6-r1.ebuild

If that seems like a long list to you, well it does to me too! Along with the security fixes, these ebuilds carry along a large number of improvements:

  • post-installation chrome registration is handled by mozilla-launcher’s update_chrome() and the -register argument to the binaries. This is a huge improvement over dropping in an “initialized” tarball during src_install and hoping it worked. (It also starts us on the road to including extensions as separate ebuilds in portage, which was completely impossible before.)

  • The multilib-awareness of mozilla-launcher and the ebuilds has been improved via the MOZILLA_LIBDIR variable that is set in the stubs. This obviates the old multilib patch that was being applied to mozilla-launcher and allows us to change the installation directories far more easily.

  • Speaking of changing the installation directories, we now install to /usr/lib/mozilla-firefox and /usr/lib/mozilla-thunderbird instead of /usr/lib/MozillaFirefox and /usr/lib/MozillaThunderbird respectively. This was a purely aesthetic change, but one that certainly demonstrates that it can be done!

  • The mozilla-launcher stub in /usr/bin is installed by install_mozilla_launcher_stub() from mozilla-launcher.eclass. That makes installation of the stub generic and rescues us from some of the duplication in the ebuilds.

  • Plugins shouldn’t clash now because we’re no longer moving them around and creating symlinks, instead we’re setting MOZ_PLUGIN_DIR in mozilla-launcher.

  • The ebuilds have been re-organized and synchronized, which means that they should be easier to maintain in the future.

Maintaining the mozilla stuff takes a lot of effort. A huge amount of effort. More effort than I can give it right now… This will be my last weekend working on the mozilla ebuilds. Starting Monday, I intend to leave their care and feeding to the rest of the mozilla team. It seems like a good time since this week has seen so many improvements and we’re mostly stable now.

Since that’s the case, here are some things that have been on my mind, which I hope the rest of the mozilla team will see through. Terminology note: I use “browser” to refer to any of mozilla, firefox or thunderbird.

Extensions in portage

Ideally, it would be nice to build enigmail (and possibly other extensions) as separate ebuilds in portage. I spent quite a while on this problem and was ultimately stumped. First, I tried to build enigmail by massaging the Makefiles to use include files, tools and libs from ${MOZILLA_FIVE_HOME}. Gradually I prodded it along, but finally surrendered to the notion that enigmail wants to be built in an unpacked mozilla or thunderbird tree.

So I pursued using a stripped-down thunderbird ebuild, based on instructions on the enigmail site, especially the later paragraphs that talk about building it more quickly. I got quite a long way with that, but could never get it actually working once it was installed. Finally, I came across a message that indicated that installing global extensions in 1.0.x is difficult/broken. I also found suggestions that 1.1 will make global extension installation easier. My personal conclusion: Wait for thunderbird-1.1 to build extensions in portage.

Some people have been very unhappy that I took enigmail out of the thunderbird build in the first place. The reason it was taken out was that the build wasn’t working in my testing, even on thunderbird-1.0.2. For some reason the build process that we’ve been using for enigmail in mozilla and thunderbird stopped working for thunderbird. It seems to still work for mozilla, so I’ve left it there.

Note to the mozilla team: Nathan Adams says he has something working. The only difference I can see between his effort and mine is that he’s using make instead of emake. It would be worth following up on this, trying to get it back into the thunderbird ebuild, until it can be properly split out for thunderbird-1.1.x

Unmerge and -unregister

Recent versions of mozilla-launcher sport a new -unregister argument and associated remove_chrome() function. The point is to be called in pkg_prerm() so that the package can be completely uninstalled instead of leaving the generated bits on the filesystem. The ebuilds haven’t been modded yet for that, though, so there’s a task for somebody.

Patches

One of the reorganization steps I took recently was to sort the patches in src_unpack() into architecture-specific, general compilation issues, and behavioral changes. Furthermore, I made the sorting mostly the same between the packages, so it should be possible to, for example, do a meaningful diff between the current versions of the browser ebuilds.

A couple things I noticed: First, there is an alpha stubs patch being applied in all the ebuilds. Is that still necessary? Is the patch the same in all the ebuilds, and therefore could it be moved to ${DISTDIR} as I’ve done for some of the other patches?

Second, not all of the architecture patches are applied in all the ebuilds. For example, there is an hppa patch that is applied inconsistently. What’s the story with that?

There are gcc4 patches being applied all over the place. Are those in reality a common patch that could be applied from ${DISTDIR}?

Presently there are a few bugs open recommending cherry-picking patches from Fedora and Ubuntu. I’d urge you guys to be careful about that. Some of the patches are worthwhile and some are not. The closer you stay to upstream sources, the easier it will be going forward. Usually Gentoo has made a point of staying close to upstream, and I think that’s the right route for the browser ebuilds. Just my humble opinion…

A final thought on patches: Right now many of the patches are applied from either ${DISTDIR} or ${FILESDIR}. It might be advantageous to roll the current set into a versioned tarball that is common for the browsers. You could keep the tree of patches in subversion along with a script for generating the tarball and putting it on the mirrors. That would reduce the number of downloads people are doing per-ebuild.

Plugins

The plugins problem is this: many packages (for example acroread and java) provide plugins written to the netscape API, which is supported by the browsers. These plugins aren’t found by the browsers by default, because they only look in ${MOZILLA_FIVE_HOME}/plugins

The original solution to this problem was architected by Azarah in nsplugins.eclass: The plugin ebuilds would call inst_plugin to create a symlink from the plugin to /usr/lib/nsbrowser/plugins. In src_install, the browser ebuilds would move their own plugins to /usr/lib/nsbrowser/plugins and make ${MOZILLA_FIVE_HOME}/plugins be a symlink to that dir.

This worked for a time, but was relatively difficult to maintain in the ebuilds, and had the negative effect that the files in /usr/lib/nsbrowser/plugins were owned by multiple packages.

Recently I attempted to solve this problem a different way: First I created a patch which I thought would add /usr/lib/nsbrowser/plugins to the default search path for plugins. Second I removed the code from the ebuilds which would move the browser-installed plugins. I was hoping this would allow both browser-installed and third-party-installed plugins to be found automatically without all the hackery.

However the patch didn’t work (I don’t know why yet…) so instead I resorted to setting MOZ_PLUGIN_PATH=/usr/lib/nsbrowser/plugins in mozilla-launcher. For all intents and purposes, this works! It only has a couple small issues, such as removing the user’s ability to set their own MOZ_PLUGIN_PATH.

Whatever solution the mozilla team chooses for this problem, I’d suggest first trying to make the patch work. It can’t be that hard, I’m just out of steam! If that doesn’t work, then consider sticking with the MOZ_PLUGIN_PATH solution. The least desirable solution is one that moves plugins around and makes more symlinks, because that perpetuates the old nsplugins.eclass complexity and makes full unmerge of the browser difficult.

Adjusting Exif Date/Time with jhead

Working on some digital photos today, I found the Date/Time field in the Exif header was set to EDT, because that’s how my camera was set (I’ve changed it now to UTC). This is problematic because I recently took a trip to Italy, and of course the times on those photos don’t make sense:

$ jhead IMG_0356.JPG | grep ^Date
Date/Time    : 2005:09:20 09:46:49

To get the times changed to UTC on my existing pictures, I used the following sequence:

$ jhead -ta+4:00 *.JPG
$ TZ=UTC jhead -ft */*.JPG

The first command advances the Exif Date/Time on all the pictures by 4 hours, which is the difference between EDT and UTC. The second command copies the Exif Date/Time to the file’s mtime, pretending that we’re presently in UTC so that it doesn’t attempt to adjust. The final result is:

$ jhead IMG_0356.JPG | grep ^Date
Date/Time    : 2005:09:20 13:46:49
$ ls -l IMG_0356.JPG
-rw-r--r--  1 agriffis agriffis 3.9M Sep 20 09:46 IMG_0356.JPG

qemu notes

I’ve been using qemu lately to test kernel changes. In particular I’m using qemu-0.7.2 running x86 virtual machines on an amd64. This configuration seems to work really well, particularly once I remembered to “echo 1 >/proc/sys/kernel/sysrq” in the host OS’s rc.local (doh!)

Today I switched from using qemu’s user mode network stack to using the tun/tap interface with masquerading. To that end, my qemu startup script now looks like this:

#!/bin/bash
# qemu.sh
sudo bash -c 'echo 1024 > /proc/sys/dev/rtc/max-user-freq'
#sudo modprobe kqemu
qemu \
    -kernel linux-2.6-qemu/arch/i386/boot/bzImage \
    -append 'console=ttyS0 root=/dev/hda ro clock=pit' \
    -hda root_fs.fc-4-base.pristine.20051026 \
    -nographic \
    -n $PWD/qemu-ifup \
    "$@"

and qemu-ifup looks like this:

#!/usr/bin/sudo bash
/sbin/ifconfig $1 172.20.0.1
iptables --table nat --flush
iptables --table nat --append POSTROUTING \
    --source 172.20.0.0/16 --destination ! 172.20.0.0/16 \
    --jump MASQUERADE
[[ -e /proc/sys/net/ipv4/ip_forward ]] && \
    echo 1 > /proc/sys/net/ipv4/ip_forward

This gives me outgoing networking from the virtual machine, plus incoming networking from the host OS. The virtual OS uses 172.20.0.2 on its virtual adapter.

Gentoo Council nomination

Since there is no public gentoo-core archive, here’s a transcription of my nomination for the Gentoo Council. For the sake of reference, the message-id is <20050628203320.GA20210@olive.flatmonk>.

Grant Goodyear wrote: [Tue Jun 28 2005, 02:45:53PM EDT]
> I’m formally nominating vapier and agriffis as council
> members. They are both skilled developers who have a deep understanding
> both of how Gentoo actually works and how it probably should work.

Thanks! I’m happy to accept this nomination. For those that I haven’t had the pleasure of getting to know yet, here is an introduction for you:

My history with Gentoo goes back to 2001. I had been developing for other distros prior to that and found my way to Gentoo by reading one of Daniel Robbins’s DeveloperWorks articles. I jumped on IRC and helped solve some early portage problems, particularly “try vs. die” (which is famous only in my own mind) and file corruption issues that were the result of writing directly on top of existing files instead of moving the old ones aside first. Daniel asked me to be a developer and I dove in head first.

Since then I’ve worn a number of hats. I’ve worked on a lot of architecture issues, contributing to alpha and ia64 in particular. I’ve maintained or co-maintained a number of packages, including bash, mozilla, firefox, thunderbird, mutt, mercurial, vim, xcut, xautomation, ctrlproxy, lilypond, librep, rep-gtk, sawfish, plucker, subversion, jpilot and baselayout. I also wrote mozilla-launcher, epm, echangelog, ekeyword, eviewcvs and votify. I didn’t originally author keychain, but all of the code in it is mine at this point. 😉 Finally, I was one of the original trustees, but I chose not to run this year because I believe my skills are more suited to making technical contributions.

Technical contributions are important for anybody wanting to facilitate cross-project decisions. Sometimes a good technical solution will allow two projects to forge ahead in peace when they were previously at odds. When that isn’t possible, it’s important to get along with other developers and find a way forward. I believe I get along with my fellow developers. I have my own opinions, but I work hard to understand the perspectives of other developers, and I’m not shy about backing down and admitting I was wrong.

I look forward to serving on the Gentoo Council if I am elected.