Libreoffice state in Gentoo and openSUSE

Lets start with the happy stuff we have in Gentoo.

Andreas (dilfridge) created the binary package for 4.1.3.2 which means we have latest upstream provided as binary for stable users. He also managed to get Patrick (bonsaikitten) to provide HW where we can build the testing tree based version too. So stay tuned the binary will soon TM arrive for testing tree.

There seem to be some issues with portage lately wrt subslots and tracking the rebuilds so if you have build collisions while emerging this just try to mess with –backtrack option and put there some value bigger than 30.

I spent some time updating the dependencies and prepped the switches for 4.2 release so we should be on good track there. But as always it would be nice to have more people in the team working on individual bumps (it is just copy from one version to another most of the time) but I can’t be everywhere and sometimes I overlook things. So if you are interested just drop by #gentoo-office irc channel and we can get you hooked. Later on the run you can became also developer as this position is suited for contributors who want to get involved more.

Now the other half about openSUSE state

Here goes the sad part. As libreoffice team moved to collabora it left openSUSE without any packager for the suite. Part of the libreoffice team responsibility was also maintaining the hunspell dicitionaries which is even worse as it impacts everybody who uses the hunspell for spellchecking (kde/gnome/…).

So we need people interested in maintaining all the stuff updated and working. Most of the tasks now are quite complex as the packages are written with compatibility for sle10 in mind. We can finally give up on that and start cleanups (which I intend to do during my free time at some point) but I could use help a lot. So if you are interested in libreoffice future on openSUSE and know how to write “bit” complex spec files just drop me a mail or ping me on irc so we can talk about it.

Some sort of plan I have with this:
1) Reduce the repositories used. As I am all alone working on this I will remove all the stable/unstable stuff and provide only :Factory repo which will be built against Factory and supported openSUSE versions (12.2/12.3/13.1). Unstable will be utilized only for testing of beta releases before next minor bump if needed (4.1.0 -> 4.2.0).
2) Cleanup all the stuff in the libreoffice package to have it maintainable with less dificulity (now it takes quite few hours just to bump from 4.1.2 to 4.1.3 which is bugfix release and that is not optiomal).
3) Probably solve the dictionaries situation for good by using only the dictionaries from libreoffice repo. Where we can say to people if they want support they just need to git commit it to libreoffice git repository. Which has nice sideefect of improving spellchecking for english and others.

Installing openSUSE Factory on laptop

First let me explain why openSUSE Factory and not in fact Gentoo.

Factory points:

  • + Much faster install/update
  • + Better out of the box experience – no need to fiddle with everything to get it running
  • + I can throw stuff to OBS and get it back fast as a binary when I am on battery
  • - It has more issues than Gentoo stable in some areas
  • - Unable to remove things like pulseaudio easily

Gentoo points:

  • + More stable on stable – this will change a lot in near future if everything goes right for openSUSE
  • + Faster system (matter of few percents)
  • + Possibility to remove stuff you really don’t want
  • - Compilation of everything – I update/install on battery often
  • - Smaller chance to fix anything while I am traveling as I have to compile it on the box

Continue reading

Working in openSUSE Team

Some of you probably noticed I work at SUSE for a while. I was working on SLE maintenance and thus it was bit hidden work, not really in touch with upstream and other projects. This all changes starting this week because 1.7. I changed my post so I will be one of the few working on openSUSE.

I have high hopes for what we can achieve in the team and I really look forward on all the coding and fixing possibilities ahead. Currently I am mostly focusing on the great beach party^W^Wconference where I will be giving few talks and handle creative autootols workshop. Writing those presentations is quite task, specially for the workshop, because I am used for 90 min lectures and this one will be even 30 minutes longer and I really want to have it interesting and educational.

After this party there is even more fun ahead for me. I will be focusing my breaking skills on openSUSE:Factory trying to improve the next release so I could happily keep using it on my and my family laptops.

As an endnote please do not expect from me to magically change all the bugs you might have on openSUSE, tho I will try my best to have it working well. :-)

Prague Installfest results

Last weekend (2.-3.3. 2013) we had a lovely conference here in Prague. People could attend to quite few very cool talks and even play OpenArena tournament :-) Anyway that ain’t so interesting for Gentoo users. The cool part for us is the Gentoo track that I tried to assemble in there and which I will try to describe here.

Setup of the venue

This was easy task as I borrowed computer room in the dormatories basement which was large enough to hold around 30 students. I just carried in my laptop, checked the beamer works. Ensured the chairs are not falling apart and replaced the broken ones. Verified the wifi works (which it did not but the admins made it working just in time). And for last brought some drinks from main track so we do not dry out.

The classroom was in bit different area than the main track I tried to put some arrows for people to find the place. But when people started getting in and calling me where the hell is the place I figured out something is wrong. This pointy was then adjusted but still it shows up that we should rather not split of the main tracks or ensure there are HUGE and clear arrows pointing in directions where people can find us.

Talks

During the day there were only three talks, two held by me and one that was not on the plan done by Theo.

Hardened talk

I was supposed to start this talk at 10:00 but given the issue with the arrows people showed up around 10:20 so I had to cut back some informations and live examples.
Anyway I hope it was interesting hardened overview and at least Petr Krcmar wrote lots of stuff so we maybe will se some articles about it in czech media (something like “How I failed to install hardened Gentoo” :P).

Gentoo global stuff

This was more discussion about features than talk. The users were pointing out what they would like to see happening in Gentoo and what were their largest issues lately.

From issues people pointed out the broken udev update which rendered some boxes non-bootable (yes there was message but they are quite easy to overlook, I forgot to do it on one machine myself). Some sugesstions went for genkernel to actually trigger rebuild of kernel right away in post stage for user with the enabled required options. This sounds like quite nice idea, as since you are using genkernel you probably want your kernel automatically adjusted and updated for the cases where the apps require option additions. As I am not aware of the genkernel stuff I told the users to open bug about this.

Second big thing we were talking about were binary packages. The idea was to have some tinderbox which produce generic binary packages available for most useflag variants. So you could specify -K and it would use the binary form or if not provided compiled localy. For this the most work would need to be done on portage side because we would have to somehow figure out multiple versions of the same package with different enabled uses.

Infra talk

Theo did awesome job explaining how infra uses puppet and what services and servers we have. This was on-demand talk which people that were on-site wanted.

Hacking — aka stuff that we somehow did

Martin “plusky” Pluskal (SU) went over our prehistoric bugs from 2k5 and 2k6 and created list of cantfix ones which are no longer applicable or are new pkg requests with dead upstream. I still have to close them or give him editbugz privs (this sounds more like it as I am lazy like hell, or better make him developer :P).
Ondrej Sukup (ACR) attending over hangout worked on python-r1 porting and I commited his work to cvs.
Cyril “metan” Hrubis (SU) worked on crossdev on some magic avr bug I don’t want to hear much about but he seems optimistic that he might finish the work in near future.
David Heidelberger worked first on fixing bugs with his lappy and then helped on the bug wrangling with Martin.
Jan “yac” Matejka (SU) finished his quizzes and thus he got shiny bug and is now in lovely hands of our recruiters to became our newest addition to the team.
Michal “miska” Hrusecky (SU) worked on update of osc tools update to match latest we have in opensuse buildservice and he plans to commit them soonish to cvs.
Pavel “pavlix” Simerda (RH) who is the guy responsible for latest networkmanager bugs expressed his intentions to became dev and I agreed with him
Tampakrap (SU) worked on breaking one laptop with fresh install of Gentoo, which I then picked up and finished with some nice KDE love :-)
Amy Winston helped me a lot with setup for the venue and also kept us with Theo busy breaking her laptop, which I hope she is still happily using and does not want to kill us, other then that she focused on our sweet bugzie and wrangling. She seems not willing to finish her quizzes to became full developer, so we will have work hard on that in the future :-)
And lastly I (SU) helped users with issues they had on their local machines and explained how to avoid those or report directly to bugzie with relevant informations and so on.

In case you wonder SU = SUSE ; RH = RedHat; ACR = Armed forces CR.

For the future events we have to keep in mind that we need to better setup those and have prepared small buglists rather then wide-range ones where people spend more time picking ideal work than working on those :-)

Lunch/Afterparty

The lunch and the afterparty were done in nice pub nearby which had decent food and plenty of beer so everyone was happy. The only problem was that it take some waiting to get the food as suddenly there were 40 people in the pub (I still think this could’ve been somehow prepared so they had only limited subset of foods really fast so you can choose between waiting a bit or picking something and going back fast).

During the night one of Gentoo attendees got quite drunk and had to be delivered home by other ogranizers as I had to leave bit early (being up from 5 am is not something I fancy).
The big problem here was with the location where one should put him, because he was not able to talk and his ID contained residency info for different city. So for the next time when you go for linux event where you don’t know much put into your pockets some paper with the address. It is superconvenient and we don’t have to bother your parents at 1 am to find out what to do with their “sweet” child.

Endword

I would like to say huge thanks to all attendees for making the event possible and also appologize for everything I frogot to mention here.

Libav going to be default provider for your codec experience

UPDATE: Added some basic migration instructions to the bottom.
UPDATE2: Removed mplayer incompatibility mention. Mplayer-1.1 works with system libav.

As the summary says the default media codec provider for new installs will be libav instead of ffmpeg.

This change is being done due to various reasons like matching default with Fedora and Debian, or due to fact that some projects which are high-profile (eg sh*tload of people use them) will be probably libav only. One example being gst-libav which is in return required by libreoffice-4 which is due release in about month. To go for least pain for the user we decided to move from default ffmpeg to default libav library.

This change won’t affect your current installs at all but we would like to ask you to try to migrate to the libav and test and report any issues. So if stuff happen in the future and we are forced to throw libav as only implementation for everyone you are not left in the dark screaming for your suddenly missing features.

What to do when some package does not build with libav but ffmpeg is fine

There are no such packages left around if I am searching correctly (might be my blindness so do not take my word for it).

So if you encounter any package not building with libav just open bugreport on bugzilla and assign it to media-video team and add lu_zero[at]gentoo.org to CC to be sure he really takes a sneaky look to fix it. If you want to fix the issue yourself it gets even better. You write the patch open the bug in our bugzie and someone will include it. Also the patch should be sent to upstream for inclusion, so we don’t have to keep the patches in tree for long time.

What should I do when I have some issues with libav and I require more features that are on ffmpeg but not on libav

Its easier than fixing bugs about failing packages. Just nag to lu_zero (mail hidden somewhere in this post ;-)) and read this.

So when is this stuff going to ruin my day?

The switch in the tree and news item informing all users of media-video/ffmpeg will be created at the end of the January or early February, unless something really bad happens while you guys test it now.

I feel lucky and I want to switch right away so I can ruin your day by reporting bugs

Great I am really happy you want to contribute. The libav switch is pretty easy to be done as there are only 2 things to keep in mind.

You have to sync your useflags between virtual/ffmpeg and the newly-to-be-switched media-video/libav. This is most probably best to do just edit your package.use stuff and replace the media-video/ffmpeg line with media-video/libav one.

Then one would go straight away for emerge libav but there is one more caveat. Libav has split libpostproc library while ffmpeg still is using the internal one. Code wise they are most probably equal but you have to take account for it so just call emerge with both libraries.
emerge -1v libav libpostproc

If this succeeds you have to revdep-rebuild the packages you have or use @preserved-rebuild from portage-2.2 to rebuild all the RDEPENDS of libav.

Good luck and happy bug hunting.

Libreoffice 4.0 and other cool stuff

During the following week there will be hard feature freeze on libreoffice and 4.0 branch will be created. This means that we can finally start to do some sensible stuff, like testing it like hell in Gentoo.

This release is packed with new features so let me list at least some relevant to our Gentoo stuff:

  • repaired nsplugin interface (who the hell uses it :P) that was fixed by Stephan Bergmann for wich you ALL guys should sent him some cookies :-)
  • liblangtag direct po/mo usage that ensures easier translations usage because the translations are not converted in to internal sdf format
  • liborcus library debut which brings out some features from calc to nice small lib so anyone can reuse them, plus it is easier to maintain, cookies to Kohei Yoshida
  • bluetooth remote control that allows you to just mess with your presentations over bluetooth, also there is android remote app for that over network ;-)
  • telepathy colaboration framework inclusion that allows you to mess with mutiple other people on one document in semi-realtime manner (it is mostly tech preview and you don’t see what is the other guy doing, it just appears in the doc)
  • binfilter is gone! Which is awesome as it was huge load of code that was really stinky

For more changes you can just read the wiki article, just keep in mind that this wiki page will be updated until the release, so it does not contain all the stuff.

Build related stuff

  • We are going to require new library that allows us to parse mspub format. Fridrich Strba was obviously bored so he wrote yet another format parser :-)
  • Pdfimport is no longer pseudo-extension but it is directly built in with normal useflag, which saves quite a lot of copy&paste code and it looks like it operates faster now.
  • The openldap schema provider is now hard-required so you can use adresbooks (Mork driver handles that). I bet some of you lads wont like this much, but ldap itself does not have too much deps and it is usefull for quite few business cases.
  • There are also some nice removals, like glib and librsvg are goners from default reqs (no-suprise for gnomers that they will still need them). From other it no longer needs the sys-libs/db, which I finally removed from my system.
  • Gcc requirement was raised to 4.6, because otherwise boost acts like *censored* and I have better stuff to do than just fix it all the time.
  • Saxon buindling has been delt with and removed completely.
  • Paralel build is sorted out so it will use correct amount of cpus and will fork gcc only required times not n^n times.
  • And last but most probably worst, the plugin foundation that was in java is slowly migrating to python, and it needs python:3.3 or later. This did not make even me happy :-)

Other fancy libreoffice stuff

Michel Meeks is running merges against the Apache Openoffice so we try hard to get even fixes that are not in our codebase (thankfully allowed by license this way). So with lots of efforts we review all their code changes and try to merge it over into our implementation. This will grow more and more complex over a time, because in libo we actually try to use the new stuff like new C++ std/Boost/… so there are more and more collisions. Lets see how long it will be worth it (of course oneliners are easy to pick up :P).

What is going in stable?

We at last got libreoffice-3.6 and binary stable. After this there was found svg bug with librsvg (see above, its gone from 4.0) so the binaries will be rebuilt and next version bump will loose the svg useflag. This was caused by how I wrote the detection of new switches and overlook on my side, I simply tried to just launch the libreo with -svg and didn’t dig further. Other than that the whole package is production ready and there should not be much new regressions.

Running owncloud on Gentoo stable

As I migrated to clean data layout (see previous post) I decided to be cool&trendy guy and fire up my own lovely cloudy service.

First my thinking was bit off regular setup, because even if we have in-tree ebuild of owncloud it hard-requires apache, which I find overkill here.

So I introduce you to secret approach how to make it work with ngnix and sqlite3. Before you say that I should use *insertothercooldbname* please think of that my deployment is only for handfull users and I tested it with 5 users connected at once each of them having access to 1 tb shared datastore and it proven fast enough.

Preparing keywords/useflags/etc

Well owncloud is testing, so unmask it:

scarabeus@htpc: /etc/portage $ cat package.keywords/own-cloud
www-apps/owncloud

We need dav for direct access and php stuff for the setup (some useflags might be useless or redundant):

scarabeus@htpc: /etc/portage $ cat package.use/own-cloud
dev-lang/php pdo sqlite3 curl xmlwriter gd truetype cgi force-cgi-redirect fpm
www-servers/nginx nginx_modules_http_dav

Now silently punt the apache away as we love nginx:

scarabeus@htpc: /etc/portage $ cat make.profile/package.provided
virtual/httpd-php-5.4

And put all this to good use by emerging required stuff:

emerge -v www-servers/nginx www-apps/owncloud

Setting up the stuff

As nginx does not have any fcgi we will use the fpm from php directly. For that we need to add it to runlevel rc-update add php-fpm default and set up a bit default number of spawned servers (config is in /etc/php/fpm-php5.4/php-fpm.conf). Also remeber to set there proper user/group there, or you won’t be able to store content in your cloud, just read from it.

Then we set up the nginx (/etc/nginx/nginx.conf and /etc/nginx/fastcgi_params). To keep this short and easy I will just post the config I used and let you to google for other nginx variables.
First the conf file:

        server {
                listen 80;
                server_name hostname;
                rewrite ^ https://$server_name$request_uri? permanent;  # enforce https
        }

        server {
                listen 443;
                server_name hostname;

                ssl on;
                ssl_certificate /etc/ssl/nginx/nginx.crt;
                ssl_certificate_key /etc/ssl/nginx/nginx.key;

                access_log /var/log/nginx/htpc.access_log main;
                error_log /var/log/nginx/htpc.error_log info;

                root /var/www/htpc/htdocs/owncloud/;

                client_max_body_size 8M;
                create_full_put_path on;
                dav_access user:rw group:rw all:r;

                index index.php;

                location ~ ^/(data|config|\.ht|db_structure\.xml|README) {
                        deny all;
                }

                location / {
                        rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
                        rewrite ^/.well-known/carddav /remote.php/carddav/ redirect;
                        rewrite ^/.well-known/caldav /remote.php/caldav/ redirect;
                        rewrite ^/apps/calendar/caldav.php /remote.php/caldav/ last;
                        rewrite ^/apps/contacts/carddav.php /remote.php/carddav/ last;
                        rewrite ^/apps/([^/]*)/(.*\.(css|php))$ /index.php?app=$1&getfile=$2 last;
                        rewrite ^/remote/(.*) /remote.php/$1 last;

                        try_files $uri $uri/ @webdav;
                }

                location @webdav {
                        fastcgi_split_path_info ^(.+\.php)(/.*)$;
                        fastcgi_pass 127.0.0.1:9000;
                        include fastcgi_params;
                        fastcgi_param HTTPS on;
                }

                location ~* ^.+.(jpg|jpeg|gif|bmp|ico|png|css|js|swf)$ {
                        expires 30d;
                        access_log off;
                }

                location ~ \.php$ {
                        fastcgi_split_path_info ^(.+\.php)(/.*)$;
                        fastcgi_pass 127.0.0.1:9000;
                        include fastcgi_params;
                        fastcgi_index index.php;
                        fastcgi_intercept_errors on;
                        try_files $uri =404;
                }
        }

For the fcgi we also need some params to make the webdav work:

fastcgi_param   SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param   SCRIPT_NAME     $fastcgi_script_name;
fastcgi_param   PATH_INFO       $fastcgi_path_info;

That should be it, now we just deploy the owncloud to our webserver by webapp-config:

/usr/sbin/webapp-config -I -h htpc -u root -d /owncloud owncloud 4.0.7

After we start up the webserver and fcgi provider, we should be up and running to open the stuff in web browsers.

Few issues I didn’t manage to sort out in owncloud

  • External module to load all system users into it does not pass the auth
  • Google sync just timeouts everytime I try it (I maybe have just damn huge content here)
  • External storage support from within owncloud didn’t work for me, I just symlinked the data folder to the proper places under each user and logged into them in browser, then waited for 3 hours (1tb of data to index) and they were able to access everything.

Migrating disk layout from mess to raid1

Imagine you are dumb guy like me, first what I did was to set up 3 1TB disks into one huge LVM copied data on it and then found out that grub2 needs more free space before the first partition to be able to load the LVM module and boot. For a while I solved this with external USB token plugged in the motherboard. But I said no more!

I bought two 3TB disks to deal with the situation, and this time I decided to do everything right and add UEFI boot instead of normal good old booting.

Disk layout

Model: ATA ST3000VX000-9YW1 (scsi)
Disk /dev/sda: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name     Flags
 1      17.4kB  512MB   512MB   fat32        primary
 2      512MB   20.0GB  19.5GB               primary
 3      20.0GB  30.0GB  9999MB  xfs          primary
 4      30.0GB  3001GB  2971GB  xfs          primary

So as you can see I created 4 partitions. First is special case and it must be always created for EFI boot. Create it larger than 200 megs, up to 500, which should be enough for everyone.

The disk layout must be set up in parted as we want GPT layout (just google how to do it, it is damn easy to use), It accept both values like 1M, 1T and percetage like 4% to specify the resulting partition size.

Setting up the RAID

We just create simple nodes and plug /dev/sda2-4 and /dev/sdb2-4 to them. Prior creating the RAID make sure you have RAID support in your kernel.

for i in {2..4}; do mknod /dev/md${i} b 9 ${i}; mdadm --create /dev/md${i} --level=1 --raid-devices=2 /dev/sda${i} /dev/sdb${i}; done

After these commands are executed we have to watch mdstat until it is prepared (note that you can work with the md disks in the meantime, just the setting of the RAID will be slower as you will be writting on the named disks.

After we check the mdstat and see that all the disks are ready for play:

croot@htpc: ~ # cat /proc/mdstat 
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty] 
md4 : active raid1 sda4[0] sdb4[1]
      2900968312 blocks super 1.2 [2/2] [UU]
      
md3 : active raid1 sda3[0] sdb3[1]
      9763768 blocks super 1.2 [2/2] [UU]
      
md2 : active raid1 sda2[0] sdb2[1]
      19030679 blocks super 1.2 [2/2] [UU]

we can proceed with data copying.

Transfering the data and setting up the system

mkfs.ext4 /dev/md2 ; mkfs.xfs /dev/md3 ; mkfs.xfs /dev/md4 # create filesystems
mkdir -p /mnt/newroot/{home,var} # create the folder struct (home and var are actually the md3 and md4 so prepare the folders for them
mount /dev/md2 /mnt/newroot
mount /dev/md3 /mnt/newroot/var
mount /dev/md4 /mnt/newroot/home

Now that we are ready we will use rsync to transfer living system and data (WARNING: shutdown everything that temper with data (like ftp/svn/git services). Only thing we are going to loose is few lines of syslog and other log services.

rsync -av /home /mnt/newroot/home # no -z as we don't need to compress
rsync -av /var /mnt/newroot/var
rsync -av / --exclude '/home' --exclude '/dev' --exclude '/lost+found' --exclude '/proc' --exclude '/sys' --exclude '/var' --exclude '/mnt' --exclude '/media' --exclude '/tmp' /mnt/newroot/ # copy all relevant stuff to newroot
mkdir -p /mnt/newroot/{dev,proc,sys,mnt,media,tmp}

After the transfer you need to edit /etc/fstab to reflect new disk layout. Update kernel (if needed to support new RAID layout) and update /etc/defaults/grub if you did RAID like me to contain domdadm line for default command.

Preparing new boot over UEFI

On your machine you need to create usb dongle which supports UEFI boot (you need to be uefi booted to setup UEFI [fcking hilarious]).

We need to download latest archboot iso 64bit (gentoo minimal didn’t contain this lovely feature).
Grab some usb disk and plug it into our machine. We will format it to 32b fat: mkfs.vfat -F32 /dev/[myusb] , mount somewhere and copy the ISO image content to the usb folder (you can enter it in mc and just F5 it if you are lazy like me, but it is working with tar, p7zip or whatever else). Shutdown the computer, unplug old disks and with manic laughter turn the machine again on.

To boot the uefi just open boot list menu and select the disk which has UEFI around its name. It will open grub2 menu where you just select first option. We should be then welcomed by lovely arch installer. Not caring about it switch to another console and open terminal. Setup again the arrays using mdadm –assemble.

for i in {2..4}; do mknod /dev/md${i} b 9 ${i}; mdadm --assemble /dev/md$i /dev/sda${i} /dev/sdb${i}; done

Then just proceed with mounting them somewhere to /mnt and chroot like you would do new gentoo install. Exact steps:

modprobe efivars # load the efi tool variables
mkdir -p /mnt/newroot/{home,var} # create the folder struct (home and var are actually the md3 and md4 so prepare the folders for them
mount /dev/md2 /mnt/newroot
mount /dev/md3 /mnt/newroot/var
mount /dev/md4 /mnt/newroot/home
mount -o rbind /dev /mnt/newroot/dev
mount -o rbind /sys /mnt/newroot/sys
mount -t proc none /mnt/newroot/proc
chroot /mnt/newroot /bin/bash
. /etc/profile
env-update

Now that we are in chroot we just install grub2 with GRUB_PLATFORMS=”efi-64″. After that we proceed easily by following wiki article.

Unmount the disk, reboot the system, unplug the flasdrive, …, profit?