sh4 binrepo started

Using mike’s SuperH lanktak I’ve setup a new binrepo for those of you crazy enough to own one of these things or still have a dreamcast in the closet and a little time to spare.

It’s painfully slow to compile on, but oh well..

machine : LANDISK
processor : 0
cpu family : sh4
cpu type : SH7751R
cpu flags : fpu ptea
cache type : split (harvard)
icache size : 16KiB (2-way)
dcache size : 32KiB (2-way)
bogomips : 266.24
master_clk : 266.66MHz
module_clk : 33.33MHz
bus_clk : 133.33MHz
cpu_clk : 266.66MHz
tmu0_clk : 8.33MHz

PROFILE: default-linux/sh/2006.1
PACKAGES: 371
ACCEPT_KEYWORDS: sh
CBUILD: sh4-unknown-linux-gnu
CHOST: sh4-unknown-linux-gnu
CFLAGS: -O2 -m4 -pipe
FEATURES: autoconfig buildpkg ccache distlocks metadata-transfer noauto sfperms splitdebug strict

sparc64 binrepo started

Weeve @ gentoo hooked me up with an account on a Sun T2K and I’ve started a tinderbox run over there.
The repo is pretty small in size right now (343 pkgs) but will be growing over time. I can see right now I’ve got lots of bugs to file and poking to do as this arch is lacking behind in stable keyword markings for a lot of standard packages in addition to lots of packages failing that are marked stable. But it will be a while before I can really dive head on into this as I’m moving next week.

Adventures with our x86 tinderbox.

For a few months now I’ve profile testing on pretty decent x86 host at OSU. As a side result of building and leaving FEATURES=buildpkg enabled I started ending up with pretty decent sized binary repositories which was pretty cool. I moved on to expanding it to make other arch test/tinderboxes start pushing the $PKGDIR from their local repos as well. Also as a result of testing I crontab a crossdev job to build nearly every known cross-compiler on the face of the planet that we could possibly support. As soon as KingTaco and I can do some disk shuffling at OSU I’ll have an amd64 to beat on in the same way as the x86 tinderbox. Currently I’m limited to profile testing on mipsel/arm/ppc/x86/hppa. x86 has the best coverage of them.

Side note I wanted to get portage-2.0.54-r2 out the door yesterday but that got delayed. Today is a bad day for me and development so I don’t think I’ll be pushing one today either.

Before I go to sleep

So it’s probably going to be a while before we can do drive swapping on for pitr/dustpuppy and dustpuppy is offline for till that happens so I’ve started a stable multilib hardened repo for amd64 using an chroot on pitr. hardened/amd64/multilib/
The repo is just shy of about 500 of the most common packages now.

It’s make.conf

CFLAGS=”-O2 -pipe -fforce-addr”
CHOST=”x86_64-pc-linux-gnu”
CXXFLAGS=”${CFLAGS}”
USE=”-nls -tcpd userlocales dlloader snmp bzip2 cli cgi session tiff jpeg png sysfs bindist boundschecking mp3 ogg vorbis png vcd mpeg xml xml2″
MAKEOPTS=”-j4 –quiet”
FEATURES=”buildpkg distclean nodoc noinfo”
PKGDIR=/usr/portage/local/hardened-multilib
CLEAN_DELAY=0
USE_ORDER=env:pkg:conf:defaults

Added quite a few more packages to the hardened/ppc repo. Current count there is 468

Per request I’m starting a ppc-uclibc repo. I started from a stage1-ppc-uclibc-2005.0 but am running into a few problems. First every single package wanted to install itself into a new SLOT. This was caused because no SLOT file existed in that stageball. Then while running $PORTDIR/scripts/bootstrap.sh uclibc itself would puke saying that the old version was built with +nls and I needed to keep it enabled. Well that’s bogus because gentoo nls support is/was non exisxtant back then for uclibc. The problem seems to be that the built_with_use function when no USE= file exists returns an incorrect value.

While updating stuff I noticed that the default icon sets for apache are really ugly. I asked around and sure enough before long somebody pointed me in the right direction for some sweet eye candy.

Pushed a new portage-utils (0.1.17) the other day.
ChangeLog

  • q: Updated stderr/stdout handling for BSD again.
  • qfile: do not abort when user passes qfile “…”
  • qimlate: New applet (Thomas Cort tcort@gentoo.org) uses portage metadata/cache directly.
  • qmerge: bug fix. dont remove vdb entries in pretend mode.
  • qpkg: new switch -P/–pkgdir to allow user defined pkgdirs.
  • quse: new switch. -N/–name-only used to only display matching entries and not the values.

If vapier has not pushed a new pax-utils already then I’ll try to get to that in the next few days. It’s got a few updates to make life suck less.

arm/cobalt binary repos and free candy for everyone

The tinderbox current pkg count is at 6517 packages across 66 repos now. If anybody has a package or profile request to be added to the tinderbox please feel free to let me know.

default-linux/arm

I added another binary repo to the tinderbox. This time it’s for the default-linux/arm/ profile. The build host is an arm netwinder and is so incredibly slow to build on it’s not funny, (a PDA would be faster) but I gotta take one for the team if I want good coverage of all the profiles. This repo for sure wont be pure arch or ~arch either.

=================================================================
Processor       : StrongARM-110 rev 4 (v4l)
BogoMIPS        : 185.54
Features        : swp half 26bit fastmult 
CPU implementer : 0x44
CPU architecture: 4
CPU variant     : 0x0
CPU part        : 0xa10
CPU revision    : 4

Hardware        : Rebel-NetWinder
Revision        : 59ff
Serial          : 00000000000020f9
=================================================================
             total       used       free     shared    buffers     cached
Mem:           123         93         30          0          4         53
-/+ buffers/cache:         35         88
Swap:         1043         15       1028
=================================================================
Portage 2.1_rc1-r3 (default-linux/arm, gcc-3.4.6, glibc-2.3.6-r3, 2.6.14.2-grsec armv4l)
=================================================================

default-linux/mips/cobalt

I was starting the initial mipsel rsync repo transfer when the raq2.mips box locked up. I don’t think anybody from the OSL will be able to reboot it anytime today so I’ll probably delete whatever I have and wait for another day and another kernel. Boxes freezing up while simply transferring files is never good.

Icons
I noticed the icons on dev.gentoo.org were not loading after the migration to the new box so I updated those with the same eye candy ones I used on the tinderbox. Looks pretty sweet imo. Hopefully nobody will bitch about it being nice.

cross compiling kernels
Still on my todo list. I really don’t have a clean way yet to map blindly which kernels I want to build using what options. Maybe next weekend.

Benchmarking the power5

Yesterday I got fwded along a forum posting over at penguinppc.org from our good friend over at the OSL [cshields@osuosl] powerpc-cpu optimizations

Gentoo PPC64 has glibc-2.3.4.20041102-r2 marked stable so first I started patching it up. While talking to another developer [vapier@gentoo] I found out that the powerpc-cpu optimizations had already been integrated in our glibc-2.4-r3 by him, so I stopped with the patching up of 2.3.x
Time for a few benchmarks. First nbench was not keyworded for PPC64 so I passed that info along to our PPC64 team and [ranger@gentoo] keyworded it for future use.

Here are the results.

Base PPC64 stable.
gcc-3.4.4 glibc-2.3.4.20041102-r2

BYTEmark* Native Mode Benchmark ver. 2 (10/95)
Index-split by Andrew D. Balsa (11/97)
Linux/Unix* port by Uwe F. Mayer (12/96,11/97)

TEST                : Iterations/sec.  : Old Index   : New Index
                    :                  : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT        :          581.52  :      14.91  :       4.90
STRING SORT         :            96.4  :      43.07  :       6.67
BITFIELD            :      1.2933e+08  :      22.19  :       4.63
FP EMULATION        :          36.512  :      17.52  :       4.04
FOURIER             :          8689.8  :       9.88  :       5.55
ASSIGNMENT          :          7.3297  :      27.89  :       7.23
IDEA                :          1503.4  :      22.99  :       6.83
HUFFMAN             :          589.62  :      16.35  :       5.22
NEURAL NET          :          14.411  :      23.15  :       9.74
LU DECOMPOSITION    :           556.8  :      28.85  :      20.83
==========================ORIGINAL BYTEMARK RESULTS==========================
INTEGER INDEX       : 22.153
FLOATING-POINT INDEX: 18.757
Baseline (MSDOS*)   : Pentium* 90, 256 KB L2-cache, Watcom* compiler 10.0
==============================LINUX DATA BELOW===============================
CPU                 : 8 CPU
L2 Cache            : 
OS                  : Linux 2.6.5-7.97-pseries64
C compiler          : 3.4.4
libc                : 
MEMORY INDEX        : 6.069
INTEGER INDEX       : 5.154
FLOATING-POINT INDEX: 10.403
Baseline (LINUX)    : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38
* Trademarks are property of their respective holder.

glibc-2.4.x requires gcc-4 so I compiled that, then recompiled nbench so we could establish any differences it makes alone.
gcc-4.1.1 with 2.3.4.20041102-r2

BYTEmark* Native Mode Benchmark ver. 2 (10/95)
Index-split by Andrew D. Balsa (11/97)
Linux/Unix* port by Uwe F. Mayer (12/96,11/97)

TEST                : Iterations/sec.  : Old Index   : New Index
                    :                  : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT        :          586.56  :      15.04  :       4.94
STRING SORT         :          102.48  :      45.79  :       7.09
BITFIELD            :      1.3137e+08  :      22.54  :       4.71
FP EMULATION        :          39.625  :      19.01  :       4.39
FOURIER             :            8742  :       9.94  :       5.58
ASSIGNMENT          :          10.188  :      38.77  :      10.06
IDEA                :          1750.9  :      26.78  :       7.95
HUFFMAN             :          775.66  :      21.51  :       6.87
NEURAL NET          :          16.845  :      27.06  :      11.38
LU DECOMPOSITION    :          605.04  :      31.34  :      22.63
==========================ORIGINAL BYTEMARK RESULTS==========================
INTEGER INDEX       : 25.276
FLOATING-POINT INDEX: 20.354
Baseline (MSDOS*)   : Pentium* 90, 256 KB L2-cache, Watcom* compiler 10.0
==============================LINUX DATA BELOW===============================
CPU                 : 8 CPU
L2 Cache            : 
OS                  : Linux 2.6.5-7.97-pseries64
C compiler          : 4.1.1
libc                : 
MEMORY INDEX        : 6.948
INTEGER INDEX       : 5.866
FLOATING-POINT INDEX: 11.289
Baseline (LINUX)    : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38
* Trademarks are property of their respective holder.

Sadly the kernel version that’s running on the OSL box is a SuSe one without the support needed. I kept getting FATAL: kernel too old while building glibc. Guess I’ll have to save the powerpc-cpu optimizations testing for another day..

cvs2git/parsecvs gentoo-x86 conversion

Recently thanks to the Oregon State University Open Source Labs I’ve been given access to an IBM OpenPower 720. This thing is a beast like no other box which I have access to. The specs are simply amazing. Anyway I noticed that Alec Warner/antarus@gentoo was having problems with running a cvs2git conversion of the gentoo-x86 tree, every box which he attempted it on ran out of memory. I figured ok well I’ve got access to the mothership and should not have any problems doing a run for him. We talked for a little while and he provided me with a quick little script to fire off the conversion process. Well it took 21 hrs consumed 100% of the CPU the entire time and then it failed, towards the end right before it died with an Out of Memory: Killed process 14671 (parsecvs) error. It had consumed 70.1G of virtual memory and 30G RSS as well as all the swap. The gentoo-x86 tree is about 1.4G worth cvs data, the parsecvs util had managed to convert that into 4.1G of git data before it got killed. Gotta say from an admin/infra point of view going from a 1.4G to +4.1G backend repo leaves little room to be desired.

None the less I don’t see us switching to git any time soon unless the backend tools for conversion get a rewrite/update so they can process the full repo as incremental parts or learn how to use the existing memory more efficiently.

In this graph you can see where I started about at 21:00 and ran till about 18:00 the following day, at about 14:00 the basic conversion process was done and parsecvs started allocating memory here pretty quickly for another 4 hours. The final spike is when it started swapping to disk before it got killed.

24h CPU Usage

Unfortunately the snmpd version running on the box does not appear to support 64bit counters so all the memory graphs are/were nil.

In the end I had fun helping him with this, and it really gave the power5 a workout 🙂
Sometime later this week I’ll start setting up ppc64 binrepos, cross compilers etc..

ia64 binrepo added/x86 livecd started/SoC server built

Well I’m still busy doing whatever it is I do. Tim Yamin [plasmaro@gentoo] hooked me up with an account on the ia64 at OSL to get a repo going for it. So been that’s pretty much what I’ve been doing all day and filing bugs for other packages and arches as I come across them on all the other hosts.

Current package count is 7011 across all repos and by the time I wake up hopefully it will be up it will be up by another 268. Then I got whatever is in this list unstable.ia64 to get keyworded and marked stable.

I do hope to get my hands on a fast mips host here in the near future so I can get that going as well. Hoping to break the 10k mark!!

jforman hooked me up with a new vhost for the tinderbox as the .x86. part is a bit deceptive. It’s now just tinderbox.dev.gentoo.org for the master repo.

r2d2 is attempting to work with the x86 hardened binrepo directly to see if he can speed up the process of livecd creation. Added a few hundred more packages to the hardened/x86 repo for that making it the largest repo of them all.

Some point over the weekend Lance and me did a lot of work on the box that is going to be used to host most of the Google Summer of Code projects. Still got a few final touchup things todo on it, but thats mostly all just a bit of configuring services. It’s still a bit to soon to tell if they are going to need more than 1 server for everything. genone’s project might take a wee bit more space than the box we had allocated initially for this. One of the things I’d really like to see some of the money Gentoo will earn from doing this years SoC is that we pick up a nice server which can be used for future SoC events and or to host the fruitful project that come out of this years (like stats/anonsvn/anon-other or so). I do have high hopes and I’ve already picked out a pretty decent dell 2850 for about ~$4500 that I’d like to see us get that includes 12G Ram/dual dualcore xeons at 3GHZ each and about 100GB of space or so. (having a good server benefits everybody)

Cool new bug.. New QA warning – detect already stripped binaries in prepstrip (I like ELF)

default-linux/hppa binrepo is up

Last weekend I got a list of packages from GMsoft(hppa lead dev) and started a big fat run on hake. Well it’s pretty much finishing up and the only packages which I did not merge from his list of 1083 packages were those ones which were either ~arch or required a configured kernel (silly kde for having a harddep on media-sound/cdparanoia). Currently there are 758 pkgs in this repo and like the others, if/when something breaks I’ll know about it and report it.

CFLAGS=”-O2 -pipe -march=2.0″
CHOST=”hppa2.0-unknown-linux-gnu”
CXXFLAGS=”-O2 -pipe”
USE_ORDER=env:pkg:conf:defaults
PORTAGE_TMPFS=”/dev/shm/portage”
PKGDIR=/packages/
MAKEOPTS=”-j3″
CLEAN_DELAY=0
FEATURES=”sandbox distclean buildpkg genpkgindex test collision-protect”
USE=’dlloader test bindist’