Jan 4, 2012

New GNU C Library Version Coming to openSUSE

The GNU C library (glibc) is the basic system library used by almost every program, it comes with the base library (libc), a math library (libm), the thread library (libpthread), support for locales and character conversions and a couple of tools. The latest release - 2.15 - was released Christmas 2011.
I have now updated glibc for openSUSE 12.2 in its development project (Base:System) to version 2.15 and it seems to work on x86, x86-64, PowerPC and even Arm.

The new version 2.15 of glibc brings mainly optimizations and bugfixes.

The main new features of the 2.15 version are:
  • a new program pldd to list loaded objects of a process.
  • nss_db support - without a dependency on berkely db.
  • integration of libm with gcc's -ffinite-math-only option.
Besides updating the upstream version, I made a couple of cleanups of the package itself:
  • The x86-64 package was falsely compiled with gcc's flag -mno-tls-direct-seg-refs - an option only needed for running under 32-bit Xen. This might speedup some threaded applications.
  • The i686 library is not compiled with -mno-tls-direct-seg-refs anymore but a special version is installed in /lib/i686/nosegneg that is compiled with this flag. So, installations using 32-bit Xen will use the nosegneg library and everybody else the normal version. This should speedup some threaded applications when not using Xen.
  • The AMD libm implementation has been removed in favour of glibc's libm implementation. It didn't integrate well with the changes that  went into glibc 2.15.
  • I've reviewed all patches we have in the package and tried to upstream as many as possible - and removed a few that were obsolete. We still have far too many, help is welcome to upstream ;)
Call for testing
I welcome contributions to the package, testing of glibc 2.15 and fixing of any bugs in glibc or the packages using it. I'm especially interested on tests under Xen to check that my optimizations did not break anything.

So far, one package was failing due to changes - unscd - and the upstream author provided quickly a fix. Thanks to Cristian Rodriguez for his tests!

I'm pushing glibc 2.15 now to openSUSE:Factory for the usual review and integration.

Jan 3, 2012

Finding subtile malloc bugs

Last year I've talked with a couple of engineers that had where hunting done some strange bug reports with memory corruption that upstream developers could not directly reproduce. The bugs included accessing data after a closedir call or buffer overflows. These kind of bugs normally lead to random behaviour at one point in time and are really difficult to debug. In the case of openSUSE Factory, the openSUSE developers could reproduceable trigger these since glibc comes with some helpers.
If  "MALLOC_PERTURB_" is set in the environment, then every malloc call will clear memory after every free - thus destroying its content - and also initialize malloc'ed memory (with the exception of cmalloc).

Ulrich Drepper who implements this said:
"The reason for this exercise is, of course, to find code which uses memory returned by malloc without initializing it and code which uses code after it is freed. valgrind can do this but it's costly to run. The MALLOC_PERTURB_ exchanges the ability to detect problems in 100% of the cases with speed."
In openSUSE, we set MALLOC_PERTURB_ to 69. So, if you find your memory location suddenly is filled with "E"s (ASCII 69 is 'E') or with "\272" (the bitwise inverse), you have found a bug in handling of malloc.

For more information, read Ulrich's blog or Jakub's blog.

This is only enabled for openSUSE Factory but if you're developing software yourself, I advise to set it yourself and use it to find bugs early.

Jan 2, 2012

Cleaning up Factory Packages

It's now the beginning of a new year (happy 2012 everybody!) and I'm writing about some changes that happened at the end of the last year and the period right after the openSUSE 12.1 release. Especially Coolo has used this to do a big cleaning up of our factory distribution touching most of openSUSE's 4000+ source packages this way.

Automake, autoconf, libtool removal from default build environment

Automake, autoconf and libtool are currently installed with every build and many packages don't need it. Removing unneeded packages from the default installation will make the build process faster. Therefore Coolo has started to add explicit BuildRequires lines to packages that need either of the tools and now removed these three packages from the default installation.

Packaging files twice is a bug

Some packages installed the same file in different sub packages or as another package - without adding a conflict on the other package. Coolo made the check fatal that checks for this situation to avoid packaging bugs.

Removing old %suse_update_config macro

As Coolo said:
"...I made the %suse_update_config macro a wrapper for autoreconf --force --install and removed it from all packages that crossed my way (with a SR to the devel project of course . The macro was used by SUSE when porting SLES7 (yes, that was the first sles actually  to s390 and power. But in the last 10 years most packages should have seen an update of their config - but the macro stayed. It's time to give the macro a rest in macro heaven."

Use SPDX license tag

We're using in the "License:" header now the SPDX definition.

Stricter checks for submission to Factory

Before a package gets checked into openSUSE:Factory, some automatic checks are done before the review team reviews the package. These checks have now been enhanced. A major change is that the URL given for source files is now used to verify that the downloaded file is really the one on the given URL. If this fails, you can run the service manually (use "osc service localrun download_files") to check for failures.

New format_spec files service

A new format_spec files service has been implemented by Coolo and was used in his submit requests introducing other changes. Included are the following changes to the spec file:
  • remove obsolete #norootforbuild lines
  • remove duplicated group and license from sub packages if all sub-packages have the same license
  • patch license field to apply to spdx.org
  • split buildrequires in single lines
He also changed a couple of packages directly where
  • license can be autoconverted (otherwise legal needs to review anyway)
  • devel project has unmodified branch
  • devel project is not in black list

Getting rid of /etc/tmpdirs.d

In the past /etc/tmpdirs.d was used with SysV init scripts and systemd used tmpfiles.d. Now the SysV init scripts call tmpfiles.d as well - and do not handle tmpdirs.d anymore. All packages with tmpdirs.d usage have been adjusted.

Obsoleted %suse_update_desktop_file

Coolo changed the way the openSUSE desktop files which are used by the application menus of the desktops can be translated and changed. Instead of manually calling %suse_update_desktop, a check script collects all desktop files and makes them available for translation. So, if a package only calls %suse_update_desktop_file without any options, the call can be removed completely.
[Update 2012-01-03] Since other distributions are using "desktop-file-install" and "desktop-file-edit", packages can use that one as well to edit the desktop files. These are part of the package desktop-file-utils that need to be installed via "BuildRequires". [End update]