Switch from syslog-ng to rsyslog

I just think it is funny for a package as fundamental as system logger to depend on glib (part of the GTK+ project). (Of course, I don’t mean glib is a bad library. I’m not implying anything like that.)

Of course there are other reasons, but they are less important:

• Many distributions also prefer rsyslog to syslog-ng because it’s more powerful and scalable. [1] [2]
• Several people complained against the performance of syslog-ng [3], although the performance of syslog is normally not an issue for individual users.
• Third, I don’t understand why the author of syslog-ng prefers listening on /dev/log as a stream socket (local counterpart of TCP) instead of a datagram one (local counterpart of UDP). (I know it can be changed in the configuration file.) None of the common issues of UDP is present in local datagram sockets: too short length limit; out of order; unreliability. On the contrary, syslog needs unidirectional channels carrying boundary-maintained messages, for which nothing can be more suitable.
• Finally, licensing issues, upstream responsiveness, etc. They are less important for individual users, though.

Sure, syslog-ng also has its advantages. For example, I like the format of its configuration files.

Tags: Linux, syslog

CPU Frequency Governor

Kernel documentation recommends “conservative” for laptops, citing latency reasons.

However, Intel explicitly recommends “ondemand” in its powertop. So does at least one Intel kernel developer.

OK. I have been using “conservative.” I decide to switch to “ondemand” from now on.

Reference

Tags: hardware, Linux

NULL can be a valid address

It is only a convention to consider NULL (0) as an invalid pointer. Technically, the operating system or hardware does not really care if a pointer is zero or not, although operating systems may restrict the use of valid null pointers as they may be a security hole.

Consider this program:

#include <stdio.h>
#include <sys/mman.h>

int main ()
{
    int *p = mmap (0, 4096, PROT_READ|PROT_WRITE,
        MAP_FIXED|MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
    *p = 2554;
    printf ("p=%p; *p=%d\n", p, *p);
    return 0;
}

It attempts to make NULL (0) a valid address and then write to it. On Linux, it runs without error as root, but crashes as a normal user.

Note: Calling mmap with NULL as its first argument usually means the kernel will choose an address. However, if MAP_FIXED is also specified, it instead instructs the kernel to use the very address 0. Only privileged processes are allowed to do so; a non-privileged process only gets EPERM (Permission denied).

This also explains why MAP_FAILED is equal to (void *)-1 instead of NULL.

Tags: C/C++, Linux, memory

Removed some distribution-specific patches

With all due respect to Gentoo developers, I really hate the patches they made for coreutils, especially the one to have uname parse /proc/cpuinfo.

The result is that uname -a displays more info, specifically the mode of the CPU, on Gentoo than other Linux distributions. Generally this is not a bad thing. But I do have concerns:

(1) In my view the utility uname should remain a simple wrapper of the homonym system call. If CPU/vendor info really needs to be returned by uname, it is better to add it to kernel. This is also part of the reason why upstream rejected this patch.

(2) If I am used to finding CPU info from uname, I will likely forget the more orthodox method (cat /proc/cpuinfo). A job interviewer may not be that patient to listen to my explanation about the patch.

Tags: Gentoo, Linux

Extract Deb files from command line

Debian and its derivatives use the .deb format to distribute their packages. To extract them, use ar – Yes, the very program we programmers use to make static libraries.

ar x sudo_1.6.9p17-2_i386.deb

Or we can directly extract things from data.tar.gz contained in the .deb file:

ar p sudo_1.6.9p17-2_i386.deb data.tar.gz | tar -xzf -

No longer a user of Debian GNU/Linux, I still have to remember how to extract .deb files. I frequently need to cross-compile a 64-bit version of my program on a 32-bit system, and vice versa; but I don’t want to cross-compile by myself so many libraries on which my program depends. Instead, I find it a good idea to download a right .deb file from the Debian Packages Repository and pick out the .so files.

Tags: compression, Debian, Gentoo, Linux

Unaligned access

Misalignment is not an error (only incurs a performance penalty) on x86 processors except for a few new instructions added in recent years. MOVDQA, for example, is an SSE2 instruction requiring alignment on 16-byte boundaries.

Textbooks have normally taught us we get a bus error if a CPU which disallows unaligned access actually encounters one.

But we observe a Linux process passing misaligned addresses to MOVDQA receives SIGSEGV (segmentation fault) instead of SIGBUS (bus error), on both ia32 and x86-64.

laptop /tmp $ cat a.c
int main ()
{
    char X[32];
    asm ("pxor %%xmm0,%%xmm0; movdqa %%xmm0,%0" : "=m"(X[1]) :: "xmm0");
    return 0;
}
laptop /tmp $ gcc -msse2 a.c
laptop /tmp $ ./a.out
Segmentation fault
laptop /tmp $ kill -l $?
SEGV

x86-64 (and ia32 beginning 80486SX) supports disallowing any misaligned access*. In that case, a normal instruction raises SIGBUS, but instructions which inherently requires alignment (e.g. MOVDQA) still raises SIGSEGV. It’s not so consistent.

* It is normally disabled. To enable it, set the AC bit in FLAGS:

pushf
or $0x40000,(%esp) (or %rsp on x86-64)
popf

Tags: assembly, Linux, segfault, x86-64

Floating point exception

It is already confusing enough that “floating point exception” may mean “division by zero” in integral arithmetic. It turns out it can also mean “overflow” in some cases, as in the following program (it’s difficult in C, so I had to use assembly):

#include <asm/unistd.h>
.code:
.globl _start
_start:
    mov $1, %eax
    mov $1, %edx
    div %eax
    mov $__NR_exit_group, %eax
    int $0x80

(Type “gcc -m32 -nostdlib a.S” to compile and link.)

In this program, EDX:EAX (0x100000001) divided by ECX (0x1) cannot be represented in 32-bit integer and thus it is an overflow. X86 CPUs raise a “division by zero” interruption (int 0) in such cases, and “division by zero” is displayed as “floating point exception” in Linux…

PS. The same assembly program in Intel style:

.code
.startup
    MOV EAX,1
    MOV EDX, 1
    DIV EAX

    MOV EAX, __NR_exit_group
    INT 80H

Tags: assembly, Linux

Ack is a good alternative to grep

When I grep -r something in a Subversion-controlled directory, I get a lot of results under the .svn.

In this sense, Git is better than Subversion. (Git creates only one .git directory, and stores data in some compressed format which gets ignored by grep -r.)

So I have switched to ack when working with Subversion. Ack is written in Perl, claimed to be “aimed at programmers with large trees of heterogeneous source code.”

Tags: dev, Linux

I can’t PGO compile Firefox

A normal build of Firefox for Linux is reportedly even slower than the Win32 binary running under Wine.

The reason is reportedly that the pre-compiled binary for Windows uses PGO (profile guided optimization), which is usually not enabled under Linux. Sure, the fact that GCC does not generate as efficient codes as VC may also be a reason.[1]

Firefox also supports PGO in Linux. However, I failed at this (3.5.1). The profile-generating binary always segfaults.

Other people have encountered the same problem, even with the official PKGBUILD from Arch Linux. It is said to be a compiler problem.

Well, gave up. Maybe I’ll try it again some time later, with a more “stable” version of GCC probably.

[1] This statement only applies to the 32-bit platform. It seems GCC does a very good job on x86-64.

Profile-guided optimization is a relatively new feature. GCC began supporting it starting version 4.0; Microsoft VC 2005; and Intel C/C++/Fortran 9 (?).

A typical PGO-enabled building requires three steps:

(1) Build a profile-generating binary;
(2) Run the binary, which automatically collects useful data – branch probability, etc.
(3) Rebuild the program, using the data (“profile”) from Step 2.

With PGO, Internet Explorer reportedly gains an improvement of 8%, and Firefox 11% in JavaScript.

Tags: browser, dev, GCC, Linux, optimization

install vs. cp; and mmap

If we hand write a Makefile, we should always stick to install instead of using cp for the installation commands. Not only is it more convenient, but it does things right (cp does things wrong).

For example, if we attempt to update /bin/bash, which is currently running, with “cp ... /bin/bash”, we get a “text busy” error. If we attempt to update /lib/libc.so.6 with “cp ... /lib/libc.so.6”, then we either get “text busy” (in ancient versions of Linux) or breaks each and every running program within a fraction of a second (in recent versions of Linux). install does the thing right in both situations.

The reason why cp fails is that it simply attempts to open the destination file in write-only mode and write the new contents. This causes problem because Linux (and all contemporary Unices as well as Microsoft Windows) uses memory mapping (mmap) to load executables and dynamic libraries.

The contents of an executable or dynamic library are mmap’d into the linear address space of relevant processes. Therefore, any change in the underlying file affects the mmap’d memory regions and can potentially break programs. (MAP_PRIVATE guarantees changes by processes to those memory regions are handled by COW without affecting the underlying file. On the contrary, POSIX leaves to implementations whether COW should be used if the underlying file is modified. In fact, for purpose of efficiency, in Linux, such modifications are visible to processes even though MAP_PRIVATE may have be used.)

There is an option MAP_DENWRITE which disallows any modification to the underlying file, designed to avoid situations described above. Executables and dynamic libraries are all mmap’d with this option. Unfortunately, it turned out MAP_DENYWRITE became a source of DoS attacks, forcing Linux to ignore this option in recent versions.

Executables are mmap’d by the kernel (in the execve syscall). For kernel codes, MAP_DENYWRITE still works, and therefore we get “text busy” errors if we attempt to modify the executable.

On the other hand, dynamic libraries are mmap’d by userspace codes (for example, by loaders like /lib/ld-linux.so). These codes still pass MAP_DENYWRITE to the kernel, but newer kernels silently ignores this option. The bad consequence is that you can break the whole system if you think you’re only upgrading the C runtime library.

Then, how does install solve this problem? Very simple – unlinking the file before writing the new one. Then the old file (no longer present in directory entries but still in disk until the last program referring to it exits) and the new file have different inodes. Programs started before the upgrading (continuing using the old file) and those after the upgrading (using the new version) will both be happy.

Tags: fs, Linux, memory