ELF Symbol table

I’ve recently been doing some hacking (for food). I need to modify some object files (.o) generated by GCC before they’re passed to linker. (Using elfutils)

I thought I was doing everything right, but ld (GNU linker) insisted on crashing with segmentation fault. Finally, eu-elflint told me what was wrong – I was unaware of the ELF requirement that LOCAL symbols must come before GLOBAL symbols in a symtab (symbol table), and that sh_info is the boundary between LOCAL and GLOBAL symbols.

Many thanks to authors of elfutils, especially for their eu-elflint!

Tags: assembly, dev, GCC, Linux, opensource

Increasing the size of a VirtualBox vdi virtual drive

VBoxManage modifyhd Win7.vdi --resize SizeInMiB
Then go into the guest OS and create new partitions or extend existing partitions.

Tags: VirtualBox

A problem with pipes in Python 3

The most disturbing change from Python 2 to 3 definitely is not the print() function; nor that some functions which used to return lists now return iterators; nor the removal of __cmp__; but the transition to Unicode.

I’m completely supportive of the transition per se, but I’m disappointed that they’re trying to compel us to use Unicode by dropping useful functionalities for byte-streams/8-byte strings. For example, bytes has no format or % in Python 3.

I have some code like this:

proc = subprocess.Popen((....), stdout=subprocess.PIPE)
for line in proc.stdout:
    ...

I found that, on Linux, this code snippet is almost 10 times slower in Python 3 than in Python 2. Then I strace‘d the code and found Python 3 is passing length 1 to read, incurring thousands of times more system calls than Python 2. Are you kidding me? I was forced to use something like proc.stdout.read(...).

I understand this is not the direct result of the transition to Unicode, but it is somehow related.

Tags: Linux, Python

Intel announces AVX2

The documentation is available for download.

The instruction set war is still there – Intel still doesn’t plan to support many XOP features of AMD; also Intel still plans to use FMA3 while AMD uses FMA4. Nevertheless, this time Intel is at least not making the war even worse. In addition to extending most SSE2/SSE3/SSE4 instructions to 256 bits (this is no surprise), they copied BMI (with an extension called BMI2) and CVT16 from AMD. If I recall correctly, Intel had never copied so many instructions from AMD at once, with the notable exception of x86-64.

Tags: assembly, hardware

Integer division

C89 and C++98 say the result of an integer division where the divisor and/or dividend is negative is implementation defined. This reflects that early hardware implemented integer divisions differently.

According to C89/C++98, we may have either (-3)/2 == -1 (round toward zero) or (-3)/2 == -2 (round toward negative infinity).

It appears round toward zero has become the overwhelming de facto standard now, adopted by both hardware and software vendors. Now both C and C++ explicitly require round toward zero in their new standards (C99 and C++2011*).

Division of negative integers has always been a complicated problem. Fortran mandated the same round-toward-zero mode much earlier than C/C++; so did Java. Python, on the other hand, has required round toward -∞ (i.e. (-3)//2 == -2) from its beginning. Everybody, nevertheless, agrees that a/b*b + a%b == a should always hold.

* C++0x has yet to be officially approved. Hopefully it will be approved within this year and known as C++2011. I’m using this name prematurely.

Tags: C++0x, C/C++, Python

A hack to strace -f

I have a multithreaded program which I would like to strace for debugging purpose. My program sometimes calls (fork and exec) an external program, which in turn calls a setuid program.

Because my program is multithreaded, I cannot omit the “-f” flag (also trace child threads and processes) when using strace. And because all children, including the setuid program, are traced, setuid fails. (Yes, I am aware that strace claims it is possible to trace setuid programs, but the trick does not work for me, probably because the setuid program is not directly executed by strace.)

Fortunately, the clone system call has many useful flags. It works fine for me when I substitute calls to fork() with:

(pid_t) syscall (__NR_clone, CLONE_UNTRACED|SIGCHLD, NULL);


(Yes, SIGCHLD, not CLONE_SIGCHLD. It’s not a typo.)

I guess there may be better solutions, without modifying the program being traced?

Tags: dev, Linux, multithread

Concatenate PDF files in Linux

Some people recommend using convert or gs. However, there is a major problem with them – all text and vector graphics become raster graphics.

pdftk (PDF ToolKit) is a better solution – it keeps text and vector graphics. We just have to use this command:

pdftk *.pdf cat output result.pdf

PDF Shuffler, written in Python and based on poppler, also does the trick, and has a nice GUI.

However, there are drawbacks for both pdftk and PDF Shuffler:

• pdftk only supports ASCII filenames. So it’s a bit inconvenient for non-English users like me.
• PDF Shuffler is way too slow. I tried concatenating several files (approx. 1000 pages), and it kept running for more than 10 minutes before I hit Ctrl-C; pdftk finished the same task in just a few seconds.

Tags: Linux

.note.GNU-stack

GCC always appends one line to any assembler file (.s) file it generates:

	.section	.note.GNU-stack,"",@progbits

Literally, it adds an empty section named .note.GNU-stack to the object file, but it actually serves a hint to the linker* that code in this object file does not require an executable stack. GNU assembler also accepts command-line option “--noexecstack”, which has the same effect.

If every object file contains a section of this name, the linker knows the whole program does not need an executable stack, and the resulting executable will run with a non-executable stack if the OS and underlying hardware support it (see also NX bit).

Why is this important? In practice, virtually no program needs an executable stack (hackers may sometimes use it, though), but buffer overflow attacks frequently insert and run code in stacks. A non-executable stack helps improve security without any overhead.

* GNU linker only.

Tags: assembly, security

Convert PDF to images in Linux

Just use convert, the universal image converter shipped with ImageMagick.

convert a.pdf a.png

And we get as many PNG files as there are pages in the PDF. They converted files are named a-0.png, a-1.png, …

We can also use it the other way around:

convert a.jpg b.png c.gif abc.pdf

This will combine the three images into one PDF file. Very flexible.

Tags: image, Linux

man 3 sleep

The man page of sleep(3) used to say:

sleep() makes the calling process sleep until seconds seconds have elapsed or a signal arrives which is not ignored. [Old version; color added]

Now it says:

sleep() makes the calling thread sleep until seconds seconds have elapsed or a signal arrives which is not ignored. [New version]

Alas, they finally did the right thing – we have waited for at least 5 years. This change happened some time between versions 3.23 and 3.26.

There is indeed no reason to use “process” anymore, at least since the emergence of Linux 2.6 and NPTL. As far as I know, this very sentence has confused many newbie Linux programmers who are not familiar with the history of Linux multithreading, and has led some to firmly believe there is no “real” thread or threads are actually processes under Linux, a statement which was probably right for the obsolete implementation LinuxThreads, but definitely wrong today.

Tags: Linux, multithread