Discussion:
Building glibc 2.19 for OS/ABI UNIX - System V
Shaun Jackman
2014-07-24 18:54:52 UTC
Permalink
I’m building glibc 2.19 on CentOS 5.10. It built, and that makes me
happy. It unfortunately doesn’t load with the system’s default loader
/lib64/ld-2.5.so because the ABI UNIX - Linux is incompatible. How can
I build glibc 2.19 that has an OS/ABI of UNIX - System V rather than
UNIX - Linux?

Thanks,
Shaun

$ LD_LIBRARY_PATH=~/local/Cellar/glibc/2.19/lib ~/local/bin/hello
/home/sjackman/local/bin/hello: error while loading shared libraries:
/home/sjackman/local/Cellar/glibc/2.19/lib/libc.so.6: ELF file OS ABI
invalid
$ readelf -h ~/local/Cellar/glibc/2.19/lib/libc-2.19.so |grep ABI
OS/ABI: UNIX - Linux
ABI Version: 0

glibc 2.19
Linux headers 3.15.6
binutils 2.24
gcc 4.4.7
CentOS 5.10

http://sjackman.ca
Carlos O'Donell
2014-07-24 18:59:29 UTC
Permalink
Post by Shaun Jackman
$ LD_LIBRARY_PATH=~/local/Cellar/glibc/2.19/lib ~/local/bin/hello
/home/sjackman/local/Cellar/glibc/2.19/lib/libc.so.6: ELF file OS ABI
invalid
You can't run your binary that way.

Please see this wiki page about building and testing your builds:
https://sourceware.org/glibc/wiki/Testing/Builds

You need to run the binary under the *new* dynamic loader, and using
the *new* core libraries.

We do not support using the old dynamic loader with new libraries.

Cheers,
Carlos.
Shaun Jackman
2014-07-24 19:14:33 UTC
Permalink
Post by Carlos O'Donell
Post by Shaun Jackman
$ LD_LIBRARY_PATH=~/local/Cellar/glibc/2.19/lib ~/local/bin/hello
/home/sjackman/local/Cellar/glibc/2.19/lib/libc.so.6: ELF file OS ABI
invalid
You can't run your binary that way.
https://sourceware.org/glibc/wiki/Testing/Builds
You need to run the binary under the *new* dynamic loader, and using
the *new* core libraries.
We do not support using the old dynamic loader with new libraries.
Cheers,
Carlos.
Hi, Carlos. Thanks for your quick response. Yes, it does work as
expected when I use the newly built loader with the newly built glibc.
I can't unfortunately replace the system's default loader in /lib, and
there's no environment variable equivalent of `LD_LIBRARY_PATH` that I
know of to set the default loader.

Is it possible to build glibc 2.19 with an OS/ABI of UNIX - System V?
If not, is there a component of my toolchain (Linux headers, binutils,
glibc) that I can downgrade to an earlier release that will build a
UNIX - System V glibc?

Thanks,
Shaun
Carlos O'Donell
2014-07-24 19:21:19 UTC
Permalink
Post by Shaun Jackman
Hi, Carlos. Thanks for your quick response. Yes, it does work as
expected when I use the newly built loader with the newly built glibc.
I can't unfortunately replace the system's default loader in /lib, and
there's no environment variable equivalent of `LD_LIBRARY_PATH` that I
know of to set the default loader.
That is correct, there is no environment variable that sets the dynamic loader.

The kernel selects it solely on the INTERP program header of the binary.
Post by Shaun Jackman
Is it possible to build glibc 2.19 with an OS/ABI of UNIX - System V?
Yes, but that's not the problem, and it won't help you.
Post by Shaun Jackman
If not, is there a component of my toolchain (Linux headers, binutils,
glibc) that I can downgrade to an earlier release that will build a
UNIX - System V glibc?
This is an X Y problem. Why don't you start by explaining what's wrong
and why you want to upgrade the core runtimes?

Cheers,
Carlos.
Shaun Jackman
2014-07-24 19:44:00 UTC
Permalink
Post by Carlos O'Donell
Post by Shaun Jackman
Hi, Carlos. Thanks for your quick response. Yes, it does work as
expected when I use the newly built loader with the newly built glibc.
I can't unfortunately replace the system's default loader in /lib, and
there's no environment variable equivalent of `LD_LIBRARY_PATH` that I
know of to set the default loader.
That is correct, there is no environment variable that sets the dynamic loader.
The kernel selects it solely on the INTERP program header of the binary.
Post by Shaun Jackman
Is it possible to build glibc 2.19 with an OS/ABI of UNIX - System V?
Yes, but that's not the problem, and it won't help you.
Post by Shaun Jackman
If not, is there a component of my toolchain (Linux headers, binutils,
glibc) that I can downgrade to an earlier release that will build a
UNIX - System V glibc?
This is an X Y problem. Why don't you start by explaining what's wrong
and why you want to upgrade the core runtimes?
I'd like to run a binary executable that was compiled on a newer
system (Ubuntu 14.04) on an older system (CentOS 5.10). To that end,
I'm compiling glibc 2.19 on the older system (CentOS 5.10). I realize
that the short answer is probably "Don't do that". All the same, call
it a learning experience for me. If it doesn't work and can't work,
I'd like to learn first hand why it doesn't work.

The current problem I'm facing is the error message `ELF file OS ABI
invalid` because the `libc-2.19.so` that I've compiled on CentOS 5.10
uses an ABI unsupported by the system's loader `/lib64/ld-2.5.so`. Is
it possible to get past this without using a newer loader?

Using a newer loader, although possible, would make running the new
executable(s) pretty cumbersome. I could either wrap all the new
executables in a shell script that runs the new loader, pretty ugly,
or I could patch the ELF header to specify the new dynamic linker (the
effect of `ld --dynamic-linker`). I'd prefer to use the old loader, if
possible.

Cheers,
Shaun
Carlos O'Donell
2014-07-24 20:10:29 UTC
Permalink
Post by Shaun Jackman
I'd like to run a binary executable that was compiled on a newer
system (Ubuntu 14.04) on an older system (CentOS 5.10). To that end,
I'm compiling glibc 2.19 on the older system (CentOS 5.10). I realize
that the short answer is probably "Don't do that". All the same, call
it a learning experience for me. If it doesn't work and can't work,
I'd like to learn first hand why it doesn't work.
It is not my place to judge your requirements or needs, your question
is a perfectly valid one, and I have some answers for you.

To run a newer application on an older system requires *all* of
runtimes that are needed by the newer application.

It does work, and I've done things like run Chrome 30 on RHEL6 but it
requires 35+ newer libraries, changing the scripts to call the new
ld.so, and because I don't want to modify the chrome binaries
(licensing reasons) I preload a DSO that interposes all of the exec*
family of functions to run any helper programs under the right
runtimes.
Post by Shaun Jackman
The current problem I'm facing is the error message `ELF file OS ABI
invalid` because the `libc-2.19.so` that I've compiled on CentOS 5.10
uses an ABI unsupported by the system's loader `/lib64/ld-2.5.so`. Is
it possible to get past this without using a newer loader?
Yes it is possible to revert the OS ABI changes, but that doesn't
solve your problem.

Let me reiterate why.

All of the core runtime, loader, libc, libm, libpthread, libdl, etc,
which are built by glibc, must be upgraded *together* or downgraded
*together*. They are coordinating between each other to provide the
implementation and it is unsupported to mix and match.
Post by Shaun Jackman
Using a newer loader, although possible, would make running the new
executable(s) pretty cumbersome. I could either wrap all the new
executables in a shell script that runs the new loader, pretty ugly,
or I could patch the ELF header to specify the new dynamic linker (the
effect of `ld --dynamic-linker`). I'd prefer to use the old loader, if
possible.
In order of least isolation to most isolation you have the following options:

(a) Copy new versions of all the new libraries you need to your
system. Rewriting the program header e.g. INTERP and adding DT_RPATH
to force the application to search in a special place first. This
option is great since you use the dyanmic loader to your advantage to
ensure the right libraries are used.

(b) Copy new versions of all the new libraries you need to your
system. Wrap all executables in shell scripts that invoke them via the
new loader and the right `--library-path` options, possibly preloading
a shared library that interposes exec*. To give you an idea of what an
interposer library would look like see the attached exec_wrapper.c (it
may make your eyes bleed) that I've used to run chrome in a contained
runtime. This option is harder but allows you to avoid changing any
binaries if licensing is a problem.

(c) Setup a Ubuntu chroot, chroot into it, and run your application.

(d) Setup a linux container (cgroup) with a Ubuntu chroot, and run
your application in there.

(e) Setup a Ubuntu VM and run your application there.

Lastly, the core runtimes are backwards compatible, and we work very
very hard to keep it that way (at least on x86*). You could upgrade
glibc to a newer glibc from say Fedora, but at that point you might
want to stop compiling applications on that system. Any newly compiled
applications might possibly use newer symbols from the newer glibc
(contaminated with a newer ABI) and you could never run those
applications on other Cent OS 5.10 boxes because they would not have
the newer glibc installed. However, nothing stops you from installing
a new core runtime... except that we don't guarantee bug backwards
compatibility or undefined behaviour compatibility, so some other old
applications may stop working (despite the fact that glibc is
correct).

The question of running newer applications on older distributions is a
very old question, and one that has subtle tradeoffs.

The industry believes the ultimate solution is going to be containers,
but that's just a synonym for "static linking" and has it's own
problems e.g. all your containers, not just all of your systems, need
updating if there is a security bug in a shipped component. So it's
much more like a light-weight VM.

In full disclosure, I work for Red Hat and I am the team lead for
glibc within the base OS tools team. I could talk all day about these
problems if you're interested, but I figure you've got a specific
problem to solve :-)

Cheers,
Carlos.
Shaun Jackman
2014-07-24 22:58:18 UTC
Permalink
Post by Carlos O'Donell
Post by Shaun Jackman
I'd like to run a binary executable that was compiled on a newer
system (Ubuntu 14.04) on an older system (CentOS 5.10). To that end,
I'm compiling glibc 2.19 on the older system (CentOS 5.10). I realize
that the short answer is probably "Don't do that". All the same, call
it a learning experience for me. If it doesn't work and can't work,
I'd like to learn first hand why it doesn't work.
It is not my place to judge your requirements or needs, your question
is a perfectly valid one, and I have some answers for you.
To run a newer application on an older system requires *all* of
runtimes that are needed by the newer application.
It does work, and I've done things like run Chrome 30 on RHEL6 but it
requires 35+ newer libraries, changing the scripts to call the new
ld.so, and because I don't want to modify the chrome binaries
(licensing reasons) I preload a DSO that interposes all of the exec*
family of functions to run any helper programs under the right
runtimes.
I'm okay with upgrading dynamic libraries. That I can accomplish with
RPATH, RUNPATH and LD_LIBRARY_PATH. The two things that I cannot
upgrade easily are the Linux kernel 2.6.18 and the system's default
loader /lib64/ld-2.5.so.
Post by Carlos O'Donell
Post by Shaun Jackman
The current problem I'm facing is the error message `ELF file OS ABI
invalid` because the `libc-2.19.so` that I've compiled on CentOS 5.10
uses an ABI unsupported by the system's loader `/lib64/ld-2.5.so`. Is
it possible to get past this without using a newer loader?
Yes it is possible to revert the OS ABI changes, but that doesn't
solve your problem.
I'd like to explore reverting the OS ABI as one possible solution. I
understand that running a newer glibc with an older ld.so is not
supported, and if it goes up in smoke, I only have myself to blame.
Could you please elaborate on how to compile a glibc that uses the
older OS ABI?
Post by Carlos O'Donell
Let me reiterate why.
All of the core runtime, loader, libc, libm, libpthread, libdl, etc,
which are built by glibc, must be upgraded *together* or downgraded
*together*. They are coordinating between each other to provide the
implementation and it is unsupported to mix and match.
I can upgrade all these components of glibc except the system default loader.
Post by Carlos O'Donell
Post by Shaun Jackman
Using a newer loader, although possible, would make running the new
executable(s) pretty cumbersome. I could either wrap all the new
executables in a shell script that runs the new loader, pretty ugly,
or I could patch the ELF header to specify the new dynamic linker (the
effect of `ld --dynamic-linker`). I'd prefer to use the old loader, if
possible.
(a) Copy new versions of all the new libraries you need to your
system. Rewriting the program header e.g. INTERP and adding DT_RPATH
to force the application to search in a special place first. This
option is great since you use the dyanmic loader to your advantage to
ensure the right libraries are used.
This worked really well! I used PatchELF to modify the dynamic loader
and the RPATH of the executable. It'll have to be done on deployment
to each system, because the location of the new glibc is in the user's
home directory, but it's possible. I don't suppose there's support to
expand environment variables like ${HOME} in the ELF header, is there?

patchelf --set-interpreter ${libc}/lib/ld-linux-x86-64.so.2
--set-rpath ${prefix}/lib foo

[patchelf]: https://nixos.org/patchelf.html
Post by Carlos O'Donell
(b) Copy new versions of all the new libraries you need to your
system. Wrap all executables in shell scripts that invoke them via the
new loader and the right `--library-path` options, possibly preloading
a shared library that interposes exec*. To give you an idea of what an
interposer library would look like see the attached exec_wrapper.c (it
may make your eyes bleed) that I've used to run chrome in a contained
runtime. This option is harder but allows you to avoid changing any
binaries if licensing is a problem.
If the patchelf solution above works, I won't explore this option.
Post by Carlos O'Donell
(c) Setup a Ubuntu chroot, chroot into it, and run your application.
chroot requires root permission, and the end user won't have root
access. The target machine is a high performance computing cluster.
Post by Carlos O'Donell
(d) Setup a linux container (cgroup) with a Ubuntu chroot, and run
your application in there.
Docker is a possible solution. It require root access to install, but
not to run. It requires a newer Linux kernel though.
Post by Carlos O'Donell
(e) Setup a Ubuntu VM and run your application there.
I'd rather not. Let's call that plan F.
Post by Carlos O'Donell
Lastly, the core runtimes are backwards compatible, and we work very
very hard to keep it that way (at least on x86*). You could upgrade
glibc to a newer glibc from say Fedora, but at that point you might
want to stop compiling applications on that system. Any newly compiled
applications might possibly use newer symbols from the newer glibc
(contaminated with a newer ABI) and you could never run those
applications on other Cent OS 5.10 boxes because they would not have
the newer glibc installed. However, nothing stops you from installing
a new core runtime... except that we don't guarantee bug backwards
compatibility or undefined behaviour compatibility, so some other old
applications may stop working (despite the fact that glibc is
correct).
I can't unfortunately upgrade the system's default libc/loader.
Post by Carlos O'Donell
The question of running newer applications on older distributions is a
very old question, and one that has subtle tradeoffs.
The industry believes the ultimate solution is going to be containers,
but that's just a synonym for "static linking" and has it's own
problems e.g. all your containers, not just all of your systems, need
updating if there is a security bug in a shipped component. So it's
much more like a light-weight VM.
I agree with you here.
Post by Carlos O'Donell
In full disclosure, I work for Red Hat and I am the team lead for
glibc within the base OS tools team. I could talk all day about these
problems if you're interested, but I figure you've got a specific
problem to solve :-)
Thanks for your help, Carlos. This discussion has already been fruitful.

Cheers,
Shaun
Carlos O'Donell
2014-07-25 02:54:47 UTC
Permalink
Post by Shaun Jackman
Post by Carlos O'Donell
(a) Copy new versions of all the new libraries you need to your
system. Rewriting the program header e.g. INTERP and adding DT_RPATH
to force the application to search in a special place first. This
option is great since you use the dyanmic loader to your advantage to
ensure the right libraries are used.
This worked really well! I used PatchELF to modify the dynamic loader
and the RPATH of the executable. It'll have to be done on deployment
to each system, because the location of the new glibc is in the user's
home directory, but it's possible. I don't suppose there's support to
expand environment variables like ${HOME} in the ELF header, is there?
patchelf --set-interpreter ${libc}/lib/ld-linux-x86-64.so.2
--set-rpath ${prefix}/lib foo
[patchelf]: https://nixos.org/patchelf.html
Given your deployment restrictions this is likely your best option.
Particularly given that you don't want to require an admin to install
the application.

To answer your other question: there are dynamic string tokens (DST)
you can use.

See: http://man7.org/linux/man-pages/man8/ld.so.8.html

However, IIRC they are expanded only for DT_RPATH, but I'd verify that
they don't also expand for INTERP.

It doesn't seem like a bad enhancement to expand DSTs for INTERP to
allow self-contained deployments like the one you are designing.

Cheers,
Carlos.
Paweł Sikora
2014-07-25 06:24:50 UTC
Permalink
Post by Shaun Jackman
I’m building glibc 2.19 on CentOS 5.10. It built, and that makes me
happy. It unfortunately doesn’t load with the system’s default loader
/lib64/ld-2.5.so because the ABI UNIX - Linux is incompatible. How can
I build glibc 2.19 that has an OS/ABI of UNIX - System V rather than
UNIX - Linux?
you need to configure gcc in non-standard way to get binaries
compatible with old ld.so.

configure \
--with-linker-hash-style=both
--disable-gnu-unique-object
--disable-initfini-array

with these additional options all should work on ancient rhel/centos.
Shaun Jackman
2014-07-26 01:05:20 UTC
Permalink
Post by Paweł Sikora
Post by Shaun Jackman
I’m building glibc 2.19 on CentOS 5.10. It built, and that makes me
happy. It unfortunately doesn’t load with the system’s default loader
/lib64/ld-2.5.so because the ABI UNIX - Linux is incompatible. How can
I build glibc 2.19 that has an OS/ABI of UNIX - System V rather than
UNIX - Linux?
you need to configure gcc in non-standard way to get binaries
compatible with old ld.so.
configure \
--with-linker-hash-style=both
--disable-gnu-unique-object
--disable-initfini-array
with these additional options all should work on ancient rhel/centos.
Hi, Pawel. The odd thing is, GCC is the only component that I
*haven't* upgraded. I'm using the stock GCC 4.4.7 20120313 (Red Hat
4.4.7-1) that comes with CentOS 5.10. I did however upgrade binutils
to 2.24, because glibc requires binutils at least 2.20. Should I
compile binutils with these options?

Cheers,
Shaun

Loading...