PROBLEM WITH SAVED IMAGES AND OTHER THINGS ON FC4 AND FC5
CAUSED BY DEFAULT SECURITY SETTINGS

News: 11 Aug 2006

Problems turned up in Fedora Core 4 due to the RedHat security modifications (selinux ?) to Linux. The same problems were found in FC 5. So I present a summary here, and a solution.

The summary, extracted from the Free Poplog website is

• For a temporary solution:
  To turn off randomization in memory allocator 'malloc' temporarily give this command as super-user:

  sysctl -w kernel.randomize_va_space=0

  To temporarily turn off Exec-shield (a security modification to the Linux kernel that makes large parts of specially-marked programs -- including their stack -- not executable) give this command as super-user

  sysctl -w kernel.exec-shield=0

• For a permanent solution turn both off as follows
  As super-user edit your /etc/sysctl.conf file to contain these lines:

  kernel.randomize_va_space=0
  kernel.exec-shield=0

I have heard that the problems don't arise on 64bit FC5.

Background information and further details

A message, describing a problem and a partial solution, was posted to the comp.lang.pop news group and pop-forum email list on 19 Mar 2006

I previously reported a mysterious problem with saved images on fedora core 4 and also some of the more recent FC 3 kernels with SWSUSP available here:

http://mhensler.de/swsusp/download_en.php

The problem was that running saved images did not work: e.g. the 'pop11' command, which is equivalent to

    basepop11 +startup

worked only randomly.

Later when I tried a machine running FC4, I thought the problem had been solved, but when a colleague installed FC4 he reported the same problem with saved images, though he could run basepop11 and compile the code for the saved images, losing a little time whenever he started. Chris Dollin also reported the problem.

Last June John Duncan wrote

> Have you seen this,
>
>     http://dag.wieers.com/howto/compatibility/
>
> Seems like many of the security enhancements in Fedora Core
> might well interfere with Poplog. You could try turning them off one
> by one using these instructions and see what happens.

Today I finally tried using the information there, but was not able to fix the problem until I made a lucky guess that worked.

That file states:

> Exec-shield randomizes VM mappings, this may confuse programs that do
> not expect this behaviour (and for that reason alone are considered
> broken).
>
> To disable it permanently system-wide, add the following to
> /etc/sysctl.conf:
>     kernel.exec-shield-randomize = 0
> And make it effective by running:
>     sysctl -w kernel.exec-shield-randomize=0

However when I tried that I got

    kernel.exec-shield-randomize' is an unknown key

After searching for other things, I had the idea of reading the man file for sysctl, where I found that '-a' shows the available values. So I did

    sysctl -a | grep rand

and got this as one of the outputs:

    kernel.randomize_va_space = 1

So I tried (nervously)

    sysctl -w kernel.randomize_va_space=0

and after that the saved images started every time!

So I have inserted the following in /etc/sysctl.conf

   # Prevent problems starting saved images
   kernel.randomize_va_space=0

I also later found this online discussion of the issue:

    http://www.ussg.iu.edu/hypermail/linux/kernel/0504.3/1451.html

which is part of quite a long inconclusive thread about how to do this on a per program basis instead of globally.

It seems that the randomize_va_space mechanism is designed to thwart some viruses. My understanding is that other linux distributions do not use it. It does not seem to be a good security mechanism for a general linux installation if it can break random programs and there is no easy way to turn it off for programs known to be 'safe'. (People reported trying to use 'setarch' without success.)

Anyhow, this information should help users of recent Fedora Core or Redhat systems who wish to use Poplog (and other systems defeated by the security settings) and are willing to turn off a security measure that is not included in other linux distributions.

Maintained by Aaron Sloman A.Sloman@cs.bham.ac.uk
Last updated: 20 Aug 2006 Aug 2006