Saturday, 16 July 2016

Creating a System z (s390x) C Build Environment on CentOS 7 Linux using the Hercules Emulator

Why Emulate s390x?

I recently had a need to set-up a Jenkins CI compile worker for build C applications on a RHEL 5.6 s390x (System z) host. Rather than use expensive kit, which was otherwise already allocated for mission critical purposes I thought I’d see what other options there were.
I came across Hercules, the System/370, ESA/390, and z/Architecture emulator and ended up trying this on our XenServer infrastructure. After a day or two, to my astonishment, it worked extremely well!
This was the system I was aiming to reproduce:
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.6 (Tikanga)

# uname -a
Linux somehost.myprovider.com 2.6.18-238.el5 #1 SMP Sun Dec 19 14:27:28 EST 2010 s390x s390x s390x GNU/Linux

# cat /proc/cpuinfo
vendor_id       : IBM/S390
# processors    : 1
bogomips per cpu: 4587.52
features        : esan3 zarch stfle msa ldisp eimm dfp
processor 0: version = FF,  identification = 00000,  machine = 0000

# /lib/libc.so.6
GNU C Library stable release version 2.5, by Roland McGrath et al.

Installation and Configuration

Presumptions

  • LAN Configuration:
    • Domain: mydomain.com
    • Network: 10.0.0.0/24
    • Gateway: 10.0.0.1
    • DNS: 10.0.0.68
    • An IP has been reserved in DHCP for: 10.0.0.169 for the emulated s390x host

Prerequisites

  • XenServer 6.5 SP1 hypervisor with a host already setup with:
    • Host OS: CentOS 7 (via kickstart, with development tools installed)
    • RAM: 3072
    • vCPUs: 4
    • HD: 40GB
    • IP: 10.0.0.28/24

Steps to Reproduce

  1. Download rhel-server-5.6-s390x-dvd.iso. You need a RHN subscription to access this ISO from RH.
  2. As root, run:
    wget http://www.hercules-390.org/hercules-3.07.tar.gz
yum groupinstall -y 'Development Tools' && yum install -y bzip2-devel.x86_64 bzip2-libs.x86_64 bzip2.x86_64 nfs-utils createrepo.noarch
tar -xf hercules-3.07.tar.gz && cd hercules-3.07/ && ./configure --enable-cckd-bzip2 && make && make install
sudo chmod u+s /usr/local/bin/hercifc
sudo chmod o+rw /dev/net/tun

export BASE=~/s390x
# If using FirewallD
#firewall-cmd --zone=internal --add-interface=tun0
#firewall-cmd --zone=external --change-interface=eth0
# If using tun/NAT device
#sudo iptables -t nat -A POSTROUTING -s 192.168.200.0/24 -d 0.0.0.0/0 -j MASQUERADE

sysctl -w net.ipv4.ip_forward=1
sysctl -w net.ipv4.conf.all.proxy_arp=1
echo net.ipv4.ip_forward=1 >> /etc/sysctl.conf
echo net.ipv4.conf.all.proxy_arp=1 >> /etc/sysctl.conf

CDMNT=/mnt/cdrom
mkdir -p $BASE/dasd $BASE/images $CDMNT /mnt/repo
# Ensure rhel-server-5.6-s390x-dvd.iso is copied to $BASE/images already
mount -o loop $BASE/images/rhel-server-5.6-s390x-dvd.iso $CDMNT
dasdinit -bz2 -linux $BASE/dasd/linux-ckd.130 3390-9 LNX000

CDIMG=$CDMNT/images/
cp $CDIMG/initrd.addrsize $BASE/images
cp $CDIMG/initrd.img $BASE/images
cp $CDIMG/kernel.img $BASE/images
cp -fr $CDMNT/* /mnt/repo/
cd /mnt/repo ; createrepo -v -s sha1 .
  3. Edit /etc/exports and add:
    /mnt/repo *(ro,sync)
  4. Run:
    service nfs restart
  5. Create: $BASE/images/generic.prm.kslvm:
    ro ramdisk_size=40000 selinux=0 text inst.zram=off ks=nfs:10.0.0.28:/mnt/repo/rh56-lvm.ks RUNKS=1
  6. Create: /mnt/repo/rh56-lvm.ks where 10.0.0.28 is the IP of the CentOS host VM:
    lang en_US.UTF-8
key --skip
install
#cdrom
nfs --server=10.0.0.28 --dir=/mnt/repo
text
skipx
auth --enablemd5
timezone Europe/London
keyboard us

zerombr
bootloader --location=mbr
clearpart --all --initlabel
part / --fstype="ext3" --size=1 --grow
part swap --recommended

logging --level=debug
rootpw redhat
firewall --disabled
selinux --disabled
poweroff

## Minimal install
%packages --ignoremissing
  7. Create: $BASE/ks-lvm.ins
    * minimal lpar ins file
images/kernel.img 0x00000000
images/initrd.img 0x02000000
images/generic.prm.kslvm 0x00010480
images/initrd.addrsize 0x00010408
  8. Create: $BASE/rh.cnf
    CPUSERIAL 002623              # CPU serial number
CPUMODEL  2064                # CPU model number
MODEL     EMULATOR            # STSI returned model
PLANT     ZZ                  # STSI returned plant
MANUFACTURER HRC              # STSI returned manufacturer
LPARNAME  HERCULES            # DIAG 204 returned lparname
CPUVERID  FD                  # CPU Version Identification
MAINSIZE  2048                # Main storage size in megabytes
XPNDSIZE  0                   # Expanded storage size in megabytes
NUMCPU    2                   # Number of CPUs
ARCHMODE  z/Arch              # Architecture mode S/370, ESA/390 or z/Arch
#ECPSVM    20                  # VM Assist : NO or Level (20 recommended)
LOADPARM  0120....            # IPL parameter
OSTAILOR  LINUX               # OS tailoring
SYSEPOCH  1900                # Base year for initial TOD clock
CNSLPORT  3270                # TCP port number to which consoles connect

LOGOPT NOTIMESTAMP      # Avoid timestamps for log messages
TIMERINT 200    # Increase timers update interval, improves performance
TODDRAG 2       # Higher TOD clock drag means less interrupts -> higher performance
PANRATE SLOW    # Slowest update
TRACEOPT NOREGS # Don't display registers

#   .-----------------------Device number
#   |     .-----------------Device type
#   |     |       .---------File name and parameters
#   |     |       |
#   V     V       V
# ----    ----    --------------------
0009    3215-C  / noprompt
000C    3505    images/kernel.img images/generic.prm.kslvm images/initrd.img autopad
001F    3270

0120    3390     dasd/linux-ckd.130
# cdrom NOT USED
#0160    9336     images/rhel-server-5.6-s390x-dvd.iso
#
# CTC network
# run "iptables -t nat -A POSTROUTING -s 192.168.200.0/24 -d 0.0.0.0/0 -j MASQUERADE"
# for allowing the RH VM to access external networks
#0600.2    3088 CTCI /dev/net/tun 1492 192.168.200.13 192.168.200.14 255.255.255.255
0700.2    3088 LCS 10.0.0.169 -m 00:11:22:33:44:58
# /dev/net/tun 1492 192.168.200.13 192.168.200.14 255.255.255.255
  9. File structure in $BASE should now look like:
    ./images
./images/kernel.img
./images/rhel-server-5.6-s390x-dvd.iso
./images/initrd.addrsize
./images/initrd.img
./images/generic.prm.kslvm
./dasd
./dasd/linux-ckd.130
./ks-lvm.ins
./rh.cnf
  10. To fire off the rh install, run ‘screen’
    cd $BASE
hercules -f $BASE/rh.cnf > $BASE/hercules.log
ipl ks-lvm.ins
  11. After initrd has unpacked you are prompted for enter the following where 10.0.0.169 is a spare (reserved) IP
    Network type: .lcs
Read,write channel: .0.0.0700,0.0.0701
Hostname: .s390xrh.mydomain.com
IPv4 address: .10.0.0.169
netmask: .24
Gateway: .10.0.0.1
DNS server: .10.0.0.68
Search domain: .mydomain.com
DASD range: .
.c
  12. Wait for the RHEL install to complete. When ‘you may safely reboot your system’ is displayed, type: ‘quit’ to exit hercules.
    Note:
    • Issues with hercules can be diagnosed via hercules.log (see pt.10)
    • Installer issues/prompts can be resolved by sending commands to stdin in Hercules by prefixing with a full-stop, e.g select menu options 8 -> .8
  13. Run:
    echo ipl 120 > $BASE/hercules.rc
  14. Now run Hercules again from the CentOS 7 host so it’s boots in to the installed OS:
    hercules -f $BASE/rh.cnf > $BASE/hercules.log
  15. SSH in from the CentOS 7 host or another host on the same network (password=redhat as set in the kickstart file):
    ssh -l root 10.0.0.169
  16. Create: /etc/yum.repos.d/RHEL_5.6_s390x_Disc.repo
    [RHEL_5.6_Disc]
name=RHEL_5.6_s390x_Disc
baseurl=file:///mnt/repo/
gpgcheck=0
  17. Run the following to install the dev tools:
    mkdir /mnt/repo && mount 10.0.0.28:/mnt/repo /mnt/repo
yum clean all
yum repolist
yum install -y strace autoconf automake binutils bison flex gcc gcc-c++ gettext libtool make patch pkgconfig redhat-rpm-config rpm-build byacc cscope ctags diffstat doxygen elfutils gcc-gfortran indent intltool patchutils rcs subversion swig systemtap

Conclusion

If you’ve followed the steps above, then your s390x development environment should now be setup and ready to compile openssl, zlib, etc… To test binary compatibility, I compiled our applications on it, transferred and tested on the real host and re-ran all the regression tests sucessfully.
Since this is an emulated host, it runs at a fraction the speed of an actual s390x host but is perfectly good enough for me (or Jenkins) to compile relatively small C applications on.
Our emulated environment now shows:
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.6 (Tikanga)

uname -a
Linux s390xrh.mydomain.com 2.6.18-238.el5 #1 SMP Sun Dec 19 14:27:28 EST 2010 s390x s390x s390x GNU/Linux

cat /proc/cpuinfo
vendor_id       : IBM/S390
# processors    : 2
bogomips per cpu: 1854.66
features        : esan3 zarch stfle msa ldisp eimm dfp
processor 0: version = 00,  identification = 000000,  machine = 2064
processor 1: version = 00,  identification = 000001,  machine = 2064

# /lib/libc.so.6
GNU C Library stable release version 2.5, by Roland McGrath et al.
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.1.2 20080704 (Red Hat 4.1.2-50).
Compiled on a Linux 2.6.9 system on 2010-12-14.
Available extensions:
        The C stubs add-on version 2.1.2.
        crypt add-on version 2.1 by Michael Glad and others
        GNU Libidn by Simon Josefsson
        GNU libio by Per Bothner
        NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
        Native POSIX Threads Library by Ulrich Drepper et al
        BIND-8.2.3-T5B
        RT using linux kernel aio
Thread-local storage support included.
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.
Notes:
  • To run the emulator, SSH to the CentOS host (10.0.0.28) and see pt. 14 to run Hercules
  • To SSH directly in to s390x, see pt. 15
All that’s left now is to hook this in to Jenkins, which I may blog about at a later date.
Anyway, I hope that saves somebody out there lots of time!
– Steve

References

Posted by at
Labels: , , , , , , ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)