Saturday, 16 July 2016

Development & Infrastructure - Hints and Tips

This post contains general bits and pieces that I’ve come across over the years, which don’t necessarily warrant a full blog posting. I’ll keep updating this as I go…

Solaris

  • Solaris 9: Enable core dumps:
    coreadm -e process 
    coreadm -i /var/core/core_%n_%f_%u_%g_%t_%p 
    
    Where:
    %n = System name 
    %f = Executable file name
    %u = User ID 
    %g = Group ID 
    %t = Decimal value of time 
    %p = Process ID 
    
  • Solaris: NFS mount gives “No such file or directory” on the solaris client, edit /etc/default/nfs, and make sure to set NFS_CLIENT_VERSMAX=3.
  • Solaris 10: network service management:
    List services: svcs
    e.g. output:
    online         14:52:15 svc:/network/svn/tcp:default
    online         Jun_13   svc:/network/ftp:default
    
    Stop service: svcadm disable svc:/network/ftp
    Start service: svcadm enable svc:/network/ftp
    Delete service: svccfg delete svc:/network/ftp
    
  • Solaris 9: show PIDS listening on a port (e.g: 35302):
    ps -ef| awk '{print $2}'| xargs -I '{}' sh -c 'echo examining process {}; pfiles {}| grep "port: 35302$"'
    
  • Solaris 10: Oracle 10g Full 64bit client install problems Log file shows:
    ld.so.1: proc: fatal: /space/oracle/ora9i/lib32/libclntsh.so.9.0: wrong ELF class: ELFCLASS32
    
    Do this after the install:
    export LIB_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/lib32:$LIB_LIBRARY_PATH
    #Setup the environment manually, e.g:
    export ORACLE_HOME=/opt/oracle/product/10.2.0/client_1
    export LD_LIBRARY_PATH=$ORACLE_HOME/lib
    export PATH=$PATH:$ORACLE_HOME/bin
    cd :$ORACLE_HOME/bin ; ./relink all
    
  • ls /proc/*/lwp
    
    * being the pid of the application. This also works:
    ps -efL
    
  • Solaris 9: Add a package, as root e.g:
    pkgadd -d ./curl-7.21.6-sol9-sparc-local
    

AIX

  • NFS mount gives “mount: giving up on…vmount: Not owner” on AIX, run the following and then try again
    nfso -o nfs_use_reserved_ports=1
    
  • AIX: gcc compile complains:
    ld: 0711-317 ERROR: Undefined symbol: __crypt_r
    ld: 0711-317 ERROR: Undefined symbol: __setkey_r
    ld: 0711-317 ERROR: Undefined symbol: __encrypt_r
    
    If symbols such as these are definitely not used and you want to ignore them, create a file, e.g crypt.exp with the contents between START/END:
    #!
    __crypt_r
    __encrypt_r
    __setkey_r
    
    Then add the following to your Makefile CFLAGS: -Wl,-bI:crypt.exp -fPIC
  • AIX: login always gives - “3004-610 You are required to change your password.”. Clear password flag: pwdadm -c myuser

Linux

  • How to forward X when sudo to a different user
    you@server$ xauth list
    [output]
    you@server$ sudo su - otheruser
    otheruser@server$ xauth add [paste output from "xauth list"]
    otheruser@server$ xterm (or other X application)
    
  • Linux kernels 2.x hard-reset:
    echo 1 > /proc/sys/kernel/sysrq
    echo b > /proc/sysrq-trigger
    
  • RHEL 7.x: mount smbfs, Windows file share:
    mount -t smbfs -o username=myUser,workgroup=myDomain //nas/someFolder /mnt/nas
    
  • RHEL 6.x: enable FTP Access when SELinux is an issue:
    /usr/sbin/setsebool -P ftp_home_dir=1
    /usr/sbin/setsebool -P allow_ftpd_full_access=1
    
  • Redhat 5.x: force disk check on reboot (as root):
    touch /forcefsck
    
  • Linux, kill an open socket:
    fuser -k -n tcp 37
    
  • Linux kernel 2.6 core dump enable:
    mkdir /var/core
    chmod 0777 /var/core/
    echo /var/core/core.%e.%p > /proc/sys/kernel/core_pattern
    ulimit -c unlimited
    
  • Run a program via the Linux Valgrind profiler to check for leaks/issues:
    valgrind --leak-check=full -v --track-origins=yes --show-reachable=yes ./myApp.Linux -someAppargs
    
  • Linux 2.6+ kernel settings to mitigate TIME_WAIT issues visible in netstat (remember to update /etc/sysctl.conf)
    echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
    echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
    echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
    
  • Linux show process threads: ps -elfT
  • RHEL 6: Install alternative e.g javac: alternatives --install /usr/bin/javac javac /usr/java/latest/bin/javac 3
  • RHEL 6: Configure alternatives e.g java: alternatives --config java

UNIX General

  • Find extended non-ASCII characters in a file
    grep --color='auto' -P -n "[\x80-\xFF]" ./file.txt
    
  • Extract a cpio file: cpio -idmv < file
  • Vim tips
    vim Bracket matching
    Shift-%
    :syntax on
    # vim enable mouse pointing
    set mouse=a
    # Pasting a block of code copied from elsewhere in Vim can be a pain. The autoindent (and, possibly, smartindent) features of Vim go crazy when you yank and slap a block of text, so:
    set pastetoggle=
    # that way you can turn auto indenting for pastes on and off with one press of a key. Basically pastetoggle takes care of “:set paste” “:set nopaste”
    
  • OpenSSL: How to show an SSL certificates contents:
    cat client.pem | openssl x509 -noout -text
    
  • Net-SNMP: Generate .index file for mib names and files
    #!/bin/bash
    rm idx
    for file in `ls *.mib`
    do
      name=`grep ' DEFINITIONS ::= BEGIN' $file | awk {'print $1'}`
      echo $name $file >> idx
    done
    
  • Create a tar with in input file list:
    cat list.txt|xargs tar cf archive.tar
    
  • Net-SNMP: Create a MIB= list from a bunch of mib files:
    grep BEGIN ./*.txt | sort | uniq | awk -F':' {'print $2'} | grep -v 'BEGIN' | awk {'print $1'} > list
    for file in `cat list`
    do
    name=$name:$file
    done
    echo $name
    
  • Check CPU endianness where 0=big, 1=little:
    echo I | tr -d [:space:] | od -to2 | head -n1 | awk '{print $2}' | cut -c6 
    
    Solaris SPARC: big
    AIX PPC: big
    Linux x86_x64: little
    s390x: big
  • Get libc version:
    Linux RHEL/CentOS: rpm -qa|grep -i glibc
    Linux others: /lib/libc.so.6
    Others: strings /lib/libc.so.6 | grep GLIB
    

Windows

  • Create a Windows service from the cmd prompt:
    sc create "MyService"" binpath= "\"C:\Program Files\MyCompany\MyService\bin\myExecutable.exe\"" displayname= "My Service" start= auto obj= LocalSystem
    
  • Windows “Edit plan settings” opens every time I click enter - Press: windows-key x
  • Windows EventViewer Security log logon check query for a time period:
    <QueryList>
      <Query Id="0" Path="Security">
        <Select Path="Security">
    *[System[(EventID=4624)
    and
        TimeCreated[timediff(@SystemTime) &lt;= 2592000000]]
    and
        EventData[Data[@Name='LogonType'] and (Data='10' or Data='2' or Data='7' or Data='3')]
    and
        EventData[Data[@Name='TargetUserName'] and (Data='someDomainUser')]
    ]
    </Select>
      </Query>
      <Query Id="0" Path="Security">
        <Select Path="Security">
    *[System[(EventID=4634)
    and
        TimeCreated[timediff(@SystemTime) &lt;= 2592000000]]
    and
        EventData[Data[@Name='LogonType'] and (Data='10' or Data='2' or Data='7' or Data='3')]
    ]
    </Select>
      </Query>
    </QueryList>
    

Network

  • Packet sniff on a UNIX/Linux box:
    /usr/local/sbin/tcpdump -s 1500 -w /tmp/pcap.dmp "host 10.0.0.220 and port 1234"
    
    load pcap.dmp in to Wireshark on Windows to view/search contents easily
  • Missing “UNIX attributes” tab when setting up NIS on Windows 2008 domain controller:
    cd C:\Windows\idmu\common
    net stop “server for nis”
    regsvr32.exe /u nisprop.dll
    regsvr32.exe nisprop.dll
    net start “server for nis”
    

Programming Misc

  • C FAQ - http://www.c-faq.com/
  • Online ISO/ANSI C reference: http://www.opengroup.org/onlinepubs/009695399/idx/index.html
  • .NET show exe/dll headers:
    dumpbin YourProgram.exe /headers
    
  • .NET assembly table editor, orca:
    C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\Orca.msi
    
  • Disable advertised shortcuts in .msi Windows installer:
    cscript "C:\Program Files (x86)\Windows Installer 4.5 SDK\SAMPLES\SCRIPTS\WiRunSQL.vbs" "$(BuiltOuputPath)" "INSERT INTO Property(Property, Value) VALUES ('DISABLEADVTSHORTCUTS', '1')"
    
  • Running Powershell script shows: cannot be loaded because the execution of scripts is disabled on this system. As administrator run powershell and then enter: Set-ExecutionPolicy RemoteSigned
  • SVN import project:
    svn import -m "Initial import for JIRA-123" myApp svn://svnserver/applications/myApp
    
  • Apache Karaf jmx location for jconsole: service:jmx:rmi:///jndi/rmi://localhost:1099/karaf-root
  • Gradle: Skip tests: gradle build -x test
  • Git: Remove all refs to a file locally:
    git filter-branch --force --index-filter "git rm --cached --ignore-unmatch src/test/resources/config.properties" --prune-empty -- --all
    
  • Maven: set project and sub-modules version Using release plugin:
    mvn release:update-versions -DautoVersionSubmodules=true -DdevelopmentVersion=1.6.0-SNAPSHOT
    
    Using versions plugin:
    mvn versions:set -DnewVersion=1.6.3-SNAPSHOT
    

Database

  • MS SQL Server 2008: Delete inuse DB:
    osql -E
    use master
    go
    alter database [besrep] set single_user with rollback immediate
    go
    drop database besrep
    go
    
  • MS SQL Server 2008: drop multiple tables like:
    DECLARE @sql NVARCHAR(MAX) = N'';
    SELECT @sql += '
    DROP TABLE ' 
        + QUOTENAME(s.name)
        + '.' + QUOTENAME(t.name) + ';'
        FROM sys.tables AS t
        INNER JOIN sys.schemas AS s
        ON t.[schema_id] = s.[schema_id] 
        WHERE t.name LIKE 'ISS_PR_%';
    PRINT @sql;
    EXEC sp_executesql @sql
    
  • IBM DB2: diagnostic messages - These are usually found under the DB2 server/ client install ($DB2PATH) ./sqllib/db2dump/db2diag.log
  • Sybase ASE 15 Linux: error “os_create_region…”
        os_create_region: can't allocate 355016704 bytes 00:00:00000:00000:2011/05/13 10:56:35.54 kernel kbcreate: couldn't create kernel region. 00:00:00000:00000:2011/05/13 10:56:35.54 kernel kistartup: could not create shared memory
    
    Fix by increasing kernel.shmmax, e.g
    /sbin/sysctl -w kernel.shmmax=700000000
    # Also set: /etc/sysctl.conf:kernel.shmmax=700000000
    

Virtualisation

  • XenServer 6.x: mount a disk as local ext based storage Go to the console in XenCenter, prepare new partition with fdisk and then:
    xe sr-create type=lvm content-type=user device-config:device=/dev/disk/by-id/DISK name-label="LOCAL SR"
    
  • XenServer 6.x: set VM timeoffset: Get the uuid from: xe vm-list
    xe vm-param-set platform:timeoffset=0 uuid=53e7b30b-3f15-f5cc-d9ec-ee64ccbed52d
    
  • Docker: remove all IMAGES: docker rmi $(docker images -qf “dangling=true”)
  • Docker: delete all docker images: docker rmi $(docker images -q)
  • Docker: kill and REMOVE all containers: docker rm $(docker kill $(docker ps -aq))
  • Docker: attach and start shell: docker exec -i -t loving_heisenberg bash

Creating a System z (s390x) C Build Environment on CentOS 7 Linux using the Hercules Emulator

Why Emulate s390x?

I recently had a need to set-up a Jenkins CI compile worker for build C applications on a RHEL 5.6 s390x (System z) host. Rather than use expensive kit, which was otherwise already allocated for mission critical purposes I thought I’d see what other options there were.
I came across Hercules, the System/370, ESA/390, and z/Architecture emulator and ended up trying this on our XenServer infrastructure. After a day or two, to my astonishment, it worked extremely well!
This was the system I was aiming to reproduce:
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.6 (Tikanga)

# uname -a
Linux somehost.myprovider.com 2.6.18-238.el5 #1 SMP Sun Dec 19 14:27:28 EST 2010 s390x s390x s390x GNU/Linux

# cat /proc/cpuinfo
vendor_id       : IBM/S390
# processors    : 1
bogomips per cpu: 4587.52
features        : esan3 zarch stfle msa ldisp eimm dfp
processor 0: version = FF,  identification = 00000,  machine = 0000

# /lib/libc.so.6
GNU C Library stable release version 2.5, by Roland McGrath et al.

Installation and Configuration

Presumptions

  • LAN Configuration:
    • Domain: mydomain.com
    • Network: 10.0.0.0/24
    • Gateway: 10.0.0.1
    • DNS: 10.0.0.68
    • An IP has been reserved in DHCP for: 10.0.0.169 for the emulated s390x host

Prerequisites

  • XenServer 6.5 SP1 hypervisor with a host already setup with:
    • Host OS: CentOS 7 (via kickstart, with development tools installed)
    • RAM: 3072
    • vCPUs: 4
    • HD: 40GB
    • IP: 10.0.0.28/24

Steps to Reproduce

  1. Download rhel-server-5.6-s390x-dvd.iso. You need a RHN subscription to access this ISO from RH.
  2. As root, run:
    wget http://www.hercules-390.org/hercules-3.07.tar.gz
    yum groupinstall -y 'Development Tools' && yum install -y bzip2-devel.x86_64 bzip2-libs.x86_64 bzip2.x86_64 nfs-utils createrepo.noarch
    tar -xf hercules-3.07.tar.gz && cd hercules-3.07/ && ./configure --enable-cckd-bzip2 && make && make install
    sudo chmod u+s /usr/local/bin/hercifc
    sudo chmod o+rw /dev/net/tun
    
    export BASE=~/s390x
    # If using FirewallD
    #firewall-cmd --zone=internal --add-interface=tun0
    #firewall-cmd --zone=external --change-interface=eth0
    # If using tun/NAT device
    #sudo iptables -t nat -A POSTROUTING -s 192.168.200.0/24 -d 0.0.0.0/0 -j MASQUERADE
    
    sysctl -w net.ipv4.ip_forward=1
    sysctl -w net.ipv4.conf.all.proxy_arp=1
    echo net.ipv4.ip_forward=1 >> /etc/sysctl.conf
    echo net.ipv4.conf.all.proxy_arp=1 >> /etc/sysctl.conf
    
    CDMNT=/mnt/cdrom
    mkdir -p $BASE/dasd $BASE/images $CDMNT /mnt/repo
    # Ensure rhel-server-5.6-s390x-dvd.iso is copied to $BASE/images already
    mount -o loop $BASE/images/rhel-server-5.6-s390x-dvd.iso $CDMNT
    dasdinit -bz2 -linux $BASE/dasd/linux-ckd.130 3390-9 LNX000
    
    CDIMG=$CDMNT/images/
    cp $CDIMG/initrd.addrsize $BASE/images
    cp $CDIMG/initrd.img $BASE/images
    cp $CDIMG/kernel.img $BASE/images
    cp -fr $CDMNT/* /mnt/repo/
    cd /mnt/repo ; createrepo -v -s sha1 .
    
  3. Edit /etc/exports and add:
    /mnt/repo *(ro,sync)
    
  4. Run:
    service nfs restart
    
  5. Create: $BASE/images/generic.prm.kslvm:
    ro ramdisk_size=40000 selinux=0 text inst.zram=off ks=nfs:10.0.0.28:/mnt/repo/rh56-lvm.ks RUNKS=1
    
  6. Create: /mnt/repo/rh56-lvm.ks where 10.0.0.28 is the IP of the CentOS host VM:
    lang en_US.UTF-8
    key --skip
    install
    #cdrom
    nfs --server=10.0.0.28 --dir=/mnt/repo
    text
    skipx
    auth --enablemd5
    timezone Europe/London
    keyboard us
    
    zerombr
    bootloader --location=mbr
    clearpart --all --initlabel
    part / --fstype="ext3" --size=1 --grow
    part swap --recommended
    
    logging --level=debug
    rootpw redhat
    firewall --disabled
    selinux --disabled
    poweroff
    
    ## Minimal install
    %packages --ignoremissing
    
  7. Create: $BASE/ks-lvm.ins
    * minimal lpar ins file
    images/kernel.img 0x00000000
    images/initrd.img 0x02000000
    images/generic.prm.kslvm 0x00010480
    images/initrd.addrsize 0x00010408
    
  8. Create: $BASE/rh.cnf
    CPUSERIAL 002623              # CPU serial number
    CPUMODEL  2064                # CPU model number
    MODEL     EMULATOR            # STSI returned model
    PLANT     ZZ                  # STSI returned plant
    MANUFACTURER HRC              # STSI returned manufacturer
    LPARNAME  HERCULES            # DIAG 204 returned lparname
    CPUVERID  FD                  # CPU Version Identification
    MAINSIZE  2048                # Main storage size in megabytes
    XPNDSIZE  0                   # Expanded storage size in megabytes
    NUMCPU    2                   # Number of CPUs
    ARCHMODE  z/Arch              # Architecture mode S/370, ESA/390 or z/Arch
    #ECPSVM    20                  # VM Assist : NO or Level (20 recommended)
    LOADPARM  0120....            # IPL parameter
    OSTAILOR  LINUX               # OS tailoring
    SYSEPOCH  1900                # Base year for initial TOD clock
    CNSLPORT  3270                # TCP port number to which consoles connect
    
    LOGOPT NOTIMESTAMP      # Avoid timestamps for log messages
    TIMERINT 200    # Increase timers update interval, improves performance
    TODDRAG 2       # Higher TOD clock drag means less interrupts -> higher performance
    PANRATE SLOW    # Slowest update
    TRACEOPT NOREGS # Don't display registers
    
    #   .-----------------------Device number
    #   |     .-----------------Device type
    #   |     |       .---------File name and parameters
    #   |     |       |
    #   V     V       V
    # ----    ----    --------------------
    0009    3215-C  / noprompt
    000C    3505    images/kernel.img images/generic.prm.kslvm images/initrd.img autopad
    001F    3270
    
    0120    3390     dasd/linux-ckd.130
    # cdrom NOT USED
    #0160    9336     images/rhel-server-5.6-s390x-dvd.iso
    #
    # CTC network
    # run "iptables -t nat -A POSTROUTING -s 192.168.200.0/24 -d 0.0.0.0/0 -j MASQUERADE"
    # for allowing the RH VM to access external networks
    #0600.2    3088 CTCI /dev/net/tun 1492 192.168.200.13 192.168.200.14 255.255.255.255
    0700.2    3088 LCS 10.0.0.169 -m 00:11:22:33:44:58
    # /dev/net/tun 1492 192.168.200.13 192.168.200.14 255.255.255.255
    
  9. File structure in $BASE should now look like:
    ./images
    ./images/kernel.img
    ./images/rhel-server-5.6-s390x-dvd.iso
    ./images/initrd.addrsize
    ./images/initrd.img
    ./images/generic.prm.kslvm
    ./dasd
    ./dasd/linux-ckd.130
    ./ks-lvm.ins
    ./rh.cnf
    
  10. To fire off the rh install, run ‘screen’
    cd $BASE
    hercules -f $BASE/rh.cnf > $BASE/hercules.log
    ipl ks-lvm.ins
    
  11. After initrd has unpacked you are prompted for enter the following where 10.0.0.169 is a spare (reserved) IP
    Network type: .lcs
    Read,write channel: .0.0.0700,0.0.0701
    Hostname: .s390xrh.mydomain.com
    IPv4 address: .10.0.0.169
    netmask: .24
    Gateway: .10.0.0.1
    DNS server: .10.0.0.68
    Search domain: .mydomain.com
    DASD range: .
    .c
    
  12. Wait for the RHEL install to complete. When ‘you may safely reboot your system’ is displayed, type: ‘quit’ to exit hercules.
    Note:
    • Issues with hercules can be diagnosed via hercules.log (see pt.10)
    • Installer issues/prompts can be resolved by sending commands to stdin in Hercules by prefixing with a full-stop, e.g select menu options 8 -> .8
  13. Run:
    echo ipl 120 > $BASE/hercules.rc
    
  14. Now run Hercules again from the CentOS 7 host so it’s boots in to the installed OS:
    hercules -f $BASE/rh.cnf > $BASE/hercules.log
    
  15. SSH in from the CentOS 7 host or another host on the same network (password=redhat as set in the kickstart file):
    ssh -l root 10.0.0.169
    
  16. Create: /etc/yum.repos.d/RHEL_5.6_s390x_Disc.repo
    [RHEL_5.6_Disc]
    name=RHEL_5.6_s390x_Disc
    baseurl=file:///mnt/repo/
    gpgcheck=0
    
  17. Run the following to install the dev tools:
    mkdir /mnt/repo && mount 10.0.0.28:/mnt/repo /mnt/repo
    yum clean all
    yum repolist
    yum install -y strace autoconf automake binutils bison flex gcc gcc-c++ gettext libtool make patch pkgconfig redhat-rpm-config rpm-build byacc cscope ctags diffstat doxygen elfutils gcc-gfortran indent intltool patchutils rcs subversion swig systemtap
    

Conclusion

If you’ve followed the steps above, then your s390x development environment should now be setup and ready to compile openssl, zlib, etc… To test binary compatibility, I compiled our applications on it, transferred and tested on the real host and re-ran all the regression tests sucessfully.
Since this is an emulated host, it runs at a fraction the speed of an actual s390x host but is perfectly good enough for me (or Jenkins) to compile relatively small C applications on.
Our emulated environment now shows:
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.6 (Tikanga)

uname -a
Linux s390xrh.mydomain.com 2.6.18-238.el5 #1 SMP Sun Dec 19 14:27:28 EST 2010 s390x s390x s390x GNU/Linux

cat /proc/cpuinfo
vendor_id       : IBM/S390
# processors    : 2
bogomips per cpu: 1854.66
features        : esan3 zarch stfle msa ldisp eimm dfp
processor 0: version = 00,  identification = 000000,  machine = 2064
processor 1: version = 00,  identification = 000001,  machine = 2064

# /lib/libc.so.6
GNU C Library stable release version 2.5, by Roland McGrath et al.
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.1.2 20080704 (Red Hat 4.1.2-50).
Compiled on a Linux 2.6.9 system on 2010-12-14.
Available extensions:
        The C stubs add-on version 2.1.2.
        crypt add-on version 2.1 by Michael Glad and others
        GNU Libidn by Simon Josefsson
        GNU libio by Per Bothner
        NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
        Native POSIX Threads Library by Ulrich Drepper et al
        BIND-8.2.3-T5B
        RT using linux kernel aio
Thread-local storage support included.
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.
Notes:
  • To run the emulator, SSH to the CentOS host (10.0.0.28) and see pt. 14 to run Hercules
  • To SSH directly in to s390x, see pt. 15
All that’s left now is to hook this in to Jenkins, which I may blog about at a later date.
Anyway, I hope that saves somebody out there lots of time!
– Steve

References