Naming for conversion routines

Date Tags

I love code that self-documents and reads naturally, so I strive to name my routines and variables appropriately. When you are faced with the implementation of a conversion method, in generally the name is in the form "toB", where B is the result of the conversion, for example

rgb_values = color.toRgb()

The inverse, when we want to convert and integrate the conversion into an object's internal state, is generally in the form "fromB"

color.fromRgb(rgb_values)

When you have a conversion function, things are slightly different: "From" assumes a different weight and meaning. To perform a conversion color object to rgb you can use either

rgb_values = colorToRgb(color)

or

rgb_values = rgbFromColor(color)

The second works best because the proximity and order of the terms resembles the one naturally established by the prepended nature of the routine call and the order of the two objects. Everything moves from right to left in the second example, and it reads naturally. The first solution is clumsier and disorganized. Similarly, the reverse conversion will be

color = colorFromRgb(rgb_values)

The conclusion is that when you write a conversion function, prefer the "From" naming exclusively, while for methods you can use "To" and "From" forms for different conversion directions.


Installing ubuntu 14.04 on the new Dell XPS 13

This is a guest post from my friend Hans

As my old MSI X340 was in desperate need of replacement, I started looking for a new laptop with similar specs (preferably below 1.5kg, 13 inch, FullHD screen or better, and long battery life). As you can guess from the title, I ended up buying the new 2015 model of the Dell XPS 13. I opted for the non-touch FullHD screen with a matte (anti-glare) finish, as it is cheaper and gives you roughly 20% more battery life. It is also a little bit lighter and doesn't behave like a mirror (I like my terminals with a black background). My model features a Broadcom wifi chip, branded "Dell Wireless 1560 802.11ac".

You can find the specs and options on the Dell website.

I planned to run Ubuntu linux on the laptop, but did not want to wait for Dell to release their "Project Sputnik" version of the new XPS 13. You can find some info on Barton's Blog.

The fun thing about linux nerds is that a lot of them also did not want to wait, and Major Kayden's blog is currently the unofficial location for discussions about the new XPS 13 and solving the current issues (keyboard and touchpad, and sound).

Most of the technical work to get everything running on the XPS is not my own, and I will try to list all sources that helped me getting forward here:

Updating BIOS and creating a restore disk

We start by updating the BIOS to version A01 (or higher). Boot into Windows, go to the Dell Support website for the XPS 13 9343 and download the bios update. Install the BIOS update, and reboot the machine. You should now create a "self-starting backup disk" with the "Dell Backup and Recovery" program. You can find this on the right side of the start menu. It requires an 8 GB usb thumbdrive (or larger), and will be very useful if you ever need to restore the Windows installation on the machine.

Installing Ubuntu 14.04 (and getting the broadcom wifi online)

I do not want to go into too much detail here. Download the 14.04 LTS release from the ubuntu website, and create a bootable USB drive from the iso file. You can do this with the "Startup Disk creator" utility on another ubuntu machine, or with the UNetbootin utility. I would like to point out that installing ubuntu is not difficult, but the XPS 13 is currently not well supported by the default installation and thus should not be your first linux installation ever.

After creating the USB drive you need to find a way to get an internet connection on the XPS 13 after installing ubuntu:

  • Get a USB wifi/ethernet dongle that you know to be working out of the box on linux
  • Replace the wifi board inside of the XPS (hardcore solution, not advised)
  • download and copy the needed packages onto the USB drive and manually install them after booting. The needed packages are: bcmwl-kernel-source dkms fakeroot libfakeroot.

Next you will need to reboot the machine, and press F12 multiple times when the Dell logo shows, until you see a yellow text appearing in the top-right corner. Select your USB drive in the Boot mode menu and press enter. In the next menu (GNU GRUB) pick the top item (Try Ubuntu without installing) and press enter. If your USB drive is not showing in the boot menu options, you will have to disable "Secure boot" in the BIOS setup.

After Ubuntu is finshed booting, you will need to get an internet connection. Plug in your USB dongle or open a terminal window (CTRL+ALT+T) to install the downloaded packages. Enter the following commands to install the wifi driver

cd /cd-rom
sudo dpkg -i libfakeroot*.deb
sudo dpkg -i fakeroot*.deb
sudo dpkg -i dkms*.deb
sudo dpkg -i bcmwl-kernel-source*.deb

After these commands you should be able to connect to your wifi network.

Now start GParted partition editor from the applications (press the Windows key and type gparted). Resize the windows partition to create some space for the ubuntu installation. (I advise against a full wipe at this point. If Dell releases firmware or BIOS updates you will most likely need Windows to install them) Apply the changes, close GParted and start the installer. Follow the setup, and select "Something else" when you get to "Installation type".

You can create a single Ext4 partition with mount point "/", have a separate home partition, and/or create a swap partition. I will use swap with one big partition for root (/) and home. Select "free space" at the bottom of the list, and click the + button. In the "Create partition" window, pick a size for your swap partition (4096 or 8192 MB), set the location to "end of this space" and pick Use as: "swap area". Hit OK. Select the "free space" again, and click the + button. Set Mount point to "/". Hit OK. Click on "Install Now" to proceed, and finish the installation.

Reboot the machine, and fix the internet connection again. If you have a USB dongle plugged in you can open a terminal and type

sudo apt-get install bcmwl-kernel-source

Press Y and enter when asked if you want to proceed.

If you downloaded the packages onto the USB drive, open the file browser and copy the 4 packages to the Downloads folder in your home directory. Open a terminal and type

cd Downloads
sudo dpkg -i libfakeroot*.deb
sudo dpkg -i fakeroot*.deb
sudo dpkg -i dkms*.deb
sudo dpkg -i bcmwl-kernel-source*.deb

You should now be able to use the wifi menu to connect to your wireless network!

Update the machine (use the software updater from the dash). Most things should be working now, because the 3.13 kernel that comes with Ubuntu 14.04 puts the touchpad, keyboard and sound card in PS2 mode. However, if you upgrade your kernel to a newer version the hardware will try to switch to I2C mode, which improves battery life and some other things (see Major's blog for some details).

Getting stuff working with a newer kernel

We will need to start with compiling a very recent kernel, with some patches to it. We will also need the latest linux-firmware. Fortunately, Rene Treffer created a git repo for this which we can clone, so we don't have to do the patching! We do need some aditional packages to build the kernel. We will also need a .config file, which is a mixture between the ubuntu kernel 3.13 default config and the .config provided by Rene. Open a terminal and enter the following commands

sudo apt-get install git build-essential kernel-package fakeroot libncurses5-dev dh-modaliases debhelper devscripts
cd $HOME
git clone https://github.com/rtreffer/linux.git
git clone git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
ln -s ../../linux-firmware/intel linux/firmware/intel
cd linux
wget http://forthescience.org/blog/wp-content/uploads/2015/03/linux-kernel_4.0rc4-config-ubu1404-xps13 -O .config
make oldconfig
make clean
make -j 4 deb-pkg LOCALVERSION=-xps13

This last command compiles the linux kernel, and after compilation creates installation packages. Kernel compilation takes quite a bit of time, so make some coffee or tea and patiently wait for it to finish. When the compilation is done you wil end up with 5 .deb images in your home folder. Enter the following commands in the terminal window to install them

cd $HOME
sudo dpkg -i linux-headers-*xps13_*.deb
sudo dpkg -i linux-image-*xps13_*.deb
sudo dpkg -i linux-firmware-image-*xps13_*.deb
cd /lib
sudo mv firmware firmware-old
sudo cp -r $HOME/linux-firmware firmware

Congratulations! You just compiled and installed the linux kernel! But wait... something broke...

koekie@XPS13:~$ sudo dpkg -i linux-image-4.0.0-rc4-xps13_4.0.0-rc4-xps13-1_amd64.deb
Selecting previously unselected package linux-image-4.0.0-rc4-xps13.
(Reading database ... 215546 files and directories currently installed.)
Preparing to unpack linux-image-4.0.0-rc4-xps13_4.0.0-rc4-xps13-1_amd64.deb ...
Unpacking linux-image-4.0.0-rc4-xps13 (4.0.0-rc4-xps13-1) ...
Setting up linux-image-4.0.0-rc4-xps13 (4.0.0-rc4-xps13-1) ...
ERROR (dkms apport): kernel package linux-headers-4.0.0-rc4-xps13 is not supported
Error! Bad return status for module build on kernel: 4.0.0-rc4-xps13 (x86_64)
Consult /var/lib/dkms/bcmwl/6.30.223.248+bdcom/build/make.log for more information.

The wifi driver fails to compile a kernel module for the 4.0 kernel. to fix this we will need to check out the source of the bcmwl-kernel-source package, apply a patch, package it and then install it. You can find the patch in comment #4 of this bug report. Start a terminal and type the following commands

cd $HOME
mkdir broadcomwifi
cd broadcomwifi
apt-get source bcmwl-kernel-source
wget https://bugs.launchpad.net/ubuntu/+source/bcmwl/+bug/1424676/+attachment/4327652/+files/0017-add-support-for-Linux-4.0.patch
cd bcmwl-6.30.223.248+bdcom/src
patch -p1 < $HOME/broadcomwifi/0017-add-support-for-Linux-4.0.patch
cd $HOME/broadcomwifi/bcmwl-6.30.223.248+bdcom
debchange --increment "patched the package for kernel 4.0 compatibility"
dpkg-buildpackage
cd ..
sudo dpkg -i bcmwl-kernel-source_6.30.223.248+bdcom-0ubuntu0.2_amd64.deb

Now you will need to do 2 restarts, and one cold boot (shutdown the machine, and power it back on).

The audio board should now be detected. You can open the sound settings from the small speaker in the top-right corner, it should list broadwell-rt286 on the output tab. Switch to the input tab to check if that also lists the broadwell-rt286, and then back to output. Open a new terminal window and start alsamixer. Press F6 and select the broadwell-rt286 device. on the playback tab (F3), set the following settings:

  • Master: 100 (all the way up, arrow keys)
  • Headphones: 00 (press m to flip between mute and on, 00 means on)
  • Speaker: 00
  • Front DAC: 00
  • Front REC: mm
  • ADC 0 Mux: Dmic (arrow keys)
  • ADC 1 Mux: Dmic
  • AMIC: 100
  • DAC0: 100
  • HPO L: 00
  • HPO Mux: front
  • HPO R: 00
  • Media0: 100
  • Media1: 100
  • RECMIX Beep: 00
  • RECMIX line1: 00
  • RECMIX mic1: 00
  • SPK Mux: front
  • SPO: 00

Switch to the record tab (F4), and set the following settings:

  • Mic: 100
  • ADC0: 100 and CAPTURE (use the space bar to set CAPTURE)
  • AMIC: 100

Congratulations, you should now have sound! Open your favorite youtube clip to check this.

You might have noticed that the bluetooth is in a weird state, it shows as working and sometimes manages to detect devices, but it doesn't work properly. Shawn Tan posted a way to fix this. Download the Windows drivers from Microsoft, open a terminal and run the following commands

sudo apt-get install cabextract
cd $HOME/Downloads
git clone https://github.com/jessesung/hex2hcd.git
cd hex2hcd
make
cd ..
mkdir btcab
cd btcab
cabextract ../20662520_6c535fbfa9dca0d07ab069e8918896086e2af0a7.cab
../hex2hcd/hex2hcd BCM20702A1_001.002.014.1443.1572.hex ../BCM20702A0-0a5c-216f.hcd
cd ..
sudo cp BCM20702A0-0a5c-216f.hcd /lib/firmware/brcm

After a reboot your bluetooth should now be working.

Some other tweaks

I prefer my bluetooth to be off by default. Open a terminal window, and edit /etc/rc.local to add "rfkill block bluetooth" before the last line. Open a terminal and enter

sudo sed -i -e 's/^exit\ 0$/rfkill\ block\ bluetooth\nexit\ 0/' /etc/rc.local

We can also improve the behaviour of the touchpad a bit, by enabling the "clickpad" setting. Open the dash and start "Startup Applications". Click add, set name to "Synaptics clickpad setting", command to "synclient ClickPad=1" (without the quotes of course), and click Add.

You should also check if your touchpad is only using the I2C bus, and not ps2 mode. Start up a terminal and enter

xinput

The Virtual core pointer should have only two items:

  • Virtual core XTEST pointer
  • DLL........ UNKNOWN

if it lists a third item with PS2 touchpad in the name, you should blacklist the psmouse module. Open a terminal and enter

echo -e "\n# remove psmouse because we want the mouse to work over I2C bus\nblacklist psmouse" | sudo tee -a /etc/modprobe.d/blacklist.conf
sudo update-initramfs -u

We have an awesome laptop with big battery, so let's make some changes to optimize battery lifetime. Open a terminal and enter the following commands

cd /etc/pm/power.d/
sudo wget http://forthescience.org/blog/wp-content/uploads/2015/03/powersaverXPS13Trusty -O powersaverXPS13Trusty

Reboot the machine to make the touchpad change, and enjoy your XPS 13!

Some more tweaks

You could try using the xorg-edgers ppa for the latest graphics drivers for your XPS. This could improve the haswell graphics, but please do read the warning notices on the ppa page.


New gcc linker behavior leaves me puzzled

Date Tags gcc

Today, together with a colleague, I found out this interesting behavior of the GNU linker. Suppose you define the following trivial (and incorrect, but who cares) program

int main() {}

if you link it and specify a library (say, libm) it does not depend on, the final executable does not depend on that library. This is a good thing, because there's no point in having a dependency against a library if you don't use any part of it.

sbo@NAS:~$ gcc test.c -o test -lm
sbo@NAS:~$ ldd test
    linux-vdso.so.1 =>  (0x00007fff673ff000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff5cc6a4000)
    /lib64/ld-linux-x86-64.so.2 (0x00007ff5cca83000)

This was not always the case. In old versions of the linker the result would have been

linux-vdso.so.1 =>  (0x00007fff317bc000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fb5d9d13000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb5d9953000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb5da02e000)

occasionally resulting in spurious dependencies. The linker would have added an entry to the ELF dynamic section, regardless of the actual need for that library.

sbo@NAS:~$ readelf --dynamic test

Dynamic section at offset 0xe40 contains 21 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

To prevent the addition of libraries when no symbols from those libraries are actually used, a flag --as-needed was introduced to the linker. It wasn't enabled by default, but in gcc 4.6 it is. In gcc 4.2 it was not. One can always restore the old behavior by issuing --no-as-needed

gcc test.c -o test -Wl,--no-as-needed -lm

The magic numbers of reboot()

Date Tags

Today I learned an interesting piece of Linux trivia. To reboot the machine, there's a system call, reboot(). The funny thing is in its signature

int reboot(int magic, int magic2, int cmd, void *arg);

Of course this routine can only be called by uid 0 (root), but you also need to pass two "magic numbers" for the call to actually work. Why?

Imagine a rogue process with uid 0 gets to screw up and jump at a random location, and this location happens to be the location of reboot(). It would trigger a reboot, something rather unpleasant. To prevent this, magic numbers provide an additional safety net. It's unlikely that the rogue program jumps _and_ has the proper magic numbers in the stack or the registers.

The comment in the kernel confirm this

192 * Reboot system call: for obvious reasons only root may call it,
193 * and even root needs to set up some magic numbers in the registers
194 * so that some mistake won't make this reboot the whole machine.
195 * You can also set the meaning of the ctrl-alt-del-key here.
196 *
197 * reboot doesn't sync: do that yourself before calling this.
198 */

Another interesting trivia is that the magic2 numbers have a special meaning. In hex, they are the birthdates of Torvalds and his daughters.

#define LINUX_REBOOT_MAGIC1 0xfee1dead
#define LINUX_REBOOT_MAGIC2 672274793   // 0x28121969
#define LINUX_REBOOT_MAGIC2A 85072278   // 0x05121996
#define LINUX_REBOOT_MAGIC2B 369367448  // 0x16041998
#define LINUX_REBOOT_MAGIC2C 537993216  // 0x20112000

Any of these values will be accepted to initiate a reboot

210 /* For safety, we require "magic" arguments. */
211 if (magic1 != LINUX_REBOOT_MAGIC1 ||
212         (magic2 != LINUX_REBOOT_MAGIC2 &
213          magic2 != LINUX_REBOOT_MAGIC2A &
214          magic2 != LINUX_REBOOT_MAGIC2B &
215          magic2 != LINUX_REBOOT_MAGIC2C))
216 return -EINVAL;

Utilities for debugging

Date

Part of my daily job involves getting dirty at the assembler level, generally to debug segfaults, FPE, deadlocks and similar stuff. It's a fascinating challenge which makes me feel extremely powerful and in control of the hard, low-level technicalities of the OS, libraries and hardware I use. Most of my effort, however, would be impossible without a few tools I learned to appreciate and love through hours of solid dependency. I'm posting them here, and while the list is certainly not exhaustive, it's a starting kit for the hardcore debugger.

Linux

c++filt

This nifty utility converts C++ mangled symbols into the unmangled form. A simple example is the following

$ c++filt _ZNK3MPI4File19Get_position_sharedEv
MPI::File::Get_position_shared() const

More complex symbols can become unreadable in the mangled form, especially when they arise from template instantiation. A quick pass with c++filt will make everything much clearer.

pmap

Another nifty utility in the Linux arsenal, this easy program reports the VM layout of a running process.

$ pmap  21102
21102:   cat
0000000000400000     44K r-x--  /bin/cat
000000000060a000      4K r----  /bin/cat
000000000060b000      4K rw---  /bin/cat
0000000001969000    132K rw---    [ anon ]
00007f3b98aa9000   7068K r----  /usr/lib/locale/locale-archive
00007f3b99190000   1748K r-x--  /lib/x86_64-linux-gnu/libc-2.15.so
00007f3b99345000   2048K -----  /lib/x86_64-linux-gnu/libc-2.15.so
00007f3b99545000     16K r----  /lib/x86_64-linux-gnu/libc-2.15.so
00007f3b99549000      8K rw---  /lib/x86_64-linux-gnu/libc-2.15.so
00007f3b9954b000     20K rw---    [ anon ]
00007f3b99550000    136K r-x--  /lib/x86_64-linux-gnu/ld-2.15.so
00007f3b99749000     12K rw---    [ anon ]
00007f3b99770000      8K rw---    [ anon ]
00007f3b99772000      4K r----  /lib/x86_64-linux-gnu/ld-2.15.so
00007f3b99773000      8K rw---  /lib/x86_64-linux-gnu/ld-2.15.so
00007fff18ae9000    136K rw---    [ stack ]
00007fff18bff000      4K r-x--    [ anon ]
ffffffffff600000      4K r-x--    [ anon ]
 total            11404K

To achieve the same with a post-mortem core file, the gdb commands "info files" and "maintenance info sections" are your friends. When running, "info proc mappings" can achieve the same.

lddtree

lddtree is like ldd, but it traverses the full hierarchy, building a tree diagram of all dependencies. It is available on Ubuntu in the package pax-utils.

$ lddtree /bin/ls
ls => /bin/ls (interpreter => /lib64/ld-linux-x86-64.so.2)
    libselinux.so.1 => /lib/i386-linux-gnu/libselinux.so.1
        libdl.so.2 => /lib/i386-linux-gnu/libdl.so.2
        ld-linux.so.2 => /lib/i386-linux-gnu/ld-linux.so.2
    librt.so.1 => /lib/i386-linux-gnu/librt.so.1
        libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0
    libacl.so.1 => /lib/x86_64-linux-gnu/libacl.so.1
        libattr.so.1 => /lib/x86_64-linux-gnu/libattr.so.1
    libc.so.6 => /lib/i386-linux-gnu/libc.so.6

Windows

I am not a Windows person, but I build and debug on it rather often. With time, I gathered a few essential tools to understand the source of your problems during build, deployment and execution.

Process monitor

Reports events of different nature from all the processes in your system, such as disk access, registry access, and so on. I found this utility fundamental to track a strange crash that occurred in a third party closed-source library. The problem turned out to be an attempt to write to a directory that was not writable. From the website:

Process Monitor is an advanced monitoring tool for Windows that shows real-time file system, Registry and process/thread activity. It combines the features of two legacy Sysinternals utilities, Filemon and Regmon, and adds an extensive list of enhancements including rich and non-destructive filtering, comprehensive event properties such session IDs and user names, reliable process information, full thread stacks with integrated symbol support for each operation, simultaneous logging to a file, and much more. Its uniquely powerful features will make Process Monitor a core utility in your system troubleshooting and malware hunting toolkit.

Process Explorer

An equivalent of fuser on Linux. From the website:

Ever wondered which program has a particular file or directory open? Now you can find out. Process Explorer shows you information about which handles and DLLs processes have opened or loaded.

VMMap

Extremely useful tool to see the Virtual Memory layout. It is precious to check potential fragmentation scenarios that could lead to invalid allocations.

From the website:

VMMap is a process virtual and physical memory analysis utility. It shows a breakdown of a process's committed virtual memory types as well as the amount of physical memory (working set) assigned by the operating system to those types. Besides graphical representations of memory usage, VMMap also shows summary information and a detailed process memory map.

Dependency walker

Performs more or less the task of lddtree on Linux and otool on Mac. It's invaluable to figure our which libraries or symbols are unresolved. I soon discovered that Windows is not very verbose when it comes to unresolved dependencies at startup. From the website:

Dependency Walker is a free utility that scans any 32-bit or 64-bit Windows module (exe, dll, ocx, sys, etc.) and builds a hierarchical tree diagram of all dependent modules. For each module found, it lists all the functions that are exported by that module, and which of those functions are actually being called by other modules. Another view displays the minimum set of required files, along with detailed information about each file including a full path to the file, base address, version numbers, machine type, debug information, and more.

Event Viewer

Provided on a standard windows 7 installation. Another invaluable tool to understand what's going on in your program, reports error and anomalous conditions that prevent an application to start, and much more.


Undefined symbols for Fortran module variables in static library on OSX. Problem (and solution)

Date

If you program in Fortran on the Mac, you might meet this odd problem. You have a module test.f90 containing nothing but public variables

module Test
    implicit none
    integer :: value
end module

and a program

program main
    use Test
    implicit none

    value = 5
    print *, value
end program

If you try and compile as follows, you have no problems

ifort -c test.f90
ifort -c main.f90
ifort main.o test.o
./a.out

However, if you package the test.o in a test.a static library

ar cru test.a test.o
ranlib test.a
ifort main.o test.a

You will get an undefined symbol error for the value symbol. What gives? Well, the problem has to do with how ranlib on Mac works. You have two solutions to this problem:

  • either you add a module procedure to the test module
  • or you use ranlib -c instead of plain ranlib.

I thank Drew McCormack for figuring this out and going back to post the solution.


Beautiful Git rebasing of version branch

Date Tags

In this post, I want to share with you a technique I learned recently from a colleague. It's a really great trick to keep your history nice and clean, while being able to work and push feature branches.

Let's start with the workflow and the problem that generates. You have a master branch that is pushed to a remote, containing the following

* f2bc01d 2014-10-07 17:57:52 +0200 Stefano Borini (HEAD, origin/master, origin/HEAD, master) - added 5 to foo
* 2ecacd7 2014-10-07 17:57:52 +0200 Stefano Borini - added 4 to foo
* d0f53d8 2014-10-07 17:57:52 +0200 Stefano Borini - added 3 to foo
* d26a4db 2014-10-07 17:57:52 +0200 Stefano Borini - added 2 to foo
* f98ef63 2014-10-07 17:57:52 +0200 Stefano Borini - added 1 to foo
* d817b10 2014-10-07 17:57:04 +0200 Stefano Borini - Added foo
* 5fdec88 2014-10-07 17:50:00 +0200 Stefano Borini - Initial commit

You want to work on a feature, so you create a feature branch from the current master

sbo@NAS:~/tmp/git-experiment-1$ git checkout -b feature
Switched to a new branch 'feature'
sbo@NAS:~/tmp/git-experiment-1$ git push --set-upstream origin feature
Total 0 (delta 0), reused 0 (delta 0)
To git@github.com:stefanoborini/git-experiment-1.git
 * [new branch]      feature -> feature

Master keeps receiving patches from other developers. At the same time, you keep pushing your feature changes to your remote, because you don't want to risk losing all your work if your hard drive crashes. The situation ends up like this

* 453bada 2014-10-07 18:06:36 +0200 Stefano Borini (HEAD, origin/master, origin/HEAD, master) - added 30 to foo
* 1546da7 2014-10-07 18:06:36 +0200 Stefano Borini - added 29 to foo
* cdfa47d 2014-10-07 18:06:36 +0200 Stefano Borini - added 28 to foo
* 00a0051 2014-10-07 18:06:36 +0200 Stefano Borini - added 27 to foo
* 8bc15dd 2014-10-07 18:06:36 +0200 Stefano Borini - added 26 to foo
* a3b542a 2014-10-07 18:06:36 +0200 Stefano Borini - added 25 to foo
... more commits ...
* 4610ec5 2014-10-07 18:06:36 +0200 Stefano Borini - added 11 to foo
* bf30e67 2014-10-07 18:06:36 +0200 Stefano Borini - added 10 to foo
* 2404746 2014-10-07 18:06:36 +0200 Stefano Borini - added 9 to foo
* 0452b5c 2014-10-07 18:06:36 +0200 Stefano Borini - added 8 to foo
* 245864a 2014-10-07 18:06:36 +0200 Stefano Borini - added 7 to foo
* 85a9f39 2014-10-07 18:06:36 +0200 Stefano Borini - added 6 to foo
| * 8f25256 2014-10-07 18:04:42 +0200 Stefano Borini (origin/feature, feature) - added 5 to bar
| * 7f5c34f 2014-10-07 18:04:42 +0200 Stefano Borini - added 4 to bar
| * 4e57707 2014-10-07 18:04:42 +0200 Stefano Borini - added 3 to bar
| * 9c12235 2014-10-07 18:04:42 +0200 Stefano Borini - added 2 to bar
| * a714ec3 2014-10-07 18:04:42 +0200 Stefano Borini - added 1 to bar
|/
* f2bc01d 2014-10-07 17:57:52 +0200 Stefano Borini - added 5 to foo
* 2ecacd7 2014-10-07 17:57:52 +0200 Stefano Borini - added 4 to foo
* d0f53d8 2014-10-07 17:57:52 +0200 Stefano Borini - added 3 to foo
* d26a4db 2014-10-07 17:57:52 +0200 Stefano Borini - added 2 to foo
* f98ef63 2014-10-07 17:57:52 +0200 Stefano Borini - added 1 to foo
* d817b10 2014-10-07 17:57:04 +0200 Stefano Borini - Added foo
* 5fdec88 2014-10-07 17:50:00 +0200 Stefano Borini - Initial commit

Now you have a problem. If you merge feature back into master, you will get a very long line connecting your last commit on feature back to the current master. This may seem trivial, but in complex scenarios it can become unpleasant to browse.

You may think that rebasing is the solution, but unfortunately, you would introduce a lot of problems if you do so. The Feature branch is already pushed to remote, and it's on all developers' machines. Rebasing would mean force a push and rewrite history, with ugly consequences.

What you would like to do is to "cherry pick" all the commits from feature on top of your current master, but there's a more elegant way of doing it. First of all, checkout your feature branch

sbo@NAS:~/tmp/git-experiment-1$ git checkout feature
Switched to branch 'feature'

Then, create and checkout a local branch from the feature branch

sbo@NAS:~/tmp/git-experiment-1$ git checkout -b feature_rebaser
Switched to a new branch 'feature_rebaser'

Now, rebase this branch against the current master. This migrates all the patches from feature_rebaser to master, but only locally.

sbo@NAS:~/tmp/git-experiment-1$ git rebase master
First, rewinding head to replay your work on top of it...
Applying: added 1 to bar
Applying: added 2 to bar
Applying: added 3 to bar
Applying: added 4 to bar
Applying: added 5 to bar

And finally, you can merge onto master. You can use --no-ff to keep the patches "visually separated" instead of in continuity with the current master timeline.

sbo@NAS:~/tmp/git-experiment-1$ git checkout master
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
sbo@NAS:~/tmp/git-experiment-1$ git merge --no-ff feature_rebaser

The resulting tree is the following

*   c15286c 2014-10-07 18:22:53 +0200 Stefano Borini (HEAD, master) - Merge branch 'feature_rebaser'
|\
| * 166bbe0 2014-10-07 18:19:45 +0200 Stefano Borini (feature_rebaser) - added 5 to bar
| * 49eb2ac 2014-10-07 18:19:45 +0200 Stefano Borini - added 4 to bar
| * a5e36d7 2014-10-07 18:19:44 +0200 Stefano Borini - added 3 to bar
| * a8ca8a2 2014-10-07 18:19:44 +0200 Stefano Borini - added 2 to bar
| * 69d5ad3 2014-10-07 18:19:44 +0200 Stefano Borini - added 1 to bar
|/
* 453bada 2014-10-07 18:06:36 +0200 Stefano Borini (origin/master, origin/HEAD) - added 30 to foo
* 1546da7 2014-10-07 18:06:36 +0200 Stefano Borini - added 29 to foo
* cdfa47d 2014-10-07 18:06:36 +0200 Stefano Borini - added 28 to foo
... more commits ...
* bf30e67 2014-10-07 18:06:36 +0200 Stefano Borini - added 10 to foo
* 2404746 2014-10-07 18:06:36 +0200 Stefano Borini - added 9 to foo
* 0452b5c 2014-10-07 18:06:36 +0200 Stefano Borini - added 8 to foo
* 245864a 2014-10-07 18:06:36 +0200 Stefano Borini - added 7 to foo
* 85a9f39 2014-10-07 18:06:36 +0200 Stefano Borini - added 6 to foo
| * 8f25256 2014-10-07 18:04:42 +0200 Stefano Borini (origin/feature, feature) - added 5 to bar
| * 7f5c34f 2014-10-07 18:04:42 +0200 Stefano Borini - added 4 to bar
| * 4e57707 2014-10-07 18:04:42 +0200 Stefano Borini - added 3 to bar
| * 9c12235 2014-10-07 18:04:42 +0200 Stefano Borini - added 2 to bar
| * a714ec3 2014-10-07 18:04:42 +0200 Stefano Borini - added 1 to bar
|/
* f2bc01d 2014-10-07 17:57:52 +0200 Stefano Borini - added 5 to foo
* 2ecacd7 2014-10-07 17:57:52 +0200 Stefano Borini - added 4 to foo
* d0f53d8 2014-10-07 17:57:52 +0200 Stefano Borini - added 3 to foo
* d26a4db 2014-10-07 17:57:52 +0200 Stefano Borini - added 2 to foo

Note how the feature branch is now "brought to present day" and inserts nice and easy. The feature branch can then be left dangling and deleted at your best convenience.

Coudn't we just rebase feature, instead of creating a new branch? Unfortunately no. I created the same situation with a new branch "feature2", pushed it to remote, and rebased feature2. Checking out master and then back to feature2 I get

sbo@NAS:~/tmp/git-experiment-1$ git checkout feature2
Switched to branch 'feature2'
Your branch and 'origin/feature2' have diverged,
and have 15 and 5 different commits each, respectively.
  (use "git pull" to merge the remote branch into yours)

oops... I changed history on my local machine, and this differs from the situation that is on the remote server. If you pull now, you will apply the same patches twice

sbo@NAS:~/tmp/git-experiment-1$ git pull
Merge made by the 'recursive' strategy.

with the following tree

*   a037791 2014-10-07 18:37:03 +0200 Stefano Borini (HEAD, origin/feature2, feature2) - Merge branch 'feature2' of github.com:stefanoborini/git-experiment-1 into fea
|\
| * 6c9e82a 2014-10-07 18:30:45 +0200 Stefano Borini - added 10 to bar
| * 4bcfced 2014-10-07 18:30:45 +0200 Stefano Borini - added 9 to bar
| * 469517d 2014-10-07 18:30:45 +0200 Stefano Borini - added 8 to bar
| * a28a360 2014-10-07 18:30:45 +0200 Stefano Borini - added 7 to bar
| * 894b24c 2014-10-07 18:30:45 +0200 Stefano Borini - added 6 to bar
* | a2b14a4 2014-10-07 18:31:44 +0200 Stefano Borini (origin/master, origin/HEAD, master) - added 10 to bar
* | acefda0 2014-10-07 18:31:44 +0200 Stefano Borini - added 9 to bar
* | 2be9bef 2014-10-07 18:31:44 +0200 Stefano Borini - added 8 to bar
* | 5d528cb 2014-10-07 18:31:44 +0200 Stefano Borini - added 7 to bar
* | a6ec80f 2014-10-07 18:31:44 +0200 Stefano Borini - added 6 to bar
* | d1dc5d2 2014-10-07 18:31:17 +0200 Stefano Borini - added 40 to foo
* | bdafcce 2014-10-07 18:31:17 +0200 Stefano Borini - added 39 to foo
* | 2dd4415 2014-10-07 18:31:17 +0200 Stefano Borini - added 38 to foo
* | 11a8f22 2014-10-07 18:31:17 +0200 Stefano Borini - added 37 to foo
* | f208658 2014-10-07 18:31:17 +0200 Stefano Borini - added 36 to foo
* | ef1e319 2014-10-07 18:31:17 +0200 Stefano Borini - added 35 to foo

Git is unable to recognize the patches' duplication, because rebasing changes the SHA. In this case, it went smooth, but in a real-case scenario, you would get plenty of conflicts.


C++ stdlib std::srand/std::rand are not repeatable across platforms

Date

I recently observed that std::srand/std::rand are not required to yield the same pseudorandom sequence across platforms. This code, for example

#include <cstdlib>
#include <iostream>
#include <ctime>

int main()
{
     std::srand(333);
     int random_variable = std::rand();
     std::cout << "Random value on [0 " << RAND_MAX << "]: " << random_variable << '\n';
     random_variable = std::rand();
     std::cout << "Random value on [0 " << RAND_MAX << "]: " << random_variable << '\n';
     random_variable = std::rand();
     std::cout << "Random value on [0 " << RAND_MAX << "]: " << random_variable << '\n';
}

Gives different results on Linux and Mac. On Linux ubuntu 12.04, it gives

Random value on [0 2147483647]: 1756860556
Random value on [0 2147483647]: 1141727289
Random value on [0 2147483647]: 551934435

On Mac OSX 10.8 gives instead

Random value on [0 2147483647]: 5596731
Random value on [0 2147483647]: 1722461096
Random value on [0 2147483647]: 1324078912

Keep this into account if you need cross-platform behavior of your PRNG.


Simulating network problems on OSX

Recently I had to simulate unreliable connections to a remote server, for testing purposes. I needed two cases: a fully unreachable host, and a host with considerable packet loss and delay. The first is easy to achieve: just pull the cable. For the second, I had to google up, as my network-fu is a bit rusty on the latest developments, especially on OSX. I am more of a Linux guy when it comes to networking.

A quick search produced this blog, with great details on how to achieve my needs. I copy the details here for future reference.

Here I add a 1 second delay and a 0.4 packet loss in both directions

bash-3.2# ipfw add pipe 1 ip from any to 10.0.0.1
00100 pipe 1 ip from any to 10.0.0.1

bash-3.2# ipfw add pipe 1 ip from 10.0.0.1 to any
00200 pipe 1 ip from any to any src-ip 10.0.0.1

bash-3.2# ipfw pipe 1 config delay 1000ms plr 0.4
bash-3.2# ipfw pipe 2 config delay 1000ms plr 0.4

bash-3.2# ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 56 data bytes
Request timeout for icmp_seq 0
64 bytes from 10.0.0.1: icmp_seq=0 ttl=64 time=2000.386 ms

This is really harsh, but it is what I need.

The ipfw utility is the traditional FreeBSD packet filter. The amount of power behind this little command is impressive and reminds me of Linux iptables. ipfw has been deprecated in OSX 10.9 in favor of pfctl. Similarly, iptables is going to be deprecated on Linux in favor of nftables.


New style, new direction

Date Tags

As you can see, I did some deep changes to the website. My hosting at bluehost is expiring and my needs have changed since I started. I am overall happy with bluehost, they are a great, reliable hosting, but I don't need their services anymore. I migrated my wordpress deployment to a static website written in reStructuredText and compiled to HTML with pelican. I then host both the reST and the HTML at github.

This approach requires a bit more manual work, but I think it's worth the savings and the simple technology behind it. In the process, I also got rid of the heavily spammed wordpress comment system and introduced google plus comments.

In the meantime, I decided to shift in focus. In the past I occasionally wrote about science in general. This will no longer happen, and I will post and focus exclusively on scientific programming and related issues, with a general bias toward python. I will try to keep some form of regularity in my posting frequency, but considering the increased manual work required, there will be more variability.

Currently, I am working on the following long term projects:

As you can see, I am overinvolved, which is quite typical for me. Since I am only one, I jump from one project to another in a round robin way. Don't assume a project is dead because you don't see changes. I am probably working on something else and I'll go back to it at a later stage.