Simulating network problems on OSX

Recently I had to simulate unreliable connections to a remote server, for testing purposes. I needed two cases: a fully unreachable host, and a host with considerable packet loss and delay. The first is easy to achieve: just pull the cable. For the second, I had to google up, as my network-fu is a bit rusty on the latest developments, especially on OSX. I am more of a Linux guy when it comes to networking.

A quick search produced this blog, with great details on how to achieve my needs. I copy the details here for future reference.

Here I add a 1 second delay and a 0.4 packet loss in both directions:

bash-3.2# ipfw add pipe 1 ip from any to 10.0.0.1
00100 pipe 1 ip from any to 10.0.0.1

bash-3.2# ipfw add pipe 1 ip from 10.0.0.1 to any
00200 pipe 1 ip from any to any src-ip 10.0.0.1

bash-3.2# ipfw pipe 1 config delay 1000ms plr 0.4
bash-3.2# ipfw pipe 2 config delay 1000ms plr 0.4

bash-3.2# ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 56 data bytes
Request timeout for icmp_seq 0
64 bytes from 10.0.0.1: icmp_seq=0 ttl=64 time=2000.386 ms

This is really harsh, but it is what I need.

The ipfw utility is the traditional FreeBSD packet filter. The amount of power behind this little command is impressive and reminds me of Linux iptables. ipfw has been deprecated in OSX 10.9 in favor of pfctl. Similarly, iptables is going to be deprecated on Linux in favor of nftables.


New style, new direction

Date Tags

As you can see, I did some deep changes to the website. My hosting at bluehost is expiring and my needs have changed since I started. I am overall happy with bluehost, they are a great, reliable hosting, but I don't need their services anymore. I migrated my wordpress deployment to a static website written in reStructuredText and compiled to HTML with pelican. I then host both the reST and the HTML at github.

This approach requires a bit more manual work, but I think it's worth the savings and the simple technology behind it. In the process, I also got rid of the heavily spammed wordpress comment system and introduced google plus comments.

In the meantime, I decided to shift in focus. In the past I occasionally wrote about science in general. This will no longer happen, and I will post and focus exclusively on scientific programming and related issues, with a general bias toward python. I will try to keep some form of regularity in my posting frequency, but considering the increased manual work required, there will be more variability.

Currently, I am working on the following long term projects:

As you can see, I am overinvolved, which is quite typical for me. Since I am only one, I jump from one project to another in a round robin way. Don't assume a project is dead because you don't see changes. I am probably working on something else and I'll go back to it at a later stage.


Scrambling the stack for fun and profit

Date Tags

Like any good respected tinkerer, I sometimes like to play with the madness and intricacies of the hardware and software I use. Recently I was tracing problems in a software I won't name but whose objective is to prevent tampering attempts, and while dumping stacks and disassembling routines, I came around this very interesting backtrace

#0  0x0000003f95e0d654 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x0000003f95e08f65 in _L_lock_1127 () from /lib64/libpthread.so.0
#2  0x0000003f95e08e63 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002b67cbdeaded in ?? ()
#4  0x000000002d0e9608 in ?? ()
#5  0x00002b67cbd1e1f2 in ?? ()
#6  0x000000000000000b in ?? ()
#7  0x00002aaaca08e410 in ?? ()
#8  0x00002aaab405d558 in ?? ()
#9  0x00002aaaadf65f48 in ?? ()
#10 0x00002aaaadf65fa0 in ?? ()
#11 0x00002aaaadf65fc0 in ?? ()
#12 0x00002aaaadf65f40 in ?? ()
#13 0x00002aaaadf65f50 in ?? ()
#14 0x000000002d0e7460 in ?? ()
#15 0x0000000026014330 in ?? ()
#16 0x00002b67cc1d08b0 in ?? ()

Interesting, I said to myself. I had no idea why I didn't have any backtrace information. What's interesting is that the addresses appeared to be incorrect, or not pointing to any code section. Note for example #6. I hardly doubt that's a correct address. I had no idea why this happened.

Nevertheless, I eventually found what I was looking for and moved on to other things, but the weird stack remained in the back of my head. Could it be that the stack has been scrambled on purpose? Maybe... or maybe the library was just stripped? Then why the weird address at frame #6? I still don't know, but the idea of scrambling the stack was appealing. So I wrote this trivial program as a proof of concept, just for fun

#include <stdio.h>

int routine4() {
    printf("hello\n");
}
int routine3() {
    routine4();
}
int routine2() {
    routine3();
}
int routine1() {
    routine2();
}
int main() {
    routine1();
}

then I compiled it with a simple g++ file.cpp, set a breakpoint on routine4, ran it, and asked for a backtrace

#0  0x0000000100000e18 in routine4 ()
#1  0x0000000100000e2f in routine3 ()
#2  0x0000000100000e3a in routine2 ()
#3  0x0000000100000e45 in routine1 ()
#4  0x0000000100000e50 in main ()

Pretty nice backtrace. We see the full information and all routine names. If we disassemble routine4, we also see the printf, which is actually reworked into a puts on OSX 10.6.8 with gcc.:

0x0000000100000e14 <_z8routine4v +0>:        push   %rbp
0x0000000100000e15 <_z8routine4v +1>:        mov    %rsp,%rbp
0x0000000100000e18 <_z8routine4v +4>:        lea    0x45(%rip),%rdi        # 0x100000e64
0x0000000100000e1f <_z8routine4v +11>:       callq  0x100000e5e <dyld_stub_puts>
0x0000000100000e24 <_z8routine4v +16>:       leaveq
0x0000000100000e25 <_z8routine4v +17>:       retq

If I instead strip the binary with strip a.out, I can't set a breakpoint on routine4 anymore, and rightly so:

(gdb) break routine4
Function "routine4" not defined.
Make breakpoint pending on future shared library load? (y or [n]) n
(gdb) break puts
Breakpoint 1 at 0x3126e978c12ef0

Thanks to strip, all symbols are gone and the debugger can only refer to addresses. I can only guess (but not a hard one) where the code is, by checking at which VM pages they are mapped to, and the entry point

(gdb) info file
Symbols from "/Users/sbo/tmp/a.out".
Mac OS X executable:
   /Users/sbo/tmp/a.out, file type mach-o-le.
   Entry point: 0x0000000100000dd8
   0x0000000100000000 - 0x0000000100001000 is LC_SEGMENT.__TEXT in /Users/sbo/tmp/a.out
   0x0000000100000dd8 - 0x0000000100000e57 is LC_SEGMENT.__TEXT.__text in /Users/sbo/tmp/a.out

In any case, with the breakpoint at puts I can get to the printf and issue a backtrace to get to our infamous condition

#0  0x00007fff86eb0ef0 in puts ()
#1  0x0000000100000e24 in ?? ()
#2  0x0000000100000e2f in ?? ()
#3  0x0000000100000e3a in ?? ()
#4  0x0000000100000e45 in ?? ()
#5  0x0000000100000e50 in ?? ()
#6  0x0000000100000e0c in ?? ()

Yet, as you can see, the stack makes sense. I cannot disassemble, but at least I can dump the contents and they make sense

(gdb) disas 0x0000000100000e24
No function contains specified address.
(gdb) x/30i  0x0000000100000e24
0x100000e24: leaveq
0x100000e25: retq
0x100000e26: push   %rbp
0x100000e27: mov    %rsp,%rbp
0x100000e2a: callq  0x100000e14
0x100000e2f: leaveq
0x100000e30: retq
0x100000e31: push   %rbp
0x100000e32: mov    %rsp,%rbp
0x100000e35: callq  0x100000e26
0x100000e3a: leaveq
0x100000e3b: retq
0x100000e3c: push   %rbp
0x100000e3d: mov    %rsp,%rbp
0x100000e40: callq  0x100000e31
0x100000e45: leaveq
0x100000e46: retq
0x100000e47: push   %rbp
0x100000e48: mov    %rsp,%rbp
0x100000e4b: callq  0x100000e3c
0x100000e50: mov    $0x0,%eax
0x100000e55: leaveq
0x100000e56: retq

In fact, you can see the whole shebang. All the calls of the routines, the stack pointer changes, and the final setting to zero of eax when main ends.

Scrambling the return address

Here is the idea: Instead of smashing the stack, I will try to scramble it. What does it mean? Well, let's see how the stack is when we are just about to be calling puts. We select the previous frame

(gdb) frame 1
#1  0x0000000100000e24 in ?? ()

Get the stack pointer at the current frame

(gdb) info registers
...snip...
rbp            0x7fff5fbff680        0x7fff5fbff680
rsp            0x7fff5fbff680        0x7fff5fbff680
...snip...

Then we take a look at what is in there

(gdb) x/10a 0x7fff5fbff680
0x7fff5fbff680:      0x7fff5fbff690  0x100000e2f
0x7fff5fbff690:      0x7fff5fbff6a0  0x100000e3a
0x7fff5fbff6a0:      0x7fff5fbff6b0  0x100000e45
0x7fff5fbff6b0:      0x7fff5fbff6c0  0x100000e50
0x7fff5fbff6c0:      0x7fff5fbff6d8  0x100000e0c
(gdb) bt
#0  0x00007fff86eb0ef0 in puts ()
#1  0x0000000100000e24 in ?? ()
#2  0x0000000100000e2f in ?? ()
#3  0x0000000100000e3a in ?? ()
#4  0x0000000100000e45 in ?? ()
#5  0x0000000100000e50 in ?? ()
#6  0x0000000100000e0c in ?? ()

Nothing unusual, it's simply the stack pointer and the return address, traditional stack contents for a routine call. When the routine returns, the old return address will be restored to the rip, and the program will continue where it left off, at the routine call. If we were to change this address in the stack, the program would jump to a different location, and that would be bad and likely lead to a crash. Note however that, in order for the stack to unwind correctly, only the frame below the current one is needed, and it's needed just before the return occurs.

So, we can technically scramble all the stack, set those addresses to something else and completely break the backtrace even of a non-stripped binary, provided that we restore the frame under the current one just before returning. The process will be:

  1. Inside every routine, we will drop at the assembly level and write a prologue section where we alter the underlying frame's return address.
  2. We do our thing inside the routine
  3. Again at the assembly level, we write an epilogue section where we restore the return address, just before issuing the return that needs it.

With this strategy in place, if you break anywhere inside the function all the frames (except the one your code is currently in) will be "scrambled" and pointing at nonsensical memory areas. Despite the completely trashed stack, the program will behave correctly because when those addresses will be needed at return, the right address has been restored just a few instructions earlier. Let's see:

int routine4() {
    asm("mov 8(%rsp), %rbx");
    asm("lea 0xdeeead(,%rbx,), %rbx");
    asm("mov %rbx, 8(%rsp)");
    printf("hello\n");
    asm("mov 8(%rsp), %rbx");
    asm("lea -0xdeeead(,%rbx,), %rbx");
    asm("mov %rbx, 8(%rsp)");
}

I altered the routine to perform the prologue and the epilogue. In the prologue, I extract the content of the stack pointer plus 8, which happens to be the return address. I put this value in rbx as it seems to be unused. Then, with lea, I add a fixed offset (oxdeeead) to the content of rbx. Finally, I write this value back in the stack at %rsp+8. In the epilogue, I simply perform the opposite operation, subtracting 0xdeeead and restoring the correct return address in the stack. If I compile and run, the program works correctly.

The gdb session is really nice:

Breakpoint 1, 0x0000000100000df4 in routine4 ()
(gdb) bt
#0  0x0000000100000df4 in routine4 ()
#1  0x0000000100000e2f in routine3 ()
#2  0x0000000100000e3a in routine2 ()
#3  0x0000000100000e45 in routine1 ()
#4  0x0000000100000e50 in main ()

Note how the stack is correct, as we haven't executed the prologue yet.

(gdb) disas

Dump of assembler code for function _Z8routine4v:
0x0000000100000df0 <_z8routine4v +0>:        push   %rbp
0x0000000100000df1 <_z8routine4v +1>:        mov    %rsp,%rbp
0x0000000100000df4 <_z8routine4v +4>:        mov    0x8(%rsp),%rbx           # prologue
0x0000000100000df9 <_z8routine4v +9>:        lea    0xdeeead(,%rbx,1),%rbx   # prologue
0x0000000100000e01 <_z8routine4v +17>:       mov    %rbx,0x8(%rsp)           # prologue
0x0000000100000e06 <_z8routine4v +22>:       lea    0x57(%rip),%rdi          # 0x100000e64
0x0000000100000e0d <_z8routine4v +29>:       callq  0x100000e5e <dyld_stub_puts>
0x0000000100000e12 <_z8routine4v +34>:       mov    0x8(%rsp),%rbx           # epilogue
0x0000000100000e17 <_z8routine4v +39>:       lea    -0xdeeead(,%rbx,1),%rbx  # epilogue
0x0000000100000e1f <_z8routine4v +47>:       mov    %rbx,0x8(%rsp)           # epilogue
0x0000000100000e24 <_z8routine4v +52>:       leaveq
0x0000000100000e25 <_z8routine4v +53>:       retq
End of assembler dump.

The current situation looks like this:

(gdb) info register
rbx            0x0   0
rsp            0x7fff5fbff680        0x7fff5fbff680
(gdb) x/10a 0x7fff5fbff680
0x7fff5fbff680:      0x7fff5fbff690  0x100000e2f <_z8routine3v +9>
0x7fff5fbff690:      0x7fff5fbff6a0  0x100000e3a <_z8routine2v +9>
0x7fff5fbff6a0:      0x7fff5fbff6b0  0x100000e45 <_z8routine1v +9>
0x7fff5fbff6b0:      0x7fff5fbff6c0  0x100000e50

Stepping instruction after instruction, we can follow the events: first the rbx register is filled with the return address from the stack:

-> mov    0x8(%rsp),%rbx
(gdb) info register rbx
rbx 0x100000e2f 4294970927

Then, we add 0xdeeead

-> lea    0xdeeead(,%rbx,1),%rbx
(gdb) info register rbx
rbx            0x100defcdc   4309581020

and finally, we store it back into the stack

-> mov    %rbx,0x8(%rsp)           # prologue
(gdb) x/10a 0x7fff5fbff680
0x7fff5fbff680:      0x7fff5fbff690  0x100defcdc
0x7fff5fbff690:      0x7fff5fbff6a0  0x100000e3a <_z8routine2v +9>
0x7fff5fbff6a0:      0x7fff5fbff6b0  0x100000e45 <_z8routine1v +9>
0x7fff5fbff6b0:      0x7fff5fbff6c0  0x100000e50

Et voila'. The backtrace is now pointing to neverland

(gdb) bt
#0  0x0000000100000e06 in routine4 ()
#1  0x0000000100defcdc in ?? ()
#2  0x0000000100000e3a in routine2 ()
#3  0x0000000100000e45 in routine1 ()
#4  0x0000000100000e50 in main ()

If we were to return now, a segfault would occur: that return address is completely invalid. It's only by performing the reverse operation that we can land safely back into routine3

-> mov    0x8(%rsp),%rbx
rbx            0x100defcdc
-> lea    -0xdeeead(,%rbx,1),%rbx
rbx            0x100000e2f
-> mov    %rbx,0x8(%rsp)
Stack 0x7fff5fbff680:        0x7fff5fbff690  0x100000e2f <_z8routine3v +9>

Now the backtrace is sane again and we are ready to return

(gdb) bt
#0  0x0000000100000e24 in routine4 ()
#1  0x0000000100000e2f in routine3 ()
#2  0x0000000100000e3a in routine2 ()
#3  0x0000000100000e45 in routine1 ()
#4  0x0000000100000e50 in main ()

Now that we can reliably alter the stack frame, we can apply the same trick to our complete call hierarchy. Here is the full code:

#include <stdio.h>

#define scramble() asm("mov 8(%rsp), %rbx"); \
                    asm("lea 0xdead(,%rbx,), %rbx"); \
                    asm("mov %rbx, 8(%rsp)")

#define unscramble() asm("mov 8(%rsp), %rbx"); \
                     asm("lea -0xdead(,%rbx,), %rbx"); \
                     asm("mov %rbx, 8(%rsp)")
int routine4() {
    scramble();
    printf("hello\n");
    unscramble();
}
int routine3() {
    scramble();
    routine4();
    unscramble();
}
int routine2() {
    scramble();
    routine3();
    unscramble();
}
int routine1() {
    scramble();
    routine2();
    unscramble();
}
int main() {
    scramble();
    routine1();
    unscramble();
}

If you compile it, it runs

sbo@sbos-macbook:~/tmp$ g++ test.cpp
sbo@sbos-macbook:~/tmp$ ./a.out
hello

and if you debug it, break at puts, and backtrace, here is the funny result:

(gdb) bt
#0  0x00007fff86eb0ef0 in puts ()
#1  0x0000000100000d82 in routine4 ()
#2  0x0000000100defc5e in ?? ()
#3  0x0000000100defc8d in ?? ()
#4  0x0000000100defcbc in ?? ()
#5  0x0000000100defceb in ?? ()
#6  0x0000000100defc05 in ?? ()
(gdb) x/10a 0x0000000100defc8d
0x100defc8d: Cannot access memory at address 0x100defc8d
(gdb) disas 0x0000000100defc8d
No function contains specified address.
(gdb) cont
Continuing.
hello

Program exited normally.

Now you can get creative. For example, you can

  • Scramble your frames according to a random number that you seed differently at every new run.
  • Scramble the whole frame content, not only the return address
  • Spread out preamble and epilogue throughout the routine code, so that it's harder to find out which opcode is devoted to actual execution, and which one is unscrambling the frame, maybe through tortuous operations full of indirections.

Of course, this stuff is extremely hard to do correctly. You have to keep into account that some stack content could be needed by callees, so you may have to unscramble any frame content at any time. It can also quickly turn into a portability nightmare, as different compilers may have different strategies to fill the stack with local variables.

Yet, it was fun, and I hope you enjoyed it.


Python and memory fragmentation

Date

If you use CPython on 32 bit architectures, you may encounter a problem called memory fragmentation. It is more likely to happen on Windows for reasons that will soon be clear, but it's not a Windows exclusive. It is also not an exclusive python problem, but tends to occur more often on CPython due to its intrinsic memory allocation strategy.

When you dynamically allocate memory in C, you do so in a particular area of the Virtual Memory, the heap. A requirement for this allocation is that the allocated chunk must be contiguous in virtual memory. CPython puts extreme stress on the heap: all objects are allocated dynamically, and when they are freed, a hole is left in the heap. This hole may be filled by a later allocation, if the requested memory fits in the hole, but if it doesn't, the hole remains until something that fits is requested. On Linux, you can follow the VM occupation with this small python script

import sys
import subprocess

mmap = [' ']*(16*256)
out = subprocess.check_output(["/usr/bin/pmap","-x", "%d" % int(sys.argv[1])])
for i in out.splitlines()[2:-2]:
    values = i.split()[0:2]
    start = int("0x"+values[0], 16) / 2**20
    end = start + (int(values[1])*1024)/2**20
    for p in xrange(start, end+1):
        mmap[p] = '*'

for row in xrange(16):
    print hex(row)+" | "+"".join(
                        mmap[row * 256:(row+1)*256]
                        )+"|"

On Windows, the great utility VMMap can be used to monitor the same information.

Given the above scenario, the Virtual Memory space can eventually become extremely fragmented, depending on the size of your objects, their order of allocation, if your application jumps between dynamically allocating large chunks of memory and small python objects, and so on. As a result, you may not be able to perform a large allocation, not because you are out of memory, but because you are out of contiguous memory in your VM address space. In a recent benchmark I performed on Windows 7, the largest contiguous chunk of memory available was a meager 32 megabytes (ow!), which means that despite the free memory being around 1 gigabyte, the biggest chunk I could request was only 32 megabytes. Anything bigger would have the malloc fail.

Additional conditions that can make the problem worse are dynamic libraries binding and excessive threading. The first invades your VM address space, and the second needs a stack per each thread, putting additional unmovable barriers throughout the VM and reducing real estate for contiguous blocks. See for example what happens with 10 threads on Linux

(gdb) thread apply all print $esp

Thread 10 (Thread 10606): $5 = (void *) 0x50d7184
Thread 9 (Thread 10607): $6 = (void *) 0x757ce90
Thread 8 (Thread 10608): $7 = (void *) 0x7b69e90
Thread 7 (Thread 10609): $8 = (void *) 0x7d6ae90
Thread 6 (Thread 10618): $9 = (void *) 0x8a4ae90
Thread 5 (Thread 10619): $10 = (void *) 0xb22fee90
Thread 4 (Thread 10620): $11 = (void *) 0xb20fde90
Thread 3 (Thread 10621): $12 = (void *) 0xb1efce90
Thread 2 (Thread 10806): $13 = (void *) 0xb2ea31c4
Thread 1 (Thread 0xb7f6b6c0 (LWP 10593)): $14 = (void *) 0xbffd1f3c

Fragmentation is eventually made irrelevant by a 64 bit architecture, where the VM address space is huge (for now ;) ). Yet, if you have a 32 bit machine and a long running python process that is juggling large and small allocations, you may eventually run out of contiguous memory and see malloc() fail.

How to solve? I found this interesting article that details the same issue and provides some mitigation techniques for Windows, because Windows is kind of special: on Windows, the 4 gigabytes address space is divided in two parts of 2GB EACH, the first for the process, the second reserved for the kernel. If you must stay on 32 bits, your best bet is to give an additional gigabyte of VM with the following recipe (only valid for Windows-7)

  1. run bcdedit /set IncreaseUserVa 3072 as administrator, then reboot the machine.
  2. mark your executable with EditBin.exe you_program.exe /LARGEADDRESSAWARE

With this command, the VM subdivision is set to 3GB+1GB, granting one additional gigabyte to your process. This improves the situation, but sometimes it's enough. If it is not, and you still need to work on 32 bit machines then you are in big trouble. You have to change your allocation strategy and be smarter on handling the fragmentation within your code.


Nitpicking on python properties

Date

I like python properties, I really do. Properties allow you to convert explicit setters and getters into lean code while keeping control of your operation. Instead of this

state.setMode(EditorMode.INSERT)
current_mode = state.mode()

with properties you can write a much more pleasant

state.mode = EditorMode.INSERT
current_mode = state.mode

In both cases, if it weren't for the magic of properties, you would get direct access to a member, but if you create your class like this

class State(object):
    def __init__(self):
        self._mode = EditorMode.COMMAND

    @property
    def mode(self):
        return self._mode

    @mode.setter
    def mode(self, value):
        self._mode = value

the two decorated methods are called automatically. This gives you a chance for validating the passed value, for example, or using smart tricks like caching if getting the value is slow, basically the same tricks you would obtain by traditional accessor methods, but with a better syntax and without violating encapsulation.

That said, I noticed a few minor things with properties that are good to point out.

1. Refactoring

Imagine that you find "mode" too poor as a name. and you decide to use editor_mode instead. If you had an accessor method, you would refactor setMode() to setEditorMode(). If any part of your code still calls setMode(), you get an exception.

With a property, the same refactoring implies a change from state.mode to state.editor_mode. However, if other parts of your code still use state.mode = something, you will not get an exception. In fact, it's trivially correct. This could produce hard-to-find bugs.

2. No callbacks

While you can store or pass around state.setEditorMode as a callback, you can't achieve the same effect with a property, not trivially at least. No, you can't use a lambda, because assignment is forbidden in a lambda.

3. Mocking

You can certainly mock a property, but requires a bit more care. Nothing impossible, but if you learn the mock module, you have to go on that extra bit if you want to cover properties.

4. Soft-returning a set operation details

Sometimes you might want your setter to return state information about the set operation. One trivial example may be True or False, depending if the operation was successful or not. You can certainly throw an exception for this specific case, but your mileage may vary depending on the specifics of your problem and what "looks better" for your design. A property does not give you flexibility to return a value during set.

5. Only one value at a time

If your setters are connected to some notification system, you might want to set multiple values at once to trigger a single notification. Once again, it's a minor problem: you can use a special property accepting a tuple. For example, if you have values v1 and v2, on class Foo, you could have something like

class Foo(object):
    def __init__(self):
        self._v1 = None
        self._v2 = None
    @property
    def v1(self):
        return self._v1
    @v1.setter
    def v1(self, value):
        self._v1 = v1
        # notify listeners
    @property
    def v2(self):
        return self._v2
    @v2.setter
    def v2(self, value):
        self._v2 = v2
        # notify listeners
    @property
    def values(self):
        return (self._v1, self._v2)
    @values.setter
    def values(self, value_tuple):
        self._v1, self._v2 = value_tuple
        # notify listeners

f=Foo()
f.values = (1,2)

6. Magic!

There's some kind of magic behind properties that you can't perceive to be there when you read client code. For example, code like this

myobj.my_foo = 5

generally makes you think of a simple assignment taking place, but this is not the case if my_foo is a property. Maybe naming convention could disambiguate? I am not a fan of the strict PEP-8 requirements on naming of methods, so one could potentially decide for

myobj.myMethod()
myobj.myProperty = 5
myobj.my_member_var = 3

I am thinking out loud here, and I don't have a strong opinion on this issue. That said, properties are definitely cool and will make your interfaces much more pleasant, so I definitely recommend their use, if no other constraints prevent you to do so.



Building my own NAS/Media center – Part 9 – BIOS update and final decorations

The final assembly is now performing its task amazingly well, except for one annoying issue: the fans are too noisy. I decided to explore the issue to conclude the setup. In particular, I observed that the fans always ran at full speed after wake-up. That sounded like a BIOS problem, so I checked for BIOS upgrades on the ZOTAC website. Turns out I was right, and there is indeed a fix for the issue I encounter. I had to flash the BIOS, an operation that generally makes me nervous, because if it goes bad, the risk of bricking the BIOS is real.

I downloaded the driver and followed the instructions I found on the XBMC website. Briefly said, it's a matter of installing freedos on a USB stick with unetbootin, which is available on the Ubuntu repository. Then, I copied the contents of the dowloaded BIOS upgrade package, and followed the readmes and pdf guides I found inside. The flashing operation didn't go smooth, I got some strange behaviors which unfortunately I didn't write down, but in the end the BIOS was flashed and the fan problem after wake-up went away.

The second operation I performed was to carve a good fan grid just above the processor fan. The case is aluminum so it's rather easy to drill, but unfortunately I only had a Dremel with no drill press, and the handmade drilling ended up rather poorly executed. One mistake I made was to drill from the outside, so I ended up with a lot of scratches on the external paint. I should have done it from the inside. After drilling, the external case was full of aluminum dust, and I took extra care of removing every drill imperfection and dust with some light sandpaper and canned air. You definitely don't want conductive debris on your electronics.

The resulting, poorly executed grid left me a bit disappointed, so I wanted to add some creative execution to the case. I decided to carve three wolf moon, a picture I honestly like, with the fan grid being the moon.To do so, I printed a thresholded picture of the wolves on a piece of paper and attached it to the case.

Then, I started carving the outline with a sewing needle inserted into a mechanical pencil.

Unfortunately I would have preferred to transfer the drawing with carbon paper, but I wasn't able to find it. After transferring the outline, I shaded the result adding proper hair-like scratches. Here is just a sample of the procedure

I then scratched the moon and added a few stars with the Dremel. I can't say I am an artist, but I had a good time doing it, and the result is better than I expected.

The full assembly

This post closes the series, although I will do another post immediately after this one as an aggregating index. I am currently working on other two long-term projects, this time software related.


Building my own NAS/Media center - Part 8 - Other Software

The next step (after managing to get out of minecraft) is to install some other fundamental software I can need. Here is a quick list of the little things that make the Linux box jump out of the home appliance/NAS realm and get into serious business.

Git

As a developer, I find git extremely useful. Additionally, I keep some important configuration files on github, so it makes sense to install git. From the ubuntu software center is just a couple of clicks away.

Vim

To install vim, I go again through the software center. I also download my vim configuration files from github.

XBMC

XBMC was easy to install via the ubuntu software center. The software is extremely good, and I found it packed with plugins to get easily on youtube. It's a bit disappointing the presentation and access to the movies (just a directory, I would have preferred a set of scrolling screenshots of each movie, but maybe I am using it wrong). Overall, I am extremely satisfied of XBMC. Premium quality software indeed.

I installed a lot of plugins, allowing me to watch euronews, browse my pictures, and listen to my favorite songs and radio. XBMC is definitely recommended, its only problems are the tendency to get stuck once in a while (nothing a ssh+kill can't fix), the abundance of broken plugins that I had to fix by hand (but they are in python, so it's easy) and a general tendency to be reliant on mouse action and the right button click.

Chrome

Chrome was obtained from the google website as a .deb. Once downloaded, clicking the package brings the package manager up, and provides an easy installation step. I am a bit disappointed with the overall quality of the fonts for the browser.

Stellarium

A great planetarium program. A toy for most people, but I find extremely beautiful to browse the current nightsky and the artificial satellites moving above my head from my comfy chair.

Psensor

Useful to monitor the temperature of my processors. I posted a lot of diagrams on this regard, and I keep monitoring the temperature during intensive tasks, as I want to be sure the hardware does not suffer in its current enclosure.


Building my own NAS/Media center - Part 7 - Minecraft and PS3 SixAxis

How can I possibly live without minecraft? Downloading the Sun JRE is easy. From the Oracle website, I got the JRE-7u21, downloaded a .tar.gz file, and unpacked it in /opt/oracle (just to keep it consistent with /opt/google/chrome)

To download minecraft, I logged in my *premium* account (heh) and downloaded the software, a straightforward operation but when I finally started the game, all I got was a black screen with the following error message

Exception in thread "Thread-3" java.lang.UnsatisfiedLinkError:
/home/sbo/.minecraft/bin/natives/liblwjgl.so:
/home/sbo/.minecraft/bin/natives/liblwjgl.so: wrong ELF class: ELFCLASS32
(Possible cause: architecture word width mismatch)

Apparently, this is due to incorrect LD_LIBRARY_PATH setup and outdated libraries. I fixed with adding the following line to ~/.bashrc export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:"/opt/oracle/java-7-oracle/lib/amd64/" and updating the LWJGL library as indicated here.

To add the minecraft icon, it was easy: I simply followed this post on askubuntu.

Setting up the Playstation 3 controller for minecraft

Playing minecraft with the mouse and keyboard is fine if you are sitting close to the screen and with a table in front of you. If you are playing from a sofa, you need a controller, and I already have one: the PS3 controller. The task now is to setup the Linux box to make the two communicate.

The PS3 controller is a real cool toy: it sports a huge number of proportional buttons, gyroscopes, and joysticks. It's a completely programmable, very standard object that talks USB and Bluetooth. Props to Sony for producing such a technological jewel, and for once not going proprietary. That said, I did some research and got somewhere, although it's far from easy. Technically, the kernel recognizes the controller as soon as plugged in. dmesg says

[182796.359735] usb 2-1.1: new full-speed USB device number 16 using ehci_hcd
[182796.469769] usb 2-1.1: New USB device found, idVendor=054c, idProduct=0268
[182796.469775] usb 2-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[182796.469778] usb 2-1.1: Product: PLAYSTATION(R)3 Controller
[182796.469781] usb 2-1.1: Manufacturer: Sony
[182796.688727] sony 0003:054C:0268.0011: Fixing up Sony Sixaxis report descriptor
[182796.714451] input: Sony PLAYSTATION(R)3 Controller as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.1/2-1.1:1.0/input/input31
[182796.714909] sony 0003:054C:0268.0011: input,hiddev0,hidraw0: USB HID v1.11 Joystick [Sony PLAYSTATION(R)3 Controller] on usb-0000:00:1d.0-1.1/input0

but this is not enough to use the sixaxis out of the box. The following instructions focused on USB communication, as my computer has no bluetooth at the moment. I installed three softwares, two of them (QtSixA and jstest-gtk) from the Ubuntu software center. I recommend installing them first, because they bring a lot of useful dependencies in.

First I tried QtSixA. I got it to see the controller, but no buttons were available. I could see the controller though, so it was a step in the right direction.

Then, I tried jstest-gtk. This recognized the controller as a joystick on /dev/input/js0. I performed the calibration moving the controller here and there, pressing all the buttons and moving all the joysticks. This step is particularly important to associate the raw values for each "axis" (as the control channels are called) coming from the driver to the values to send to the applications (in an interval apparently between -32767 and 32768). After the step was completed, I could see the actual values fluctuate in jstest-gtk as I moved the controller around.

|Screenshot from 2013-05-24 20:22:26|

Now that I have a correct driver setup, I need to configure the mapping between the joypad events and the sending of actual game-useful events (such as mouse movements, keyboard presses). For this, I downloaded qjoypad. This application is not available on the ubuntu software center, so I had to compile and install it at the command line. Once ready, I had to run it with the --notray option, because with the icon in the tray I don't get any popup to configure the pad. This produces a nasty floating window with a joystick in it, but at least it works. QJoypad must be kept running when I play, being the one responsible for mapping pad events to keyboard/mouse events.

qjoypad is a bit complex to configure, but it does its job and it's highly configurable. I created a new layout called "minecraft" and then, for each axis, configured the events. For axis 1 (the left-right movement of the left analog stick) I wanted to have the player movement, so I set in the popup dialog as follows

|Screenshot from 2013-05-24 20:25:15|

Note the blue and red markers in the dialog. When you move the analog stick, a grey area increases. Hitting the blue marker makes it enter in a "fire" area, where the event is produced, in this case, the A letter is sent. I put the blue marker very high so that I don't get accidental movements if I just touch the analog stick.

The red marker becomes important for proportional movements, such as looking around (assigned to the right analog stick, Axis 3 and 4). In this case, I want to send mouse events. The blue marker is considered the zero. the red marker is considered the highest value for mouse action. I put my markers as follows

|Screenshot from 2013-05-24 20:25:58|

so that I don't get accidental movement by just brushing the stick (blue marker), and I get full speed a bit before hitting the maximum excursion with the stick (red marker). I had to reduce the mouse sensitivity, and mapped the excursion linearly (that is, the farther I move the stick off-center, the faster the visual panning goes, with speed being in a smooth, progressive association to the stick displacement).

Full configuration took me a while, and I had to test it for practicality in minecraft, because some actions must be accessible at the same time (for example, jumping and digging, jumping and moving, and so on). I am still refining the associations. Obviously, the select is a good button for inventory (it's kind of a standard in PS3 games), as well as the two sticks for movement and look-around. The arrows buttons left and right could be used to cycle the in-hand item, although this prevents inventory change while moving. A good alternative could be the shoulder trigger buttons, but they are not easy to reach and have a bad tactile feedback, so I am trying not to use them. Shoulder buttons can be used for digging or jumping.

After a bit of trials, I came up with this redundant association. It's still experimental, but I really enjoy being able to set the digging action sticky by pressing square.

Joystick 1 {
 Axis 1: dZone 16768, +key 40, -key 38
 Axis 2: dZone 13492, +key 39, -key 25
 Axis 3: gradient, dZone 6167, maxSpeed 10, tCurve 0, mouse+h
 Axis 4: gradient, dZone 5589, maxSpeed 10, tCurve 0, mouse+v
 Button 1: key 26
 Button 3: key 65
 Button 4: key 9
 Button 6: mouse 5
 Button 8: mouse 4
 Button 9: mouse 4
 Button 10: mouse 5
 Button 11: mouse 3
 Button 12: mouse 1
 Button 14: key 9
 Button 15: key 65
 Button 16: sticky, mouse 1
}

Trying Bluetooth

With the addition of a Bluetooth dongle, I started playing with the possibility of using the controller as a wireless device. I followed this page, but despite my best efforts, I failed to pair the controller and the computer. I give up on this because it's not as important, and I can play with the cable just fine.


Building my own NAS/Media center - Part 6 - NAS Software, WakeOnLAN

At the moment, I have a stable system and I can start installing the stuff I need. So let's start.

NFS, SMB, SSHFS, AFP?

The main role of the NAS is to be a NAS, obviously, and this means being accessible from my laptop for storage. With this, comes the choice of which service/protocol to use among NFS (mostly used in the Unix world), SMB (windows), and AFP (Apple), and other less common alternatives, such as SSHFS and iSCSI. The choice is not necessarily exclusive: the same files or directories can be exported through all or some of these channels simultaneously, but staying simple, which one should I use?

I am a Mac user, and I am looking for a solution that is satisfying for both Linux (as a server) and Mac (as a client). I also have a Windows laptop, so it might put an additional requirement. Mac technically supports both NFS and SMB protocols, although its native protocol is AFP. When it comes to performance, NFS is the winner. SMB is slower, and can't take advantage of multithreading, but in all fairness I don't really care too much. Another option is SSHFS, which is basically a sftp made filesystem. All the traffic is encrypted and it's relatively straightfoward to setup. Finally, there's also iSCSI, something I may try out later, but unfortunately iSCSI is not supported by Mac, unless you use this iSCSI initiator software. A bit pricey, but I might be tempted to try and see how it goes just for fun.

Trying out SMB

I am the only user of the system, and I started with what I thought was the fastest to setup: SMB. I followed this excellent guide, and configured a directory "Media" where I will keep all my media files. It took only a few minutes. On the OSX side however, it turned out to be a bit more tricky. I thought the computer would appear in the "Network" entries automatically, but I was wrong. Instead, I had to connect manually using "Go -> Connect to server" and specifying smb://ip-address/" in the dialog. Easy enough, but I honestly don't understand why the computer does not appear in my Network. My other laptop (with ubuntu 12.04 as well) is found by my Mac immediately, and I didn't do anything special except basic install.

After playing for a while with the shares, I realized the interaction was weird. Occasionally, the machine would actually appear in the "Shares" section of the Finder, then disappear, then appear again, with no apparent relation to my actions on the server or on the Mac. Occasionally, I was also unable to connect at all. I browsed around, and asked around, but apparently the only way to make the system detectable by the mac auto discovery was to change the avahi configuration to add the following smb.service

<?xml version=”1.0” standalone=’no’?><!—*-nxml-*—>
<!DOCTYPE service-group SYSTEM “avahi-service.dtd”>
<service-group>
    <name replace-wildcards=”yes”>%h</name>
    <service>
        <type>_smb._tcp</type>
        <port>139</port>
    </service>
<service>
<type>_device-info._tcp</type>
<port>0</port>
<txt-record>model=RackMac</txt-record>
</service>
</service-group>

Still, this didn't fix the occasional hiccups of the service, that kept returning an "unable to connect to the share" error messaged occasionally, and a general strange behavior of what and how things are exported to a guest. Although I am sure there is a solution to my problems, I gave up on SMB, and moved on to SSHFS.

Trying SSHFS

On paper, SSHFS is trivial: a sftp session is established to the remote host, and the content is made available as a filesystem via FUSE. This makes the access to the remote NAS extremely transparent, secure (the traffic is encrypted), and easy to setup. The problem, however, is that if the connection is suspended (for example, if I put my laptop in sleep mode) it won't restart when I turn it on again. Minor problem, but still...

Going with AFP

After so much experimenting, I decided to try out AFP support, and worry about other computers later on. There this an excellent tutorial, albeit old, if you want to use AFP on a Linux box, another, much shorter one, and a blog post. I went with the first one, to learn something new.

To enable AFP and serve files, I followed the tutorial step by step: installed netatalk (version 2.2.1 gets installed, so it should use encryption), configured the netatalk file /etc/default/netatalk with

ATALKD_RUN=no
PAPD_RUN=no
CNID_METAD_RUN=yes
AFPD_RUN=yes
TIMELORD_RUN=no
A2BOOT_RUN=no

Then added the home and share volume to /etc/netatalk/AppleVolumes.default

:DEFAULT: options:upriv,usedots
~/ "$u home" cnidscheme:cdb
/home/Share/Media allow:sbo cnidscheme:cdb options:usedots,upriv

And finally putting the following line in /etc/netatalk/afpd.conf (and it should be the only line in the file)

- -tcp -noddp -uamlist uams_randnum.so,uams_dhx2.so -nosavepassword -advertise_ssh

It is important that you use uams_dhx2.so, NOT uams_dhx.so, otherwise you will not be able to login. Also, I created an empty /etc/netatalk/afppasswd file, just to be sure. Apparently, Ubuntu does not ship the command afppasswd to administer this file. The authentication info should come from /etc/passwd anyway.

I restarted the daemon (service afp restart), then it's time to configure avahi to advertise the service. As in the case of SMB, it's a matter of creating a proper XML file. I created /etc/avahi/services/afp.service

<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
<name replace-wildcards="yes">%h</name>
<service>
<type>_afpovertcp._tcp</type>
<port>548</port>
</service>
<service>
<type>_device-info._tcp</type>
<port>0</port>
<txt-record>model=Xserve</txt-record>
</service>
</service-group>

And restart avahi with service avahi-daemon restart. The service immediately appeared in my finder, but I could not access my Media directory, with the following error message "Something wrong with the volume's CNID DB". These problems were solved by following the last part of this blog post: I removed all the .AppleDB and .AppleDouble directories in my exported dirs, then changed the cnidscheme to dbd.

AFP now works like a charm, except for a couple of things: first, it`comes with a range of caveats <http://netatalk.sourceforge.net/2.0/htmldocs/configuration.html#CNID-backends>`_, the most striking one is the problem handling symbolic links. I don't know how hard this can hit me, but I don't think it's a problem at the moment. Second, there might be problems in using the Share for anything mac-related such as the iPhoto library and similar stuff. The reason is that these tools rely on metainformation that is supported by the native HFS+ file system, but not by the underlying EXT4 filesystem my Ubuntu box is based on. I don't know if HFS+ support on linux would solve this problem, but for now, I just transfer my iPhoto library there as-is, and see what happens. Worst case scenario, I can create a HFS+ volume as suggested in the post, and mount it remotely from the Mac.

UPnP/DLNA

A quick note on UPnP/DLNA server. My TV technically supports this protocol, so I tried to install a DLNA server, mediatomb. There's a good guide on this blog. After installing it and trying it out, I decided to remove it for various reasons. The first is that mediatomb is not really intuitive, not particularly fancy in its setup. The configuration file is overly complex, and setting up the file database is counterintuitive and requires web access. Additionally, the daemon scans the directories, but not having any understanding of the actual content, you find a lot of spurious information, such as the thumbnails of iPhoto. Also, UPnP is very slow: accessing my photo directory took 5 minutes and, although I hope it's cached, I don't find the whole thing deserving of attention when I can use XBMC.

Wake-On-Lan

It would be very convenient to be able to turn on the system from my laptops, in case am in bed, I am lazy and I need to access some files. To this purpose, there's Wake-On-Lan, a mechanism that allows to turn on the system remotely by sending a "magic ethernet packet" to its ethernet adapter. To enable this behavior, support in both the BIOS and the ethernet adapter must be present. When the system is off, the network adapter keeps listening for the magic ethernet packet, and when it receives it, it asks the BIOS to power up the system.

Technically, WiFi Wake-On-LAN does exist, but it's not supported by my motherboard. This is unfortunate, as I did everything through WiFi until now. I bite the (minor) bullet and connect the system via ethernet cable.

I found a couple of good tutorials on how to setup WoL ([1], and [2]), and I will detail the main points. First, I entered the BIOS, and in Advanced -> ACPI settings I set Wake On Lan to Enabled. While I was there, I also set "Restore from AC Power Loss" to "Power Off". I don't want my computer to start up arbitrarily if there's a power outage, and this solves the minor inconvenience of having it power up when I turn on the PSU.

For the network adapter side, I need to start up Linux and do some tinkering. This post describes how to do it. I installed ethtool and peeked the current setup with ethtool eth0

Supports Wake-on: pumbg
Wake-on: g

So it appears that Wake-on-Lan is already enabled, and it will be triggered by the magic ethernet packet (option "g").

To test it from my Mac, I downloaded WoM. This tiny but amazing program allows you to send the WoL packet with just a click. I took note of the IP address and ethernet address of my NAS, configured WoM appropriately, and turned off the NAS. With a click on "Wake", I was able to start it successfully, and I am now a happy nerd.