Building my own NAS/Media center – Part 5 – Make Ubuntu 12.04 play nice on NVIDIA

Following the adventures in OS setup from last time, here is a step-by-step guide to make Ubuntu 12.04 play nice on a nvidia card setup:

  1. Plug your screen on the sockets associated to the nvidia card.
  2. Get into the BIOS -> chipset -> Display configuration -> Initiate graphic adapter. Set it to PEG/PCIEx4
  3. Boot the ubuntu 12.04 installation disk or usb key
  4. Install the system normally.
  5. reboot into the new system
  6. Login. Open a terminal. Ignore any popup, and close any software manager emerging. Do
    sudo apt-get update
    sudo apt-get upgrade
    sudo apt-get install linux-image-generic-lts-quantal linux-headers-generic-lts-quantal
  7. Tea break. When the three commands are completed, reboot.
  8. Back into the system. Open a terminal
  9. Go into System settings -> Additional drivers. Select and activate the “version current” of the NVIDIA driver.
  10. Do not reboot. Instead, open a terminal, do
    cat /etc/X11/xorg.conf

    If it contains something like

    Section "Device"
           Identifier "Default Device"
           Option "NoLogo"    "True"
    Endsection

    this is wrong, and you have to issue the following command

    sudo nvidia-xconfig

    Disregard any VALIDATION ERROR messages. Check if the new xorg.conf file contains (among a lot of other things) something like this

    Section "Device"
           Identifier "Device0"
           Driver     "nvidia"
           VendorName "NVIDIA Corporation"
    Endsection
  11. Reboot.
  12. Now you are using the Nvidia drivers. Test them with the following from a terminal
    sudo apt-get install mesa-utils
    glxgears

    You should get a nice spinning gearset. Close the glxgears window

  13. Edit the /etc/rc.local as root with the following command
    sudo vi /etc/rc.local

    and add the following line before the “exit 0″

    /usr/bin/nvidia-smi -pm 1
  14. Reboot. You are done.

The magic line nvidia-smi -pm 1 deserves some explanation. The problem with the corrupted screen and inability to get back to lightdm once logged out is due to the fact that X releases the nvidia driver. with -pm 1 you force the module to be persistent and not be unloaded, The reason why X did not start is probably due to a race condition between the instantiation of the driver and the access of the new X session started by lightdm after logout.

Posted in Hardware. Tags: , , , , , . No Comments »

Building my own NAS/Media center – Part 4 – OS setup

As soon as I plugged the NAS to the electric socket, something was fishy. Apparently it starts up without any signal from the power button. The only way to turn it off is to use the PSU. This is worth more investigation, but for now I will move on to set up the BIOS and installing Ubuntu.

Downloading Ubuntu

I downloaded the Ubuntu 12.04 LTS ISO image from the website. Nothing peculiar here, except for the fact that I need to install from USB key, as I don’t have a CD reader. I followed the instructions from the Ubuntu website and obtained a USB memory stick which I plugged into the USB port of the NAS.

First light and BIOS setup

It was a nice surprise to see that, once I started the NAS and turned on the monitor, Ubuntu was already booting. Apparently the BIOS is already configured to boot from USB stick, which is a plus, but I think it deserves some tinkering just because of the nature of the project. Also, I want to see if the CPU temperature is appropriate. I might have had some troubles due to the difficult seating of the pins.

Apparently, the CPU temperature is a bit weird. It keeps rising from 52 C to a stable 55 C. The CPU fan has four pins, so it’s adaptive and can be controlled via the BIOS. I tinkered the BIOS values to make it work more, and see how the temperature changes with time, but I didn’t get any appreciable difference in the fan speed. The CPU fan is working at 1000 RPM, and the system fan at 4000. I discovered later on that while in the BIOS, the CPU never enters low-power mode, so it’s normal for the temperature to increase.

IMG_3108

Nevertheless, I might have to monitor the situation. According to the Intel website, the maximum temperature (Tcase) for the Core i3 is 69.1 C. The conductive paste has a break-in period, which should lower the temperature of 2-5 C. This might be the reason why I have readings that are higher than what I feel comfortable with. For now, the temperature is not above the limit and I can try to install a basic Ubuntu.

A quick note on noise. The PSU and the processor fans are extremely silent. The GPU fan is instead quite loud in the high-pitch range. Putting the lid on the case doesn’t really solve. I’ll address this problem later.

Installing Ubuntu

The system boots naturally from the USB key. I am impressed at the ease of installation of Ubuntu. Literally 3 clicks and the system is installed. I didn’t bother doing excessive configuration of the partitions, I just let the installer choose for me. The 160 GB disk is probably oversized for the task of keeping the system, and I might consider replacing it with a small SSD or even a plain USB key. I don’t expect a lot of stuff to go on the system partition. The bulk of the space will be provided by the additional hard drives.

After the installation completed, I rechecked the BIOS CPU temperature monitor, which is now at 48 degrees, again increasing. As I said, staying in the BIOS makes the CPU work hard, thus the increase in temperature. I tried monitor the CPU from the system, and it stays rather cool, in the 30 C range. This implies that my setup is apparently fine.

Troubles with Nvidia and Intel HD cards

The nvidia drivers introduced some trouble. I installed the nvidia-current from the software center, but even after reboot I didn’t have a GLX display. I tried running nvidia-xconfig, but it trashed the X window config file. After rebooting, I was left in text mode. I struggled with the problem for a while, until I found a (partial) solution.

The first thing to note is that there are two graphic cards available: the Nvidia (called PEG in the BIOS) and the integrated Intel (called IGD). On Linux, using the command lspci -vv reported two entries with VGA capabilities.  The idea at the hardware level is that the Intel is a low-consumption graphic card. When it needs accelerated rendering, it delegates it to the nvidia card and multiplexes the data. This is called hybrid graphics, and it’s not supported fully by Linux (the usual drill). Some efforts are being made, but they are not for the faint of heart or those who just want something that works (you know, like on the Mac).

Long story short, as Linux can’t use the hybrid mechanism, you have to choose, and the presence of two cards makes trouble, because by default, Linux might use the integrated Intel. Some options are possible, namely to use vga_switcheroo, but it works only if you use the nouveau driver (an opensource implementation of the nvidia drivers). I wanted to use the closed source nvidia ones, making this solution not feasible. As far as I’ve been told, the opensource drivers are still too radioactive to be worth of consideration.

The second point is that there are four sockets for video on the board. two DVI, one DisplayPort, and one HDMI. One DVI (the top one) and the DisplayPort are connected to the Nvidia card. The remaining DVI and the HDMI are connected to the Intel. When I started the system the first time, I plugged in the DVI, but then I realized there was no sound. I thought DVI didn’t carry the sound signal (wrongly), so I connected the HDMI to the monitor (my TV). Unfortunately, this meant that X windows always ran on the Intel, making all efforts to use the Nvidia drivers fruitless (GLX not available when running an OpenGL program, nothing available under the nvidia-settings program, and so on).

To solve, I “switched off” the Intel graphics chip. You can’t technically turn off the integrated Intel, but the way I did is the best you can do to make it irrelevant to Linux. From the BIOS as suggested here I chose option Initiate Graphic adapter, selected PEG/PCIEx, instead of PEG/IGD. Additionally, I connected the second DVI port (the nvidia one) to the monitor via a DVI-HDMI adapter cable. Apparently, this works: lspci now reports only one VGA card, the nvidia, and the drivers now install and work correctly. glxgears runs accelerated at 60 frames per second. I found suggestions to blacklist the driver as an additional line of defense, but as far as lspci tells me, I am running full nvidia now. For additional reference, here is another ubuntu page that might address additional problem someone google visitor may face.

Unfortunately, I don’t get any sound, and I assumed it was because DVI does not carry sound information. There’s plenty of useful information on the topic of audio on this forum, where it says that it’s technically possible to have audio out of the (Nvidia) DisplayPort with a DP->HDMI cable. If you want to know more about DVI, I suggest this excellent article from tomshardware. I thought about feeding sound with a proper cable DVI+Audio Out -> HDMI, when I realized I was probably using a different output. I changed it in the Ubuntu system configuration and I am now getting audio.

I have two problems left, one minor, one major: the first, minor one is that the noise of the GPU fan is extremely annoying; the second, major one is that I can’t log out. If I do, the system hangs. I’ll address these two problems one after another.

Changing Fan behavior from the BIOS

To make the fans more silent, I simply changed the appropriate values for temperature startup and running speed in the BIOS. I settled for a maximum 80% fan speed  (the “fan duty” value). This reduces the noise considerably, while keeping the temperature within an acceptable maximum.

Problems with lightdm at logout

Solving this problem was much harder: I quickly realized I could not perform a sane logout once I started using the nvidia card. When I got out of the X session, instead of going back to lightdm, I got a strange garbled text-only display instead.

IMG_3128

I followed the advice from user jazztickets, and tried various proposed solutions in the ubuntu forums, such as adding an entry for udev to lightdm configuration, checking other users in similar conditions, dropping lightdm for gdm in the hope that it was a lightdm issue, adding a sleep 5 just before the exec lightdm in /etc/init/lightdm.conf, and also putting DEVPATH=*card0 instead of card0 in the “start on” entry of the same file. None of these worked.

After plenty of reboots and no clear clue, I managed somehow to stop the X server, get the garbled text, and get access to the text consoles, something that was not possible before. Magic, or chance, I’ll never know. In any case, I noted that I could not restart X manually due to the following error

NVIDIA: could not open the device file /dev/nvidia (Input/output error)

That gave me a hint. I tried to add vmalloc=256M in grub. Didn’t work, and I then found it’s only for 32 bit machines anyway. I also tried to remove xorg.conf and recreate it. No luck.

I started reinstalling Ubuntu again and again, trying different options. It took me 3 days and a lot of googling, but I finally found the sequence of events that works, which will be the subject of the next post.

Formatting and encrypting the big hard drive

The 3 terabyte big hard drive is correctly seen at boot and the entry is present in dmesg

[ 2.402539] scsi 1:0:0:0: Direct-Access ATA WDC WD30EZRX-00M 80.0 PQ: 0 ANSI: 5
[ 2.402606] sd 1:0:0:0: Attached scsi generic sg1 type 0
[ 2.402621] sd 1:0:0:0: [sdb] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)

In order to partition and format it, I had to use parted, since fdisk apparently does not support disks so big. I found this post extremely useful and I will give here just the essence of the operation

(parted) mklabel gpt
Warning: The existing disk label on /dev/sdb will be destroyed and all data on
this disk will be lost. Do you want to continue?
Yes/No? yes
(parted) unit TB
(parted) mkpart primary 0.00TB 3.00TB
(parted) print
Model: ATA WDC WD30EZRX-00M (scsi)
Disk /dev/sdb: 3.00TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number Start End Size File system Name Flags
1 0.00TB 3.00TB 3.00TB primary

Now that I have a partition, it is time to bring truecrypt to work. I want to encrypt my disk for security. In case of theft, my data would be easily accessible, and I don’t really like this. I will use truecrypt to encrypt the whole partition, then mount it manually when needed by typing in the password. Although it appears to be a bit annoying, it will be done only once the computer boots up. I plan to keep it asleep when I’m not using it, and this should not compromise the truecrypt mount. If the computer gets stolen, it must necessarily be unplugged, and the data won’t be accessible later on.

Once again, another blogger provides me an easy (though outdated) guide to achieve my goal. Truecrypt is easily installed from the website. The downloadable .tar.gz package extracts to an executable script that can be run and performs installation graphically. After the installation is completed, I had a “truecrypt” application in my application menu. Then I created the encrypted partition with

truecrypt -t -c /dev/sdb1

and answered the questions. I chose my preferred encryption and hash method, my password, and ext4 as a partition. The operation worked through the night.

Unfortunately, in the morning I got a 100% completion but also a I/O error. Being the disk new, I started being worried of a potential faulty drive. I checked the kernel logs with dmesg and got plenty of messages of this kind

[168480.771645] sd 1:0:0:0: [sdb] Unhandled error code
[168480.771647] sd 1:0:0:0: [sdb]  
[168480.771647] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[168480.771648] sd 1:0:0:0: [sdb] CDB: 
[168480.771649] Read(10): 28 00 c1 cf 57 d8 00 00 08 00
[168480.771653] end_request: I/O error, dev sdb, sector 3251591128

bad news. I ran badblocks, but that turned out to be a very bad idea. The program ran for 9 hours with CPU at max power, getting only to 50% of the task, and I got my other hard drive filled with kernel error messages. I was able to recover the situation by killing the process on a dying system, and freeing some space. I am not convinced it’s a hardware issue, and before returning it, I decided to do some investigation.

My first suspect is temperature. While the process gets pretty hot (maximum 70 degrees, operative around 65) despite the big fan, the WD drive was rather cool, so I’d exclude temperature as a potential troublemaker.

The second suspect is power supply. Is 350 W enough? I both asked on superuser, and checked on the eXtreme power supply calculator. For my configuration, around 170 W should be enough. I have plenty of wiggle room when it comes to power, and other components would fail if power were an issue.

At this point, I decided to install gsmartcontrol, and run the SMART diagnostics. Right after the incident, the drive was “Unknown”. After the reboot, it is correctly detected. All tests are successful. What gives, then?

I decided to do nothing and ignore the error. I have the suspicion the problem is a kernel issue, and if it doesn’t bother me in ordinary use, I won’t really care. What I did was to set the driver on the SATA II channel (instead of the SATA III), change the cable, and simply do a quick truecrypt creation with

truecrypt -t -c --quick /dev/sdb1

This prevents encryption of the free space. I am not that paranoid. The result is that the filesystem is created and formatted instantly, and I can successfully mount it with

truecrypt /dev/sdb1 /media/test

I decided to trim down the reserved space for root to 1%, since this is not a boot disk

sudo tune2fs -m 1 /dev/mapper/truecrypt1

And finally, I have an encrypted filesystem on a really, really big disk.

Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/truecrypt1  2.7T  201M  2.7T   1% /media/test

I also set the scheduler from “deadline” (the default) to “noop” first and “cfq” later. I had a lot of problems rsyncing a large external hard drive to the NAS. The whole computer froze, sshd dropped connections, and the processor temperature started raising. I simply did

echo "noop" >/sys/block/sdb/queue/scheduler

and the problems stopped, although I am not sure the scheduler was the real source of it.

I confirm that both temperature and the disk are acceptable and working. I stressed both the disk and the processor for hours, filling it completely with random generated data. No problems at all, so I start to cast some personal doubts over the “deadline” scheduler, at least with my current setup.

Posted in Hardware, Ubuntu. Tags: , , . Comments Off »

Building my own NAS/Media center – Part 3 – Assembling

Before I start, please keep in mind that all images are linked to a high resolution version, in case you want to see more details. For the impatient, here is a pic of the final assembly

IMG_3508

Preparing the case

To start my adventure in assembling, I opened and prepared the case. There’s plenty of space, for now.

IMG_3009

The case has three bottom intakes (one covered by the hard drive brace), and one potential intake on the back. I say potential, because this is actually the space for the PCIe cards, and the plate can be removed with some minor hack. I am not planning to use any PCIe card, so at least in theory, I can put a fan in there.

The case came well equipped with cables. The connectors you see for the front panel are for the frontal USB ports and the IR receiver/front panel buttons. The case came with a remote, so I assume it’s possible to generate USB keyboard events and hook them to specific software actions. Unfortunately, my motherboard comes with only one 5 pins plug for USB2, so I had to choose between the frontal USB panel and the IR receiver. I chose the former.

One possible solution to this dilemma is that the motherboard also provides USB3 pins. These pins accommodate a 2 ports USB3 connector, shaped to fit the PCIe reserved area in the back. I could, in principle, keep this connector inside the case, then use this USB to PIN adapter to plug the IR receiver in. It’s a side project I will attack if I have time. For now, I need to assemble the motherboard in place.

Putting the processor in the socket

The Zotac Z68 ITX motherboard is very cute but packed with action.

Zotac Z68-ITX Supreme Hardware

I removed the red warning label, and opened the CPU socket by moving the lever on the side. The amount of pins found in a modern processor is mind-bending.

IMG_3026

Once I unpacked the processor, fitting the processor required some care. First, I had to be sure it was correctly aligned (see red arrows). Then I had to clean it from the inevitable fingerprint smudges that end up on it after I fumbled for a while trying to find the correct alignment.

IMG_3027

All that was left to do was to close the metal brace and put the lever back in the slot. It required some “Damn I’m going to break it” strength, but it fitted nicely.

IMG_3028

Assembling the processor fan and cooling

The fan and the cooling system came preassembled. I had a lot of different mounting braces available, and I had to choose the proper one.

IMG_3031

Unfortunately, I had to use the push pins, the black and white plastic things you can see in the picture. These things are evil, and I’ll get into that in a moment.

Before that, I had to apply the heat paste. I visited the Arctic Silver website, where they have clear instructions for the various processors. For my i3, the vertical strip method is the best choice. I added a small amount (a bit more than the size of a rice seed)  to the cooling plate

IMG_3034

and spread it thin and uniform with a plastic spatula. This fills eventual gaps and provides better grip to what comes next.

IMG_3036

Now it’s time for the processor. I spread a uniform line in the appropriate direction, 2-3 mm thick.

IMG_3039

and I can now put the cooling system on. Here comes the push pins. These things are extremely hard to put on, especially the last one. Supposedly, you push them and a click they are set. In practice, I had to push so hard for the last two that I almost broke my thumb. To remove them, it’s even worse, because you have to rotate the upper part, but the pressure makes it hard to rotate and you don’t have enough leverage. If you add to this the cramped space of a mini-ITX board, the evening ends in tears, swearing, and more than 30 minutes of failed attempts to settle the last pin. The bastard is here pictured

IMG_3040

Assembling the RAM

Adding the two RAM modules is easy. Just open the white locks

IMG_3053

align the modules in the sockets, putting the notch in the right place, and push down vertically with some force. The white locks will close automatically. Clearly, I started from the slot most difficult to reach, that is, the one near the processor. The CPU cooler prevents easy access to this slot, but it’s a minor inconvenience and everything slides into place with relative ease.

Adding the Hard Drives

For testing and initial boot, I did not use the large hard drive, but an old spare I had laying around. I originally changed it due to some worries about its condition, so I might just discard it, but I used it for a while and it’s not giving me any problems. I just plugged the SATA cable and the power cable. Initially, I let it loose, but when I started the final assembly I used Powerstrips to lock it into a reasonable position.

IMG_3341

For the big hard drive, I used the 3.5 inches slot provided by the case. The remaining 5.25 inches is currently empty, and technically made to accept a dvd player, but I am keeping the option open for a second hard drive.

IMG_3334

Final result

Apparently, jumpers went the way of the dodo and the dinosaurs. The Motherboard has only one jumper, to reset the BIOS. Nothing to be done on this regard.

I set my fan to be blowing into the case from the bottom, so to keep a higher pressure inside (reduces dust accumulation). Hot air escapes from the back through the grid. This setup seems to be fine, but if I find additional problems, I can add a second fan to the PCIe grid with some hacking.

When it comes to electrical consumption, the computer in standby  (suspended) consumes 2 W, probably to keep the ethernet powered up for the WakeOnLan. When turned on and idle, the consumption clocks at 50 W, which raises to 60 W when watching a movie. Playing minecraft drains up to 100 W.

Here is again the picture of the final assembly. Now it’s time for software configuration, the subject of the next post.

IMG_3508

Posted in Hardware. Tags: , , , . Comments Off »

Building my own NAS/Media center – Part 2 – Hardware

To select the hardware, I mostly followed indications from various sources. This blog had me started on the basics, together with the followup by the same author. Most of the open questions and possible troubles have been solved through aggressive googling, but most of all precious contribute from a friend of mine, Hans.

Here is a quick panorama of the various components I bought, together with the unit price, the shop I used, and a general description for the choice I made.

Motherboard: Zotac Z68-ITX WiFi Supreme

Zotac Z68-ITX Supreme

Price: 110.99 EUR on salland.eu

This little mini-ITX motherboard has everything I need. Extremely compact and yet very powerful.

Zotac Z68-ITX Supreme Hardware

It supports Intel core i3/5/7 processors, so it leaves my options open when it comes to processor choice. It has an on-board nvidia (great for experiments with CUDA) and support for on-chip Intel HD, which is provided by my Intel Core i3. This is normally known as a hybrid graphics system, and it can be a source of trouble with Linux. More details will follow on the subsequent posts.

There are plenty of connection options in the back panel.

Zotac Z68-ITX back panel

Notice the presence of many different video output ports. This will become important later on, during graphics setup. There are also plenty of USB ports, both on the panel and on-board, and an eSATA port.

Processor: Intel Core i3 3.3 GHz tray

Intel Core I3

Price: 107.99 EUR sold by salland.eu

A core i3 is probably all I need. If I find it inadequate, the motherboard can take up to an i7. I probably erred in choosing a high clock frequency, as the higher it is, the higher is the heat dissipated. I was surprised at the shape and weight of the processor. The last time I had a processor in my hands, it was a bulky but light Pentium II 333 MHz cartridge. The i3 is small, tough and heavy.

I decided to take the tray version. The boxed version includes the cooler/fan (or so I’ve been told), but I was not sure of the cooler size, and I wanted a low profile, low noise one. Vertical space in the case is at a premium.

Processor cooling:  Scythe Shuriken Rev.2 – SCSK-1100

Schyte Shuriken Rev. B

Price: 28.95 + 19.99 shipping = 48.94 GBP = 57.07 EUR sold by ADMI Limited

This processor fan is dead silent and with a reduced height, making it the perfect candidate for a HTPC case form factor.

Thermal paste: Arctic Silver 5 3.5g Thermal Paste

Arctic Silver

Price: 6.05 + 5.02 = 11.07 GBP = 12.91 EUR  Sold by ADMI Limited

This is the best thermal paste you can get, according to my friend Hans. The Scythe came with a small amount of unnamed paste, but I preferred to have my own Artic Silver syringe.

Case: Techsolo TC-2200 Casing M-ATX HTPC 350 W Aluminium

Techsolo mediacenter

Price: 80.43 + 14.75 shipping = 95.18 GBP = 110.99 EUR from ALB Computer Germany via Amazon.co.uk

I chose this case for many reasons: first, it was cheap. Second, it included a Power Supply that was already sized appropriately to my needs. Third, its small form factor. There’s not a lot of disk slots available (only two, a 3.5 and a 5.25), I am not planning to build a disk farm.

Techsolo TC-2200

The case is also quite stylish and with plenty of features, including a remote control (which I am not using at the moment). Overall, I think this case as excellent for my needs.

Memory: Crucial SODIMM 2GBx2,204-pin,DDR3 PC3-10600,Cl=9,1.5v

Crucial SO-DIMM

Price: 27.40 + 7.02 = 34.42 GBP = 40.14 EUR by: mopodo-uk

I did not have particular requirements for the RAM, I just wanted a decent amount. 8 GiB were probably too much. I opted for the same amount I have on my laptop, and I will upgrade later on if I really want to. RAM is cheap, but less RAM is cheaper.

Hard drive - WD Caviar Green 3000 GB, SATA, 64 MB, 5400 RPM

WD Caviar Green

Price: 121,99 EUR from salland.eu

I chose a Western Digital Green with a whooping 3 Terabytes of space. After the starving of the SSD, I don’t want to risk space exhaustion anymore.

Case Fan - Scythe Glide Stream Zwart, 120 mm, 1000 Rpm

Scythe Glide Stream

Price 13,00 EUR from salland.eu

I needed a good fan to improve air circulation in the case. I chose the dead-silent, big Scythe Glide Stream. The fan is big, and fits perfectly on the bottom vent of the case.

Bluetooth dongle – Konig Micro USB

Bluetooth dongle

Price: 13,00 EUR from salland.eu

A simple bluetooth dongle to connect to a wireless mouse and keyboard. Also useful if I were to implement a proximity detection so that the media center turns active when I come back home.

OS Hard drive – Unnamed Fujitsu

Fujitsu

Price: Unknown. Recovered from my Mac after replacement.

This is the old disk I extracted from my mac and replaced with an SSD. I changed it due to worries about hardware failure, but apparently the disk is fine, and my woes at the time might have been due to battery issues. The disk is only 160 GB, which is enough to keep the OS and some additional software I might install.

The reason for the powerstrips will become clear in later posts.

Total price for the system and final remarks

With the exclusion of the latter item, the final price for the system is 588.08 EUR. The basic MacMini is priced at 649,00 EUR, and has a smaller hard drive (500 GB), no nvidia card, and less software flexibility.  I am overall extremely pleased with the result I obtained.

As a final note, I am deeply satisfied of the online shop salland.eu. They have excellent products in stock, they are quick at shipping and provide good and fast email service. I can confidently recommend them, if you can deal with the page in Dutch.

In the next post, I will start assembling the hardware.

Posted in Hardware. Tags: , , , . Comments Off »

Building my own NAS/Media center – Part 1 – Introduction

A few weeks ago, I took a look at my MacBook’s disk occupation and got a bit worried. The great little software Disk Inventory X made extremely clear that my work directory was a bit too packed with pictures, and checking the total disk occupation left me with an uneasy feeling of scarcity. This is the life of the “lucky” owners of SSD disks: fast, small, and unreliable. In fact, I had some worrying events on my laptop, like requiring multiple on and offs to finally show the “apple” logo, or getting stuck while installing updates. To be fair, it could also be the battery’s fault (which OSX is asking me to service), but these factors combined created a worry and an itch I must scratch.

Building my own Network Attached Storage system

Clearly I needed a NAS where I could put my stuff, keeping the laptop light, and get access through the local network. I checked around for interesting products, and I found some: I really like Drobo, but to me, the absolute best is the Synology product selection. Yet, the idea of having a dedicated product for nothing else than storage started to feel a bit restrictive. I have a TV with HDMI input. Why can’t I play Minecraft on it? The NAS project slowly became a NAS + “Media center” project, and I embraced the complexity. I quickly realized it was impossible to find this kind of product off-the-shelf, so I started planning for self-assembling.

The last time I built my own computer, the processor available was a Pentium II 333 MHz. It was as cool as it can get, with its cartridge-like shape and its holographic watermark. Despite having previous experience in hardware assembling, it took me a couple of days to build and even more to install, due to driver problems with Linux. Overall, it was a good experience but I was more interested in the software side. A few years later, I abandoned the Intel/Linux world and switched to PowerPC/OSX, and the rest is history. Things just work in the Mac world, and I like it that way in some circumstances, when my target is to use a computer with minimal fuss. Yet, I am not over-zealot as a Mac user, and this is especially true after they decided to drift more and more towards markets I either don’t care about (iPhone stuff) nor support (App stores). I also recognize a chance for flexibility and ad-hoc tinkering when I see one, and I’m all for it.

“Stefano” – I hear you say – “Why don’t you buy a Mac mini with an additional disk and get over the problem? It’s a full computer, it’s small, it has OSX and it does all you need”. Indeed this is true, but it’s also more expensive. Also, the needs are different: I don’t want a closed system. I need a flexible, highly configurable system that allows me to do both network storage and general computing in any potential software direction, and change the hardware when needed, adding new disks or upgrading the old ones. Having some additional nice features like an Nvidia chip for some grotesque CUDA experiment is a plus. From where can I start, with no experience in modern hardware?

Re-learning the world of consumer hardware

Not having a lot of hardware experience, I had to re-learn or update on a few things: processor types, socket types, RAM types, Motherboard types, case types, and power supply requirements. For example, the processor world is confusing at best. Here are some of the questions I had

  1. Is an LGA 1155 socket compatible with an LGA 1156. No, but they are for cooler purposes
  2. How many processors are available today, and which one should I get? Too many. In addition to the plenty of i3/5/7 models, Intel also has restored the Pentium name for recent processors.
  3. Sandy bridge, Ivy bridge? Ok, bored now.

Overwhelmed with the details, I asked a friend for help and checked some blogs. He gave me an extremely useful course in modern hardware, and an extremely useful price-comparison website, tweakers.net. It’s in dutch, but having spent a year in the most badass country in the world, I understand enough to use it. For the blogs, an authentic trove of information came from this one. In fact, I almost built his configuration, with proper changes.

In the next part, I will detail the products I bought and their price.

Posted in Hardware. Tags: , , , . Comments Off »

The von Neumann method revisited: a letter from a reader

It’s always a source of extreme satisfaction when someone contacts you about a post you made, and even more so when the mail actually answers an open question I had about generalizing the von Neumann method. If you don’t recall what’s about, it was about getting a fair coin throw from an unfair coin. I wondered if it was possible to generalize it to any number of faces (a coin is a 2 face die), but left the issue unsolved and moved on.

A few days ago, I received a mail from Albert Rafetseder, which I copy here verbatim except for minor presentation adjustments. Thank you Albert for the interest and the cool research

Hello Stefano,

by chance I came across your blog (via some Python PEP discussion you participated in), finding your post from August on the von Neumann method. I have pondered the question of a von Neumann die, too, and my solution is this: Observe that the von Neumann coin method can extract an unbiased coin flip from those strings of flips of length 2 that are permutations on the set S of all possible outcomes of the biased coin S = {0, 1}:

Permutations P(S) = {01, 10}.

There are Length(S)! possible permutations, i.e. the extracted unbiased “random gadget” is a coin again.

If you consider a die with S = {1, 2, 3, 4, 5, 6}, then P(S) = {123456, 123465, 123564, 123546, …. }, and the extracted unbiased random gadget is a Length(S)! = 6! = 720 facet thing. Unless, of course, you simply group the permutations by some property to reduce the number of unbiased results, e.g. grouping all 120 of them that had a “1″ in the first roll, and so on for every s in S.

By different grouping, I believe you could fabricate an unbiased random gadget for every divisor of 720, too; similarly, you could use a biased n-sided die rather than a six-sided one. And maybe you could increase the efficiency of the scheme by taking into account how the pips on opposing sides usually add up to seven ([that is:] probabilities of opposing sides of a real die are somewhat complementary, although they usually do not complement in an absolute sense, i.e. not q=1-p). This conjecture is problematic: first, I have nothing to its support other than that I think the physics would work this way in a real die. Second, this (sub-)complementary property does not hold in general for a simulated die as the probabilities for each side could be anything. Third, no joy for “dies” with an odd number of sides, obviously.

As it often goes, you and I are not the first people to ponder this question. Here are other’s peoples approaches: this one and especially this which lists a rich source of prior as well.

Also, optimization of von Neumann’s coin method exist that don’t throw away so many 00 or 11 results. [This one] has a multi-level method for extracting unbiased bits from longer strings of flips, indeed asking at the very end “Now, how do you simulate an unbiased die with a biased die”. (Well, we know now). This is a collection of further reading material on Stackexchange. I haven’t read this one, but it seems to take up the efficiency/optimization idea too.

After stealing so much of your valuable time, allow me to mention one further paper that describes computations that can be done using random numbers of unknown probability distribution. I personally find late Philippe Flajolet’s work always worth a read.

With that, thank you for your blog post reminding me of this interesting topic (and stimulating me to write up my thoughts on it).

Best regards,
Albert.

I will hopefully explore in more details with actual code after my next series is finished. I built a NAS from scratch and I have a long series of posts ready to ship starting next month. I want to keep the post chain uninterrupted, but I really wanted to publish your mail as soon as I could. For this reason, I couldn’t research any further on your insights, but they are definitely worth of interest for future posts. Stay tuned!

Posted in Probability. Tags: . Comments Off »

Old CentOS “4 is not a valid release or hasnt been released yet” error

If you have to handle an old Centos 4 machine and you try to use yum to get some packages, you will get an error “4 is not a valid release or hasnt been released yet”. This is because CentOS 4 is end of life, and the default repo does not provide updates anymore. Yet, you might still be interested in getting the latest packages for the end of life release.

You can solve the problem by switching to the vault as a repo, doing the following

  1. open /etc/yum.repos.d/CentOS-Base.repo
  2. comment out all the mirrorlist entries
  3. uncomment and set all the baseurl entries to http://vault.centos.org/4.9/os/$basearch

 

Posted in Linux. Tags: . Comments Off »

Git hangs at clone? It could be a Type of Service issue.

Today I was trying to download some code via git, and I got into a strange problem. When I ran git clone, it simply reported cloning into ‘repo’ and was stuck there, until timeout. Same thing for pull. I could not understand what was happening, and it took me a while to figure it out but here is the solution, at least to my problem. Add this to your .ssh/config

Host *
  IPQoS 0x00

What was the problem? Here are the symptoms:

  • git worked on my NAS (Linux, connected to the router via cable), but not from my laptop (OSX, connected via WiFi)
  • ssh worked from the laptop. I could ssh to the NAS and to other external ssh machines.
  • However, I observed that the interactive ssh behaved strange when I ssh’d to an external machine. For example, if I asked for an ls, it kind of got stuck, and I had to press enter every time to get further data. It was like acting with a constant “more”.
  • using GIT_TRACE, I found out that ssh negotiation was completely successful and it got stuck at invoking git-upload-pack via ssh.

Google didn’t help. I changed versions of ssh and git, checked my router for strange setups, but nothing could solve. Until I smelled something about the way packets were exchanged. Don’t ask me how I did it. I don’t know. I just had the feeling that strange “enter to get more stuff” behavior had something to do with it. The fact that the laptop had the problem, while the NAS didn’t, made me think it was either a OS issue or a NAT configuration issue.

So I started checking something about NAT, adding NAT to the google query, and here I found a post of a Dane who had similar troubles, apparently caused by the router messing up due to Quality of Service. This sounded just about right. Quality of Service influences how packets are considered for priority routing. I did more searching until I found this other post, where I found the solution given above.

It was a fun hour.

Posted in Linux, MacOSX. Tags: , . Comments Off »

A raytracer in python – part 6: cameras

In the latest commit for the raytracer, I added cameras. The design changed so that now the responsible for rendering is the camera object. Actual cameras are specializations of an abstract class BaseCamera, which holds common information about positioning. The BaseCamera is then specialized into two concrete classes:

  1. PinholeCamera is a camera where rays are shot as diverging from a single point, called the eye_point. This allows perspective, which was not present previously as the rays were emerging from the ViewPlane pixels.
  2. LensCamera is a camera that simulates depth of field, that is, focus/out of focus. Contrary to the PinholeCamera, where everything is in focus, LensCamera allows different focusing. Objects that happen to be on the “focal plane” are in focus, while objects that are outside (either closer or farther from the camera) present less defined details proper of an out-of-focus object. To perform this effect, we need the random sampling on a disk implemented in the previous post.

The following picture shows how LensCamera performs. A set of hemispheres are deployed along a line. The camera is above them, slightly angled and with a roll angle appreciable from the horizon. In all three cases, the orange central sphere is focused, as the focus plane has been set to fall on the sphere’s position. Note how other objects are in focus for a Pinhole camera (left picture) which has no depth of field by construction, and become more out of focus as the lens size increases (1.0 in the center picture, 5.0 in the right one)

Focusing

From left to right, PinholeCamera, LensCamera with lens size 1.0, LensCamera with lens size 5.0

Other cameras may technically be possible: the book goes further in deploying fisheye and stereoscopic cameras, but I am not interested in them. I think the pinhole and lens camera are flexible enough for quality renderings and my desire to learn.

One important feature of the Camera system is that it requires the definition of local coordinates on the camera itself. The three vectors defining this set of coordinates, called u, v, w in the book, are obtained by building an orthonormal basis using the cross product between the observation vector (the vector between the “eye” of the camera and the look_at point) and an “up” vector, our default being in the same direction as the y axis. Doing the cross product of these two vectors (observation and up) produces the third remaining vector of the orthogonal basis centered on the camera. However, if the camera looks straight up, or straight down, the cross product is zero and we obtain a singularity, losing one degree of freedom (a condition also known as gimbal lock). The book proposes to detect this condition and treat it accordingly, by either overriding the specification and setting the vectors to an arbitrary, well defined alternative, or by “juggling” the up vector out of alignment so that the third vector is still defined. I decided for the third option, ignore the problem, as I am not going to use gimbal locked configurations for now, but it’s definitely a problem to add to the todo list.

With this post, I take a temporary break from the raytracing business. I may add optical effects such as reflections, refractions, materials, lights, but the point is that the amount of rays that must be propagated for these effects to show tends to be very high. I want to venture into CUDA, and therefore I will switch my attention to CUDA programming from now on, integrate it with the raytracing later on, then go back to light effects at a later stage. I will implement light effects first in python, then use CUDA to achieve the same results. My aim is to have fun, test CUDA/C/Python integration, compare performances, and provide a fully python raytracer with optional C/CUDA high-performance code to achieve the same task. For CUDA tinkering, I will switch back to my old friend, the mandelbrot set.

Posted in Python, Raytracing. Comments Off »

How to convert a QString to unicode object in python 2?

I had this problem to solve, and I tried to find the safest way. This program illustrates the solution

from PyQt4 import QtCore, QtGui                                                                                                                       
import sys                                                                                                                                            

app = QtGui.QApplication(sys.argv)                                                                                                                    
ed = QtGui.QLineEdit()                                                                                                                                

def editingFinished():                                                                                                                                
    # The text() returns a QString, which is unicode-aware                                                                                            
    print type(ed.text())                                                                                                                             
    # This returns a QByteArray, which is the encoded unicode string in the utf-8 encoding.                                                           
    print type(ed.text().toUtf8())                                                                                                                    
    # Since unicode() accepts a sequence of bytes, the safest and fully controlled way of performing                                                  
    # the transformation is to pass the encoded sequence of bytes, and let unicode know it is utf-8 encoded                                           
    print unicode(ed.text().toUtf8(), encoding="UTF-8")                                                                                               

QtCore.QObject.connect(ed, QtCore.SIGNAL("editingFinished()"), editingFinished)                                                                       
ed.show()                                                                                                                                             

app.exec_()

So the solution is

unicode(qstring_object.toUtf8(), encoding="UTF-8")

Maybe there’s another, simpler way that it’s also safe, but for now this solution is good enough.

Posted in Python. Comments Off »