Python / Cython / Java / Go / Rust

I’m in love with Python. At first I had to use it because I had to make and maintain some Subversion hooks, and hated its indentations, that reminded me of Fortran.

Some years ago I rediscovered it, and I love it as a programming language, it is a language you can have fun programming with, powerful, logical and complete. And it is open source. Recently I discovered Django, and it has become my web framework of choice.

But this Python has a problem. It is not fast as a rattle snake, but slow as a snail. As you may know, Python is interpreted. Unlike C or C++, you cannot compile a machine code executable that the CPU can run directly. The interpreter takes the source file and generates a “*.pyc” byte-code file that is then read for execution. It is a step further than simply interpreting the source code, like BASIC, but it is not real machine code compilation. There are other languages, like Java, that use the same technique.

So let’s do something to accelerate a language we love. Can we compile Python? We have “Cython”, a tool we can use to generate machine code from a Python source code.  Cool! So we have it! Fast Python code, the panacea. Well, it depends. Let’s have a look.

Let’s take a little piece of code that concatenates strings, in Python:

def test_fun():
 s = ""
 for i in range(100000):
 s = s + "/" + str(i)

def main():
 test_fun()

if __name__ == "__main__": main()

If we execute it with python interpreter:

$ time python test.py
real 0m3.766s
user 0m1.812s
sys 0m1.955s

With cython, a C source can be generated, that you can compile with gcc. Let’s give it a chance:

$ cython -2 --embed test.py
$ gcc -pthread -fPIC -fwrapv -O2 -Wall -fno-strict-aliasing -I/usr/include/python2.7 -lpython2.7 -o test test.c
$ time ./test
real 0m6.895s
user 0m3.293s
sys 0m3.552s

Oh, no! Compilation <> acceleration. I’m sure this test is too simple, optimizations can be done and all that, maybe cython is good at some specifical tasks, etc. But as you can see, the time as almost double. Not very promising. C code generated by Cython is not optimized at all. In this particular case, it is pesimized.

If we want fast programs, I’m afraid Python is not the choice. But if you want to program fast, Python is your language.

Let’s try to fall in love with some other languages. Well, I have been a Java programmer for years, and I used to love it. I suspect Java will be faster. Let’s translate that simple program to Java. I don’t want to use StringBuffer, instead I want to do it as clear and simple as it is in my Python example above:

package javatest;

public class Test {
   public static void main(String []args) {
     String s = "";
     for (int i=0;i<100000;i++){
        s = s + "/" + Integer.toString(i);
     }
   }
}

How faster will it be?
time will tell us the truth:

$ time java javatest.Test

real	0m36.517s
user	0m37.533s
sys	0m0.277s

Oh, no! My good old Java is slower than Python for this simple task! What can I do now?
It is not two times slower, but TEN times slower. I’m sure that with StringBuffer we can do something better, maybe another day (or you can give me the answer in the comments below).

I’m afraid I will have to learn a new programming language. No problem, I like it. I have heard of two languages with cool names: Go and Rust

Go is a quite new language by people at Google. It is interpreted, very easy to learn and quite interesting. The translation will be something like this:

package main

import (
            "strconv"
        )

func main() {
   s := ""
   for i := 0; i < 100000; i++ {
        s = s + "/" + strconv.Itoa(i)
   }
}

The mecanism for running Go is similar to that of Python: you generate a byte-code file and run it. So let’s Go:

$ time go run test.go 

real	0m2.789s
user	0m2.363s
sys	0m0.093s

Good! Faster than Python, 1 second below Python’s mark. Very promising.

Our other option for today’s little benchmark is Rust. Rust is compilable, like C.   I am not a Rust expert, so I’m sure my translation below could be improved in many ways. In addition to that, Rust development is very active and the language definition is changing to some degree.

fn main() {

    let mut count: int = 0;
    let mut s = "".to_string();
    let mut count_s = "".to_string();
    let bar = "/".to_string();

    loop {
       count += 1;
       count_s = count.to_string();
       s = s + bar.as_slice() + count_s.as_slice();
       if count == 100000 {
            break;
        }
    }
}
$ rustc test.rs
$ time ./test

real	0m0.038s
user	0m0.037s
sys	0m0.001s

What? 38 milliseconds? Rust is similar to C in performance, and this result was expected. Really good result in terms of performace, and Rust is being developed to be a enjoyable language, at least more than C. I agree that C is THE language if you want to program close to the machine, but Rust can be a good alternative if you want to have fun while programming. You have even some web frameworks for it: nickel and Iron, for example.

Web frameworks for Go are also available, and they are very active. Go will go very far (I didn’t want to make more word plays with the name, I swear, but it is too easy…).

I know this benchmark is a I-do-not-know-what-to do-before-going-to-bed-let’s-do-a-benchmark, not very comprehensive, not very accurate, not scientific at all. Comparing Rust with the rest of the languages is not fair. But it served me to have an overview of these programming languages in terms of performance, and wanted to share it with the community.

In summary:

  • Python: 3.8s
  • Cython compiled: 6.9s
  • Java: 36.5s
  • Go: 2.8s
  • Rust: 38ms

Linux on AIRIS Kira N7000

Yesterday I installed Linux (a distribution called Kirbian, excellent job by these guys: http://kirbian.wordpress.com, thanks!!! ) on a AIRIS Kira N7000, an arm based netbook.

Just to help anyone who wants to do the same, I managed to make Wikidpad work on this device. The original wxPython library did not work in this hardware, it threw an “illegal instruction” exception, but after compiling the library on the device, it worked perfectly. Even with 256MB of RAM, it is perfectly suitable for my purposes.

I would like to have Dropbox in the device, but there is still no dropbox for arm. If you are a dropbox user, you can vote for this development in: https://www.dropbox.com/votebox/358/linux-arm-support, as I mentioned in my previous post.

So I installed btsync in both my laptop and kira, and will synchronize Dropbox between them, until there is some arm support from dropbox…

Image

OpenSuSE not booting after upgrading from 12.1 to 12.2

Hi all, this post has nothing to do with Raspberri Pi, but following its advices will be very valuable in any Linux system.

The problem.

I upgraded my PC with OpenSuSE 12.1 to 12.2 following this guide: http://en.opensuse.org/SDB:System_upgrade

Nothing new, I have been upgrading this system from 11.2 following more or less the same procedure.

After upgrading, the system does not boot. Well, it did boot, but got stuck at some point and did not show neither XWindow nor shell.
I tried booting in single mode (in OpenSuSE, just write “single” when the grub menu shows up), and the same symthoms.
This system, since 12.1 upgrade, is booting with systemd instead of sysvinit, so I tried to go back to old sysvinit (press F5 in grub menu or add init=/sbin/init to boot command line). It said that I did not have any /etc/inittab, maybe I lost it during the upgrade. So let’s try the shell boot.

On boot menu, write “init=/bin/sh” or press F5 and choose “Shell”. That drives you to a system with no network, no graphical interface, a very limited system with no services at all, but with a PROPMT!
On this shell, you can see, as I saw, that the special device files for a mirror disk had changed from /dev/md1 to /dev/md127. Linux, did not find my mirror. Also, I found that special device files for raw disks had also changed, but fortunately I had them mounted using either LVM or /dev/disks/by-id device files.

The solution
Well, using this shell boot you can edit fstab and make the system boot again. Once you have booted (or before, but you don’t have copy paste features 🙂 ), use udevadm to find information about mounted disks, this way:
# udevadm info -q all -n /dev/sda1 | grep DEVLINKS

You get something like:
E: DEVLINKS=/dev/disk/by-id/ata-ST9320423AS_5VH3W3P8-part1 /dev/disk/by-id/scsi-SATA_ST9320423AS_5VH3W3P8-part1 /dev/disk/by-id/wwn-0x5000c50029cb080f-part1 /dev/disk/by-path/pci-0000:00:1f.2-scsi-0:0:0:0-part1

Use one of the by-id special device files instead of /dev/sdXP because the latter can change due to upgrades, race conditions or simply hardware relocation of disks. Using /dev/disk/by-path device files can yield the same problems, because it depends on the hardware path of the disk, and it can change if you have a screwdriver at home.

I also got rid of md mirroring and migrated the disk to LVM, for one thing, I didn’t have a mirror at all, just a “one disk mirror”… historical reasons… LVM could have mounted this disk, because lvols device files do not depend on disk location or udev discovering. For instance, /dev/vg01/lvol1 will always be /dev/vg01/lvol1 as long as the vg metadata is not corrupted, so you can trust LVM and use lvol device files in fstab.

You can also write udev rules in order to call your disks anything you want, depending on serial number, size of the disk, etc. and use that name in fstab.

Trusting any kind of raw special device files is not a good practice at all. This is the same that trusting /dev/dsk/cXtYdZ device files in HP-UX, they depend on the path to the disk. On the other hand, /dev/disk/diskX special device depend on the disk itself. As long as you do not change WWID of disk, this device file does not change even if you change the hardware path of the disk (in HP-UX 11.31 at least). However, it is a good practice to use LVM in HP-UX because it isolate you from hardware details, you can even migrate disks physically and keep the same fstab, because the LVM device files depend on the metadata, not the hardware paths.

Raspberry supernintendo emulation, working recipe

After messing around with libretro-snes9x and not getting video through hdmi at all, I tried to compile retroarch and pocketsnes from scratch, ending with a working configuration.

Get retroarch:

$ git clone https://github.com/Themaister/RetroArch.git

Compile:

$ ./configure
$ make

If you get errors like these ones:

CC gfx/vg.c
gfx/vg.c: In function ‘vg_init’:
gfx/vg.c:102:4: error: too many arguments to function ‘vg->driver->update_window_title’
gfx/vg.c: In function ‘vg_frame’:
gfx/vg.c:390:4: error: too many arguments to function ‘vg->driver->update_window_title’

Just edit gfx/vg.c and change update_window_title(true/false) with update_window_title() at those lines mentioned above.

after that:

$ sudo make install

Compiled retroarch is at /usr/local/bin/retroarch

Get pocketsnes:

$ git clone https://github.com/ToadKing/pocketsnes-libretro.git

Compile:

$ make

This Makefile does not have an “install” target, so you have to copy library by hand:

$ sudo cp libretro.so /usr/lib/retroarch-pocketsnes.so

Configure RCB so this binaries are called, for instance:
config.xml

...

/usr/local/bin/retroarch
"%ROM%" -f -c ~/.config/retroarch/retroarch-snes.cfg
...

retroarch-snes.cfg

video_driver = "gl"
libretro_path = "/usr/lib/libretro-pocketsnes.so"
video_fullscreen = true
audio_enable = true
audio_out_rate = 48000
audio_driver = sdl
...

I’ll try with different drivers, but this configuration does work. Sound is not very good, but I’m affraid that, unless I overclock my RPi I will not get much better results. And I want my little berry to last very long…

Raspbian + XBMC + retroarch : Gamepad does not work!

I did buy a new brand gamepad for 21€ in Amazon and connected to my Raspberry Pi. Dmesg didn’t report many details about it, just:

[    3.342426] usb 1-1.2: new full-speed USB device number 4 using dwc_otg
[    3.476789] usb 1-1.2: New USB device found, idVendor=044f, idProduct=b326
[    3.502087] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    3.529629] usb 1-1.2: Product: GPX Gamepad
[    3.551359] usb 1-1.2: Manufacturer: Thrustmaster
[    3.582106] usb 1-1.2: SerialNumber: 00000001

Well, at least it exists for the kernel.

Executing retroarch-joyconfig:

$ sudo retroarch-joyconfig
Cannot find joystick at index #0, only have 0 joystick(s) available ...

Oh, my God, my wife is going to kill me!!!

Don’t panic, maybe I need a driver or something. I had tried an old Joystick before and everything was ok but its lacking of buttons and axises and some calibration problems made me go for a new gamepad.

This is the Joystick:

411evC+bYYL._AA160_This is a GP XID from Thrustmaster, and I had bought it with the promise of being XBox compatible, and I remembered something about XBox gamepads being compatible with retroarch. After some googling I found a driver for this devices:

http://pingus.seul.org/~grumbel/xboxdrv/

Developed by Ingo Ruhnke (thanks Ingo).

Installed with: apt-get install xboxdrv (it’t in raspbian repository, fortunately).

It has an user space part, so you have to run a command:

$ sudo xboxdrv
xboxdrv 0.8.4 - http://pingus.seul.org/~grumbel/xboxdrv/
Copyright © 2008-2011 Ingo Ruhnke <grumbel@gmx.de>
Licensed under GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html&gt;
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it under certain conditions; see the
file COPYING for details.

-- [ ERROR ] ------------------------------------------------------
No Xbox or Xbox360 controller found

Oh, no, I’m lost again!!!!

No, you have to RTFM. The FM is REALLY GOOD for this product, and you find some things like:

–device-by-id : force the driver to drive a device by vendor / productid

–type xbox360 : if you use the above parameter, you have to tell what kind of thing it is

–trigger-as-zaxis: it helped me to map the triggers, it tells the driver to manage the triggers as a Z axis instead of two independent axises.

So, with this command:

$ xboxdrv --silent --device-by-id 044f:b326 --type xbox360 --trigger-as-zaxis
xboxdrv 0.8.4 - http://pingus.seul.org/~grumbel/xboxdrv/
Copyright © 2008-2011 Ingo Ruhnke <grumbel@gmx.de>
Licensed under GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html&gt;
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it under certain conditions; see the
file COPYING for details.

Controller:        unknown
Vendor/Product:    044f:b326
USB Path:          001:004
Controller Type:   Xbox360

Your Xbox/Xbox360 controller should now be available as:
/dev/input/js0
/dev/input/event2

Press Ctrl-c to quit

Voila! It works, and now you can map the buttons and axises with retroarch-joyconfig. In my case, I obtained the following configuration:

# Defines axis threshold. Possible values are [0.0, 1.0]
# input_axis_threshold = 0.5
input_player1_joypad_index = "0"
input_player1_a_btn = "0"
input_player1_b_btn = "1"
input_player1_x_btn = "2"
input_player1_y_btn = "3"
input_player1_l_btn = "4"
input_player1_r_btn = "5"
input_player1_l2_axis = "-2"
input_player1_r2_axis = "+2"
input_player1_l3_axis = "-2"
input_player1_r3_axis = "+2"
input_player1_start_btn = "7"
input_exit_emulator_btn = "6"  # Added my hand, found the button with jset-gtk
input_player1_select_btn = "8"
input_player1_left_btn = "h0left"
input_player1_up_btn = "h0up"
input_player1_right_btn = "h0right"
input_player1_down_btn = "h0down"
input_player1_l_x_plus_axis = "+0"
input_player1_l_y_plus_axis = "+1"
input_player1_l_x_minus_axis = "-0"
input_player1_l_y_minus_axis = "-1"
input_player1_r_x_plus_axis = "+3"
input_player1_r_y_plus_axis = "+4"
input_player1_r_x_minus_axis = "-3"
input_player1_r_y_minus_axis = "-4"

You can experiment with different –type parameters (please, RTFM).

In my case, I added the xboxdrv execution to:

/home/pi/.xbmc/addons/script.games.rom.collection.browser/applaunch.sh

So it gets executed only when I need it. At the end of the script, I kill (pkill) the xboxdrv process.

Good luck with your gamepads!!

Raspberry Pi enclosure with an 8mm videotape case

This is a picture of my Raspberry Pi model B 512MB in an 8mm videotape case:

enclosure with an 8mm videotape case

The finishing is not perfect, I am not very skilled in plastic works. At first, temperature reached 58ºC. After making some holes on top of the SoC, temperature lowered to 50ºC. I suspect that the Lan chip is hotter than the SoC, as you can see in this post:
http://www.zipfelmaus.com/blog/raspberry-pi-thermal-images/

XBMC + ROM Collection Browser + retroarch: sound and joystick calibration

A new modification to applaunch.sh, so you calibrate and map the joystick and force mixer before launching the emulator.
Don’t take the arguments as is, they are just an example. Well, the amixer arguments force audio output to HDMI.


#!/bin/bash

# Check for arguments
if [ -z "$*" ]; then
echo "No arguments provided."
echo "Usage:"
echo "launcher.sh [/path/to/]executable [arguments]"
exit
fi

echo "Trying to stop xbmc "
sudo /etc/init.d/xbmc stop
# Deactivates HDMI output
sudo /opt/vc/bin/tvservice -o
# Activates HDMI output again
sudo /opt/vc/bin/tvservice -p

# Wait for the kill
sleep 1

# Joystick calibration and mapping
jscal -s 3,1,0,82,82,8388352,8134160,1,0,99,99,6882750,8388352,1,2,84,84,7254791,7063875 /dev/input/js0
jscal -u 3,0,1,6,4,288,289,290,291 /dev/input/js0
# Force audio output
amixer cset numid=3 2

echo "$@"

# Launch app - escaped!
"$@"

echo "Emulator terminated"

# Done? Restart XBMC
sudo /etc/init.d/xbmc start

XBMC – ROM Collection Browser – retroarch: problem with screen

When trying to setup ROM Collection Browser2.0.0  plugin in XBMC 2:12.0-1 under Raspbian 7.0, kernel 3.6.11, I had problems related to screen display.

At first, emulator was launched correctly, but xbmc stayed at the top left corner, on top of the game. After searching in forums and trying several things  I tried Solo mode, but the situation was worse, the emulator display didn’t even show up.

Digging in the Python and shell script code, I found this solution: launcher.py, when solo mode is selected, runs applaunch.sh script, but this script tries to kill xbmc.bin process. This does not seem to work, at least in my configuration, XBMC continues being displayed even it retroarch is launched.

My modified applaunch.sh script does a sudo /etc/init.d/xbmc stop (I have set up sudoers file) instead of killing the xbmc.bin process. In the first version this did not remove XBMC last displayed screen, and the emulator didn’t appear on screen. The trick is to deactivate the HDMI output and activate again, like this:

#!/bin/bash

# Check for arguments
if [ -z "$*" ]; then
echo "No arguments provided."
echo "Usage:"
echo "launcher.sh [/path/to/]executable [arguments]"
exit
fi

echo "Trying to stop xbmc "
sudo /etc/init.d/xbmc stop
# Deactivates HDMI output
sudo /opt/vc/bin/tvservice -o
# Activates HDMI output again
sudo /opt/vc/bin/tvservice -p

# Wait for the kill
sleep 1

echo "$@"

# Launch app - escaped!
"$@"

echo "Emulator terminated"

# Done? Restart XBMC
sudo /etc/init.d/xbmc start

This way xbmc disapears and the emulator appears. Not very fast, but it works. If someone finds another better way, I would like to know…

xbmc stops working after raspbian upgrade

The precompiled binary works!

At first the screen appeared zoomed, but I read this post in Michael Gorven page:

11.0~git20121018.ff434fe-1: Upgrading to this version will probably result in the GUI resolution being zoomed as described in this forum post. To fix this edit userdata/guisettings.xml and remove the contents of the <resolutions> element.

Thanks, Michael!