Blog Jovialyteam

Update of the linear algebra libraries in Debian

2010-04-06

In the numerical computing world, the cornerstones libraries are BLAS and LAPACK. They have been used in most of the numerical software for decades (like Scilab, R, numpy, OpenOffice with calc, etc).

During that time, many implementations appeared to improve the performances taking advantages of clusters, multicore, SEE{1,2,3,4}, various levels of cache...
Between the reference BLAS (refblas) to an optimized one like ATLAS or MKL (Math Kernel Library by Intel - non-free), it is not rare to have a 15 factor.

In Debian, we use by default the reference implementation of BLAS (168 reverse dependencies) and LAPACK (178 reverse dependencies). If the results are usually bad, they are pretty easy to use. What is hard to use, is switch between highly optimized libraries.
For now, the main one in the archive is ATLAS. ATLAS build process will launch many computations to know what will work best on the architecture. Results are usually excellent.

1) Upload of a refactoring of the ATLAS package.
I have been working on this for a while and after 19 uploads into Debian Experimental and I am happy (and kind of relief) to upload into debian unstable the release 3.8.3 of ATLAS.

The new key elements in this release are:

Package of the release 3.8.3 ... Long overdue
Much more packages for recent architectures (sse3, core2sse3, etc)
A simplified maintenance
Easy to build a custom package: fakeroot debian/rules custom
Easy upgrade to version 3.9.X when it is stable
12 bugs closed in Debian (including 4 RCs)
6 bugs closed in Launchpad.
MMX optimized package removed

Note that, as before, all prebuilt binaries of ATLAS will be always slower than if you built them on the target architecture (but using Debian binary packages will save a few kilograms of Uranium).

And one of most important feature is the capability to switch to any ATLAS implementation.

2) Switch between the different implementation
The problem in Debian (and Ubuntu) was that it was hard to switch between the ref BLAS/LAPACK and the optimized libraries. The user has to play with the LD_LIBRARY_PATH to use the various optimized packages and since there is no convention between the various distribution, the upstream developer has to develop crappy tricks to handle such things.

It is why I implemented the following proposal: Handle different versions of BLAS and LAPACK.

The main idea is to use the update-alternatives system to allow a quick and easy switch. For example:

# update-alternatives --config libblas.so.3gf 
There are 3 choices for the alternative libblas.so.3gf (providing /usr/lib/libblas.so.3gf).

  Selection    Path                                           Priority   Status
------------------------------------------------------------
* 0            /usr/lib/atlas-core2sse3/atlas/libblas.so.3gf   55        auto mode
  1            /usr/lib/atlas-base/atlas/libblas.so.3gf        35        manual mode
  2            /usr/lib/atlas-core2sse3/atlas/libblas.so.3gf   55        manual mode
  3            /usr/lib/libblas/libblas.so.3gf                 10        manual mode

# update-alternatives --config liblapack.so.3gf
There are 3 choices for the alternative liblapack.so.3gf (providing /usr/lib/liblapack.so.3gf).

  Selection    Path                                             Priority   Status
------------------------------------------------------------
* 0            /usr/lib/atlas-core2sse3/atlas/liblapack.so.3gf   55        auto mode
  1            /usr/lib/atlas-base/atlas/liblapack.so.3gf        35        manual mode
  2            /usr/lib/atlas-core2sse3/atlas/liblapack.so.3gf   55        manual mode
  3            /usr/lib/lapack/liblapack.so.3gf                  10        manual mode

Thanks to this, it is just trivial to switch from one to the other...

Conclusion:
I just pushed the changes into Debian unstable for blas, lapack and atlas.
I have been testing a lot these deep modifications and I fixed all the problems that I found. However, in case I missed something, please report a bug...

Posted by Sylvestre at 23:23:00 into the following categories: Linux, Scilab, Debian

3 comments | Permalink

A few news around Scilab (packaging & other stuff)

30.12.08

Here is a quick list of new things around Scilab (note that it is slightly modified message of the one I sent on the dev mailing list).

Sagemath - it is a software which combines the power of various opensource software.
A Experimental "Scilab/Sage package" is planed for Sage 3.4 and an experimental package by Jaap Spies is already available
Debian/Ubuntu
Packages are available on my Scilab homepage
Debian packages are up-to-date (5.0.3-2). I will to upload the new Ubuntu's packages in 2009 (for now, it is 5.0.3-1 which is working too). I might backport them to Debian Lenny (future stable) & Ubuntu Hardy.
Mandriva
Tomasz Pawel Gajc (a regular Mandriva contributor) created a package available on zarb.org
Opensuse
A Scilab package for Opensuse has been created by Andrea Florio.
It is available on packman
and should be included in the next version of opensuse.
Note that Mandriva & Opensuse packages have been created for Scilab 5.0.3 and I applied most of their patches (or update some part of the code) for Scilab 5.0.4.
Redhat/Fedora
The work is still going on.
They are also doing a great work packaging the misc dependencies but they are a bit stuck about the JOGL packaging (jogl and glugen should produce two different packages ... which I should also do in Debian/Ubuntu too)
Arch Linux
It is also available under Arch Linux by Simon Lipp (one of our former trainee).
Gentoo
A bit stuck for now but some activities have been seen lately around on jrosetta (one of the dependency introduced by Scilab 5).
Slackware
Scilab has been packaged by the Italian Slackware community. It is available on their website. I don't know if it is going to be included in Slackware by default or not;

Posted by Sylvestre at 19:42:10 into the following categories: Linux, Scilab, Debian

| Permalink

-bash: /bin/ls: No such file or directory

19/09/06

If you have done a mistake/lost the /lib/ directory. Example :

mv lib lib.

And you are stuck without anything working :

# ls
-bash: /bin/ls: No such file or directory

Don't waste your time in a reinstallation, there is a trick :

/lib./ld-linux.so.2 --library-path /bin/ mv /lib. /lib/

It invokes the program loader directly and the problem will be fixed (of course, you can use this command to call many programs).

Posted by Sylvestre LEDRU at 07:11:11 pm into the following categories: Administration, Linux

1 comment | Permalink

Ma vie (piouf)

25.05.06

De retour en France début avril, ces deux derniers mois ont été consacrés à la recherche d'un boulot et d'un appartement.
J'ai maintenant les deux et qui ont tout pour me plaire.
Niveau de l'appartement, je me suis installé avec mon amie dans un trois pièces situé à Paris (20 ème) près de la place de Nation. Super appartement, content et bien chez nous dedans. Bon, le seul problème est que l'on a pas encore l'ADSL... Donc je squate la connexion chez Pimentech.

Pour le boulot, après de longues recherches (cf message précédent), j'ai finalement trouvé exactement le boulot que je recherchais. Je vais intégrer le prestigieux institut qu'est l'INRIA - Institut National de Recherche en Informatique et Automate (pour ceux qui ne connaissent pas, c'est comme le CNRS mais appliqué à l'informatique). Régulièrement classé dans le top 3 de 01 informatique des entreprises préférées des informaticiens.
Je vais être amené à travailler sur le projet Scilab, logiciel de calcul scientifique. Mon travail consistera principalement en les développements multi plates-formes en vue de la réalisation des fonctionnalités de Scilab 5 et la responsabilité des versions UNIX et GNU/Linux du logiciel Scilab. Je sens que je vais bien m'amuser !

Ce fut un retour et une réinstallation en France un peu plus longue que je ne l'avais imaginé mais qu'il est au final bien passé !

Posted by Sylvestre LEDRU at 14:44:55 into the following categories: Perso, Développement, Linux, Travail

2 commentaires | Permalink

OpenCascade : How to make money with free software ?

11.04.06

I wrote this article 6 mois ago. I was waiting for some stuffs around this. However, it is too late so I release it.

My main concern about free software as a way of doing business is "how do you make money out of it ?".
Unfortunatelly, I still don't see any solution ... Of course, it is possible by selling services, maintenance, help, more (costly) advanced version, side development (usually proprietary) but it is not doing money only on free software/development.

Let's me describe on of the solution that uses Opencascade, a former department of Matra.

They produces for a quite long time (about 10 years) a big project called "Open CASCADE". It is a framework used to modelise, visualise 3D models (and many other things that I hardly understand)... It has an aspect of gas factory ("Usine à gaz" as we say in French) because the size/age/complexity of the project. However, it is a very convenient way of doing 3D (many famous corporations use it).

At work, I am working on a project launched by the BRGM (Bureau de recherche géologique et minière ... the equivalent of the french CNRS but for earth sciences) about 10 years ago. The aim of the project is to provide to geologists and geophysists a way of representing the underground. The corporation where I work is doing the work of adding new features, packaging and selling the software. For this project, we use Opencascade.

My work on this project was to provide a visualisation of drillholes in a 3D space (holes that are dug in the ground to get the geological structure of the underground). Basically, I was supposed to represent a (drunk) worm hole from a serie of points.
I tried many way (from the ugliest to the sharpest) but I found some problems with my favorite. When I used too much points, the framework start to reject me (technical description is following my message). As I don't like when a software rejects me, I spent time one this issue (I thought that it was my fault because it is a quite basic feature in the software). I finally successed to isolate the issue and send a kind of SOS/bug report to someone at OpenCascade.
This request cost us 2 units of support from our contract.

Then, one week after, they get back to us with a "There is a bug in Pipe algorithm.". OK, cool. I was a bit proud to find a bug in this kind of big software (for the courageus/crazy, the reason of the bug is : "This bug is caused by complex structure of the curve approximated from 14 (15) points (some discontinuities of derivatives of high levels). Such structure of the curve causes turbulence of the local coordinate system of section along the curve. That is why the algorithm can not build a pipe where the section is orthogonal to the curve in each point all along the curve.", obvious no ?). However, I was expecting a small patch which will fix the issue and a "thank for the bug report". Instead of this, I saw :

The bug fix production can be started after your confirmation.
Please note that the standard price of a query is 15 units of your support program.

OK, now, I know how they make cash (at least a part of it).
You submit what you think to be a bug, it costs you 2 units. They confirm that is a bug, you have to pay 15 units to get the fix. Otherwise, you have to wait for the new release planned whenitwillbeready.
I am not saying it is a bad solution (and I do understand why they do that ... it is called support) but it could be more respectful for the user who spent many time to isolate a bug and report it and consequently helped to improve their software...
(For their defense, they provided me a workaround for my issue).

Lire la suite!

Posted by Sylvestre LEDRU at 09:00:00 into the following categories: Développement, Linux

| Permalink

:: Page suivante >>