Sunday, May 29, 2011

Xvfb+x11vnc outperforms Xvnc

(update 2011/08/29: NX offers a much better aproach to remote X control).

If anyone of you has ever tried to run a remote X desktop using Xvnc you will probably have got quite dissapointed. The performance is poor and the response to user accions is annoying. Do not despair. There is an much better alternative. Just combine Xvfb (the "virtual in memory" X server) plus x11vnc (the vnc that attach to an existing X server). Extracted from the wikipedia.


export DISPLAY=:1
Xvfb :1 -screen 0 1024x768x16 &
fluxbox &
x11vnc -display :1 -bg -nopw -listen localhost -xkb


(Another probe in favor of the "Do one thing and do it right!" Unix philosophy)

Sunday, May 15, 2011

KILL THE NULL, KILL'EN ALL

The magical "NULL" pointer has been with us for many years (decades actually). Some developers will argue that null values are useful and even needed when writing software. Nulls allows to declare a variable without a defined value and that's something many people will think is needed when we don't known in advance the real value of an entity.

Imagine for example a database whose squema is upgraded adding a new column "second name" to a table. Since we don't know in advance what value "second name" has to be for each row, and since many new rows possible will have not "second name" it's quite easy to let it default to NULL.

In fact, I will try to show that using NULL values is always a bad idea. The first trouble many people will have trying to get rid of nulls is solving the problem in the previous paragraph. If I don't know the value of a variable of if a variable has not value at all (maybe because performance reasons force to denormalize a database), there is no other way that assigning NULL.

Right? wrong. It's always possible to assign a default value to tag a field as NULL. In the previous example we could simple assign the string "NULL" to indicate a non existent second name. It looks an arbitrary decisition that doesn't improve our code quality at all and we are wasting extra bytes. In fact it does make our application safer. Let's see why.
Imagine a typicall web app in which a form sends the "second name" to our server in a get/post request. Something like second_name=Armstrong.
Then our php server code will look something like:

$second_name = $_REQUEST["second_name"];

Look now what happens if accidentally we misspell second_name in the html form or the php code:
$second_name will be assigned a NULL value. Not the "NULL" string we want in case "second_name" is not used.

The previous example is just one of the many possible ways in which a variable can be silently set a wrong NULL value. As you can see NULL values are quite risky, since languages tend to leave variables in a "null" state when something goes wrong. For example Java, probably the most widely used programming language around has a big design mistake:
try-catch exception demarcation bounds are used also as visibility demarcation bounds. That means that we can not write a code like:

   try{
MyClass myObject = new MyClass();
...
   } catch(Exception e){
...
   }
myPreparedStatement.set(0,myObject);
myPreparedStatement.execute();

The java language syntax force to do something similar to:

MyClass myObject = null;
   try{
myObject = new MyClass();
...
   } catch(Exception e){
...
   }
...
myPreparedStatement.set(0,myObject);
myPreparedStatement.execute();

As we can see in the previous code, If an exception is thrown at object initialization and we are not cautious enough, we will insert a false wrong NULL into the database.

(Note: In the previous example it's possible to move the two last lines inside the try-catch and that will avoid propagating a null to the database. In a larger code it could be impossible and even if it could, it's up to the user writing the code to do so and we are searching a way of writting code that doesn't depends on programmer's skills and experience).

With C#, the situation is even worse, since C# designers were drunk enough to not only copy the same Java pitfalls, but to encourage the use of nulls "everywhere"

In fact, NULL values are the biggest mistake ever made in software history, and as the time pass, the dammage augment.

The problem with NULL is that NULL actually is/means/maps to *nothing*. Null is not "unknown", Null is not "not applicable", not even "not available", not even "right" o "wrong". Null bears no information at all. If we were physicists NULL will just compare to a singularity in our ecuations, a clear symptom that something is really, really wrong.

In the example of the database we can assign the string "NULL" to indicate an unknown value. In fact we can assign the string "Not known" to indicate we don't known it's value, even if it exist, and we can also assign the string "Not applicable" when the row, due to break of normalization doesn't use that value. Our code will now be able to compare the string and apply a bussiness logic or another since now it carries information and state.

Convention let us replace "null" values with a given value or even better, with a given set of values ("not known","not aplicable","" (empty),...) . That's probably what we really want. That means a little bit of extra coding since we will need to create default rows/objects for each entity in our UML diagram, but the extra work is always worth the effort when our code grows beyond the thousand lines of code. Magically null pointers will vanish. Errors will be detected much early and correct exceptions thrown properly.

And last but no least, here comes a trick to avoid NULLable values in a database in a safe way:

  • Imagine you have a set or "core" tables in your squema, for example "client" and "product".

  • Imagine now that due to performance issues or whatever the reason you have another not-normalized table "notNormalizedTable" with a foreing key column "fk_client_id" and "fk_product_id" that many times doesn't have a given set value either for client or product or both.

  • If you want to avoid the risky NULLs the first step is to create default client and product rows. For example:

    client:
    =======
    id | name
    01 | "Not Known"
    02 | "Not Applicable"
    ...
    product:
    =======
    id | name
    01 | "Not Known"
    02 | "Not Applicable"
    ...


  • A new problem arises now. It's possible to accidentally delete either client or product "meta-rows" that represent "Not applicable"/"Not known"/... values, and then our strategy for NULL replacement will go to an end.
    Noneless, the solution is quite easy. Just create a new meta-table "nullKiller" with a foreign key for each table for which we want "meta-rows" with the "ON DELETE NO ACTION ON CASCADE NO ACTION" clause like:

    create table `nullKiller`(
    fk_client_id
    fk_product_id
    ...
    fk_entityN_id
    CONSTRAINT `nullKiller_const1` \
      FOREIGN KEY (`fk_client_id`) \
      REFERENCES `client` (`id`) \
      ON DELETE NO ACTION ON UPDATE NO ACTION,
    CONSTRAINT `nullKiller_const2` \
      FOREIGN KEY (`fk_product_id`) \
      REFERENCES `product` (`id`) \
      ON DELETE NO ACTION ON UPDATE NO ACTION,
    ...
    CONSTRAINT `nullKiller_constN` \
      FOREIGN KEY (`fk_entityN_id`) \
      REFERENCES `entityN` (`id`) \
      ON DELETE NO ACTION ON UPDATE NO ACTION
    );

    then just insert the new rows:

    INSERT INTO `nullKiller` \
      (fk_client_id,fk_product_id,...,fk_entity_N_id) \
      VALUES (1,1,....1), (2,2,....2);


    Once the `nullKiller` rows are inserted we are sure that a reserved set of ids for each entity are reserved for default values thanks to the "ON UPDATE NO ACTION" clause, and we are also sure those rows will not be accidentally deleted thanks the "ON DELETE NO ACTION" clause.


So, you know what to do now: "KILL THE NULL, KILL'EN ALL"

Tuesday, May 3, 2011

Monitoring uninterruptible system calls.

Here comes a "keep it simple and stupid" one-line script to watch for any process that could have trouble beeing blocked by an uninterruptible system call.

#$ while true; do ps -e fo stat,cmd | grep ^D && echo "-------"; sleep 0.1 ; done

D< \_ [scsi_eh_0]
D \_ hald-addon-storage: polling /dev/sr0 (every 2 sec)
-------
D \_ hald-addon-storage: polling /dev/sr0 (every 2 sec)
-------
D< \_ [md0_raid1]
D< \_ [kjournald]
-------
D< \_ [md0_raid1]
-------
D< \_ [md0_raid1]
-------
D< \_ [md0_raid1]
-------
D< \_ [md0_raid1]
-------
D< \_ [md0_raid1]
Ds /sbin/syslogd
-------
D< \_ [md0_raid1]


The output in this case is normal. [md0_raid1],kjournald and syslogd are making frequent disk writes.

Wednesday, April 20, 2011

Wait, wait!!

After years of shell scripting I wasn't aware of the useful "wait" command. It makes the parent process wait until all child processes finish. I mean:


#!/bin/sh

(sleep 5; echo "good bye from child 1") &
(sleep 10; echo "good bye from child 2") &

wait
echo "good bye from parent process"


When executed, this script will output:

good bye from child 1
good bye from child 2
good bye from parent process

Arggg, so many lost hours writing custom code to do what was only one line away.

Monday, March 28, 2011

JQTJSF: Java-quick-thread-job-splitting/forking

(Sorry about the stupid title but "laws of nature" force me to prefix anything related to Java with an annoying, uggly, empty of content and counter-intuitive acronym).

Reading about closures I found the next quite useful paragraphs (Copy&Paste from the Wikipedia):

Java allows defining "anonymous classes" inside a method; an anonymous class may refer to names in lexically enclosing classes, or read-only variables (marked as final) in the lexically enclosing method.
class CalculationWindow extends JFrame {
     private volatile int result;
     ...
     public void calculateInSeparateThread(final URI uri) {
         // The expression "new Runnable() { ... }" is an anonymous class.
         new Thread(
             new Runnable() {
                 void run() {
                     // It can read final local variables:
                     calculate(uri);
                     // It can access private fields of the enclosing class:
                     result = result + 10;
                 }
             }
         ).start();
    }
}


Some features of full closures can be emulated by using a final reference to a mutable container, for example, a single-element array. The inner class will not be able to change the value of the container reference itself, but it will be able to change the contents of the container.

According to a Java 7 proposal[11], closures will allow the above code to be executed as:
class CalculationWindow extends JFrame {
    private volatile int result;
    ...
    public void calculateInSeparateThread(final URI uri) {
        // the code #(){ /* code */ } is a closure
        new Thread(#(){
            calculate( uri );
            result = result + 10;
        }).start();
    }
}
Update 2018-01-10: Many time has past since the original post. Time has changed,  async code is every where, NodeJS is fighting the Java throne, Promises rocks, so do Futures and CompletableFuture is not yet the future but the present (or is it that maybe RXJava has put the Future into the past?). Ah, do not forget the lovely lambdas.... Cool things of Java 8 now that Java 9 is officially out in the wild !!!
Time to make our code prettier and younger:

class CalculationWindow extends JFrame {
    // ...
    public CompletableFuture calculateInSeparateThread(final URI uri) () {
        final CompletableFuture result = new CompletableFuture();
        new Thread(() -> {
              try{
          calculate(uri);Ç
          result = result + 10;
          result.complete(result);
              }catch(AnnoyingCheckedException e) {
          result.completeExceptionally(
              new RuntimeException("Not implemented"));
              }
        }
        }).start();
        return result;
    }
To be continued ...

Friday, March 18, 2011

Limiting Flash CPU usage in Linux (Firefox only)

Firefox makes it easy to enable/disable plugins like Flash.

The problem is that there are many webs out there. Some of them use Flash legitimately and with good intentions, while some others try to invade us with ads banners without taking a minimal effort to keep resource usage low. Switching on/off the flash plugin is not always an option or at least not a confortable one. Ideally it would be great to have Flash running while simultaneously get certain that it will not peak CPU usage slowing down web navigation.

Newer versions of Firefox use an external process named "plugin-container" to execute external plugins like Flash. Fortunately that makes it quite easy to control the maximum CPU percentage we want to assign to Flash. All we need is to follow the next 2 steps:

- Install the command line utility "cpulimit". How to do it must be straightforward if its already in our favority linux distro repository.

- Edit (as root) /etc/rc.local and add a line similar to:

cpulimit -l 20 -e plugin-container &

Once we reboot our system 'cpulimit' will take care that external firefox plugins (aka flash) don't get more than 20% of our CPU. That can be too slow for people who want to check videos on youtube, so they will probably want to use a higher percentage. 20% is just a personal option. (Don't forget also that VLC allows to play youtube videos and it performs much better than the Flash plugin: http://www.google.com/search?q=vlc+youtube).

In my particular system, a "not so 3-years-old" laptop, I'm used to set cpu frequency to the minimum available clock, 800Mhz. That let's me use the laptop as a real LAPtop, I mean without my laps being burn by the heat dissipated by the CPU. Before using cpulimit I had no other option that turning off the plugin flash since randomly some "aggresive" flash banner knocked down my web browser. After cpulimiting the plugin those banners will need to manage themselves how to share the 20% cpu left for them. Not my problem anymore.

What about Chrome? I think, for what I have read about, that Google Chrome is ussing some type of plugin container too, but I found no easy way to control it. "top" just show a single process for the browser and the plugin so I didn't find any easy way to let cpulimit control it. Any hint/idea is wellcome!

Friday, March 11, 2011

God save the VIM!: My personal .vimrc

set number            <-- A must 
set ignorecase        <-- quite useful
set encoding=utf-8
set tabstop=4          
set shiftwidth=4      <-- 8 is too much and annoying
set expandtab         <-- replace tabs with spaces. That makes layout independent of tab's setup.
set foldmethod=indent <-- Ummm, depends on taste. Automatic folding based on indentation
Once a file is open it may be better to change it to foldmethod=manul
set cursorline        <-- Visual help (cursorcolumn is useful sometimes) 
syntax off            <-- Syntax highlighting never works to me.
set cindent           <-- C indent rules                                           
set smartindent       <-- Make vim smart about indentation
set autoindent        <-- Enable autoindent 
 
" associate file extension (lp) with syntax (lua)
au BufNewFile,BufRead *.lp setlocal ft=lua
 
" shorcut for if-then-else in C-style languages
ab ite 
\if () {
\
\} else if () {
\
\} else if () {
\
\} else {
\
\} end  

 
" shorcut for if-then-else in lua
ab itel 
\if ()  then -- {
\
\elseif () then -- } { 
\
\elseif () then -- } { 
\
\else -- } { 
\
\end -- }
 

" 
fu! SaveSess()
    execute '!mkdir .vim'
    execute 'mksession! .vim/session.vim'
endfunction

fu! RestoreSess()
execute 'so .vim/session.vim'
if bufexists(1)
    for l in range(1, bufnr('$'))
        if bufwinnr(l) == -1
            exec 'sbuffer ' . l
        endif
    endfor
endif
endfunction

autocmd VimLeave * call SaveSess()
autocmd VimEnter * call RestoreSess()
 
" indentation rules for FFmpeg
set shiftwidth=4
set softtabstop=4
"set cindent         < -- useful for C alike languages
"set cinoptions=(0   < -- useful for C alike languages

autocmd FileType make,automake set noexpandtab shiftwidth=8 softtabstop=8 < -- " Allow tabs in Makefiles.
highlight ForbiddenWhitespace ctermbg=red guibg=red < -- hihglight forbidden Trailing whitespace and tabs
match ForbiddenWhitespace /\s\+$\|\t/
autocmd InsertEnter * match ForbiddenWhitespace /\t\|\s\+\%#\@<!$/ < -- Do not highlight spaces at the end of line while typing on that line.
 
 
Update 2012-04: The RestoreSess/SaveSess (Grotty copy&paste from http://stackoverflow.com/questions/5142099/auto-save-vim-session-on-quit-and-auto-reload-session-on-start) is a really great addition. It saves and automatically restore all the vim session (open file, tabs, window layouts,...). One more reason to say Eclipse "Good Bye!".

Update 2014-05: Unashamedly copied Makefile related stuff from ffmpeg documentation project.

Update 2014-11:  associating file extensions with language syntax and shorcuts for if-then-else.

External Links: