Chuletario de pócimas y recetas

The circle of observability

2012-01-07T07:11:00.000-08:00

I though RAM didn't waste energy ....

2011-11-23T14:31:00.000-08:00

... I was so wrong!!!

How to improve disk I/O

2011-08-24T06:46:00.000-07:00

Someone called "Unnikrishnan P." from India posted the next question in the Linked-In Linux Group:

how to improve disk I/O

I have a server with 128GB RAM and I cannot find the swap memory (120G) being used even when the disk utilization is at %util =95% (using iostat -d -x 5 3). What might be the root cause? What can be done to make the server perform much better?

As is ussually the case, the thread of answers started to grew in time. Some of them were really interesting so I decided to post here a summarized view of them extracting the most interesting answers and avoiding repeated comments:

Phil Quiney:

The bottleneck is going to be disk I/O performance: Offloading disk operations onto dedicated hardware (RAID controllers - proper ones, not the ones fitted to motherboards) will speed things up. Sun were doing this years ago with SCSI drives as they had enough intelligence to do disk to disk transfers without going via the host, IDE was/is too dumb to do that.

Jim Parks:

With a minimum of four spindles, you could do a RAID 10 setup. RAID 10 is a non-standard RAID level, but it provides both redundancy and higher performance, as it uses multiple striped volumes in a mirrored configuration.

From around 12 spindles on up, you're probably better off with a RAID 6 (striping with double-distributed parity) setup. There will almost certainly be people who want to argue with me about this. They have their ideas, I have mine, and I'm not going to engage in a drawn out discussion about it, but the short version is that the more spindles you have, the more RAID 6 will make sense. RAID 6 will allow you to get more usable space than RAID 10. For example, a RAID 10 array comprised of 16 1TB drives will provide 8 TB of usable space. The same 16 1 TB drives in a RAID 6 configuration would provide 14 TB of usable space -- that's 75% more useful diskspace, with the ability to survive the loss of two drives without data loss.

Micheas Herman:

If /tmp is mounted to tmpfs then the large swap space may be reasonable, but otherwise you might try turning the swapoff temporarily to see if that doesn't reduce the diskio.
If you do have millions of session cookies in /tmp and you are using tmpfs moving the swap to a solid state drive might help a lot.

If you are trying to get the swap space to be used more aggressively you can check what the swappiness of your system is

by running: cat /proc/sys/vm/swappiness

Which will return a number from 0 to 100

If the number is less than 100 and you want to increase the swap usage as much as possible you can run the following:

echo 100 > /proc/sys/vm/swappiness

To make the change persistent across reboots run:

echo "vm/swappiness=100" >> /etc/sysctl.conf

If you want to reduce the swap usage to the absolute minimum do the same using 0 instead of 100.

If you are having a database with a lot of commits disk tuning may not be the best way to improve performance.

Removing unnecessary indexes can greatly reduce the amount of disk access. using transactions to increase the quantity of data written at one time can also improve performance. It can also make it much worse.

If the server is a mail server there tends to be a lot that can be done but there tends to be very little general advice other than to make sure that the disk cache is not being flushed after every mail received.

David Pye

I don't see any information regarding hardware configuration here, have I missed something???

Is the issue really disk utilization - have you investigated the system as a whole?

If disk I/O is suspect, then there are a number of things which could be addressed, correct RAID config is one, no. of I/O channels is another - how many, fibre / copper, etc, can you get FC direct to the disk?, fastest possible disk hardware?, more extreme areas could be to use only a small % of outermost disk surface via partitioning in order to achieve less head latency, examine the nature of the disk activity,

James Sutherland:

Also check the filesystem in use - given where this question is posted, probably something like ext3? If so, check the journalling mode: data=ordered, data=journaled and data=writeback all perform different in different situations.

John Lauro:

With 128GB of RAM you are probably better off seeing if you can move some of that I/O to a tmpfs RAM disk, often mounted under /dev/shm for RH/Centos/SL.
...
If you don't need to record when a file is accessed (many people don't even know you can get that info), you can mount the filesystem with noatime. That can make a big difference on the I/O utilization especially for a web server that has lots of small files all over the place.

If some of your I/O is being generated by mysql or other database, perhaps there is some temporary tables such as for session tracking that you might want to consider converting to memory tables. The rows will be emptied if you have to restart the database, but generally that just means the user has to relogin.

Enrique Arizon Benito (aka, that's me!):

http://en.wikipedia.org/wiki/Hdparm
man hdparm

Also, maybe some app has "too much" log output enabled and it is just a matter of modifying the /etc/... config file. Inotify can be your friend in this case:

* http://linux.die.net/man/7/inotify

Or you can simple use something like:

find /var -mmin -1 -size +0.5G
(find files 'inside' /var changed/modified in the last minute that are bigger than 0.5Gbytes)

Bond Masuda:

first of all, you need to figure out what's consuming the I/O? get a tool like iotop to help:

http://guichaz.free.fr/iotop/
...
I usually use ballpark figures like assuming most modern spinning hard drives can do about 100MB/sec sequential I/O and about 40-60MB/sec random I/O for SATA type drives and add about 10-20% higher numbers for SAS type drives. SSDs of course can have much higher numbers. using those as "guidelines", try to figure out what disk configuration will meet your requirements and add about 20% for overhead. for example, if i have a 8 disk RAID-5 array, which gives me 7 effective spindles, I would assume this can roughly meet a sequential I/O requirement of about 583MB/sec (7x100MB/sec / 1.2 = 583).

with multi-disk arrays, you'll want to consider the "best on average" stripe size for your array... if your I/O pattern is reading sequentially large files, then a larger stripe size is in your favor. if you're doing random I/O and reading small chunks here and there, then a moderate to smaller stripe size a better choice.

also, in situations where you are looking are really large arrays that might span multiple controllers, you'll need to consider the bottlenecks of the bus. older PCI-X buses are half-duplex and limited to about 1GB/sec. more modern servers (like yours with 128GB of RAM) will likely use PCI-E 4x or 8x buses which can do full-duplex and 2GB/sec (8x).

Rob Strong: not attempting to read all these comments, but take a look at the output of iostat "%iowait" field. If this number starts getting up, say past... 20% or more then you possibly have an issue with slow disk and/or bad raid setup.

Oleksiy Dovzhanytsya:

...
http://www.xaprb.com/blog/2009/08/23/how-to-find-per-process-io-statistics-on-linux/

Unnikrishnan P. (original author of the question) added some (important) details to its setup:

This is an NFS server which is used by other servers which runs tools used by the developers. I had moved some users to a dedicated server now and found that performance on this server has increased. Previously there were 4 servers accessing this NFS share. Alot of read/ write happening. I was using RAID 5 (LSI controller) on this server. May be the throughput of RAID 5 is the problem, as that wont support heavy rw operations.

James Sutherland:

That information helps - yes, RAID5 can be a bottleneck with heavy write activity, particularly scattered writes, since it has to read the corresponding blocks from the other disks to recalculate parity every time.

What filesystem is it running, ext3? You may find changing the data journalling, or switching to a separate journal device, will help here: mount with data=writeback (risky, if you don't have a very good UPS!) or data=journal. Also see if your LSI card is doing write back caching or not, that can help significantly.

John Lauro:

What IO scheduler are you running on your disks? If you have at least 128mb on your controller make sure you don't use the often default of cfq as the IO schedulerer as it is the worst with a raid controller...

for a in /sys/block/*/queue/scheduler ; do echo $a : `<$a` ; done

Unnikrishnan P, adding some more info:

I have a server with 128GB RAM and I cannot find the swap memory (120G) being used even when the disk utilization is at %util =95% (using iostat -d -x 5 3). What might be the root cause? What can be done to make the server perform much better?

Jeremy Page:

I'm reasonably certain relatime is now the default for most mount commands.

Cat /proc/mounts and see what options your clients are. I'm using squeeze so probably not a good test but (edited out my personal stuff)
home.site.company.com:/home/pagej /home/pagej nfs rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.28.17.214,mountvers=3,mountport=4046,mountproto=udp,local_lock=none,addr=10.48.33.214 0 0

John Lauro:

reltime is standard on EL6 and nearly as good as noatime for reducing I/O, but it is not the default for EL5 which still common enough that I wouldn't assume someone isn't running it

Wayne Monfries:

As mentioned above by Bond Masuda, you may obtain some easy gains by increasing the read ahead buffer size (probably most helpful for sequential loads)
blockdev --setra 2048 /dev/sda # increase readahead buffer size
Another area that is often overlooked with external raid arrays is that they are capable of processing much greater numbers of IO transactions than a single disk - so check out the qdepth on your adapters and devices.

Somo tips to make XML and HTML+JS files easier to read and mantain

2011-08-22T04:53:00.000-07:00

Here comes a couple of tips/conventions to easify xml edition. Anyone who has used Ant overengineered parametrization config files and then changed to the beautiful Maven default-by-convention config files will be aware by now how useful conventions are. For the rest of you, believe me, conventions are the best friend of anyone writting XML files.

Convention 1: Use upper case for ids global/shared constants. Those constants that are normally defined first in file and used then "everywhere". That's and standard convention in most programming languages.

Convention 2: (This is the important one). XML tags further referenced by other XML tags are usually identified by an id="arbitrary_name" attribute. This arbitrary name is later used to refer to the tag element. This second convention tell us to use the next rule for the arbitrary name:
Prefix it with "type_" where "type_" ussually equals to the tag name that "owns" the id attribute or a real type if the tag name is not descriptive enough / too generic.

Behind is shown two (Ant) XML files before and after applying conventions:

BEFORE APPLYING CONVENTIONS        ||   AFTER APPLYING CONVENTIONS                                

                                     ||                                                             

<project name=....>                  || <project name=....>

...                                  || ...

<property name="src.java.dir"        || <property name="DIR_JAVA_SRC"

   value="src" />                    ||    value="src" />

<property name="lib.dir"             || <property name="DIR_LIB"

   value="../../../lib" />           ||    value="../../../lib" />

<property name="build.dir"           || <property name="DIR_BUILD"

   value="bin" />                    ||    value="bin" />

                                     || 

<path id="project.classpath">        || <path id="PATH_PROJECT">

  <fileset dir="${lib.dir}">         ||   <fileset dir="${DIR_LIB}">

    <include name="**/*.jar" />      ||       <include name="**/*.jar" />

  </fileset>                         ||   </fileset>

</path>                              || </path>

                                     || 

<patternset id="conf">               || <patternset id="PATTERNSET_CONF">

  <include name="**/*.xml" />        ||   <include name="**/*.xml" />

  <include name="**/*.properties" /> ||   <include name="**/*.properties" />

  <include name="**/*.conf" />       || </patternset>

</patternset>                        ||

                                     || <patternset id="PATTERNSET_IMAGES">

<patternset id="images">             ||   <include name="**/*.png" />

  <include name="**/*.png" />        ||   <include name="**/*.jpg" />

  <include name="**/*.jpg" />        ||   <include name="**/*.gif" />

  <include name="**/*.gif" />        ||   <include name="**/*.gif" />

</patternset>                        || </patternset>

...                                  || 

<target name="copyconf">             || <target name="copyconf">

  <mkdir dir="${build.dir}" />       ||   <mkdir dir="${DIR_BUILD}" />

  <copy todir="${build.dir}">        ||   <copy todir="${DIR_BUILD}">

    <fileset dir="${src.java.dir}">  ||     <fileset dir="${DIR_JAVA_SRC}">

      <patternset refid="conf" />    ||       <patternset refid="PATTERNSET_CONF" />

    </fileset>                       ||     </fileset>

  </copy>                            ||   </copy>

</target>                            || </target>

                                     || 

</project>                           || </project>

The first great advantage of using the second convention is that now we can use word/code completion in our favourite text editor.

Notice the next points in the previous example:

- Once conventions are applied, if we want to edit a dir value we just write:

<tagName dir="{$DIR_

and using word_completion in our editor (for example Ctrl+X n in vim) will show a list with the avaible dirs constants defined.

Notice that <property name="arbitrary_name" ...> is a little bit assimetric (Property is actually used as a meta-tag to define constants string values for later replacement in our XML text file). It doesn't follow the convention:

<tagName id="arbitrary_name" ...>

using "name", not "id" for the tag attribute. Also we can't use the tag name, "property" as the prefix since it itself doesn't provide the type of the object (it's always an string type value). We will use something like "DIR_" as in the previous example to indicate that the property value is a directory. Other useful and descriptive prefixes could be "PATH_", "CLASSPATH_", "URL_", ... where the "PREFIX_" is a descriptive string describing the actual type of its value.

- For the "patternset", that follows the standard '<tagName id="arbitry_name" ' we will write:

<patternset idref="PATTERNSET_ (Ctrl+X n for autocompletion)

And word/code completion will offer now a list of available candidates:

- We could now continue to use code completion with any defined PATH_... element or any other type.

- The convetion used is also exceptionally useful when mixing HTML (sort of pseudo-XML) and Javascript. For example instead of identifying a table like:

<table id="results" .... >

...

<table id="buttons" .... >

we can use:

<table id="table_results" .... />

...

<table id="table_buttons" .... />

again code completion will be at our disposal when handling the html element (table, div, form,...) through Javascript. Now the editor can help us writting our "risky and error prone" javascript code as the next screenshot probes:

Basically the "TYPE_" prefix convention is adding manual type safety to our non-type-safe XML/HTML+JS files.

"Magically" now we are half-way between the non-type-safe languages and the compile-check type-safe ones. We don't yet have a compile-check to advice us of code mistakes, but at least the editor helps us now with code completion (that actually is certainly an indirect check safety measure).

This is not a joke, this is Java EE.

2011-08-06T05:09:00.000-07:00

Extracted from the Book "Real World Java EE Patterns":

@Stateful
@Local(BookFacade.class)
@TransactionAttribute(TransactionAttributeType.NEVER)
public class BookFacadeBean implements BookFacade {
    @PersistenceContext(type=PersistenceContextType.EXTENDED) 
    private EntityManager em;
    private Book currentBook;
    public Book find(long id) { 
        this.currentBook = this.em.find(Book.class, id); 
        return this.currentBook;
    }

    public void create(Book book){
        this.em.persist(book);
        this.currentBook = book;
    }
    public Book getCurrentBook() { return currentBook; }

    @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW) 
    public void save() {
        // nothing to do here
    }
}
The transaction setting of the BookFacade may appear a little esoteric at the first glance.
On the class level the TransactionAttributeType.NEVER is used. Therefore it is guaranteed, that no method of the bean will be invoked in an active transaction. The method save, however, overwrites this rule with the setting TransactionAttribute.REQUIRES_NEW at the method level, which in turn starts a new transaction every time. The method create in turn was not annotated with TransactionAttributeType.REQUIRES_NEW even though the EntityManager#persist methos is invoqued inside. This invocation will not fail with the TransactionRequiredException. It will not fail because on an EntityManager with an EXTENDED Persistence context, the perssit,remove,merge, and refresh methdos may be called regardless of whether a transaction is active or not. The effects of these operations will be committed to the database when the exteneded persistence context is enlisted in a transaction and the transaction commits. It actually happens in the empty method save.

So in order to avoid a few extra lines of intuitive standard java code in charge of transaction demarcation the author has developed a counter-intuitive code that just work when calling empty function (and just if invoqued in the correct order). What's is worse, the author looks really excited with this way of working.

I'm completly sure no one except those in charge of developing Java EE containers will think calling and empty function with metadata is the proper way to work.

Fortunatelly for the rest of us, we have Spring.

Xvfb+x11vnc outperforms Xvnc

2011-05-29T02:39:00.000-07:00

(update 2011/08/29: NX offers a much better aproach to remote X control).

If anyone of you has ever tried to run a remote X desktop using Xvnc you will probably have got quite dissapointed. The performance is poor and the response to user accions is annoying. Do not despair. There is an much better alternative. Just combine Xvfb (the "virtual in memory" X server) plus x11vnc (the vnc that attach to an existing X server). Extracted from the wikipedia.


export DISPLAY=:1
Xvfb :1 -screen 0 1024x768x16 &
fluxbox &
x11vnc -display :1 -bg -nopw -listen localhost -xkb

(Another probe in favor of the "Do one thing and do it right!" Unix philosophy)

KILL THE NULL, KILL'EN ALL

2011-05-15T10:32:00.000-07:00

Some will argue that null values are useful and even needed when writing software. Nulls allows to declare a variable without a defined value and that's something many people will think is needed when we don't known in advance the real value of an entity.

Imagine for example a database whose squema is upgraded adding a new column "second name" to a table. Since we don't know in advance what value "second name" has to be for each row, and since many new rows possible will have not "second name" it's quite easy to let it default to NULL.

In fact, I will try to show that using NULL values is always a bad idea. The first trouble many people will have trying to get rid of nulls is solving the problem in the previous paragraph. If I don't know the value of a variable of if a variable has not value at all (maybe because performance reasons force to denormalize a database), there is no other way that assigning NULL.

Right? wrong. It's always possible to assign a default value to tag a field as NULL. In the previous example we could simple assign the string "NULL" to indicate a non existent second name. It looks an arbitrary decisition that doesn't improve our code at all. In fact it does make our application safer. Let's see why.
Imagine a typicall web app in which a form sends the "second name" to our server in a get/post request. Something like second_name=Armstrong.
Then our php server code will look something like:


$second_name = $_REQUEST["second_name"];

Look now what happens if accidentally we misspell second_name in the html form or the php code:
$second_name will be assigned a NULL value. Not the "NULL" string we want in case "second_name" is not used.

The previous example is just one of the many possible ways in which a variable can be silently set a wrong NULL value. As you can see NULL values are quite risky, since languages tend to leave variables in a "null" state when something goes wrong. For example Java, probably the most widely used programming language around has a big design mistake:
try-catch exception demarcation bounds are used also as visibility demarcation bounds. That means that we can not write a code like:


   try{
MyClass myObject = new MyClass();
...
   } catch(Exception e){
...
   }
myPreparedStatement.set(0,myObject);
myPreparedStatement.execute();

The java language syntax force to do something similar to:


MyClass myObject = null;
   try{
myObject = new MyClass();
...
   } catch(Exception e){
...
   }
...
myPreparedStatement.set(0,myObject);
myPreparedStatement.execute();

Great! If an exception is thrown at object initialization and we are not cautious enough, we will insert a false null into the database.

(Note: In the previous example it's possible to move the two last lines inside the try-catch and that will avoid propagating a null to the database. In a larger code it could be impossible and even if it could, it's up to the user writing the code to do so and we are searching a way of writting code that doesn't depends on programmer's skills and experience).

With C#, the situation is even worse, since C# designers were drunk enough to not only copy the same Java pitfalls, but to encourage the use of nulls "everywhere"

In fact, NULL values are the biggest mistake ever made in software history, and as the time pass, the dammage augment.

The problem with NULL is that NULL actually is/means/maps to *nothing*. Null is not "unknown", Null is not "not applicable", not even "not available", not even "right" o "wrong". Null bears no information at all. If we were physicists NULL will just compare to a singularity in our ecuations, a clear symptom that something is really, really wrong.

In the example of the database we can assign the string "NULL" to indicate an unknown value. In fact we can assign the string "Not known" to indicate we don't known it's value, even if it exist, and we can also assign the string "Not applicable" when the row, due to break of normalization doesn't use that value. Our code will now be able to compare the string and apply a bussiness logic or another since now it carries information and state.

Convention let us replace "null" values with a given value or even better, with a given set of values ("not known","not aplicable","" (empty),...) . That's probably what we really want. That means a little bit of extra coding since we will need to create default rows/objects for each entity in our UML diagram, but the extra work is always worth the effort when our code grows beyond the thousand lines of code. Magically null pointers will vanish. Errors will be detected much early and correct exceptions thrown properly.

And last but no least, here comes a trick to avoid NULLable values in a database in a safe way:

Imagine you have a set or "core" tables in your squema, for example "client" and "product".
Imagine now that due to performance issues or whatever the reason you have another not-normalized table "notNormalizedTable" with a foreing key column "fk_client_id" and "fk_product_id" that many times doesn't have a given set value either for client or product or both.
If you want to avoid the risky NULLs the first step is to create default client and product rows. For example:
client: ======= id | name 01 | "Not Known" 02 | "Not Applicable" ... product: ======= id | name 01 | "Not Known" 02 | "Not Applicable" ...
A new problem arises now. It's possible to accidentally delete either client or product "meta-rows" that represent "Not applicable"/"Not known"/... values, and then our strategy for NULL replacement will go to an end.
Noneless, the solution is quite easy. Just create a new meta-table "nullKiller" with a foreign key for each table for which we want "meta-rows" with the "ON DELETE NO ACTION ON CASCADE NO ACTION" clause like:
create table `nullKiller`( fk_client_id fk_product_id ... fk_entityN_id CONSTRAINT `nullKiller_const1` \ FOREIGN KEY (`fk_client_id`) \ REFERENCES `client` (`id`) \ ON DELETE NO ACTION ON UPDATE NO ACTION, CONSTRAINT `nullKiller_const2` \ FOREIGN KEY (`fk_product_id`) \ REFERENCES `product` (`id`) \ ON DELETE NO ACTION ON UPDATE NO ACTION, ... CONSTRAINT `nullKiller_constN` \ FOREIGN KEY (`fk_entityN_id`) \ REFERENCES `entityN` (`id`) \ ON DELETE NO ACTION ON UPDATE NO ACTION );
then just insert the new rows:
INSERT INTO `nullKiller` \ (fk_client_id,fk_product_id,...,fk_entity_N_id) \ VALUES (1,1,....1), (2,2,....2);

Once the `nullKiller` rows are inserted we are sure that a reserved set of ids for each entity are reserved for default values thanks to the "ON UPDATE NO ACTION" clause, and we are also sure those rows will not be accidentally deleted thanks the "ON DELETE NO ACTION" clause.

So, you know what to do now: "KILL THE NULL, KILL'EN ALL"

Monitoring uninterruptible system calls.

2011-05-03T03:23:00.000-07:00

Here comes a "keep it simple and stupid" one-line script to watch for any process that could have trouble beeing blocked by an uninterruptible system call.


#$ while true; do ps -e fo stat,cmd | grep ^D && echo "-------"; sleep 0.1 ; done

D<    \_ [scsi_eh_0]
D         \_ hald-addon-storage: polling /dev/sr0 (every 2 sec)
-------
D         \_ hald-addon-storage: polling /dev/sr0 (every 2 sec)
-------
D<    \_ [md0_raid1]
D<    \_ [kjournald]
-------
D<    \_ [md0_raid1]
-------
D<    \_ [md0_raid1]
-------
D<    \_ [md0_raid1]
-------
D<    \_ [md0_raid1]
-------
D<    \_ [md0_raid1]
Ds   /sbin/syslogd
-------
D<    \_ [md0_raid1]

The output in this case is normal. [md0_raid1],kjournald and syslogd are making frequent disk writes.

Wait, wait!!

2011-04-20T05:05:00.001-07:00

After years of shell scripting I wasn't aware of the useful "wait" command. It makes the parent process wait until all child processes finish. I mean:


  #!/bin/sh
  
  (sleep  5; echo "good bye from child 1") &
  (sleep 10; echo "good bye from child 2") &
  
  wait
  echo "good bye from parent process"

When executed, this script will output:


  good bye from child 1
  good bye from child 2
  good bye from parent process

Arggg, so many lost hours writing custom code to do what was only one line away.

JQTJSF: Java-quick-thread-job-splitting/forking

2011-03-28T04:50:00.000-07:00

(Sorry about the stupid title but "laws of nature" force me to prefix anything related to Java with an annoying, uggly, empty of content and counter-intuitive acronym).

Reading about closures I found the next quite useful paragraphs (Copy&Paste from the Wikipedia):

Java allows defining "anonymous classes" inside a method; an anonymous class may refer to names in lexically enclosing classes, or read-only variables (marked as final) in the lexically enclosing method.

class CalculationWindow extends JFrame {
      private volatile int result;
      ...
      public void calculateInSeparateThread(final URI uri) {
              // The expression "new Runnable() { ... }" is an anonymous class.
              new Thread(
                      new Runnable() {
                              void run() {
                                      // It can read final local variables:
                                      calculate(uri);
                                      // It can access private fields of the enclosing class:
                                      result = result + 10;
                              }
                      }
              ).start();
      }
}

Some features of full closures can be emulated by using a final reference to a mutable container, for example, a single-element array. The inner class will not be able to change the value of the container reference itself, but it will be able to change the contents of the container.

According to a Java 7 proposal[11], closures will allow the above code to be executed as:

class CalculationWindow extends JFrame {
        private volatile int result;
        ...
        public void calculateInSeparateThread(final URI uri) {
                // the code #(){ /* code */ } is a closure
                new Thread(#(){
                    calculate( uri );
                    result = result + 10;
                }).start();
        }
}

Limiting Flash CPU usage in Linux (Firefox only)

2011-03-18T10:26:00.000-07:00

Firefox makes it easy to enable/disable plugins like Flash.

The problem is that there are many webs out there. Some of them use Flash legitimately and with good intentions, while some others try to invade us with ads banners without taking a minimal effort to keep resource usage low. Switching on/off the flash plugin is not always an option or at least not a confortable one. Ideally it would be great to have Flash running while simultaneously get certain that it will not peak CPU usage slowing down web navigation.

Newer versions of Firefox use an external process named "plugin-container" to execute external plugins like Flash. Fortunately that makes it quite easy to control the maximum CPU percentage we want to assign to Flash. All we need is to follow the next 2 steps:

- Install the command line utility "cpulimit". How to do it must be straightforward if its already in our favority linux distro repository.

- Edit (as root) /etc/rc.local and add a line similar to:

cpulimit -l 20 -e plugin-container &

Once we reboot our system 'cpulimit' will take care that external firefox plugins (aka flash) don't get more than 20% of our CPU. That can be too slow for people who want to check videos on youtube, so they will probably want to use a higher percentage. 20% is just a personal option. (Don't forget also that VLC allows to play youtube videos and it performs much better than the Flash plugin: http://www.google.com/search?q=vlc+youtube).

In my particular system, a "not so 3-years-old" laptop, I'm used to set cpu frequency to the minimum available clock, 800Mhz. That let's me use the laptop as a real LAPtop, I mean without my laps being burn by the heat dissipated by the CPU. Before using cpulimit I had no other option that turning off the plugin flash since randomly some "aggresive" flash banner knocked down my web browser. After cpulimiting the plugin those banners will need to manage themselves how to share the 20% cpu left for them. Not my problem anymore.

What about Chrome? I think, for what I have read about, that Google Chrome is ussing some type of plugin container too, but I found no easy way to control it. "top" just show a single process for the browser and the plugin so I didn't find any easy way to let cpulimit control it. Any hint/idea is wellcome!

God save VIM!: My personal .vimrc

2011-03-11T03:12:00.000-08:00

set number            <-- A must 
set ignorecase        <-- quite useful
set encoding=utf-8
set tabstop=4          
set shiftwidth=4      <-- 8 is too much and annoying
set expandtab         <-- replace tabs with spaces. That makes layout independent of tab's setup.
set foldmethod=indent <-- Ummm, depends on taste. Automatic folding based on indentation
Once a file is open it may be better to change it to foldmethod=manul
set cursorline        <-- Visual help (cursorcolumn is useful sometimes) 
syntax off            <-- Syntax highlighting never works to me.
set cindent           <-- C indent rules                                           
set smartindent       <-- Make vim smart about indentation
set autoindent        <-- Enable autoindent

External Links:

Tips for using vim as an ide

Add skip-name-resolve to default my.cnf

2010-12-21T07:44:00.000-08:00

Copying and pasting myself from an entry in the bugzilla system:

Description of problem:

The default mysql setup (/etc/my.cnf) indicates to make an inverse dns lookup
for new network connections. Mostly this is a quite primitive way to strength
database security  (IMHO it provides no added security at all) but promotes
buggy behaviour and is quite risky. For example mysql connections will fail if
DNS fails. Even if it works, connections will really slow down (around x100
times slower!!) if using an external DNS or the DNS is overloaded.

Other potencial problems. After granting access with the command:

 GRANT ALL PRIVILEGES ON *.* TO 'myuser'@'%' IDENTIFIED BY 'MyPassword' WITH
GRANT OPTION;

 connections will be allowed to 'myuser' from any host, EXCEPT from localhost
(127.0.0.1). An extra command will be needed.

 GRANT ALL PRIVILEGES ON *.* TO 'myuser'@'%' IDENTIFIED BY 'MyPassword' WITH
GRANT OPTION;

What's much worse. Allowed connections can stop working if suddenly /etc/hosts
is altered. (That really hurts in production environments).

KVM weekly backup, the easy way

2010-12-17T02:46:00.000-08:00

For the joy of all my readers here it comes the master-of-the-universe weekly script backup for KVM:

# Redirect stdout/stderr to custom log.

exec 1>> /var/log/custom_$(basename ${0}).$(whoami).$(date '+%Y%m%d').log 2>&1
pushd /media/backup/MyServer1/
# Keep copies of the VM for the last 4 weeks.
mv kvm.qemu.gz.1 kvm.qemu.gz.2
mv kvm.qemu.gz.0 kvm.qemu.gz.1
mv kvm.qemu.gz kvm.qemu.gz.0

# Stop KVM instance through telnet.
# KVM has to be started with the option:
# -monitor telnet:127.0.0.1:9942,server,nowait
echo "stop" | nc -q 10 127.0.0.1 9942 # Freeze KVM instance

#Here we do the real backup of the KVM instance
cat /VM-Images/kvm.qemu | gzip > /media/backup/MyServer1/kvm.qemu.gz
if [ $? != 0 ]; then
echo "WARN: KVM backup failed"
fi
echo "c" | nc -q 10 127.0.0.1 9942 # Continue KVM instance
popd

For the previous script to work correctly we have to start the KVM instance with the option -monitor telnet:127.0.0.1:9942.

In my particular system I start KVM instances at startup in /etc/rc.local. This is script is also quite interesting so I extracted all the kvm related stuff:

/etc/rc.local:
# Setting up the bridge (tip: apt-get install bridge-utils)
brctl addbr ofi1
brctl addif ofi1 eth1
ifconfig eth1 0.0.0.0 promisc up
ifconfig ofi1 192.168.2.100 netmask 255.255.0.0 up
ifconfig ofi1:1 172.16.1.3

...
kvm -net nic,macaddr=52:54:00:19:34:56 \
-net tap,script=/etc/qemu-ifup -hda /VM-Images/kvm.qemu \
-boot c -vnc :5 1>/var/log/custom_kvm.qemu.log \
-monitor telnet:127.0.0.1:9942,server,nowait 2>&1 &
...
# ionice/renice down our virtual machine
PID=$(sof /VM-Images/kvm.qemu | grep -v ^COMMAND | while read cmd pid staff; do echo $pid; done)
ionice -c 3 -p ${PID} &
renice 10 -p ${PID} &

/etc/qemu-ifup:
#!/bin/sh
# ofi1 is the choosen name in /etc/rc.local
/usr/sbin/brctl addif ofi1 $1 ;
ifconfig $1 0.0.0.0 up;

TIP: KVM console has lot of interesting options not shown here, including those for migrating a virtual machine.

eternal spirit of contradiction

2010-12-06T01:12:00.000-08:00

Visual/graphics applications tends to be developed with text based programming languages, while text/voice applications tends to be developed with visual based ones.

Another probe of our natural bias toward self-contradiction.

A mysql inmigrant in an Oracle world

2010-10-09T09:17:00.000-07:00

Last few years I've been living in MySQL. A quiet city full of good food and nice people.

Two months ago I had to relocate to Oracle, a "fast-food-city" where people forget what they are and become slaves of projects, budgets, estimates and more parasitic MBA vocabulary.

Life style is quite different, even if both cities are close to each other. Just one switch away.

Beeing I MySQL citizen the first thing I tried to do was to go to the Oracle's city council and sign up for a new database. What a surprise when the public servant ignored me. I was so upset that I didn't known what to do. Fortunatly a gentle citizien next to me laughted and heading to me with a smile in his face said:

- "I couldn't avoid hearing your conversation. You are new in the city, right?"

- "Yes am I" replied. "I just arrived in the city a few hours ago and ..."

- "and you tried to sign up for a new database as you ussually did in MySQL", interrupted him with an increasing smile and then continued ...
- "you can't do that in Oracle. People here don't like it. Don't worry about that, just do next thing: Tell the officer you want a new user for the existing database. That will automatically provide you with your private schema. The schema will have the same name than your user and you will be able to do whatever you like in it as you previosly could in your private MySQL database."

-"Thanks you, I was completly lost!" I replied.

-"You are wellcome. I came time ago from the beatiful PostgreSQL countryside and it was taugh to get used to a bloated big poluted city like this. Still, it's not that bad once you get used to it. have a nice day"

Now it's time to find a good server to hire! Servers in Oracle are quite expensive when compared to the prices in MySQL. I don't want an luxury one with swimming pool and a garden just something small to pass the night and good neighbours, you known, I don't like conflictive hard-disks next door. Guess me good luck!

There is live after dead...

2010-05-01T09:51:00.000-07:00

... at least if you have been cached by Google before dying:

Windows Licences comming to a dead-end

2010-03-30T02:47:00.000-07:00

The image above reflect real prices for the Zotac ION. The snapshot is dated 2010-03-25.
Prices start at 164.37Eu (220Dolars) and with such price we get a noise-free nettop with an 1.6Ghz Atom and an Nvidia ION able to play 3D content and FullHD videos. It comes also with a useful handle to hang it on the back of any VESA monitor or TV with HDMI input so in practice, it wastes no space at all.

The Atom is not big deal but is more than just enough for 90% of common office tasks and the Nvidia ION chipset can do magic.

Another "feature" is that these equipments come without CD/DVD. Just an internal laptop HD with 160Gb. Enough to install any OS. Anything else must go into the SD slot or conected through USB. The price is so adjusted that it doesn't even come with a Windows Licence. When you go bellow the 200€ price per unit that's not possible. A bunch of extra euros is too much when you have put all the efforts in getting a final low-price.

This is an start. But something that is going to be more and more normal. Prices will drop down the 150 barrier, then 100. Finally, PCs will be for free for final consumers in maybe one or two years, and most people will get them from its Internet provider.

People will still be able to buy an expensive quad-core and waste 200 Watts power to write a letter or check their emails, paying a "funny" extra bill each month to its power-supply company, bear with the annoying fan sounds and the hot air comming from the inside of the bloated cages that reminds of a old diesel engine. Yes, the will, but most will not. Today is even possible to get Atom equipments for around 100€ by dropping all the goods of the Nvidia ION.

In this context I see no place for the Microsoft tax. Whether they like it or not, it's something not in their hands. Hard times are comming for the old Windows monopoly. Microsoft will need to fetch money from anywhere else. I guess they will try to get it increasing licences fees from their loyal (and somethinglike naïve) customers. They will raise server licences, SQL licences, "whatever" licences. Loyal customers who bet for Microsoft and ignored opensource alternatives, just because Microsoft was a giant in the stock market will now have to confront a world dominated by opensource alternatives. Loyal customers will pay more and more for less and less.

A final note. When using Linux, it's quite easy to make the Zotak ION start through the PXE network protocol. Since network swithches with 48x100Mbs + 2x1Gbs Uplink ports are around 300€ (and droping) that mean we can drop the internal HD and get even a more adjusted price per unit. And having all network based means software updates and central administration comes for free, even for a farm of hundreds (maybe thousands) of PCs.

Just one extra piece is needed. A central NFS server acting as root file system. And today such server can even be placed in a virtual machine. Great, isn't it?

The King paper has died, God save the new King

2010-02-01T07:39:00.000-08:00

It has been many years of joy reading news and books using such a great invention that our ancient egipcians called "per-peraâ", later on renamed as "papirus" by the greeks, "papyrus" but the roman empire and more recently "paper" by the english who speak English.

But now it's time to say "goodbye paper!".

Modern electronic-ink and books will of course compete, but this weekend I found a much more lethal enemy for our lovely paper. The digital camera.

I was thinking how to get rid of all those books I've been collecting over the years and that have slowy been deposited in my bookself. You know, all those books you don't think you are going to read in the next comming years/life but are too good to leave appart. But our 3D space is finite and sooner or later we must surrender to the evidence. There is no place anymore for them.
And so, no without pain, I started to think about how to digitalize all their pages before burying them into the obscure junk room where they will rest in peace.

My intuition told me that if I wanted to digitalize something I had to scan it, and nothing is better to scan than an scanner... but my intuition was wrong. I tried with my first book and *slow torture* is the only way to define my experience. Nearly 30 seconds per page, and that multiplied by 300pages on average per book was something I just want for my worst enemy. Even more, placing the correct frame per page was all but easy and around one over fives times, I failed to properly place the page in the scanner so I had to reposition the page in the scan and repeat.

Frustated as I was I deciced a second strategy. A few weeks ago I bought a cheap camera for arround 120€. Nothing great for such price, but still with a maximum 12Mpx marketing resolution. I was amongst those who thought 12Mpx is an aberrant pixel definition for a camera, but for once such an army of megapixels made my day. I tried to take a shot at a page and compare the result with that of the scanner. Not without surprise, the result was much better. The foreground black text looked sharp with a yellowish background. The shot took about 7 second per page and the previous problems to place correctly the page in the scanner vanished in the air. Scanning with the camera was so fast that it took me no more than 1 hour to digitalize a 300 pages book, and with jpg compression the final space was about 200 Mbytes or around 1/2500 of my 500 external USB HD.

But the real test came next. Reading the digitized text on my 24" screen and comparing with a normal lecture on paper. It was then that I realised paper had become past (and great) history. The difference in favor of the electronic option was so big that I can't think to go back to normal paper in the rest of my life. My eyes were much less fatigated and I could read the pages at nearly two meters of distance. Suddendly all the pages had become a PowerPoint. Next come a picture that tries to reflect what I'm speaking about. Up is the text in the 24" monitor, down in blueish color that same text in my old friend "the paper".
(The picture camera senses the monitor 60Hz refresh rate and some strange distorsion effects appear on this picture that are actually invisible to the human eye, specially the aurea around the letters).

Mi primer circuito integrado.

2009-12-02T13:47:00.000-08:00

Allí está. Con un par. Si no me lo dice nadie, me lo digo yo: "Torero", "campeón", oeeeeh, oeeeh, oeeeeh, oeeeh, ..

Voilá, no sin muchos aims y más efforts aquí están la foto del recien nacido:

Ha salido a su padre. Tiene mis mismos ojos y esa sonrisa ... Snifff, snifff. Todavía no sé que nombre ponerle. Estaba pensando en Z80, como el Zilog de 8bits de mi primer ordenador.

¡Larga vida a Wikibooks!

2009-11-28T04:36:00.000-08:00

Siempre estoy apuntando chuletarios por aquí y por allá. (¿Alguien ha notado el nombre de este blog?). Unas veces en un fichero de texto, otras en papel, de vez en cuando en la intranet o en el blog. Ummm, ninguna me convencía. No era el lugar correcto. Pero al final y tras largos años de angustía y lenta agonía lo encontré. La solución es Wikibooks.

Lo siento Google, no es nada personal, pero la wiki te vuelve a ganar.

Las cuatro cifras del sistema binario

2009-04-17T14:30:00.000-07:00

La lógica binaria tal y como se explica en los libros de texto se representa con dos cifras, el cero y el uno.

En el mundo real, la lógica binaria se apoya sin embargo en cuatro símbolos para poder describirla correctamente:
Los dos primeros símbolos representan el cero y el uno bondadosos. Los dos últimos describen a sus gemelos malvados.

Para que todo funcione correctamente es necesario elegir el primer par y mezclarlos como deseemos, sin importar si es un cero o un uno.

La web se impone sobre el escritorio.

2008-05-19T07:59:00.000-07:00

Leía hace un par de días que Google había hecho mejoras en su sistema de correo web.

Según comentaba la noticia, tras un detenido estudio se había conseguido cachear mejor los datos utilizados por el usuario hasta alcanzar un 20% de aumento de velocidad.

La noticia acaba allí, maravillada por el 20% de incremento de la velocidad. Lo que no cuenta la noticia es quizás el hecho más destacable. Los usuarios de Google mail aprovecharán las mejoras instantáneamente. En el momento en que vuelvan a visitar la web todas las mejoras estarán presentes para todos y cada uno de los millones de usuarios del servicio y sin que ninguno de ellos haya tenido que realizar esfuerzo alguno para aprovechar los cambios, tan solo logearse con su usuario y password.

Esta es la mayor ventaja de la web. Los programas y los datos están centralizados y el navegador se limita a ejecutar la parte cliente mediante javascript. ¡Que diferencia con las aplicaciones habituales!. No es anormal ver oficinas donde usuarios con la ultimísima versión de Office comparten documentos con usuarios de Office (19)97. Y no sería extraño que dentro 20 años veamos oficinas donde usuarios con la ultimísima versión de Office compartan documentos con usuarios de Office (19)97. Un disco duro roto o un nuevo ordenador significa perder horas reinstalando todo lo que teníamos, todas las configuraciones y todas las cuentas, usuarios, passwords, perfiles, e incompletos backups con todos nuestros documentos. A veces, recuperar el CD original con el programa de instalación le hace sentir a uno Indiana Jones en busca del Santo Grial.

El futuro está claro y pertenece a la web, igual que este blog.

Hoy es un día histórico para Linux

2008-03-29T11:05:00.000-07:00

Durante años llevo anotando periódicamente las ofertas laborales relacionadas con Linux, Windows y Unix en Jobserve.com, uno de las mayores web de empleos para informáticos. Realmente creo que estas webs son el mejor reflejo de la realidad del mundillo del "Information Technologies" ya que los análisis de los ¿analístas? se pueden calificar como curiosos, graciosos, impactantes pero desde luego nunca objetivos, realistas o independientes:

Baste decir, por poner un ejemplo, que el rimbombante grupo Garner (queda muy "cool" si se pronuncia con acento yanqui) está parcialmente liderado por compañias de inversión con importantes "brazos" en empresas de informáticas sobre las cuales se supone que mantiene la neutralidad. Silver Lake Parteners, que ostenta el 33% de Garnet cuenta con Michal Dell, Bill Gates, Larry Ellison y otros entre sus inversores. (*http://www.informationweek.com/story/showArticle.jhtml?articleID=178601879).

Volviendo al tema de interés. ¿Qué nos cuenta Jobserve acerca de Linux ahora mismo?. Bueno, pues ni más ni menos que por primera vez Linux supera el 20% de cuota de ofertas laborales, respecto a un 43'19% de Windows y un 36.45% de Unix (entendiendo por Unix todo y cuanto se parece a Linux sin ser Linux).

Empecé a anotar la relación en Linux, Windows y Unix allá por Abril del 2001. En aquél entonces Linux contaba con un 3.8% del mercado de ofertas laborales, frente aun 48% de Unix y Windows aproximadamente. Durante 2 años Linux fué creciendo suavemente al tiempo que Unix bajaba, mientras que Windows se mantenía e incluso crecia un poco. Por fín, en diciembre del 2003 Linux se colocaba en el 10%. Durante los 3 años siguiente Linux continuó progresivamente su escalada a costa de Unix y en menor medida de Windows. En diciembre de 2006 Linux contaba con un 19.0%, Windows con un 43.5 y Unix con un 37.4%. Y entonces de repente, Linux perdió cancha, pasó todo el 2007 sin subidas ni bajadas. Un més subía un 1% y al mes siguiente bajaba otro 1% y por fín, hoy 29 de Marzo de 2008, un año y 3 meses más tarde supera el 20%. A comienzos del 2001 la relación entre ofertas relacionadas con Windows y linux superaba el 10 a 1. 7 años más tarde está proxima a 2 a 1.

¿Qué paso el 2007? ¿Acaso Linux perdió fuelle? Bueno, yo creo que la respuesta es sencilla, 2007 fué el año de la virtualización. Fué el año en que realmente Linux dejó de alojarse en máquinas físicas y empezó a ejecutarse en entornos virtuales como Xen, Virtuozzo, VMWare, KVM, etc... Esto significó menos administradores para más servicios. Por contra Windows parece haberse quedado un poco a la zaga en el tema de la virtualización, por un lado porque el sistema operativo no ofrecía el soporte que Linux ofrece desde mediados del 2006 (principalmente gracias a Xen) y quizás también porque el perfíl técnico de los trabajadores de linux es más "geek" y absorve antes las "novedades" (novedades que no son tales ya que existen desde hace mucho en sistemas unix de gama tan alta como cara).

Por supuesto, Microsoft y sus millones no han desaparecido todavía, y esto significa que mañana volveremos a leer los artículos de "expertos" consultores de Gartner hablando de Linux como un sistema minoritario, destinado a un público joven y granos en la cara así como las ventajas y "return of investment" que supone "Güindouusssss" para el businessman de 21th-century.

Dos útiles utilidades

2008-03-24T11:35:00.000-07:00

Supongo que no soy el único que al actualizar su Debian Linux y ver un listado de 3 páginas con aplicaciones pendientes de actualizar se limita a decirle "Sí" a todo y espera media horilla a que el PC y el ADSL hagan su trabajo.

El otro día y sólo por curiosidad dediqué un rato a mirar qué era todo aquello que los señores de Debian habían decidido instalar en mi sistema y me llevé un par de agradables sorpresas.
Curioseando por las opciones de configuración (Sistema -> Preferencias -> Teclado) ví una opción que me permitía establecer descansos de escritura. Esta útil utilidad simplemente se limita a dar una "colleja virtual" avisando de que llevamos mucho tiempo pegados a la pantalla y nos aníma a descansar 3 minutos antes de continuar.

Por otro lado ví que entre las diversas aplicaciones que se mostraban en el menú de aplicaciones de oficinas había una nueva llamada "AiksaurusGTK". La misma permite mostrar sinónimos "a tuti plen" de cualquier vocablo inglés que venga a nuestra mente ordenándolos por el contexto en el cuál aparecen. Cuando uno no es muy laudo en la lengua de Shakespeare y las Spice Girls pero se ve obligado a escribir en inglés un día sí y otro también pocas utilidades se agradecen tanto como esta.

Mis primeros pinitos con el Mandarín.

2008-01-06T08:53:00.001-08:00

Si China va a dominar el Mundo va siendo hora de empezar a estudiar Mandarín.

El Chino Mandarín puede paracer críptico e inaccesible a primera vista al común de los mortales.

Sin embargo he podido comprobar experimentalmente que esto es un falso tópico. Os aseguro que a segunda, tercera y cuarta vista sigue pareciendo críptico e inaccesible.

La solución para no tirar la toalla es fácil. Hay que estudiar chino como su fuese un juego rompecabezas. Cualquier otro acercamiento puede resultar mortal para nuestrar pobres y desgastadas neuronas.

Trás comprarme un manual rápido del chino-4-dummies decidí tras algo más de hora y media y tras conseguir llegar al final de la primera página que ya era de poner en práctica la ingente cantidad de conocimientos adquiridos.

Sin más dilación puse rumbo con mi navegador hacia los confines que llevan a Baidú (http://www.baidu.com). He de advertír al ocioso y probablemente adormilado lector que pierde el tiempo leyendo mi blog que los siguientes párrafos requieren tener instaladas las fuentes chinas en su sistema para poder captar la magnificiencia de la milenaria lengua.

Nada más amarrar fuí saludado cortesmente y se me brindó la oportunidad de escoger entre varias opciones:

新闻网页贴吧知道 MP3 图片

Primera sorpresa. Tras hora y media de estudio no tenía problemas en traducir MP3 del Chino al Español (Esto pinta bien pensé).

Con una pequeña ayuda de algún desconocido que puso a su disposición un traductor de chino
(http://www.mandarintools.com/worddict.html) no tardé mucho en descifrar parte de las demás opciones.
新闻 -> hojear noticias
网页 -> web page
贴吧 -> ¿Búsqueda apróximada?
知道 -> ¿Búsqueda por materia?
图片 -> Imágenes

Algo que siempre me he preguntado es qué imagen guardan los chinos en su cabeza del antaño imperio español donde nunca se ponía el Sol. Por tanto no resistí la tentación de buscar en imágenes fotos sobre España.

El resultado fué http://image.baidu.com/i?ct=201326592&cl=2&lm=-1&tn=baiduimage&pv=&word=spain&z=0

Un momento, ¿Qué demonios es el mapa que aparece en la segunda imagen? Un click y varios segundos de espera aclaraban el misterio:

Este es mi nuevo blog.

2008-01-02T04:12:00.000-08:00

Pues sin más dilación voy a ver qué tal funciona este invento de blogger.

Yo blogeo, tú blogeas, él blogea...
blogearamos o blogeasemos,

Ummm, cuando menos parece fácil de conjugar.