Friday, January 29, 2021

Having fun with bzip2

  bzip2, an standard compression utility in Linux, Mac and friends has a very efficient compression ratio for low entropy files. I just tested to compress a big 89Gigabytes "all-white-spaces" file. The final compressed size was about 0.7Megabtyes, with a compression ratio of about x130.000. This is not very impressive from a mathematical point of view, since all the information could be packed in just 6 few bytes (a single byte for the white space plus 5 more bytes for the total size or around 16.000 million compression ratio). But the important thing is that bzip2 packs data in an standard format, and by just looking at the bz2 extensions, many software knows what to do with it.


 Now the funny part. 89 Gigabytes is much more that the storage capacity of most old/unattended servers and low end cloud servers, that curiously, are the main target of worms infecting the Internet. I have a personal web server and it is a bit frustrating checking that around 99% of visits are (were) attacks trying to find holes in some WordPress, php services that could be installed on the machine or just trying good luck searching for passwords files and crypto wallets.


 I was wondering what would happen if I uploaded my compressed 0.7Megabyte bz2 file, renamed it as "password.bz2", and redirected visits like /login, /wp-content, /backup, /administrator, /back/wallet.dat to it. The preliminary results looks to indicate that Internet worms like to download the bz2 file, but then they have some problems to "digest" the uncompressed 89 GB file. One day after installing the file, "non-polite" visits decreased to just 60% (from 95%+ one day before). Two days later, they just represented 40%. With some more tuning for forward-rules I guess results can improve.

Thursday, July 27, 2017

TXT rocks!

I've been involved in an open-source project for the last year or so called Interledger (The Internet of Ledgers).

Keeping up with the project progress is not an easy task since it's in active development and new ideas surge every week, interfaces change and many people work in parallel with different targets and perspectives. To make it more difficult most of the project is developed by the internal staff of Ripple, which I'm not part of, so getting out of sync is even easier. And to scale to new levels of difficulty, the Ripple guys have to keep the pace with all the fast-moving environment around them (block-chains, side-chains, banks, regulations,  ...) and some code like routing payments requires lot of research before getting stable (just notice that routing protocols in the Internet Protocol is still a work in progress decades later). Not happy with that, the time I can dedicate to it is just plain random.

There has been no few efforts to document everything around the project, with great progress to formalize around RFCs, but in such fast-moving development context documentation is always behind the code and still really needed to keep going.

I just noticed that keeping docs up to date with code would be a titans-work so I thought about an alternative. What about trying to group related info as close as possible and let our brain link the info into something that make our neurons work?
I needed something that visually could match related topics close to each other and simultaneously link to external links, as well as providing "goodies" for formatting text, tables and some minimal active tools help navigate more easily.
Since this is a software project I also wanted a clear distinction between the separate the different software cycle scenarios (documentation, development, testing, deployment,...)

I tried with different tools (CMS, Google Drive, ...) until I noticed that the best tool at hand was no other than plain TXT (with some spots of Javascript).


The result is available at:

https://earizon.github.io/txt_world_domination/.

(Update: 2023-02: the previous HTML version has been upgraded with the even lighter version using just TXT)

Saturday, January 21, 2017

A weekend theory about evolution of mammals. When the frog became a prince (and much more)

Many theories and hypothesis arise about where first mammals came from.

The general accepted main-stream theory claims that mammals evolved from reptilians and that reptilians, in turn, evolved from amphibians.

I think this is just wrong. I have an alternative theory that match better with many experimental facts.

Be prepared .... Here it goes:

"Mammals evolved directly from amphibians, and the first mammal to arise in planet Earth was no other than a tiny cute bat".

Where this arbitrary claim come from?. Curiously, to arrive to it I took the opposite direction applying some spots of deductive and inductive reasoning and starting with some evidence nobody will deny.

Simply put: "Bats fly".

It could look a curiosity at first sight, but a very weird one. It's weird because there is no doubt than flying is something *hard*. Flying is difficult, very difficult. Flying requires lot of energy, lot of changes in the brain and lot of physiological changes in the body of a bat. It's also weird, extremely weird that bats are the only mammals that can fly.

Bats are closely related to rodents and in particular to their mouse cousins, up to the point that is quite normal to associate bats to flying mouses. This is where some deductive reasoning takes place.

Darwin's theory of evolution explains how living beings evolve through *small changes*. Random *small changes* in the ADN provides some *adaptive advantage* that makes the new specie to survive. The words *small changes* and *adaptive advantage* are highlighted because here is where things starts to fail with the current theories:

 According to established theories bats evolve from rodents but this is very hard to explain with Darwin's theory of evolution. There is no small change that suddenly allows a mouse to start flying. There is no single change in the ADN that magically lets a rodent fly. Even if magically a mouse would be born with wings, its brain would not be adapted to flying. Even if magically a mouse would be born with wings and also magically its brain would change to let him fly its digestive system will not be adapted to consume all the energy needed to fly. Even if magically a mouse would be born with wings, with an adapted brain and a digestive system allowing it to fly, it will need changes to its breathing system, and some sort of mechanism not to loose heat throught its wings, and many other adaptations just because as already said, flying is hard to achieve, even for human engineers designing an airplane it's very, very complex (and humans count with the advantage of being able to burn tons of petrol to make an airplane fly, something forbidden to living beings).

Could it be the case that through thousands or hundreds of thousands or millions of years some species of  rodents could evolve through small changes until, at some point, they were able to fly?

  The problem with that hypothesis is that it goes directly against Darwin's theory of evolution.  All the *small changes* required for a bat to fly provide no *adaptive advantage* until such a bat actually flies. Quite the opposite a mouse with "big wings" but unable to fly will be doomed to die. It will not be able to move and will loose heat very fast. This is exactly the same for many other adaptive changes.

And so, if the evolutionary step driving from rodents to bats are hardly difficult to explain, what about the opposite? What if rodents evolved from bats?

Curiously this question is much easier to defend while respecting Darwin's theory of evolution of species. Maybe just at some point the bat's wings started to become atrophied or the bat augmented its weight or its metabolism didn't allow them to consume as much energy as needed to keep flying. Not being able to fly will be a disadvantage that could be *immediately*  be compensated by the fact that such a bat will waste much less energy by just "walking". This is something we have already observed in many birds (Does it sound familiar to you some naturist that observed some albatross in Galapagos island with atrophy in its wings?).

So let's assume the arrow of time moved forward from bats to rodents. What then? Where do bats came from, if not from rodents? What other mammal could evolve to the point of creating such curious flying mammal? They easy answer is:  NONE. 
The same arguments that lead us to the conclusion that rodents can't be the parents of bat would apply to any other mammal.

There is no such mammal. Bats are actually the first mammals living in Planet Earth.

Who was them the parent of first bats? Well, I have not a clear answer to this question but intuitions tells me that, with high probability, some sort of flying frog were their parents. I base it on the next coincidences:

- Both frogs and bats emit sound and they do with similar patterns. No other amphibian or reptilians have such a feature. Emitting sound as frogs and bats do require important changes and new "modules" in the brain to be able to control it.   This is an extremely sound coincidence for such small primitive creatures with small brains and in fact the audible sound emitted by bats resembles to a certain point to that of frogs (https://www.youtube.com/watch?v=ppLsu5Z2Np0)
- In general both species are much more active during the night.
- The physical appearance of flying frogs and bats have many coincidences, specially their "hypertrophied" hands-wings.
- Both have insects as main source of proteins.
- Size and weight match.

To hypothesize that a flying frog could at some point be able to "just fly" is not hard to believe and there are adaptive advantages in passing more and more time flying. It could be that at some point this flying frog could pass most of the time flying. This will trigger many fast evolutionary steps. Its lungs will change fast to allow to breath more oxygen from the air and pass more time flying. At some point being able to carry foetus inside will allow that flying amphibian to traverse many distances in search of food or just escape from predators with easy. Flying would favor warm-blooded to support sudden changes in temperature due to the wing. (Curiously the other warm-blooded species in animal kingdom apart from mammals are birds who also fly).

At some point this flying frog changed and a first mammal, a primitive bat, was born!.



If a theory is to be taken seriously it must also have some predictive power. Let's examine what this theory can predict:

- If bats were actually the first mammals on Earth there must be many different subspecies of them after so many years of evolution. This is in fact true, and as today bats is the second more diversified specie in the mammal kingdom living in all sort of climates and places. Curiously they are only surpassed by rodents, that will be nothing more than adapted bats.

- If bats were actually the first mammals there must exists fossils where no other o very few other species of mammal actually exists. This is actually true, and fossils of bats have been found in places where mammals were difficult to find.

- If all mammals come from bats there must be some trace easy to identify. This is not so evident, but in fact bats needs lot of brain power to control their wings. Curiously bats wings are nothing more than hands in newer mammals, and curiously many mammals have extraordinary skills to control their hands than no other terrestrial specie do have. Mammals brain also have a great power to understand the world in "three dimensions", something that could have appeared while bats learn to move in three dimensions.


And finally, as a gift for curious researches with no plans for the weekend, this is a non-yet-proved predictive fact of this theory. It's well known that bats use sound to localize and hunt insects. My hypothesis is that some (or many) subspecies of frogs could be using such same mechanism to localize insects, and not just as a way for males to get the attention of females. It's even possible that they will be using the same frequencies and we were not noticing it just because we didn't even thought about it.

Update 1, 2017-08-09: 
I'm enjoying summer's holidays now. Time for beach and caipirinhas in the sunny coast.... or even better, time for updating my theory on bats and mammals evolution.
I just landed by pure chance to a web page with a complete collection of bat-faces. The first thing that anyone will notice is how many distinct bat faces can be. But more interestingly and related to this post is the fact that many of those faces resemble many other different branches of mammals. Some fruit-bats clearly resemble to a lemur, an sub-specie of primate, that curiously have the same feeding habits that the bat it resembles to.
Please, I'm a bat! not a lemur


Then we have the Hammer-headed bat whose aspect is quite close to a primitive ruminant and again, another "random" coincidence, I read that those species of male bats  "possesses an enormous head for producing loud honking calls" like ruminants, and that they live in zones of Africa that match those of African ruminants with similar lek mating habits !!!

Why not speaking about the naked buldog bat, whose movements resemble those of a feline. Again new coincidences. Those naked bats are typical of Asia, where the first skulls of felines were found.

The face difference between some other bats and hedgehog (who are also insectivore) are hardly to distinguish.

In conclusion, we have a family of mammals, the bats, that share many physiological and behavioral characteristics with many other family of mammals (primates, ruminants, felines, rodents), with little resemblance between them. This point outs to bats as common denominator (or in evolution terms, as the root of evolution for mammals).

Update 2, 2017-08-09:
I also read on page 67 of this book that "the earliest known fossil of a primitive bat, an insectivorous species from the Eocene period was found in America. Bats had certainly existed for the previous 50 million or more years, but their evolutionary history is poorly known since their small, fragile skeletons do not lend themselves well to fossilization". This "fragile skeletons" explanation is difficult to sustain, considering that fossils for much more fragile structures have been found, even for bacteria. I have a much simpler explanation. Fossils were in fact found, but they were confused with those of an amphibian (a frog or similar), since by just looking at the skeleton it would be impossible to know whether it was still a "frog" or a primitive "mammal".


Update 2, 2021-12-30:
More than 4 year ago I predicted that if bats had evolved from frogs it could also be possible for some type of frog to be using sounds to hunt their preys. Today I just got aware of this link speaking about a weird Torrent frog, able to listen sounds in the 128 kHz ultrasonic frequency, raising the question of whether frogs might use ultrasonics for echolocation, the method bats and dolphins use to find their way around.
Digging a little bit further I found this other link where it is written """(Torrent frog) Males have ear canals with recessed tympanic membranes at the end of the canals, like mammals but unlike all other anurans except for Huia cavitympanum (Feng et al. 2002). """ (yet another curious coincidence with mammals), ...
"Vomerine teeth are present " (like in bats),
...

Update 3, 2024-04-21:
Yet more "coincidences". A new article published by Ubiratã Ferreira Souza, Guilherme Augusto-Alves, Mariana Retuci Pontes, Lucas Machado Botelho, Edélcio Muscat & Luís Felipe Toledo, mention the discovery of a frog using ultrasounds as a defensive strategy. My guess is that such frog is just hunting "a la bat".

Saturday, October 22, 2016

Energy saving issues with Lubuntu (and how to fix it with powertop)

I'm a big fan of Lubuntu (Light Ubuntu), the Ubuntu without all bloated software since, on average, it saves up to 1 or 1.5 Gigas or RAM when compared to standard Ubuntu will of the fancy (and useless) 3D effects.

Something that annoys me anyway is that the default setup in Lubuntu (I ignore if that's the case in Ubuntu) is quite energy-hungry.

I use a Dell XPS 13 laptop equiped with brand new hardware that "theoretically" must work in silence most of the time, but when I just boot up Lubuntu and a couple of apps (Firefox, Eclipse) the fan starts to run at max speed.

The easy (patch) solution is to run next command:

sudo powertop

then go to the "Tunables" tab, and switch all from "Bad" to "Good", except most probably for the USB keyboard and mouse and the wifi or ethernet. (For those really concerned with the integrity of their HD data, don't swith also al SATA related stuff).

Actually, activating the power saving for the PCI Intel Core processor Graphics Controller is usually "good enough" to make the fan settle down, but switching on all the others makes no harm and our computer will be happier with us (don't forget that sooner than later IA equiped macro-computers will punish those poor humans that through the years tortured their indefense CPUs, so it's better to get friends as soon as possible before it's too late).

Friday, December 4, 2015

Swift got it closer, but still fails

Continuing my religious and long war against null beings.

I just read about Swift improvements to avoid null pointers. In Swift a variable can not be assigned a null (nil in Swift nomenclature) unless it's declared optional. And this nil is actually an object of a defined type with a presence in memory, not a pointer to a forbidden memory address (0 in practice).

Still, while Apple's guys got closer to a solution, they didn't get the point.While the solution proposed by Swift fixed the Null pointer exception, it doesn't fix all the semantic inconsistencies of using Nulls. A nil object in Swift is still something empty of information, and so the "information lost" is still present.

The solution to fix the problem was much, much simpler and much, much more effective:

- DO NOT ALLOW NULL VALUES IN THE LANGUAGE.
- FORCE THE DEVELOPER TO USE SENSIBLE DEFAULT VALUES OR RAISE AND EXCEPTION IF THAT'S NOT POSSIBLE!!!

See more about it http://chuletario.blogspot.com.es/2011/05/kill-null-killen-all.html).

Tuesday, June 30, 2015

Downloading youtube Videos with VLC (command line)


Ladies and gentlemen, we present you the fast and easy way to download Youtube videos using VLC and a shell script:

The script:

  # INPUT is the URL as seen in the browser
  INPUT="https://www.youtube.com/watch?v=...."
  
  # OUTPUT is any arbitrary file.
  OUTPUT="200-UpdatingApplication.mp4"
  
  vlc --intf dummy "${INPUT}" --sout=file/ts:${OUTPUT} vlc://quit


A problem appears if there are many related videos to download (for example a 50 video tutorial).
 Using Firefox I just select them all on the "right column" (related videos), right-click to select "see source code selected" and then clean the html to let only the input URL and the video title. 

Then a few Vim macros is all that needed to convert them in a shell script.

World Domination is an step closer now!

Tuesday, January 20, 2015

Science is not Politics


“Let’s be clear: the work of science has nothing whatever to do with consensus. Consensus is the business of politics. Science, on the contrary, requires only one investigator who happens to be right, which means that he or she has results that are verifiable by reference to the real world. In science consensus is irrelevant. What is relevant is reproducible results.” – Michael Crichton

Looks unfortunate that most software developers have forgot this point.
Many modern programming languages just try to reach a consensus instead of doing things right. Let's observe how Javascript, the most widely used language today still keeps "null" and "nondefined" types around even if science says they must be dropped (backward compatibility is just an excuse, not a reason).

(Edit 2015/02/13: ECMAScript 5 Strict Mode will throw a ReferenceError when trying to evaluate a non-existing variable).