Inside Out

Few months ago, we discussed about a mail notification method for fetchmail/procmail and mutt combo. As you can see in that post, the script we wrote to handle notification was dependent on ratpoison. Well it makes life bit tough for my non-ratpoison X sessions. I thought it would be a good idea to make the script window manager independent. Enter osd.sh

The script uses osd_cat. It comes in the XOSD package for most of the unices. It is similar to the cat command for the console. It can read a file, string or the stdin and output the text onto X display. It has a bunch of config options which let you choose font, location etc. for the text. I’d recommend a RTM (man osd_cat).

Here is the code:

#!/bin/bash
# Displays a string on the screen
# Last Modified: Mon 22 Jun 2009 02:59:55 AM IST

echo $text | osd_cat --color=$color --delay=$delay --age=$age --font=$font --offset=$offset --shadow=\
$shadow --lines=$lines --align=$align --pos=$pos

This script can be used for any notification purpose. For using this with procmail, replace use the following construct in procmailrc:

:0 h
| grep "From:" | /home/arun/bin/osd.sh

This will display alerts as “From: Foo Bar ” when a new mail is received.

Enjoy!

Well you guessed it right, the “global” command (or :g in command mode) will let you select lines that match a particular pattern globally in the file and lets you operate on the selected text. The :v operates similar to grep -v which selects all texts which do not match the pattern.

Syntax: :g/pattern/command where pattern is any regular expression, command is any vim command

Here’s the scenario which taught me the global command:
I have a file with ~29K lines. All lines must begin with “\\”. However somelines are broken into two, in which case the lines which do not start with “\\” need to be appended to the previous
line
Sample text:
\\foo\bar
\\food
bar
\\x\y
\z

Should be converted to:
\\foo\bar
\\foodbar
\\x\y\z

And we solved it this way:
:v/^\\\\/exe "normal i\<C-H>\<ESC>"

All we did was to execute a backspace(C-H) for all occurences of pattern in normal mode

Let’s end the post with a theorem:
Statement:VI is perfect
Proof: VI in roman numerals is 6. The natural numbers less than 6 which
divide 6 are 1, 2, and 3. 1 + 2 + 3 = 6. So 6 is a perfect number. Therefore, vi
is perfect.
– Arthur Tateishi

And a corollary
VIM in roman numerals might be: (1000 – (5 + 1)) = 994, which happens to be
equal to 2*496+2. 496 is divisible by 1, 2, 4, 8, 16, 31, 62, 124, and 248 and
1+2+4+8+16+31+62+124+248 = 496. So, 496 is a perfect number. Therefore, vim is
twice as perfect as vi, plus a couple extra bits of goodies.

That is, vim is better than perfect.
– Nathan T. Oelger

Have an awesome 2009!

Problem: I have my local mail setup with mutt, fetchmail and procmail. And I would need a way to run an arbitrary script when new mail arrives and _not_ loose the mail from my inbox.

Solution: Procmail supports a nice feature called nesting. With nested blocks a procmail recipe can be assigned more than action.
e.g: Here’s a recipe to filter all my email from gmail:
:0:
* ^To:.*gmail\.com
{
:0 c
$MAILDIR

:0 h
| grep “From:” | ~/src/_scripts/ratpoison/newmail.sh
}

And the newmail.sh script
#!/bin/bash
read from
ratpoison -c "echo Mail $from"

Discussion: Here’s the breakup

• *^To:*gmail.com Catch all mails which have their To: field containing gmail.com
• Nesting: specify the actions inside { .. }
• :0 c - Copy the mail into $MAILDIR
• :0 h – Pipe the header into grep. And pipe the output (”From:*”) into newmail script
• In newmail.sh script, read the piped content into variable $from and display that in ratpoison message bar

If your window manager has a notification tray, you may be interested in this. New to procmail, go here.

/me is having bit overdose of cookbooks these days

I had posted long back about a bash wrapper over youtube-dl to fetch youtube videos using your own download manager. The youtube-dl script had changed long since then. And good news is now we don’t need to change the script in any way to get the download url. So here we go!

Grab the latest youtube-dl from here
Minor update to the linux youtube-dl script:
#!/bin/bash
if [ $# -ne 2 ]
then
echo “Usage : $0 ”
echo “e.g : $0 http://www.youtube.com/watch?v=D1R-jKKp3NA steve_jobs”
else
outputfile=”.avi”
todnload=$(youtube-dl -g $1) # youtube-dl is in path, right?
echo “Got the file..”$todnload
axel -n 100 $todnload -o $2″.flv” # wget -c $todnload -O $2″.flv” .if you don’t use axel
echo “Download Completed…”
ffmpeg -i $2″.flv” $2$outputfile # get the avi file
fi

Enjoy
Thanks Jay for pointing this out!

Lemme shout it aloud: I am a big fan of free webhosts Until recently, one of my friends made me realize(brainwashed!) that probably I had had my part of the freebies And its been long since I last did something php-ish, so the thought of getting hands dirty with web programming made me take this bad decision of getting a host.
And dude, choosing the right host _is_ much tougher than expected. It ate up some 10s of my sleep-hours and a few man-weekends Finally, yesterday I gave up(damn it! set IDontCare=true) on my FindBestHost greedy approach and purchased a hosting package.
A few suggestions for those looking out for webhosts:

1. Apply filters
   Define your purpose. Why you need a web host? What kind of websites you plan to host? Any special requirements? How much resources(space/bandwidth) do you think you’ll spend?
   As an example, here are my answers: I need a webhost to host a blog(pleaseeeee not one more!). Mostly the website will be personal and will host my dev work, some projects. Since I am a self-proclaimed web hobbyist, I would need flexibility in terms of scripts (RoR?). At max, I will put in 5-10 domains in the web space. Considering a GB per site, will take up 5GB and assuming a moderate crowd will use my web resources, lets allow 40GB per domain per month, that would make bandwidth around 200GB per month.
2. The search begins!
   Use your favorite search engine. Ask around. SetGTalkStatus/Twitter/Spam/Scrap/Write-on-wall your contacts. If they can refer, your hard job is done. Don’t waste time on more search. Jump to the next step.
   In case you’re shy to ask people, you can look in some good webhosting forums (e.g: WebHostingTalk). They’re good. Lots of webhost guys roam around there. I got around 14k results for a simple search query “recommend host” I think first 3-4 pages should suffice. This is a good starting place as well.
3. Test the host!
   At this time, we should have a shortlist of 4-5 hosts. Now for each host try this:
   • Go to Webhosting.info. Search for the webhost. Have a good look at the number of domains coming into the host and going out each week. You can country wise breakup of top hosts. For me the most important criteria was domain in/out analysis and total domains hosted.
   • Do a “You suck” test. Query your favorite seach engine for negative reviews of the host. E.g: Go to your favorite search engine. Query for “[webhost] sucks”. Safely ignore the [Web Host Name]reviews.com(e.g: [TheBigCheapAndBestWebHost]Reviews.com) websites. A good choice would be query in Blog Search.
   • Consider having a chat with the sales/support people at the webhosting live chat. Sometimes the time they make you wait will immediately give you a feel of their customer service
4. Beware of these!
   • The “foobar” offers. Before you sell out cash on *any* offer that gives you substantial discounts, make sure you’re aware of their normal(non-foobar) pricing. Unless you plan to change host every billing cycle, you will be charged against the normal pricing. E.g: A webhost offers 4.95USD as foobar price and normal pricing is 7.95USD.
     And I never understood the rationale behind the *.95USD pricing rule.
   • Overselling. I was really astounded at the number of companies offering unlimited resources. So I went around bugging their sales/support guys on this. They say you can’t store personal files(Does it translate to “all your files must be web accessible”?). I doubt if they have shared hosting customers who distributes legal 1gig files! I wish I could copy/paste the chats I had with these guys Clearly these people are overselling. They’ll put in your site in some server with probably a few hundred more poor domains. IMO unlimited bandwith can make sense for people who publish streaming content. But does the resource usage policy of these shared hosts allow you to stream enough?
     I would suggest to take a limited resource plan. I agree we all love greedy algos, but lets truly have an estimate of the resource usage before getting lured away on such offers.

Hope this might save a few minutes while deciding for a webhost. Good luck!

E17 stores the background images in edje file format. Edje (*.edj) is a binary file format which compresses the images quite a bit and has many additional features (animations, layers etc.). Creating an animated background for e17 is quite easy (check here). There is already the wallpaper module in e17 which can set the current wallpaper to any image.

What if we want to write a cron job to switch the wallpapers (think slideshow module?). Adding a bit more complexity to it : e17 doesn’t like fake transparency and may not change its mind in near future (source) and our terminals(mrxvt?)/system tray(stalonetray) need fake transparency. There’s a nifty hack called Esetroot which sets fake transparency but doesn’t change the e17 background.

Even there’s a way to achieve both : Install the e_utils package and photo module. Right click on any image in photo module and you can set it as your background.Actually the photo module calls e17setroot program in e_utils and sets the background and then calls Esetroot to set fake transparency. If you already have e_utils up and running, you don’t need to read further

A script to emulate e17setroot

• Get the files : e17setroot and edc_skeleton
• Put e17setroot in $PATH (e.g: in ~/bin) and change the line #31 to the location of edc_skeleton file
• To set your wallpaper use e17setroot null /path/to/image/file

How this works ?
The main script e17setroot uses edje_cc to create the edj file. Edje (edje_cc) in fact needs an .edc file which contains few parameters to create the edj file. Our script takes the skeleton of an .edc and replaces the image name. And the resulting edj is placed in the ~/.e/e/backgrounds directory.

/me leaves the use of e17setroot to /reader’s creativity

Add the following lines to the ~/.Xdefaults to get a transparent urxvt terminal:
urxvt*depth: 32
urxvt*background: rgba:0000/0000/0000/dddd

The depth option tells urxvt to use 32 bits to define a color. Generally a color is represented as 24 bits (8 bits each for Red. Blue and Green). We add 8 more bits to store the Alpha(transparency) information. More info

Then we set the background option to denote the 4 different color information : Red/Blue/Green/Alpha.

To change the shading change the Alpha value in background option. 0000 = full transparent and ffff = no transparency.

For e17 users: Use the bling module for composite effects.

DNS(Domain Name System) is used to get the ip addresses of the domains. Basically whenever we enter an address in our browser, the browser sends a request to the “Nameserver” for the ip address of the domain. After the nameserver returns the ip address, the browser opens sockets(http, ftp etc) to the particular ip address and sends data to the server and waits for response. On response from the server, the browser shows us the content. They teach all this in a semester CS course

Lets get into the “Name Resolution” part once again. So our browser would send a request to the Nameserver for the ip address of the domain we type in the address bar. A Nameserver is a box in your local network/ISP’s network which listens on a certain port for DNS requests and responds to each request with the IP address of the domain. If your browser gets stuck in the “Looking up hostname : xxx” part, then know that your ISP sucks and the nameserver box is too busy to cater to your browser’s name resolution request

Generally what ISPs do is : forward all their clients dns requests to nearest high level nameserver. And each of these requests by the clients are cached in the ISP’s nameserver to fasten up queries about the same domain name. So if some guy A queries up the ip address of “www.google.com”, the first time nameserver would forward this request to another high level nameserver and the return the ip address. But when another guy B asks for the ip address of “www.google.com”, our nameserver would return the ip address from its own cache.

But by whatever misses, there are times when you know that your ISP sucks. We will look into a few solutions to this :

1. Use an alternative Nameserver : This is where the OpenDNS people have done an amazing work. To use their nameservers, just put these two lines in your /etc/resolv.conf
   nameserver 208.67.222.222
nameserver 208.67.220.220
nameserver your_ISP's_nameserver_here

2. Use a local DNS cache : Ah, this is the topic of this post. I’ll show the procedure for Arch Linux and OpenSuSE. So lets start -
   • Install BIND : They call BIND as Berkeley Internet Name Domain. For us BIND is a simple daemon program which will listen for dns requests on any box and send the ip addresses. Our idea is to run it in “localhost”, i.e, my own box and use my own box as my nameserver. This would save me the name resolution query time.
     For Arch : pacman -S bind
     For OpenSuSE : search for package “bind” in Yast and install it.
   • Setup the forwarders : Now localhost must access some highlevel nameservers to get the ip addresses of the domains requested for the first time. After getting the ip address, it is stored in BIND’s cache. To setup the highlevel nameservers (they call them forwarders) :
     * Open /etc/named.conf as root.
     * See the “options” block and insert/overwrite this line in there :
       forwarders { 208.67.222.222; 208.67.220.220; isp_nameserver; };
     Well
   • Start BIND as a service everytime system starts :
     For Arch : Change this in /etc/rc.conf Add “capability” to the MODULES list and add “named” to the DAEMONS list. Here’s my rc.conf :
     MODULES=(blah blah blah soundcore capability)
DAEMONS=(syslog-ng network named !netfs !crond alsa mpd)
     For OpenSuSE : Type in yast2 runlevel in terminal with root. Then change the runlevel settings for “named” there.

Blogged with Flock

Tags: opensuse, archlinux, dns, bind, named, resolv, name+resolution

This article will describe how to send packets destined to port x through a port y. Let me start off with the situation that led to port redirection stuff (and hence this article)

The Problem
I have an application X which connects to a remote website xyz.com on port 5000 and then allows me to do some data transfer. However my system is behind a firewall that blocks all outgoing connection on port 5000. Fortunately the website xyz.com accepts incoming connection on port 23 (which is surprisingly not blocked by my university firewall). Thats cool, I can connect to xyz.com on port 23 and do whatever. But the application X doesn’t allow me to change the port settings, i.e, X only connects to xyz.com:5000 ..so sad. And X is “closed source”

The Idea
Well if by some mechanism I can redirect all my outgoing connections to xyz.com on port 5000 to xyz.com on port 23 before the packets leave my system, then my problem should be solved. In other words, I’ll change the destination address of all packets to xyz.com from xyz.com:5000 to xyz.com:23 !!
I think using a proxy(e.g : tinyproxy) in my localhost will do that. But running a new proxy process for just this small application X din’t seem to be a good idea.
Aha ! here comes the solution (yet another *nix oneliner )
Note : In case anything goes wrong just delete the rule and do an iptables-save
iptables -t nat -A OUTPUT -p tcp -d a.b.c.d --dport 5000 -j DNAT --to a.b.c.d:23
iptables-save

Explanation
-A : Adds a chain-rule to the nat table
OUTPUT : For altering locally-generated packets before routing
-p : Specifies the protocol
-d : Give the destination ipaddress
–dport : Destination port
-j : Jump if the rule matches
DNAT : It specifies that the destination address of the packet should be modified
–to : The destination ipaddress:port

A few more oneliners (just for reference, in case i might forget them..)

• To DELETE the above rule
  iptables -t nat -D OUTPUT -p tcp -d a.b.c.d --dport 5000 -j DNAT --to a.b.c.d:23
• To check how many packets are redirected
  iptables -L -t nat -nv

Note: In this article, xyz.com = freechess.org

Tags: iptables, linux, network, packet, redirect, freechess.org, portredirect, dnat, nat, tcp

In this article I will outline a few methods to download YouTube Videos to your system and getting them to play in your favorite player. We will go in a step-wise way :

A. Getting the Video

1. Online Tools

   There are some sites like KeepVid.com which allow users to input a YouTube link and they would give you the download link. So get the download link from this site and download the flv file.

2. Firefox Plugin
   There is a firefox extension which allows you to download YouTube videos as you watch them. I haven’t tried this.
3. Command Line Tools
   This is the safest and best way (for CLI junkies ). Download the script YouTube-dl (thanks to Ricardo Garcia Gonzalez for this nice script). Just put that script in your ~/bin or /usr/bin directories or somewhere in the application path.
   To download the flv video do this :
   youtube-dl
e.g : youtube-dl http://www.youtube.com/watch?v=D1R-jKKp3NA

B. Playing the video

1. Use VLC Player or MPlayer or Kaffeine to play the flash video.
2. If the above method doesn’t work out (It din’t in my case)..Then go the difficult way :
   • Get ffmpeg for your system. SuSE users can get it here.
   • Convert the flv video to avi (or any other format) using ffmpeg :
     ffmpeg -i my_flash_video.flv the_avi_file.avi
   • Now drag and drop the avi file into your favorite video player n watch the video

Still unsatisfied ?! “Dude I’m a *nix user. I can’t sit and do all these steps to get a simple video playing !!” No probs. We will write a shell script to do all this work for you.

Note : I found the downloader in youtube-dl quite slow. So I tweaked it a bit(I don’t know python ) to use an external downloader (axel rocks!). Open the file youtube-dl in your favorite editor and change the following lines :

282 # Abort here if in simulate mode
283 if cmdl_opts.simulate:
284     sys.exit()

to the following :

282 # Abort here if in simulate mode
283 if cmdl_opts.simulate:
284     cond_print(video_data.geturl())
285     sys.exit()

That will make the script output the downloadable video link to the standard output (when run in simulate mode). Now write the shell script :
#!/bin/bash
if [ $# -ne 2 ]
then
     echo "Usage : $0 <youtube_file_link> <output_file_without_extension>"
     echo "e.g : $0 http://www.youtube.com/watch?v=D1R-jKKp3NA steve_jobs"
else
     outputfile=".avi"
     todnload=`youtube-dl -s $1 | tail -n 1 -`
     echo "Got the file.."$todnload
     axel -n 100 $todnload -o $2".flv" # wget -c $todnload -O $2".flv" .if you don't use axel
     echo "Download Completed..."
     ffmpeg -i $2".flv" $2$outputfile  # get the avi file
fi

Done. Now paste that code into a file utube in your path. Then run utube as :
utube http://www.youtube.com/watch?v=D1R-jKKp3NA steve_jobs

This will give 2 files steve_jobs.flv and steve_jobs.avi
Just drag the *.avi to kaffeine and enjoy the speech by Steve Jobs !

Technorati Tags: youtube, google, linux, axel, download, videos, shell, cli, command line,