Linux Journal/2008.12

Linux Journal Contents #176, December 2008

Features

• Hacking the Nokia Internet Tablet by Bill Childers

It's not just an ordinary PDA; check out some cool things the Nokia Internet Tablets can do!

• The BlackBerry in a World without Windows by Carl Fink

Sync your BlackBerry with Evolution.

• A Look at the Kindle by Daniel Bartholomew

It runs Linux, and it's hackable.

• Linux Device Roundup by James Gray

The world of Linux devices is becoming ever more dynamic and interesting.

Letters - HP Media Vault Review

Greetings. I just received my October 2008 issue of LJ, and one of the wonderful articles I saw was the review of the HP Media Vault mv series product (mv2xxx and mv5xxx products). Having played with one for a few months now, I was surprised at the amount of research that did not go into the review.

For example, take the failure to mention the rather extensive hacking guide posted at www.k0lee.com/hpmediavault and written by one of the HP engineers responsible for this product. How can a review of these devices fail to mention this site? It has links to the source code for the product, how to replace a drive, re-flash instructions and so on.

Otherwise, it's nice to see an open-source-friendly NAS being reviewed—especially one that is open and hackable.

UpFront - diff -u: What's New in Kernel Development

kernel drivers

Jonathan Corbet has written lots of extremely useful kernel documentation, including O'Reilly's Linux Device Drivers. Recently, he wrote a fairly long intro to kernel development, intended for developers employed by companies who support their kernel work. The goal is to make sure those companies understand what to expect from the developer community and from the relationship between their engineers and that community. This is an excellent document, filled with detailed advice and explanations to help newcomers understand how best to get their features into the kernel. Jonathan has submitted the work for inclusion in the Documentation directory of the kernel sources, though it also may appear on kernel.org at some point.

firewire wiki

The old FireWire wiki, having been overrun by spammers, is being replaced. Stefan Richter created ieee1394.wiki.kernel.org, which is already more up to date than the old spammy one, and it's better maintained as well. Those pesky spammers! When we all have nanotech brain implants, will the spammers get into those as well?

UpFront - The Dell IdeaStorm Index

The Dell IdeaStorm site (www.dellideastorm.com) was an inspired move by the company, providing a way for the market to tell a major supplier what to do, rather than the reverse, which has been the default for the whole Industrial Age.

When the site first went up, it sustained what we might call an Insistence on Service Attack by Linux and open-source geeks. Since then, however, the pressure hasn't let up. At the time of this writing (on September 10, 2008), the same kind of demand is there.

UpFront - They Said It

Our commitment to Linux has not changed....What's changed is that customers will no longer be able to order Lenovo ThinkPads and ThinkCentres with pre-installed Linux via the lenovo.com Web site.

—Ray Gorman, Lenovo spokesman, in an e-mail to Computerworld, computerworld.com/action/article.do?command=viewArticleBasic&taxonomyName=hardware&articleId=9114485&taxonomyId=12&intsrc=kc_top

Apple views tennis-shoe DRM as a way to head off what it sees as a potential plague of sneaker hacking.

—Nicholas G. Carr, www.roughtype.com/archives/2008/09/apple_declares.php

At the Forge - Memcached

Want to make sure your application will scale? Consider memcached, which allows you to speed up response time, as well as reduce the load on your database server.

One solution is to use multiple database servers. There are solutions for hooking together multiple servers from an open-source database (for example, PostgreSQL or MySQL), not to mention proprietary (and expensive) solutions for commercial databases, such as Oracle and MS-SQL. But, this is a tricky business, and many of the solutions involve what's known as master-slave replication, in which one database server (the master) is used for data modification, and the other (the slave) can be used for reading and retrieving information. This can help, but it isn't always the kind of solution you need.

But, there is another solution—one that is simple to understand and relatively easy to implement: memcached (pronounced “mem-cash-dee”). Memcached is an open-source, distributed storage system that acts as a hash table across a network. You can store virtually anything you like in memcached, as well as retrieve it quickly and easily. There are client libraries for numerous programming languages, so no matter what framework you enjoy using, there probably is a memcached solution for you.

This month, we take a quick look at memcached. When integrated into a Web application, it should help make that application more scalable—meaning it can handle a large number of users, spread across a large number of servers, without forcing you to rewrite large amounts of code.

Memcached

As I mentioned previously, you can think of memcached as a network-accessible hash table. Like a hash table, it has keys and values, with a single value stored per key. Also like a hash table, there aren't a lot of ways to store and retrieve your data. You can set a key-value pair; you can retrieve a value based on a key, and you can delete a key.

This might seem like a limited set of functions. And, it is, if you think of memcached as your primary data store. But, that's exactly the point. Memcached never was designed to be a general-purpose database or to serve as the primary persistent storage mechanism for your application. Rather, it was meant to cache information that you already had retrieved from a relational database and that you probably were going to need to retrieve again in the near future.

In other words, memcached allows you to make your application more scalable, letting you take advantage of the fact that data is fetched repeatedly from the database, often by multiple users. By first querying memcached and accessing the database only when necessary, you reduce the load on your database and increase the effective speed of your Web application.

The main cost to you is the time involved in integrating memcached into your application, the RAM that you allocate to memcached and the server(s) that you dedicate to memcached. How many servers you will want to allocate to memcached depends, of course, on the size and scale of your Web site. You might need only one memcached server when you start out, but you might well need to expand to ten, 100 or even several hundred memcached servers (as I've heard Facebook uses) to maximize application speed and efficiency.

Using Memcached

I was able to install memcached with:

apt-get install memcached

Then, I started memcached with:

/usr/bin/memcached -vv -u reuven

The -vv option turns on very verbose logging, allowing me to see precisely what is happening from the server's perspective. The -u flag lets me set the user under which memcached will run; it cannot be run as root, for security reasons.

memcache port:

localhost:11211

You can specify one or more servers; in this case, we indicate that there is only one, running on localhost, on port 11211. It might surprise you to learn that although memcached is described as a distributed caching mechanism, the various memcached servers never speak to one another. Rather, it is the client that decides on which server it will store a particular piece of data, and it uses that same algorithm to determine which server should be queried to retrieve that data.

What happened to our value? Didn't we set it? Yes, we did, but we told memcached to expire the value after three seconds. This is one important way that memcached makes it easy to be integrated into a Web application. You can specify how long memcached should continue to see this data as valid. By passing no expiration time, memcached holds onto the value forever. Allowing the data to expire ensures that cached data is valid.

It might seem strange for me to be describing memcached as a repository for complex objects, such as orders or people. And yet, memcached is fully able to handle such objects, assuming they are marshaled and unmarshaled by the client software.

Resources

The home page for memcached is at www.danga.com/memcached. This site contains links to software (server and client), documentation and articles about memcached.

Cooking with Linux - Really Useful Gadgets...Sort of

BSOD generator

the Blue Screen of Death, lovingly crafted for Linux by Folkert van Heusden.

Get the source from www.vanheusden.com/bsod, extract it and then simply type make (or make install) to build it. To run it, type bsod. Your console, or terminal window, displays the Blue Screen of Death. There are no options or flags, so it's very easy to use.

ASCIIQuarium

it's nowhere near as interesting as, say, watching fish in an aquarium. We've got that taken care of as well with our next gadget. In keeping with our low-tech, low-end gadget needs, this aquarium doesn't require a graphics card. It's Kirk Baucom's ASCIIQuarium (Figure 3). The program displays a variety of fish, the occasional sea monster or man-eating shark, all in glorious ASCII.

There is no building to be done because ASCIIQuarium is a single Perl script. It requires only that you have the Curses and Term::Animation Perl modules installed.

xfishtank

If ASCII seems just too, ahem, quaint for a desktop gadget, you'll be happy to know that you can get a different kind of aquarium with some nicer graphics. Most modern software gadgets tend to be small programs that run on your desktop background or wallpaper. Sometimes they become the background. One such program is xfishtank (Figure 4), written by Eric Bina. Once again, this is an easy program to find in your distribution's repositories.

Running xfishtank on most systems is as simple as typing the program name. You also can fire up your program launcher (Alt-F2), and type xfishtank to populate your aquarium. Whether you see something right away depends somewhat on the desktop environment you are running. Most environments, GNOME included, don't require any additional steps, but KDE does need to check with you before allowing programs to run on the desktop background. Right-click on your desktop, and select Configure Desktop from the pop-up menu. When the dialog box appears, click the Behavior icon in the left-hand sidebar. A three-tabbed window appears on the right-hand side. Look near the top on the General tab, and you'll see a check box with the words Allow programs in desktop window. Click that check box, and then click OK.

I mention this now because you may need it again with some of our other gadgets. A lot of the newer background gadgets are small programs that take up a small portion of your screen, quietly displaying useful information, such as system load, memory usage or network traffic.

xpenguins

wouldn't you rather see Tux running around on your screen, walking across your windows, skateboarding or parachuting down to your taskbar? Me too. You can thank Robin Hogan for writing xpenguins to help us out of that productivity conundrum. When you run xpenguins, Tux, in all the forms I mentioned, suddenly takes over your screen

Should you decide your screen isn't busy enough, you can increase the default number of penguins by using the -n flag. That default is defined in the current theme. Theme? Did I say theme? If those wonderful little penguins vying for your attention aren't enough, you are running the right program, mes amis. One of the really fun things about xpenguins is that it comes with multiple themes. To discover those themes, type xpenguins -l at the command line:

xpenguins -l

Bill, the famous hacker from Red Mond, wanders across your screen taking away Linux systems and replacing them with his own brand of OS. Yes, this is a takeoff of the (in)famous xbill game:

xpenguins -t Bill

Even more themes are available. Visit the xpenguins Web site, and check out the user-contributed themes at xpenguins.seul.org/contrib. Before we move on, I want to mention one last flag available with xpenguins—the -s flag. That one makes it possible for you to squish the characters with your mouse cursor. If you find yourself a little squeamish at the result, the -b option means no blood.

KDE Plasmoids

Let's get off the nostalgia bus now and take a look at the modern state of desktop gadgets. KDE 4.1's impressive desktop features a new desktop shell called Plasma. Plasma is, in a way, the ultimate gadget—it's a gadget that runs gadgets. Inside Plasma, you run programs (or widgets or gadgets) that appear on the desktop. Each of these programs is commonly referred to as a plasmoid. Plasmoids are more than programs, however. Each is a containment that can contain other plasmoids, all of which are, technically, able to communicate with one another. Plasma, the desktop shell, is just one big containment that contains other plasmoids. The panel at the bottom of the screen with its system-tray icons, taskbar, clock and program launcher is yet another containment. Shakespearean fans can think of it as a play within a play.

Plasmoids use scalable vector graphics (SVG). These graphics can be zoomed and rotated smoothly, meaning that plasmoids can live pretty much anywhere on your desktop, in any size and any orientation. The result is super-sweet eye candy of the gadget variety.

Note: Yes, we have a real issue with this whole gadget thing. It's called language. Some people call them gadgets, and others refer to them as widgets. KDE 4.1 calls them both widgets and plasmoids. Other environments refer to these things as toys. Gadgets and widgets and plasmoids and toys, oh my!

To add a plasmoid to the KDE 4 desktop, click on that cashew icon in the top right-hand corner of your screen. A small pop-out menu appears. If it says Unlock Widgets, make sure you click that first, then recall the menu. Now, you should see Add Widgets at the top of that menu.

When you click Add Widgets, a window labeled Add Widgets appears (Figure 9). It contains a list of all the plasmoids installed on your system, and each one has a description below its name. Some of my favorites include Dictionary, a live desktop word lookup; Luna, a moon-phase display; and the Twitter Microblogging applet. I also enjoy a variety of clocks, including a classic analog clock as well as a binary model. Those little yellow sticky notes also are handy. There's even a plasmoid that pulls in and displays your favorite comic strips right on your desktop. Figure 9 shows a number of different plasmoids running on my desktop.

While the plasmoids are unlocked, you can pause over any of them to fade in the controls (Figure 10). Each has a rotate handle, a resize handle and a button to close the plasmoid. Many, though not all, also are configurable and offer a settings icon.

Running all these cool desktop gadgets is great, but what if you've got a dozen windows open, and you want to re-read today's comic? Minimizing all those windows can be a pain, but it's one you don't need to suffer. Press Ctrl-F12, and the Plasma dashboard jumps to the forefront of your running windows, letting you see and interact with any of your plasmoids.

Google Gadgets

The last item on tonight's menu comes from those gadget-crazy people over at Google who come to us with the aptly named Google Gadgets. Unlike plasmoids, you can't rotate them, and they live only on your current virtual desktop, but the sheer number of gadgets, not to mention coolness factor, makes Google Gadgets a must. I was able to install Google Gadgets for my system from the Mandriva repositories, so check yours first. You also can get the latest from code.google.com/p/google-gadgets-for-linux.

When you install Google Gadgets for Linux, you'll find that there are two versions of the code: one for the Qt toolkit (KDE) and another for GTK (GNOME). When you first run the program (with a shortcut command named ggl), an icon appears in your system tray. To add gadgets to your desktop, right-click the icon and select Add Gadgets. Figure 11 shows a sampling Google Gadgets running on my desktop. There's a nice flowerpot that requires you to water and care for the flowers in order for them to grow (ignore the flowers and they wither and die). If, like me, you never can have enough trivia, check out the Absolut Trivia gadget (yes, that Absolut), which displays a new piece of trivia every few seconds. To help me make decisions, I've got a Magic 8 Ball. The weather, always important, shows up in a cool weather globe. And, of course, when I've been working too long, the RSI Break gadget tells me to take a break.

Resources

ASCIIQuarium: robobunny.com/projects/asciiquarium/html

BSOD: www.vanheusden.com/bsod

Google Gadgets for Linux: code.google.com/p/google-gadgets-for-linux

KDE Plasma Wiki: techbase.kde.org/Projects/Plasma

xcockroach: xcockroach.free.fr

xpenguins: xpenguins.seul.org

Marcel's Web Site: www.marcelgagne.com

Tech Tip - Speed Up Multiple SSH Connections to the Same Server (pg 31)

If you run a lot of terminal tabs or scripts that all need to make OpenSSH connections to the same server, you can speed them all up with multiplexing: making the first one act as the master and letting the others share its TCP connection to the server.

If you don't already have a config file in the .ssh directory in your home directory, create it with permissions 600: readable and writeable only by you.

Then, add these lines:

Host *
   ControlMaster auto
   ControlPath ~/.ssh/master-%r@%h:%p

ControlMaster auto tells ssh to try to start a master if none is running, or to use an existing master otherwise. ControlPath is the location of a socket for the ssh processes to communicate among themselves. The %r, %h and %p are replaced with your user name, the host to which you're connecting and the port number—only ssh sessions from the same user to the same host on the same port can or should share a TCP connection, so each group of multiplexed ssh processes needs a separate socket.

To make sure it worked, start one ssh session and keep it running. Then, in another window, open another connection with the -v option:

~$ ssh -v example.com echo "hi"

And, instead of the long verbose messages of a normal ssh session, you'll see a few lines, ending with:

debug1: auto-mux: Trying existing master
hi

Pretty fast.

If you have to connect to an old ssh implementation that doesn't support multiplexed connections, you can make a separate Host section:

Host antique.example.com
   ControlMaster no

For more info, see man ssh and man ssh_config.

-- Don Marti

Work the Shell - FilmBuzz Trivia Goes Live

Last month, I ended by showing you a rudimentary solution to sending out twitters on the command line that looked like this:

#!/bin/sh
# tweet - command line twitter interface
user="filmbuzz"; pass="acctpasswd"
msg=$(echo $@ | sed 's/+/%2B/g;s/ /+/g')
$curl --silent --user "$user:$pass" --data-ascii \
  "status=$msg" "http://twitter.com/statuses/update.json" \ 
  > /dev/null

With that available, sending Twitter updates is as easy as typing:

$ tweet "My favorite film? Probably Lawrence of Arabia"

Paranoid Penguin - Samba Security, Part II

See Part I [1]

See Part II [2]

See Part III

Build a secure file server with cross-platform compatibility.

Global Settings

Continuing through these global settings, obey pam restrictions implies that Samba will honor PAM (Pluggable Authentication Modules) settings. But in practice, if encrypt passwords remains set to yes, Samba will ignore PAM altogether.

passdb backend specifies what type of database Samba should use to store its password hashes. The default (tbdsam) is usually the best choice.

how

guest account is the local Linux account that will be used for clients who fail authentication, as I described earlier when talking about map to guest. passwd program, passwd chat and unix password sync involve how and whether Samba mediates end users' attempts to change their passwords via Samba (Windows file sharing) sessions. Leave these at the default settings unless you don't want users to be able to change their passwords that way.

password change

By now, you may be wondering, what's the difference between Samba's password database and the list of hashes stored in /etc/shadow, given the fact that they correspond to the same set of local user accounts? The short answer is, Samba (SMB/CIFS) uses an authentication protocol with which UNIX password hashes are not compatible.

The bad news is that Samba's password database is, thus, totally redundant with Linux's, and it creates the potential for users having to remember two different passwords. The good news is that if passwd program and passwd chat are set correctly (which they should be by default, if you use your Linux distribution's official Samba packages), and unix password sync is set to yes, Samba automatically will update users' Linux passwords every time they change their Samba password. (I talk about this more in the next section.)

Moving on, valid users allows you to specify a list of Linux/UNIX user accounts to which you want to grant access to Samba shares. The default value "" (null) results in all local Linux accounts being valid. For our example scenario, I've set valid users to mick, knute, pepe, skippy and nobody.

admin

admin users allows you to grant superuser privileges on all shares for one or more local user accounts, regardless of Samba or Linux file permissions on that share. Be careful with this setting! It has the effect of executing local commands as root on behalf of such users. In Figure 2, I've specified mick as an admin user, because I often use that account for system administration tasks anyhow.

read list allows you to specify which users should have default read-only permissions on shares. As you can see in Figure 2, I've set our read list to knute, pepe and skippy.

Similarly, write list specifies a list of users who should have read-write privileges by default. I've set that value to mick.

host deny

printer admin is out of the scope of this article for now (though I may cover printer shares later in this series). hosts allow and hosts deny, however, are noteworthy. They allow you to create TCP Wrappers-style access control lists. hosts allow is a whitelist of IP addresses, network addresses, hostnames or domain names that should be allowed to connect by default (assuming successful authentication, of course).

hosts deny is a blacklist, also consisting of IP addresses, network addresses and so forth, whose members won't even be permitted to attempt authentication. Samba will break any connection attempted by any host matching this list. The hosts_access(5) man page provides complete information about the syntax of the values of these two variables.

Setting Up User Accounts

For example, to create Pepe's account, I could use the following command. Note the sudo, necessary for Ubuntu. On other distributions, su to root before executing these commands, and omit the sudo that each begins with here:

bash-$ sudo useradd -c "Pepe" -m -g users pepe

This creates the user account pepe with the comment Pepe, automatically creates a home directory (/home/pepe) and assigns it to the group users. To be extra paranoid, you could insert the string -s /bin/false after -g users, which will disable normal Linux logins for Pepe's account, making it useless for anything other than Samba access.

Step two is to set each user's Linux password, like this:

bash-$ sudo passwd pepe

Obviously, you need to communicate whatever password you set here to Pepe in a secure fashion, and Pepe will need to change this password to something you don't know. (But that part happens in step four.)

Step three is to use the smbpasswd command to create each user's Samba password database entry, like so:

bash-$ sudo smbpasswd -a pepe

Finally, you'll want Pepe to log in to the system (assuming you didn't set his shell to /bin/false) and issue the following command:

pepe@casademick$ smbpasswd

Pepe will be prompted for his old password, his new password and confirmation of his new password. Assuming all three of those are good, Samba will change both Pepe's Samba password and his Linux password accordingly. Note that this synchronization does not occur when you create a new Samba password entry as root, using the -a flag.

Kenneth Question - how do you change password remotely??

Resources

Christopher R. Hertel's On-line Book Implementing CIFS, a Comprehensive Source of Information on All Things CIFS/SMB-Related: www.ubiqx.org/cifs

“The Official Samba 3.2.x HOWTO and Reference Guide”: us1.samba.org/samba/docs/man/Samba-HOWTO-Collection

Terracotta

Terracotta

"Terracotta is an open source clustering product for Java. With Terracotta, JEE Applications scale simply and reliably without databases, EJBs or other complex infrastructure."

Terracotta

The open-source Java clustering solution Terracotta has added a tick to the tenths column, landing at Version 2.7. The makers of Terracotta claim their application lowers costs and simplifies Web application deployment by reducing development effort and easing the load on application servers and databases, making it a solution well suited for scaling critical applications. Because Terracotta offers “the performance of local memory with the high availability of a database”, it eliminates the “unyielding performance and reliability trade-offs that constrain many Java applications today”. Version 2.7 also extends support of the Spring framework and the Glassfish application server, plus features that enhance scalability, performance and operational visibility.

www.terracotta.org

OpenNetAdmin—AJAX/Browser-Based Network Manager

OpenNetAdmin—AJAX/Browser-Based Network Manager (opennetadmin.com)

Network administration made clean and simple with OpenNetAdmin.

Born out of dissatisfaction with expensive commercial tools and the direction taken by most network admin projects, OpenNetAdmin (ONA) takes a different approach to network administration while making the task of administration a little bit nicer in the process. Project founder Matt Pascoe found commercial tools, such as Lucent QIP, Infoblox and Bluecat, to be okay, but they're expensive and clunky for certain tasks, and they don't follow the *nix principal of modular functionality. All of the open-source tools he found, such as IP-Plan/IP-Track, had big usability issues, and the Java interfaces always annoyed him, so a Web-based AJAX interface made more sense. After coming up with a bunch of cool ideas and methods with his former coworkers, Matt couldn't let all of them go to waste, so he re-created his own variant that would work in a general sense for the Open Source community.

ONA is meant to play a more authoritative role in your environment. Many tools want to go into a discovery mode and tell you what is in your network, while all the time adjusting your data. In contrast, ONA tells the network what it should have in it. This way, you can (hopefully) trust your own data to help you configure your environment the way you want it, but still utilize things like DHCP and its dynamic nature. ONA also is designed to help with auditing your network, and it's geared toward helping configure your routers/switches/firewalls/nagios/cacti or pretty much anything for which you want to create an output template. The GUI also is an important element of ONA, designed to flow easily with familiar elements, such as pop-ups, search as you go and so on.

Installation

First, you need a basic LAMP installation of Apache, MySQL and PHP, or you'll be going nowhere fast. Matt recommends installing the following packages:

• libapache2-mod-php5
• php5-mysql
• php5
• apache2
• mysql-server

Once you've got the LAMP side of things sorted out, head to the ONA Web site, grab the latest tarball and save it somewhere locally. Once the download has finished, open a terminal in the directory where you saved the tarball, and enter the following commands as root or using sudo:

# tar -C /opt -zxvf ona-v00.00.00.tar.gz
# ln -s /opt/ona/www /var/www/ona
# touch /var/log/ona.log
# chmod 666 /var/log/ona.log
# chown www-data /opt/ona/www/local/config 

And, for those who want to jump in and try it without going through all the nasty installation stuff, check out the on-line demo (demo.opennetadmin.com).

Kuklomenos

Kuklomenos—Weird Space Shooter? (mbays.freeshell.org/kuklomenos)

Kuklomenos has to be one of the weirdest games I've ever come across. I'm not doing a full review here, because I simply can't figure it out! If you want to compile it, go right ahead; it's easy with the usual ./configure, make, make install routine. But, once you get into the game, be prepared to be puzzled. I think it's a space shooter, because the background is black, but that's just a guess! Your goal is to fight off blobs with a strange control scheme that involves zooming, rotation and fire accuracy—all in a playing style that's like Asteroids rewritten by a French existentialist on crack. Intrigued? Check it out!

GnoMint

GnoMint—Graphical Certification Authority Management, X.509 (gnomint.sf.net)

Unfortunately, I couldn't bring this project to you properly this month, as project maintainer David Marín Carreño ran into some security holes that needed plugging before it was ready for the mainstream. However, the end results of this project look promising. Imagine you're trying to establish an IPsec VPN. If you want some actual security, you need X.509 certificates for all employees. At this point, you either can buy the certificates from an external CA (which costs money), or you can establish your own CA. Establishing your own CA always has been a bit of a pain. With OpenSSL, you need to use a console and log commands with a lot of obscure parameters. You could use other programs for managing CAs, but most of them are Web-based, and all you really want is a simple GUI application. GnoMint steps up to the plate here with a simple app that fits the bill nicely.

Tech Tip - Treating Compressed and Uncompressed Data Sources the Same (pg 56)

Occasionally, you need to process a number of files—some of which have been compressed and some which have not (think log files). Rather than running two variations, one compressed and one not, wrap it in a bash function:

function data_source ()
{
 local F=$1

 # strip the gz if it's there
 F=$(echo $F | perl -pe 's/.gz$//')

 if [[ -f $F ]] ; then
   cat $F
 elif [[ -f $F.gz ]] ; then
   nice gunzip -c $F
 fi
}

which nicely allows:

for file in * ; do
 data_source $file | ...
done

Whether you're dealing with gzip'd files or uncompressed, you no longer have to treat them differently mentally. With a little more effort, bzip files also could be detected and handled.

-- David Sinck

The Power workstation is back

Power Up!

A Review of the YDL PowerStation.

On June 6, 2005, hell froze over, and Apple announced it was abandoning the PowerPC architecture it had helped develop in favor of processors from Intel, a company Apple had actively mocked for years. By August 2006, the transition was complete, and the largest maker of computers based on the PowerPC (or Power) architecture had become an Intel-only shop.

This transition affected one company more than almost anyone else. Terra Soft Solutions of Loveland, Colorado, has been working with Linux on PowerPC hardware longer than just about anyone. Its flagship product is Yellow Dog Linux (YDL), and for years, Terra Soft's major business was selling Apple PowerPC hardware with YDL pre-installed on it. Terra Soft actually had the distinction of being the only Apple reseller authorized to sell Macintosh hardware with something other than Mac OS installed on it. With Apple now out of the picture, Terra Soft's primary business had to change.

For the past couple years, Terra Soft has focused a lot of its attention on server products from IBM and on the PlayStation 3 from Sony. Now, with the PowerStation, Terra Soft is taking a step into the hardware business Apple vacated. Its Web site says it all in a single sentence: “The Power workstation is back.”

Chips based on the Power architecture are found in many devices and products—from cars to mainframes to robots. Customers who relied on Apple for PPC-based workstation hardware were left in the lurch with Apple's Intel switch. For those that need it, being able to run PPC code without emulation on their local workstation is a big plus. The PowerStation was created to provide these developers with a high-quality open-source-friendly workstation. Not only is it more powerful than any PPC-based Power Mac from Apple, it also is more open and expandable.

The Software - Yum

The PowerStation comes with Yellow Dog Linux 6 pre-installed. YDL began life in 1999 as an alternative to the Mac OS on Apple's PowerPC hardware. It is based on Red Hat Enterprise Linux and Fedora. Like every Red Hat/Fedora derivative I have ever used, it uses RPM for package management. Yum, which started as a Yellow Dog-specific add-on for simplifying package updates (and has been adopted by most RPM-based distributions) is naturally included, along with the graphical yum updater, Pup.

A couple packages I wanted to use on the box, such as the renameutils (from www.nongnu.org/renameutils), were not available as pre-built RPM packages (as far as I could see, anyway). In the case of renameutils, I was able to download, compile and install the package manually.

One big thing I had to get used to on the PowerStation was the lack of GRUB. Yaboot is the bootloader for the PowerStation.

Being unfamiliar with Yaboot, I elected not to tinker with it or even spend much time looking at it. The system booted fine, and I didn't want to render the box unbootable inadvertently. The Yaboot configuration does look marginally similar to GRUB's, and I'll leave it at that.

The Performance - Phoronix test suite

For testing, I installed the Phoronix test suite. Unfortunately, although I was able to install it without trouble by following the directions on the Phoronix Test Suite Web site and run most of the tests, a few of them, including compiling the Linux kernel and calculating Pi to 32 million digits, failed. In the case of the Linux kernel compilation test, Phoronix reported that the test completed in 4.12 seconds. This compares to a time on my laptop of 4,407.53 seconds. Now, I am the first to admit that the PowerStation is much faster than my old laptop, but it is not a thousand times faster.

root

The reason I needed the root password brings me to my last issue. Admittedly, this is in the realm of stylistic preference and not a “real” issue. Whenever you run an application that requires root privileges, you actually have to enter the root password. I never have liked this way of doing things. A much better option, in my opinion, is to have admin-level users run admin programs using sudo or gksudo. The fewer the number of people who actually know the root password, the better. I'm happiest when I never have to use the root password or log in as root. As I said before, this is more of a style issue, not a problem or showstopper in any way.

Resources

PowerStation Web Site: www.terrasoftsolutions.com/products/powerstation

Tech Tip - Slice and Dice Images with ImageMagick | Linux Journal (pg 62)

You can use the convert command that comes with ImageMagick to extract parts of an image.

You can cut out a 100-pixel-wide chunk from somewhere in the middle of an image:

$ convert -crop 100x+0+0 orig/wrapperbg775.gif slice0.gif
$ convert -crop +200+0 orig/wrapperbg775.gif slice1.gif
$ convert +append slice0.gif slice1.gif wrapperbg675.gif

You can duplicate a 100-pixel-wide chunk from somewhere in the middle of an image:

$ convert -crop 100x+100+0 orig/wrapperbg775.gif slice100.gif
$ convert +append slice0.gif slice100.gif slice 100.gif
 ↪slice1.gif wrapperbg875.gif

Note that there was no need to specify the height of the image in any of the above commands. If you need to adjust the height instead of the width, the steps are similar, but use -append instead of +append to paste the slices vertically.

-- Janos Gyerik

Hacking the Nokia Internet Tablet

For Further Hackery

Getting more information on the tablets and their extensibility is easy, thanks to excellent documentation on the Internet about these units. The first stop when looking for information should be the maemo.org site. This site is the home for all things tablet-related, from the Nokia wiki, to the software repositories, as well as the software updates and source code archives. It's all on maemo.org—except for the true hacks. Those can be found in the archives of the Internet Tablet Talks forums. The people there have come up with some truly amazing hacks and other really neat ways of using the tablets, and there's something new almost every day there.

Resources

Maemo.org Home Page: maemo.org

Internet Tablet Talk Forums: www.internettablettalk.com

How to Flash the Latest Nokia OS Image: wiki.maemo.org/Updating_the_tablet_firmware

Gaining Root Access to the Tablet: wiki.maemo.org/Root_access

Booting the Tablet from a Flash Card: wiki.maemo.org/Booting_from_a_flash_card

The Palm “Garnet” VM for the Nokia Tablets: www.access-company.com/products/gvm

How to Enable USB Host/OTG Mode on an N800: www.harbaum.org/till/n800_usb/index.shtml

Using Gizmo and Grandcentral on the Nokia IT: www.internettablettalk.com/forums/showthread.php?t=14536

A Look at the Kindle

Yes, it runs Linux. Yes, you can hack it.

specs

First and foremost, the Kindle is defined by its screen. The E Ink display immediately sets it apart from LCD and CRT displays. The best word to describe it is steady. I can stare at it for hours without my eyes growing tired like they do with LCD displays. Yes, it is only black and white with a few levels of gray, but for something designed for reading, it is ideal, or nearly so. The current generation of electronic pager displays isn't perfect—the blacks aren't truly black, and the whites are more of a light gray—but it's pretty close.

The Kindle is powered by a PXA255 XScale processor and has 256MB of internal Flash memory (with 180MB available for books and other content). Under the back cover of the Kindle is an SD card slot, the reset hole and the battery.

functions

All documents on the Kindle behave more or less the same. There's no scrolling; instead, you page through the text. You can change the font size and use the scroll wheel to look up words in the built-in dictionary or follow links to other places in the document. You can bookmark a page by moving the scroll cursor to the top of its track and virtually folding down the top-right corner of the page. You also can add notes to the text and highlight passages by drawing boxes around them.

The Kindle's Features

One of the earliest complaints leveled against the Kindle was that it is tied to Amazon.com and its storefront. Along those lines, the two most common fears were “If my Kindle loses its memory, will I lose all my books and have to buy them again?” and “Is Amazon my only source for content?” The answer to both of those questions is no.

eBook sites

Several eBookstores have eBooks for free download or purchase, including ManyBooks.net, WebScription, Mobipocket.com and many others (see Resources).

DRM

Amazon adds DRM, unfortunately, to the otherwise Mobipocket-formatted eBooks it sells through Amazon.com and the built-in-to-the-Kindle bookstore. True to the real intent of DRM, this does little to stop piracy and everything to punish and annoy honest citizens. But, and this is a big one, the Kindle reads unencrypted Mobipocket files just fine. All of the sites listed above offer books in Mobipocket and other formats. My favorite of the bunch is ManyBooks.net, because it specializes in public domain books—meaning the books available for download on its site are not only free, they're also free (if you know what I mean).

wireless

One of the Kindle's neatest features is its wireless capabilities. The Kindle cannot connect to your Wi-Fi network, but it doesn't need to. Instead, it uses a built-in EVDO modem to connect to what Amazon calls its Whisper Net, but in reality, it's just Sprint's CDMA network. There is no charge for using this network, even for Web browsing. Instead, the costs are rolled in to the price of the Kindle itself, and the books, magazines and services you buy from Amazon.

send to

Actually, I shouldn't include “services” in the above list, because right now, the only service Amazon charges for is its document-conversion service. You can e-mail Word, HTML or image documents to <yourname@kindle.com>, and they are converted and sent directly to your Kindle for $0.10 each. There's also a free version where you can e-mail documents to <yourname@free.kindle.com>, and you'll get a link to the converted document sent back to you (getting it onto your Kindle is your responsibility). The yourname part of the e-mail can be set and changed at Amazon.com in the Manage Your Kindle section.

free vs paid

I tested the conversion functionality with several documents, and I tried both the no-cost and regular services. There wasn't any difference in the time it took to convert the documents. The only difference was that one was sent to my Kindle automatically and the other arrived in my e-mail and had to be transferred manually to my Kindle.

pdf not officially supported

My first test involved sending some .pdf files for conversion. The text converted fine, but I lost the graphics, some of the formatting, internal links and the .pdf's table of contents. I can't be too upset about what failed though, as .pdf isn't an officially supported file format. With all the PDF documents I have, it's nice that it works, even with some limitations.

mp3

Playing music on the kindle works for MP3 files. No other formats work. There's also no playlist support and no user interface apart from the Play button on the Experimental page. There are two undocumented keyboard shortcuts you can use: Alt-F to skip to the next song and Alt-P to Play/Pause the music. It's not the most useful of music players, but it does play music.

Hacking the Kindle

One of the more interesting pieces of code on the Kindle is BusyBox. Its presence suggests there is support for a command-line interface of some sort. It turns out there is, but it's not easy to access.

If you take the back cover off the Kindle, there is a little covered access port next to the battery. This access port can be removed with a small flat-head screwdriver. Under the cover is a small ribbon connector port, which functions as a console port. See Resources for links to the full details of the hack.

shortcuts

Finally, there are several global shortcuts that come in handy. The first is Alt-Shift-R, which reboots the Kindle. Next is Alt-Shift-., which restarts only the Kindle GUI. This last one is the most useful, for me anyway—Alt-Shift-G is a global screenshot shortcut.

Some Annoyances

The Kindle is underpowered, especially with larger books or when it's busy indexing or doing some other background task.

Next, the Kindle crashed a few times during my testing. Granted, I was running several apps that don't officially exist, but I don't feel I should have had to use the reset button as often as I did. Amazon still has some work to do there.

The Mobipocket format is another annoyance. It is an old binary format from the days when the Palm was known as the Pilot. It's not a very well documented format, and all of the tools for converting documents to it are proprietary and Windows-only. Conclusion

The big question regarding the Kindle is whether it is actually worth $350. My thought is it is, if you read a lot.

And, I do. I carry around lots of books and printouts and miscellaneous scraps of paper—some for enjoyment and many for my job. I used to try reading things on my computer, but found my eyes quickly tired, so I switched to printing out longer articles and documentation I wanted to read. Apart from being environmentally wasteful, all that loose, printed material has to be organized or it grows into a big mess.

The Kindle has eliminated a lot of the mess. Now, when I head back to the server room, the only thing I need to carry is the Kindle—no stacks of notes and no reams of product documentation. It's all in the Kindle, along with a new novel to read while waiting for the server to finish its install. And, my desk is cleaner than any time in recent memory.

Is it worth $350? For me? Yes.

Resources

Kindle Source Code: www.amazon.com/gp/help/customer/display.html?ie=UTF8&nodeId=200203720&tag=particculturf-20

A Discussion of the .mobi File Format: www.mobileread.com/forums/showthread.php?t=16514

Hacking the Kindle, Parts 1–3: igorsk.blogspot.com/2007/12/hacking-kindle-part-1-getting-console.html, igorsk.blogspot.com/2007/12/hacking-kindle-part-2-bootloader-and.html and igorsk.blogspot.com/2007/12/hacking-kindle-part-3-root-shell-and.html

MobileRead—a Forum Devoted to eBooks: www.mobileread.com

Non-Amazon Places to Get Kindle-Ready Books: manybooks.net, www.baen.com/library, www.webscription.net and www.mobipocket.com

Linux Device Roundup

Four Linux device experts offer their opinions on the state of Linux devices and tell you about their must-have favorites.

Rick Lehrbaum is the founder and editor of the popular site DeviceGuru.com, an independent blog devoted to new and emerging device technologies. In addition to founding LinuxDevices.com—now a part of DeviceForge.com—Lehrbaum cofounded Ampro Computers and consults for companies in the embedded market.

Kingman: Any Linux device released without source code or a promise to provide it is a dud in my book. There tends to be more GPL license violations in the device world, I guess because people think that no one will notice or want to modify software that's “embedded” inside a device. But, it's pretty obvious which devices out there run Linux. Usually, you can tell from a glance at the spec sheet—let alone any of the more-technical telltale fingerprints

Resources

LinuxDevices.com: linuxdevices.com

DeviceGuru: www.deviceguru.com

Bill Weinberg's Blog: linuxpundit.wordpress.com

Shawn Powers' Video Reviews at LinuxJournal.com: www.linuxjournal.com/video

Shawn Powers' Web Site, The Brain of Shawn: The Thinks I Think: www.brainofshawn.com

Netbooks to Take the Market by Storm: www.deviceguru.com/2008/08/23/netbooks-to-take-the-market-by-storm

What's the Difference between a Netbook and a Nettop?: http://download.intel.com/pressroom/kits/events/idfspr_2008/Netbook-Nettopbriefing.pdf"/>

Rick Lehrbaum's Review of the Roku Netflix Player: www.deviceguru.com/2008/05/20/100-netflix-dvd-downloader-runs-linux

Roku's Netflix Player

Roku's Netflix Player (www.roku.com/products/netflixplayer)

“Instant movie gratification” coos Henry Kingman of Roku's Linux-driven Netflix Player, a networked video device that delivers Netflix streaming content directly to your television. It provides access to a library of more than 12,000 on-demand titles from Netflix. The Netflix Player is HD-ready and has all the connections you need to connect to a TV, HDTV, home theatre or A/V receiver, including HDMI. The device includes Ethernet and Wi-Fi (802.11b/g), allowing one to play, pause, fast-forward and rewind movies directly from the Internet over a home network.

Roku Netflix

Roku has used Linux on the Netflix Player has since its inception. Roku's David Westerhoff, Director of Software Engineering, says his company chose Linux because it has “come a long way” and allows it to “focus on developing [its] application and helps keep the costs down”. Westerhoff adds that having the source code gives his team the flexibility to “go deep if necessary to debug, troubleshoot and optimize our software for the best user experience”. During product development, Roku developers found and fixed about a half-dozen distinct bugs in the build toolchain, plus some driver-specific bugs. However, the 2.6.19.1 Linux kernel has been very stable and required no modifications to the product.

The device uses the MIPS-based PNX8935 SoC from NXP Semiconductors for application and video processing. The application is written primarily in C++ and runs a Linux 2.6.19.1 kernel. Roku uses DirectFB to provide an abstraction layer for the graphics and video services on the platform and Qt 4.3 to provide a framework for UI development. The device has no hard disk, just 256MB of DDR RAM to provide the memory needed for its applications, plus the buffering necessary to support streaming video playback.

“Robust video streaming over home networks takes a significant amount of effort to get right”, adds Westerhoff. Therefore, the Player uses dynamic bandwidth detection to select the best possible stream for the user's network and then monitors it continuously during playback to provide the best user experience possible. If the available bandwidth changes, the device responds by selecting a new stream at a bitrate appropriate for the situation.

Automate System Administration Tasks with Puppet

Use Puppet for configuration management.

If you have more than one UNIX box in your care, you know how duplication happens. Every machine needs a common set of settings. Package upgrades need to be deployed. Certain packages need to be on every server.

You also want to make sure that any changes to your systems happen in a controlled manner. It's one thing to start off with two servers that are similarly configured; it's another thing to know they're the same a year later, especially if other people are involved.

Puppet is a system for automating system administration tasks (in the author's own words). In the Puppet world, you define a policy (called a manifest) that describes the end state of your systems, and the Puppet software takes care of making sure the system meets that end state. If a file changes, it is replaced with a pristine copy. If a required package is removed, it is re-installed.

It is important to draw a distinction between shell scripts that copy files between systems and a tool like Puppet. The latter abstracts the policy from the steps required to make a system conform. Puppet is smart enough to use apt-get to install a package on a Debian system and yum on a Fedora system. Puppet is smart enough to do nothing if the system already is conformant to the policy.

The Puppet system is split into two parts: a central server and the clients. The server runs a dæmon called puppetmaster. The clients run puppetd, which both connects to, and receives connections from, the puppetmaster. The manifest is written on the puppetmaster. If Puppet is used to manage the central server, it also runs the puppetd client.

The best way to begin with a configuration management system like Puppet is to start with a single client and a simple policy, and then roll it out to more clients and a more complex policy. To that end, start off by installing the Puppet software. Puppet is written in the Ruby scripting language, so you need to install that before you begin (Ruby is available as a package for most distributions).

Puppet vs. the Alternatives

My first experience with configuration management was with a product called cfengine. With cfengine, I was able to manage a Web cluster of 14 servers easily and reduce the time to install a new node from several hours to a matter of minutes. Puppet's author has a great deal of cfengine experience and built Puppet to address many shortcomings of cfengine.

Resources

Puppet's Home Page: reductivelabs.com/trac/puppet/wiki

Annotated Links on Using Puppet: del.icio.us/SeanW/puppetlj

EOF - The Browser Platform

Because Effect

What Netscape was to Web 1.0, Google is to Web 2.0. Like Netscape, Google is Net-native, pioneering, hacker-friendly, generous and likable. It charges for some stuff, but it gives away the most popular stuff for free. That's because it groks the Because Effect: you make money because of what you give away for free. Netscape made money with server software because it gave away the browser. Google makes money with advertising because it gives away search—and a growing portfolio of other services and applications that create vast new environments where advertising can be placed.

Google has seemed content to let its spin-off, Mozilla, with Firefox, gradually eat away at Microsoft's dominant browser share—both by being a good product and by serving as host to an endless variety of extensions and plugins.

That is, until early September 2008. That's when Google announced Chrome—a new browser that really does serve the role of an operating system. Google explained Chrome through a 39-page series of illustrations in comic book style by the brilliant Scott McCloud (www.google.com/googlebooks/chrome). With Chrome, tabs aren't just for Web pages. They're for processes, “each having its own memory and its own copy of the global data structure”. Sound familiar? The doc adds, “We're applying the same kind of process isolation you find in modern operating systems. Separate processes rendering separate tabs.”