PHP Script to trim IMAP Mailbox (e.g. trash)

I am lazy! Wait, that came out wrong. I like efficiency and automating things, and I am notorious for having scripts for anything and everything. Well, today, I have a new one to share with you.

I know most people these days use things like Gmail, but I am old fashioned and still like to own my own data, which is why I stick with IMAP. The problem with that, is that searching can get a little slow, or even time out on folders that might get a little too big.

For example, occasionally I’ll need to go and search for something in my “Trash” folder, but the search times out because of the thousands of messages I have there. The reality is, I only need messages going back about a month or so, if I only I could something that would automatically purge anything older than, say, 30 days…. I could not find anything, so I wrote my own PHP script, ready to run as cron job.

The script takes configuration parameters for:

  • IMAP server hostname (or IP) and port
  • Mailbox username and password
  • The folder of interest (e.g. Trash, Sent…etc)
  • Threshold for keeping messages newer than X days, which defaults to 30.

This is exactly what the script does:

  1. Connect to IMAP server
  2. Find the exact path of the target folder (e.g. Trash)
  3. Loop through all the messages in that folder
  4. Examine the date on each message, if older than threshold (e.g. 30 days) then delete, otherwise move on
  5. Purge deleted message (aka expunge in IMAP terminology) and disconnect

If you capture the output of the cron (e.g. have it emailed to you), then you’ll get a list of deleted messages and/or any errors encountered (if any).

View the PHP source code of the IMAP mailbox trimning cron script – Enjoy!

Wake-on-LAN is an easy, practical way to save on power

If you have ready my post on the new, power efficient power supplies for computers, you may start to notice a trend here! I have a desktop machine that, once upon a time, was doing quite a bit for me (web server, SVN repo, MySQL, PosgtreSQL, Samba shares, UPnP streaming, CUPS print server…etc), which required it to be on at all times. However, my needs have changed and these days I only need this machine turned on occasionally.

I had previously connected that machine to an LCD energy meter and figured it was probably costing me $5/month. Not too much, but not too little either. I wanted to find an easy, convenient way to turn this machine on and off (i.e. without having to physically push the power button). The turning off part was pretty easy, since I can do that over SSH or VNC, so it was just a matter of figuring out a way to turn it on remotely.

I started my search in the wrong direction, looking at power bars with wireless switches (Belkin makes one), or even Bluetooth connected power bars. All of which would be too expensive and an overkill for my simple purpose.

Eventually, a light bulb went on in my head! An old technology that I knew about for years, but never explored it, was the solution! It is called Wake-on-LAN (WOL) and it allows the network adapter, upon the reception of a “magical” trigger packet, to turn on the whole machine.

Sounds too good to be true? That’s what I thought, too. Then I Googled for examples of how to configure it and found an excellent guide at Life Hacker. It turned out to be even easier than I thought! In my case, all the configuration was perfectly lined up out of the box, so it worked on the first attempt without having to change anything.

As for where or what to use to send that magic packet, there are at least three very easy choices:

  • Your router, especially if you’re running an aftermarket firmware like DD-WRT or Tomato
  • A smartphone app
  • Websites offering it as a free service, although, you’d have to be OK with opening UDP ports 7 or 9 – or both if you don’t feel like searching!

I am surprised that I don’t hear much chatter about WOL in the IT circles. Personally, I think it is a great technology that can save a lot of energy and money, and it has been around for a while so most computers out there should have it built-in. It certainly has worked quite well for me!

Efficient power supplies are actually efficient!

I have heard a lot about the new more efficient power supplies, and I was eager to compare for myself but at the same time too cheap to buy a new one :) Luckily (for me only) a friend’s PC died all of a sudden, so he kindly gave me his fancy NZXT 650 watts to replace my generic. old 550 watts PSU. The new one was a “80 plus bronze” rated PSU.

The new PSU went into my server at home. A dated machine with AMD Athlon II X4 635, internal graphics, four 3.5″ Hard drives and an optical drive. This machine does quite a few things for me: CUPS Print server, DLNA Streaming, Samba sharing, Development web server (Apache for PHP, Tomcat for Java, MySQL and PostgreSQL), and I am probably forgetting other things. Although this is not a scientific experiment, I tried to keep things a bit scientific by changing nothing in the machine’s configuration but the PSU.

I had attached this machine to my power meter for a few weeks before and after swapping the PSU. The difference was quite clear as outlined below:

  • Running idle (not sleeping): 90 watts old vs. 72 watts news. 20% more efficient.
  • Under heavy load: 115 watts old vs. 85 watts new. 27% more efficient. Note: Heavy load involved more disk IO than CPU usage – it is pretty hard for me to max out those four cores!
  • Maximum load recorded: 150 watts old vs. 134 watts new. 11% more efficient. These number were typically scored during startup.

Considering this systems runs idle or almost idle most of the time, the total consumption saw a 20% overall drop (recorded over a few weeks).

Although that sounds like a lot of electricity, unfortunately, that does not translate as well into a dollar amount because electricity is still quite cheap (at this scale of consumption). At the current rates where I live, this translates to savings of $10-$12 per YEAR. In other words, if I had to buy this new PSU instead of getting it for free, it would have taken six years for the “investment” to pay off.

 

 

Let Selenium’s robots do the testing for you!

Recently I have stumbled upon Selenium, which according to their site’s title is a “Web application testing system.” What I love about Selenium, is that it is extremely easy to begin using, it takes absolutely no time to get it going and therefore, you really have no excuse not to be using it to automate much of the testing of your web app!

What makes Selenium special? Two very strong points:

  1. Creating tests is easy for most tasks, and when you need more sophistication, you can achieve that (more on that later).
  2. Selenium performs the tests in any or all major browsers of your choice (IE, FF, Chrome, Safari and Opera). Selenium uses Javascript to run tests within the browser and make assertions, so you can test almost all compatibility issues (except for things like flash).

Selenium IDE

To create tests, simply install selenium’s firefox plugin, which they call Selenium IDE. Once it is installed you’ll be able to invoke it through Tools -> Selenium IDE. When you invoke this tool, it is in recording mode by default, which means it is watching what your are doing and recording a log of your actions, which allows them to be repeated later.

At this point you can go ahead and hit your site, navigate around and when you are ready to make “assertions” (i.e. confirm that one or more things is/are right), right click on the element or text you want to verify, and you’ll see a list of actions available like verifyTextPresent, which is the simplest one.

Once you are done recording your test, you can have Selenium IDE playback to test at a slower speed, to confirm that it works as recorded. You can save this test and re-open it later for editing or export into an alternative format, if needed for unit testing or bromine integration (more on this coming right up!).

You’ll notice that Selenium IDE keeps the website separate from the test case, that’s because it allows to provide the website immediately before running a test, which means you can very quickly switch between websites to test.

What’s next?

There mainly two ways to make use of those tests, you can choose either or both! Firstly, you can integrate those tests with your unit tests, and have them run whenever your unit tests are usually performed, e.g. when creating a new build or on every SVN commit…etc.

Secondly, you can install Bromine, a web interface for managing projects and test cases. Bromine allows you to invoke tests from the web interface. Those load of those test is equally distributed to pre-defined nodes within Bromine. Each node defines what platform is its (Windows, Mac, Ubuntu…etc) and what browsers it has available. A machine can run multiple instances of the server software on different ports to parallelize the execution of the tests.

In both these scenarios, the nodes are to run Selenium Remote Control (RC), which is a Java-based server. It runs by default on port 4444, but you can (and you are encouraged to) run multiple instances of it on each machine on different ports, by passing -port 4445 as an example when invoking that JAR file.

What else?

A few tips and suggestions to make your Selenium experience more sparkly:

  • If you don’t like the way Selenium IDE finds your elements, you can use other alternatives like the XPath of the element.
  • You can make different assertions than the pre-defined set you see in the IDE.
  • The machine(s) which runs the tests doesn’t have to be super fast. Almost anything will work! In my case I started with one 1 GHZ machine, which ran four instances of the servers on Windows XP and handled them just fine.
  • In Bromine, you can define for each group of testcases which browsers and on what platforms are you concerned with. That is done with a grid-like so you can test with Firefox on Windows and Mac for example.
  • Bromine supports plugins if you need any extra functionality.
  • If your test requires logging in, make sure you logout at the end of the test. This way it won’t break the next time it runs and the session/cookie persists.

Record a podcast of your favourite radio show using PHP & Shell

Following my previous post on mixing PHP and shell scripting under Linux, I have a more practical and complete example coming up in this post, which will come after a little introduction, so bear with me!

If you are like me, you probably have run into situations where your favourite radio show(s) is/are not available as a podcast, either because the station doesn’t do podcasting or because they don’t podcast that particular show. That got me thinking, couldn’t I just make my own podcast of the show out of their stream? Of course that is quite possible, but requires a bit of work.

Before we go further, I must give you a bit of a legal disclaimer! Please do not use any methods described or provided here to do anything illegal. Sometimes the shows are not available as podcasts for legal reasons beyond the station’s control, some other times the stations use streaming technologies where the fine print in the terms of service make it illegal for you to tune in using “alternative” methods like the ones described here. Last but not least, keep recorded material for personal use and DO NOT redistribute, whether for profit or not.

Back on track, I imagined a solution that can tune in at the correct time, record my show that runs for X minutes then convert into MP3 for easy playback on my phone or mp3 player. Luckily, Linux has a plethora of tools and commands that can achieve that exact goal, it is just a matter of piecing it together!

In short, here’s how I got it to work:

  1. Cron job-schedules will be used to fire off the recording at the right time. Check wikipedia’s excellent cron page for more information on this tool and how you can use it.
  2. mplayer will perform the tuning in and decoding of the stream into raw wave format. Mplayer can be used both as a command line and as a GUI tool, it supports pretty much every format out there.
  3. Last but not least, lame will do the conversion into mp3.

Then I used PHP to piece everything together, here’s a quick breakdown of events:

  1. PHP script gets invoked by a cron or manually from the command line. It takes three arguments: URL to stream, length in minutes and file name/path to use for the mp3 file.
  2. PHP invokes mplayer using a pipe so it runs in the background, sleeps for the length of the show.
  3. PHP wakes up, terminates that specific instance of mplayer (i.e. you can safely run multiple instance of this script simultaneously).
  4. Lame is invoked to convert into mp3 and the temporary raw wave file is removed.

Of course there are many things your imaginative mind could tinker with to spice things up, such as:

  • With a bit of work, you could use this on a video stream, too!
  • You could swap lame for mencoder or ffmpeg, which convert audio and/or video from pretty much anything to pretty much everything! You could create the output files using AAC for example.
  • If you still prefer mp3 like me, there are many many settings and flags you can pass to lame to adjust quality settings and encoding options.

Here is a link to my “podcaster” script, it is free to download and use for personal use (keeping in mind the disclaimer above). I am sure there many improvements that can be made to it. If you make any, then kindly email me or post in the comments section below.

Last but not least, for the more novice readers, who are wondering how to get the direct URL to the stream. Here are some tips:

  • If your station’s stream usually comes up in a dedicated window, look at the HTML code (Ctrl+U in Firefox) and in most cases you’ll find a reference to stream URL, or the playlist containing that URL we’re after.
  • If you are working with a playlist that you found through the first tip or your station provides a direct link to the playlist, you’ll find most playlist formats (e.g. m3u, asx) to be text or XML-based. You could use something like wget to download the file, then a text editor like vim to read the content.
  • Some playlists are nested (if supported by the format), where one playlist contains a link to another playlist, which in turn contains the actual link to the stream URL.
  • The stream URL is most likely using one of three protocols: HTTP, MMS, RTSP.

I hope you’ve found this helpful!

Form validation tips and tricks using jQuery

I’ll keep this article short and sweet… I love jQuery! The end.

Just kidding! I do love jQuery though. Recently I have written a quick and nice (in my opinion at least) Javascript function to do simple form validation on the client side for me. I am not including the whole function here because I am sure others have posted better solutions online, but I will include the part I thought was cool, thanks to jQuery.

First, a bit of background. My function is invoked upon form submission and expetcted to return true or false, obviously. I am currently passing the list of field ids to it using a global array variable. So I loop through the array to do my checks, and for the ones that don’t pass my validation, I do this:

$('label[for="'+elemId+'"]').attr('class','error');
missing[missing.length] = $('label[for="'+elemId+'"]').text();

Where “missing” is an array I declared within my function, and class “error” is a CSS class with extra styling to make the label stand out, in my case that extra styling is a simple “color: red”. As you may have noticed, this will only work if your form has proper markup, namely if fields have unique ids each, and labels are properly linked to them using the “for” attribute.

Lastly, at the bottom I have:

if( missing.length > 0 ) {
alert("You missed filling out the following required fields:\n- "+missing.join("\n- "));
return false;
}
return true;

You can easily change it so the error message shows up in an existing div you have for that purpose, instead of using alert as I did in my example.

That’s all, I hope someone finds that useful!

Nice looking thumbnails without image manipulation

In this article I’ll explain how you can include the full-size images in your HTML, and use simple CSS to make them look like thumbnails. From my experience this works great for images up to say 1024×768 pixels, and for up to 30 or so images (depending on the size, ofcourse).

Even if you have access to image manipulation functions that make it easy for you. this function can be, in some cases, better than generating thumbnails. One popular example is if you are using lightbox (or a similar solution, and there is plenty of them), sure those clever scripts can pre-fetch full-size images as the user is browsing, but browsing through photos, which the browser downloaded on page load, is even faster!

So, how does this work? Simple! you want to wrap your image with an anchor (link) as you usually do, then wrap the anchor with a div or span element, giving each a unique class name. Example:

<div class="photoContainer" >
<a class="photoLink" title="Photo caption" rel="lightbox" href="/photos/1.jpg">
<img class="photoImage" src="/photos/1.jpg" alt="Photo caption" />
</a>
</div>

Here’s the CSS that goes with that:

div.photoContainer {
float: left;
height: 65px;
width: 100px;
overflow: hidden;
margin: 5px;
}
a.photoLink {
text-decoration: none;
}
img.photoImage {
height: auto;
width: 115px;
}

That’s all! The widths and heights I provided in the example work well with 4:3 and 16:9 aspect ratios, as well as portait photos. Feel free to adjust as necessary but I recommend you test with a variety of photos to confirm they all look good. Enjoy!

Mixing PHP and shell scripting (under Linux)

PHP plays very well with command line. On one hand, you can execute PHP from the command line using the, wait for it, “php” command! On the other hand, inside PHP you have many options for executing commands on the command line, which is the one I want to focus on here.

Before you proceed reading this article and get all excited about PHP and command line having babies together, I must warn you that I find most hosting companies don’t allow their users to access command line from within PHP. So, you might want to check on that first!

First example is the one I use the most. At work I maintain close to a hundred deployments of this one product, so when I need to run quick deployments, I can do this:
$sites = explode("\n",trim(`ls -1d /var/www/*`));

Note the tick character (`) in PHP means execute this string in shell. Also one thing people always confuse are the flags I am passing to the command “ls” (list, aka dir), the first is a digit one (1) and it means one column containing names only, the second, means return directories only.

At the end of the day, this will give me an array of all folders under /var/www, which I loop through in PHP, connect to the database to do some work and/or perform tasks in the file system. You could get fancier using a more advanced command to look up your targets such as find.

The second example is very useful for people who work with audio and/or video files. You can take advantage of awesome Linux commands such as ffmpeg and mencoder to allow your users to upload pretty much any kind of audio/video and convert it with ease into the format you prefer for playback and storage. The best part is, you can run the conversion command in the background so it doesn’t hold the page from loading (asynchronously, like AJAX). Here’s one example:

$cmd = 'nice lame -q 9 --resample 44100 '.escapeshellarg($source_file).' '.escapeshellarg($dest_file)." &\r\n";
$handle = popen($cmd,'r');
pclose($handle);

Nice is a prefixing command (like sudo), which basically sets a lower priority on this process not to lock up your server. Lame is the popular mp3 encoder. escapeshellarg() escpaes arguments for the command line interface (important!). The ampersand (&) puts this as a background process. Note that you could not use exec(), which is why I am opening up a pipe using popen(). Last but not least, the carriage return and new line characters (\r and \n) simulate hitting enter after a command when you are typing in a shell session, which is what the pipe does.

I will stop here since this post is getting too long. I should mention that the only limitation I know of with working with shell in PHP, is that you can never work with interactive prompts (e.g. being asked for a password when executing a command like ssh, scp, rsync or even svn update). However, there are ways to work around most of those scenarios (certificates, authentication caching…etc), which may not be best practices, but are available none the less if you really need to.

On the other hand, if you are building a PHP CLI solution (e.g. interactive installer the command line), you can make use of interactive prompts using existing shell libraries for PHP.

Introduction to threading in J2ME

I know from experience the idea of threads can be a little intimidating at first. So I will begin by addressing what is a thread and why you need to use it.

A thread is a way to hand out some time-intensive tasks to a separate process within your application, without keeping the “main” thread/process busy, this main thread is the primary entry point to your application, exactly like the function main() is in many programming languages such as C and Java. Think of it to be like hiring a contractor to finish your basement. Even though you are handy enough to accomplish that task, you simply have no time for such a task!

What kind of tasks are we talking here?  Well, some tasks such as TCP/IP communication (e.g. working with some API over HTTP), some big loop (e.g. building an index of the user’s MP3 collection) or even displaying an animated GIF, those tasks would take too much time on a device with limited process power, this is time on the scale of CPU cycles or microseconds, not our scale of seconds or minutes.

So, why do I have to use threads for those tasks? Well, I think this is better explained if we explored what happens if you don’t use threads and try to run some time-intensive task in the main thread:

  1. Your application will freeze and appear unresponsive until the task is accomplished, and you won’t be able to give your users any means to cancel this task while it is executing (again, because your application is locked up).
  2. On devices like the BlackBerry, your application will be terminated. Some (if not most) devices do not allow the “main” thread to consume too much CPU for too long (a.k.a become blocking), otherwise they assume your application compromises the device’s stability and shut it down.

It is always a good idea to keep the main thread for GUI and hand off grunt work to threads. One thing you should keep in mind, those threads do not communicate with the main thread unless you write code to do so.

In J2ME, creating a thread is pretty simple. The easiest and quickest way is to extend the “Thread” class and overload the run() function on-the-fly, then “start” the thread as follows:

Thread myThread = new Thread() {

public void run() {

// do whatever you need to do here

}

};

myThread.start();

Remember to properly catch and handle exceptions in your thread, otherwise they could bubble up and cause your application to terminate.

You’ll get an exception if you try to manipulate the GUI within your thread, because GUI is assumed to be a responsibility of the main thread only. This how you can do it, which can also be used to report progress to the main thread for something like a progress bar:

UiApplication.getUiApplication().invokeLater(new Runnable() {

public void run() {

// do whatever you need here!

}

};

getUiApplication() gets us the class containing the main() function, i.e. the entry point of your application and the main thread. invokeLater() is a way of “scheduling” your code within Runnable.run(). I put scheduling in double quotes because technically your code has to wait until all pending events on the main thread are processed, but practically that takes no time. However, if this becomes a problem you could use invokeAndWait() instead of invokeLater() to block your thread until the main thread processes your request.

That’s all, enjoy and remember to drop us a line if you find this to be helpful!