Render a Wicket page to a string for HTML email

Something that\’s very desirable to do in Apache Wicket is create HTML emails using Wicket\’s brilliant component-oriented markup.

I\’ve been working on this problem on and off for ages — it\’s tricky because of teh way that markup rendering is so deeply tied to the requestcycle, which in turn is deeply dependent on the httpservletrequest — with good reason, too. That\’s where Wicket gets its autoconfiguring magic from!

So in order to use Wicket to create HTML emails, we need to fake the request/response cycle. I wrote this convenient method that renders a bookmarkable page (pageclass + pageparameters) to a string:

protected String renderPage(Class<? extends Page> pageClass, PageParameters pageParameters) {

        //get the servlet context
        WebApplication application = (WebApplication) WebApplication.get();

        ServletContext context = application.getServletContext();

        //fake a request/response cycle
        MockHttpSession servletSession = new MockHttpSession(context);
        servletSession.setTemporary(true);

        MockHttpServletRequest servletRequest = new MockHttpServletRequest(
                application, servletSession, context);
        MockHttpServletResponse servletResponse = new MockHttpServletResponse(
                servletRequest);

        //initialize request and response
        servletRequest.initialize();
        servletResponse.initialize();

        WebRequest webRequest = new WebRequest(servletRequest);

        BufferedWebResponse webResponse = new BufferedWebResponse(servletResponse);
        webResponse.setAjax(true);

        WebRequestCycle requestCycle = new WebRequestCycle(
                application, webRequest, webResponse);

        requestCycle.setRequestTarget(new BookmarkablePageRequestTarget(pageClass, pageParameters));

        try {
            requestCycle.request();

            log.warn("Response after request: "+webResponse.toString());

            if (requestCycle.wasHandled() == false) {
                requestCycle.setRequestTarget(new WebErrorCodeResponseTarget(
                        HttpServletResponse.SC_NOT_FOUND));
            }
            requestCycle.detach();

        } finally {
            requestCycle.getResponse().close();
        }

        return webResponse.toString();
    }

One other thing that\’s desirable to do is change all relative links in the email to absolute URLs — something that Wicket makes super-easy, if you know how. That will be the subject of my next post.

Free SMS service notifications using Google Calendar

Today I had a small revelation.

I was wracking my brains trying to figure out the SMS messaging provider to use to send myself service outage notifications for my clients\’ web sites. Given that I have just a handful of clients so far, it makes no sense to use a provider that requires a minimum monthly or yearly spend.

Ideally of course, I\’d like to spend nothing at all, and in exasperation I finally threw my hands in the air (they\’re detachable) and whined: “Google sends SMS\’s for free – why is it so hard for everyone else?”

(answer: not everyone has billions of dollars)

And then came the revelation: Why not create a command-line tool that uses Google\’s Calendar API to create events 6 minutes in the future that have an SMS notification set for 5 minutes prior to launch? That way, within a minute you get a notification sent to your phone for free within 1 minute. Sweet!

So, here\’s the code (it\’s in Java… sorry)

/**
* Simple command-line notification command that uses Google Calendar ATOM API to create
* a single event 6 minutes in the future with a 5 minute SMS reminder
*
* @author Daniel Walmsley
*
*/

import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.Date;
import java.util.List;

import com.google.gdata.client.calendar.CalendarService;
import com.google.gdata.data.DateTime;
import com.google.gdata.data.PlainTextConstruct;
import com.google.gdata.data.calendar.CalendarEntry;
import com.google.gdata.data.calendar.CalendarEventEntry;
import com.google.gdata.data.calendar.CalendarFeed;
import com.google.gdata.data.extensions.Reminder;
import com.google.gdata.data.extensions.When;
import com.google.gdata.data.extensions.Reminder.Method;
import com.google.gdata.util.AuthenticationException;
import com.google.gdata.util.ServiceException;

/**
* This is a test template
*/

public class GCalNotifier {

public static void main(String[] args) {

    /**
     * Command line args: 
     * 
     * username
     * password
     * calendar name (e.g. "Notifications")
     * TimeZone offset (in hours)
     * event start offset (in minutes)
     * event end offset (in minutes)
     * title
     * description
     */

    try {

        // Create a new Calendar service
        CalendarService myService = new CalendarService("GCal Event Notifier");
        myService.setUserCredentials(args[0], args[1]);

        String calendarName = args[2];
        Long tzOffset = new Double(Double.parseDouble(args[3])).longValue() * 60 * 60 * 1000;
        Long startOffset = new Integer(Integer.parseInt(args[4])).longValue() * 60 * 1000;
        Long endOffset = new Integer(Integer.parseInt(args[5])).longValue() * 60 * 1000;
        String title = args[6];
        String description = args[7];

        // Get a list of all entries
        URL metafeedUrl = new URL(
                "http://www.google.com/calendar/feeds/default/allcalendars/full");
        System.out.println("Getting Calendar entries...n");
        CalendarFeed resultFeed = myService.getFeed(metafeedUrl,
                CalendarFeed.class);
        List<calendarentry> entries = resultFeed.getEntries();
        for (int i = 0; i < entries.size(); i++) {
            CalendarEntry entry = entries.get(i);
            String currCalendarName = entry.getTitle().getPlainText();
            System.out.println("t" + currCalendarName);

            if (currCalendarName.equals(calendarName)) {
                sendDowntimeAlert(myService, entry,
                        title, description, startOffset, endOffset, tzOffset);
            }
        }
        System.out.println("nTotal Entries: " + entries.size());

    } catch (AuthenticationException e) {
        e.printStackTrace();
    } catch (MalformedURLException e) {
        e.printStackTrace();
    } catch (ServiceException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    }
}

private static void sendDowntimeAlert(CalendarService myService,
        CalendarEntry entry, String title, String description, Long startOffset, Long endOffset, Long tzOffset) throws IOException,
        ServiceException {

    String postUrlString = entry.getLink("alternate", "application/atom+xml").getHref();

    URL postUrl = new URL(postUrlString);//was: "http://www.google.com/calendar/feeds/jo@gmail.com/private/full"

    CalendarEventEntry myEntry = new CalendarEventEntry();

    myEntry.setTitle(new PlainTextConstruct(title));
    myEntry.setContent(new PlainTextConstruct(description));

    Date now = new Date();

    Date startDate = new Date(now.getTime()+startOffset);
    Date endDate = new Date(now.getTime()+endOffset);

    DateTime startTime = new DateTime(startDate.getTime()+tzOffset);

    DateTime endTime = new DateTime(endDate.getTime()+tzOffset);

    When eventTimes = new When();
    eventTimes.setStartTime(startTime);
    eventTimes.setEndTime(endTime);
    myEntry.addTime(eventTimes);

    // Send the request and receive the response:
    CalendarEventEntry insertedEntry = myService.insert(postUrl, myEntry);
    System.err.println("Got response for: "+insertedEntry.getTitle().getPlainText());
    for(When when : insertedEntry.getTimes()) {
        System.err.println("When: "+when.getStartTime()+" to "+when.getEndTime());
    }

    //5 minute reminder
    Reminder reminder = new Reminder();
    reminder.setMinutes(5);
    reminder.setMethod(Method.SMS);
    insertedEntry.getReminder().add(reminder);
    insertedEntry.update();
}

}

Don\’t forget, you\’ll need to download the Google Data APIs and put their JARs in your classpath before this will work!

Personally I use this with Nagios. I always use the same args for the calendar offsets, so I\’ve encapsulated most of my settings (except title and body) in a script.

!/bin/sh

export SCRIPTDIR=/opt/calAlert
export USERNAME=username@gmail.com
export PW=mySecurePassword
export CAL=Notifications
export TZOFFSET=10
export STARTOFFSET=7
export ENDOFFSET=12
export TITLE=$1
export BODY=$2

export CURRDIR=pwd

export CLASSPATH=”${SCRIPTDIR}/calAlert.jar”

assumes GData libs are in “libs” subdirectory of SCRIPTDIR

for jarfile in $(ls “${SCRIPTDIR}/lib”)
do
CLASSPATH=”${CLASSPATH}:${SCRIPTDIR}/lib/${jarfile}”
echo lib/${jarfile}
done

echo “CLASSPATH=${CLASSPATH}”

export CLASSPATH

java GCalNotifier ${USERNAME} ${PW} ${CAL} ${TZOFFSET} ${STARTOFFSET} ${ENDOFFSET} “${TITLE}” “${BODY}”

Training Junk Mail filter using Apple Mail and GMail IMAP Connector

Like most people, I get literally thousands of spam messages a month. I never see them, of course, because they\’re filtered out by Gmail\’s incredible spam filtering system.

This is all well and good if you\’ve got a Gmail account, but what if you want Google-quality filtering for your business or other mail accounts?

Well, thanks to Gmail IMAP support, we now have a massive and incredibly accurate data set for training Apple Mail\’s junk filter. Gmail IMAP lets you browse folders other than your inbox. So once you\’ve added it to your Apple Mail account list, simply browse to the Spam folder, select all, then mark those messages As Junk.

This simple act will train Mail\’s bayesian spam filtering system on everything in your GMail spam folder, no doubt instantly improving its performance somewhat. In my case, the spam folder usually has 3000-4000 messages in it, which is fairly hefty and it would certainly take a lot of clicking to get that kind of data by hand.

Thanks Google!

Jackrabbit, Wicket, Tomcat, Maven2… hell.

What follows is lessons learned migrating to the potentially magnificent Maven2 for dependency management.

Put <scope>provided</scope> on Tomcat shared resources in your pom.xml

If you deploy jars as a shared resource on Tomcat (i.e. put the jars in common/lib) then be sure to add the <scope>provided</scope> to those dependencies in your project\’s pom.xml. Otherwise, you\’ll get absolutely daft class-cast errors on shared resources like:


2007-10-21 12:42:19,425 ERROR 0-SNAPSHOT] - Servlet /myExample2-1.0-SNAPSHOT threw load() exception
java.lang.ClassCastException: org.apache.jackrabbit.core.jndi.BindableRepository cannot be cast to org.apache.jackrabbit.core.jndi.BindableRepository

Hahahahahahaha I think I want to kill myself. The problem is that Tomcat\’s shared libraries are loaded by a different classloader than your web-app\’s shared libraries (which is nice in a way, because it means you can use different versions of log4j or whatever).

So the lesson here is: Anything you want created by Tomcat and loaded by name (e.g. “jcr/repository”), be sure to exclude from your WEB-INF/lib when you deploy.

You can load the same shared resource by name for all apps

Deploying a Maven2-enabled app using Codehaus Mojo is a breeze… unless you want to deploy a context with it. And a context is the only way to load up named shared resources like a Jackrabbit repository. The solution?

$TOMCAT_HOME/conf/Catalina/$HOSTNAME/context.xml.shared

The contents are loaded for all contexts. Brilliant.

Class blahblah violates loader constraints

Oh no. This was awful. For me it was:


2007-10-21 13:16:26,331 ERROR 0-SNAPSHOT] - Exception starting filter DataServlet
java.lang.LinkageError: Class org/slf4j/ILoggerFactory violates loader constraints

I needed to scour the dependencies that Maven was loading into my webapp automatically and explicitly label them as provided.

Make the web disappear

What is it that compels people to publish their life online?

Blogs, Moblogs, Photoblogs, Vlogs, iCalendar, iTunesBlogger, flickr, del.icio.us, geourl. Social bookmarking, social networking, aggregating and subscription. What is it that drives people to publish their life online? And what is it that compels them to read about others’ lives?

One explanation might be that people seek to present an idealised avatar of themselves, a “virtual me” that can be published free of the flaws and foibles of the physical version. Another explanation is that they merely seek to intersect with others in dimensions that are harder to express in meatspace – who else in the world like this strange band? Who else is going to this concert? Who has a male cat of the same breed as my fertile female cat? And what sort of a person are they?

It is overly simplistic to think of the web as a glorified pick-up joint. The urge to express oneself runs deeper than that.

I like to think of the new, highly-linked web as adding additional dimensions to our navigation through the world. We already have a simplistic version with the telephone book and telephone. The semantic web (which is what we’re talking about) will revolutionise the world in a directly analogous way, as you shall see.

When you want to talk to a friend on the phone, you know their name and possibly where they live. So you pick up the telephone book and look them up by name, suburb and possibly street. This provides you with a number, which you then punch into the telephone to call them. If they’re home, they pick up. If they’re not, you can leave a message (assuming they have an answering service).

What you have done is a single, one-way index lookup and then transcribed that information into a tool that allows you to perform the desired task. You are restricted to looking up the information in a single dimension, and once it has been looked up it has only one use. It’s a massive advantage over pre-telephone days because you can transcend space to contact the person you are after, albeit in a fairly restrictive way.

Now imagine if you could look them up by interest, location, or age. Suddenly, you can look up more than just people – you can find images, movies, articles, books. You can search by almost any attribute – physical location, date, time, degree of trust, rankings by your friends. Imagine if you could look up photos from a single location throughout recorded history, as though you had a time-travelling camera. Such things are possible – now – with the semantic web.

Now, people have been babbling on about this for years, particularly the boring and bureaucratic XML consortium, and people have typically said “Huh? That sounds like a fancy specification but what the Hell does it mean for me, sitting here in traffic in the stinking heat, wondering whether the desire to kill myself will pass by lunchtime?”. Actually, that’s not true. Typically people have ignored them.

Here’s what I think. I am assuming here that you want to know. Possibly a little presumptuous on my part.

I think that all this extra metadata – information about information, such as the coordinates at which a photo was taken – is slowly but surely entangling the online world with the real world. Dozens of books have fantasised about launching ourselves into “cyberspace”, flying through fanciful virtual worlds populated by people with supernatural powers. What is actually happening, which is even more exciting, is that the virtual world is leaping through that barrier the other way. Tendrils of the vast information universe that we’ve built online are extending into our world – your iPod carries around your “virtual” music as invisible files, as well as your contact and calendar information. If I make an appointment on my phone, my Apple Powerbook makes sure it appears on my website – invisibly. Calendars that people have built on the other side of the world can cause my mobile phone to beep and remind me that the new Tim Burton movie is about to be released. And all this is happening now.

Soon it will be possible, through the power of GPS and widespread (and cheap) wireless internet, so tag real objects in the real world with arbitrary digital information. You’re in a restaurant. You decide you like the food, so you push a couple of buttons on your phone which publishes a star-rating of the restaurant online, on your web presence, tagged with the GPS location. You don’t need to enter the name or address because that information is already associated with that lat/long location (provided by your phone) in Google. You don’t need to enter the time and date because your phone knows it. You don’t even need to indicate that you’re in a restaurant.

These star ratings could be like a heat-seeker. Hold up your phone and just ask it where in the surrounding blocks you’re likely to have the best time.

Of course, everyone’s different and you shouldn’t just trust some stranger’s opinion. That’s where trust networks come in. Over time, you develop a trusting relationship with your friends, and they with their friends, and so on. This provides some weightings for the informational tree that allow you to balance the information in favour of those whose opinions you respect, enhancing the quality of information you receive about your environment.

The question is: How do we make this invisible? How do we leverage the power of existing information and integrate it at every stage to make the whole process as seamless as just knowing what you need to know while barely needing to think about it?