Monday, May 26, 2008

Predicting Whether You Need to Reply to an Email

Every couple of months, I try to catch up on email research work from academia. This weekend, I found this paper [1] by Drezde et al. from UPenn.

The problem they want to solve is predicting whether an email in your inbox needs to be replied to or not. This is very relevant research as it would allow users to quickly scan their email for items they actually need to act on. The result could just be a simple "needs reply" indicator in the list of incoming items.


How do they do this? The authors used a number of attributes to classify emails, such as whether the user frequently replies to the author, whether the email contains any question marks, and combinations of words appearing in the text.

The results aren't quite there yet. For the best test corpus, they achieve are 77% recall - which means that they find 77% of the emails that need replying. However, they come in at 76% precision, which means that 24% of emails they mark as "needs reply" don’t actually need replying.

Thus, reply prediction remains exciting. I'm hoping that they come up with a better classifier, and that someone then turns this into an industrial-grade email application.

[1] Mark Dredze, Tova Brooks, Josh Carroll, Joshua Magarick, John Blitzer, Fernando Pereira: Intelligent Email: Reply and Attachment Prediction, Intelligent User Interfaces 2008, Spain. [PDF]

Sunday, May 25, 2008

iPhone, the Second Coming

I'm probably the last one to pick up on the rumors that a new version of the iPhone is coming: The online store says "currently unavailable", Wired has the scoop, and everyone seems to think that a new version will be out on June 10.


They better be prepared: Everyone without an iPhone at Xobni is considering getting one. Even my friends are ready for iPhone 2.0, as they refuse buying any 1.0 products. Apple could sell millions of these devices by just slapping "2.0" on the existing model.

These folks sure have learned how to hype up new products.

Tuesday, May 20, 2008

The Right Stuff: Building The Team

This is the last post in a series of three articles on Xobni’s launch. Check out the first episode on the ideas behind Xobni, and the second post about the journey to building the right product. Co-written with Marie Baca.

This is the story of how we are building Xobni's team and culture, and the things we've learned along the way. I think the jury's still out on whether what I'm describing is the right way, but I’ve certainly learned some things along the way.

Origins

In Xobni's application for YCombinator funding, the company was called "EmailDM," where the "DM" stood for "Data Mining". As Matt describes here, Adam was also toying with the name "InboxAdvisor" when Paul Graham suggested Xobni – the word "inbox" name backwards. The domain name was still available and so we snapped it up.

I met Adam and Matt during a 2006 trip to the East Coast. What was initially meant to be a friendly brainstorming quickly morphed into a caffeine-fueled two-day coding session at 16 Elmer Street in Cambridge, MA (Xobni's original headquarters). Over the course of 48 hours, I coded a new project for Xobni. While that product was never released, the concept of coding assignments during interviews would become a Xobni tradition.

Hiring

Some months later, Adam and Matt moved to San Francisco by driving Adam's rebellious Toyota Avalon across the country. I joined as the VP Engineering, and one of my first duties was to define a hiring process for engineers.

Everyone seems to know Joel Spolsky's motto of "Smart and Gets things Done", but finding a way to easily identify that in job candidates is a different animal altogether. Initially, I modeled our interview process off of Google: tough interviews that test "smartness." In the spirit of "seeing the juggler juggle", we added a crucial component: a coding assignment where candidates are given a workstation and a project to complete. This has helped us tremendously in identifying productive developers. Instead of testing for knowledge of algorithms and ability to solve brain teasers, you can see a person's coding style, maturity, and whether they were able to complete the project on time with a reasonable set of features.

Quality Assurance

My biggest mistake during my first months at Xobni was that I didn’t bring in top-notch quality assurance talent early enough. While it
s true that Xobni’s range of functionality is not as complex as, say, Microsoft Word, the product is still exposed to a wide variety of configurations and environments: Different versions of Windows, Outlook, user permission settings, email account types, and storage types. It's no coincidence that Microsoft allegedly has a 1:1 ratio of developers to quality assurance staff.

We needed someone who would think about QA night and day. After our private launch at TechCrunch 40, we learned our lesson, and I'm happy that we found Ryan and Tyler, who have taken ownership of making Xobni rock-solid. They of old laptops, set up test accounts and mail servers, virtual machines, and wrote test plans. I only wish we had brought them on earlier!

Data, Data, Data

Historically, we've made our best decisions when data and statistics were readily available. This isn’t always possible in every decision making process, but it's a good rule of thumb to try to surround yourself with as much relevant data as you can. For example, when you’re deciding what to have for lunch, it helps if you see what others are getting, and what you ate last time. When you’re debugging an odd bug from the field, it’s best to have log files, exception counts, and more. When you're making decisions about how to grow the user base, it helps if you have data about past user growth, uninstalls, and reasons why people like or dislike your product.

At Xobni, we’ve built systems and dashboards for all of the data listed above, from Greg’s exception robot to Bryan’s product dashboard (not to mention Ryan's lunch ordering system). When encountering a problem, we like to think about when we’ll need the same data in the future, and if the answer is yes, we build a system, not a one-off solution. There’s a lot going on behind the scenes.

The Future

Xobni the product, and the engineering team behind it are growing. It's interesting to participate in this process because with every new person, our engineering culture morphs as new hires bring new viewpoints, insights, and abilities. Jeff, our CEO, has said that with every 20 people, the company feels different, which means that we are getting close to experiencing the next chapter of the Xobni story. Oh: we're also hiring.

Monday, May 12, 2008

Xobni's Journey to the Right Product

This is the second of a three-part series on the Xobni launch. Come back on next Monday to check out Part III of the story. Co-written with Marie Baca. An abridged version of this article was featured yesterday on GigaOm.

I often speak with entrepreneurs applying for funding from YCombinator, and when I do I give them this advice: the most important decisions you make are the ones in the beginning of the process. Choices like what product to build and what market to serve determine whether you’re headed for failure or a multi-million dollar exit. Marc Andreessen, founder of Netscape, calls this "product/market fit" and says that it’s the only thing that really matters when it comes to building a company.

It's fair to say that at Xobni, is that we stumbled into product/market fit. We initially built the wrong product, but then we quickly corrected our course.

Building the Wrong Product

When Adam first wrote up our YCombinator application, Xobni was called "EmailDM"—the DM stood for "Data Mining." Adam included demos of how EmailDM would summarize emails, highlight important passages, and so on. Eventually this vision morphed into Xobni Analytics, which lets you analyze email usage by creating a diagram of your message traffic.



The initial plan was to offer this program to companies as a productivity tool. Workers would install the software into Outlook and use the data to find ways to become more productive. We would then offer this product to corporations so managers could remotely monitor employee email productivity.

This Boston Globe article, the first major coverage of Xobni, illustrates why this model was a bad idea: employees would feel like they were being spied upon. But even beyond the spying problem, Xobni Analytics was the wrong product to build for a more important reason: users would install the software, browse it for a bit, and then close it and quickly forget about it. Analytics was not a sticky application that tempted users to come back again and again.

The Xobni sidebar is completely different. There are many features beyond the analytics that add value for the user. Plus, the sidebar sits right next to the inbox, so users are exposed to it for the 2 hours they spend checking their email every day.

Our mistake became even clearer when we shopped early versions of Xobni around to venture capitalists. We first tried to remedy this problem by designing something called Xobni Feeds, which allowed users to create different types of XML queries that we would show up in the sidebar. Only later did we realize that this way too much flexibility for the average user, and we decided that it was a much better idea to show relevant, people-based default information.

In short, we discovered that our product was most successful when it provided focused information about people, not numbers. It is essential that entrepreneurs remember that they are serving an audience of individuals who need simple solutions to their everyday problems.

Why Outlook?

Unlike the initial problems with product development, Xobni made the right decision early on to target the Outlook market. There are an enormous number of Outlook users out there (around 350-400 million, depending on whose numbers you trust), and yet there is also major dissatisfaction with the program.



After Outlook, users most often requested that we build Xobni for Gmail and Apple’s Mail.app. However, both of those clients have much lower pain levels than Outlook. In addition, the addressable market is much more interesting: Outlook users are more likely to have business credit cards so if we ever switch to a freemium model, we can charge them. Also, there’s a strong demand for advertising partnership opportunities within Outlook, an application that users interact with for hours each day.

I like to think of it this way: there are a huge number of Outlook users out there who are so frustrated with the program that they feel like there hair is on fire. What Xobni offers is a cold bucket of water.

A Generalized Architecture

One of Xobni’s major differentiators is that we didn’t build a simple addin for Outlook, but an email application platform. Whenever we write software, we aim for a more general solution than what is required by our current needs. We can easily build new applications and build integrations into other email clients, as these leaked screenshots illustrate.



Building this platform took a significant amount of time and effort, but I’m happy that Adam made this decision early on: It allows us to quickly iterate on functionality, and build new, exciting products in the future!

Choosing a Look

Here are some mockups I created more than a year ago that illustrate the looks we had in mind for Xobni. The mockup on the right was our original choice: it blends nicely into Outlook, almost as if Xobni was built into it from the start. We created the third one almost as a joke, to show how the sidebar would look if we used a funky color palette.



We decided to go with the bright, saturated colors, because it would have what we call the “What the heck is that?” effect. Jane would walk by Jack’s desk, and see a bright blob on the right side of his Outlook screen, and she would blurt out: “What the heck is that?” Jack would have to explain Xobni and what it does, and Jane would probably download it once back at her desk.

Search as an Afterthought

Ironically, Xobni’s search function was initially an afterthought – Greg Duffy implemented it just a few days before we launched our private beta at TechCrunch 40. Even considering the rapid pace at which startups do product development, this was an immense accomplishment: Xobni’s search is twice as fast as Outlook’s, thanks to highly optimized and efficient search indexes. That combination of speed and usefulness make search our top-clicked feature.

Launch!

Let’s say you have a great product. How do you know when you’re ready to launch? That’s what I asked Gmail creator Paul Buchheit two months before our launch date. At the time, Paul was pretty skeptical, but he inspired us with an idea he repeated at Y Combinator's Startup School: Before launching Gmail, Eric Schmidt gave him the assignment to find 100 happy users. After talking to users, and eliminating their top feature requests and bugs, they met that number and thus, Gmail was ready to fly. We followed a similar spirit by adding the Are You Happy box to the product, and by listening closely to customers via our support team and users we knew personally.

We originally launched a private beta at TechCrunch 40, a conference where 40 startups launch their product within 2 days. The schedule was very tight, and the days leading up to it were a chaotic blur. In retrospect, we probably launched a little earlier than we should have. The product hadn’t been heavily tested at that point, yet we grew our user base by 50x that very day. With our open beta launch on Monday, we were finally ready in terms of speed and stability, and the feedback since has been extremely positive:
"Xobni is such a simple idea, but it is sure to radically change how you handle email." - Tom Spring, PC World
"For those who work in the corporate world - where Outlook is still heavily used - Xobni's public beta will be of great help, allowing them to quickly find and expose the data trapped in their inbox." - Sarah Perez, Read Write Web
"This plugin looks like it would be insanely helpful. Gmail, are you paying attention? :)" - Comment by Ross M, Lifehacker
"Given that Outlook is pretty much de rigueur for most corporate e-mail systems, this should be a welcome addition for those of you trapped in Outlook at work." - Scott Gilbertson, Wired
"Oh Microsoft, you’ve missed the boat yet again it would seem." - Zach Epstein, The Boy Genius Report
"In just a short few days, I can already share my belief that Xobni is a must download for Outlook users." - Kevin Tofel, kjOnTheRun
"Designed to make it easier to handle the deluge of daily messages, Xobni integrates with Outlook and - to quote our original review - becomes an invaluable weapon in the daily war with email." - Tim Danton, PC Pro

Thanks to Evan Solomon and Sean Ellis for reviewing drafts of this article. Be sure to check out Part III of this series, where I’ll talk about how we built a great company, assembled a superstar team, and built a culture that encourages building solid products.

Further Reading

If you enjoyed this post, be sure to check out my earlier blog entry, The Days before the Xobni Launch which talks about the days before launching our private beta at TechCrunch 40.

If you haven’t done so yet, read the first part of the series "Hello World, Meet Xobni".

Sunday, May 04, 2008

Hello, World: Meet Xobni

This is the first of a three-part series on the Xobni launch. Come back on Thursday to check out Part II of the story. This series was co-written with Marie Baca.

Every day, millions of people are forced to deal with the inefficiencies of Outlook. Almost 50,000 people have tried the early versions of Xobni's private beta. Today, we are opening the floodgates and allowing anyone to download a beta version of Xobni's eponymous product for free.

You can read our official announcement here. The New York Times details our launch in this article.

I've devoted this post to explaining why we built Xobni's software the way we did. The other posts in this series will document the journey up to the launch.

Email Overload

Experts say that there are two types of email users: Cleaners and Keepers. Cleaners receive only a few emails a day, and they meticulously file each email into a specific folder. Keepers, on the other hand, receive copious amounts of email, and although they may start out with a good organizational system, it is quickly abandoned. We designed Xobni for the Keepers — the everyday people who need a product that will help navigate their flooded inbox.

The average Xobni user deals with a whopping 30,000 stored emails and communicate with some 1,900 people. For many, this means sifting through several hundred messages every day. It’s only going to get worse: the Radicati Group estimates that by 2009, people will spend up to 41% of their workday dealing with emails. We are experiencing bona fide email overload, and the challenge for us "power users" is to find a way to process and organize large volumes of information over a short period of time.

A People-Focused System

One of the key insights the Xobni team had early on is that users think about email in terms of people and relationships, not abstract tasks. For example, think about the last time you went hunting through your inbox for an attachment. What was the subject line of that email? Can’t remember? Well, what about the name of the person who sent it to you? I bet that you were able to recall that bit of information far more easily. Indeed, the majority of searches inside email clients are for names of people, and it’s those same names that help us identify the relative importance of a particular message. It’s this idea of a people-centered email system that drove nearly every aspect of our development process.

A Smarter System

Let’s take a look at a few of Xobni's features and discuss the rationale behind them.

  1. Super-fast email search. Other than acting as a holding pen for messages, one of the most important functions an email client can perform is allowing the user to quickly search through your emails to find the information they’re looking for. It's such a fundamental need, and yet Outlook’s search is often painfully slow. That’s why we designed Xobni with as-you-type search, so that as soon as you’ve typed "Jan," Xobni has already pulled up all the emails from Jane Smith, as well as all the emails where she is mentioned.
  2. Threaded Conversations. Research indicates that one of the biggest problems people experience with their email systems is being unable to put their messages into context. In a standard inbox, messages are sorted by arrival time, which adds very little meaning to what is being said in the body text. Gmail has an effective method for grouping emails, and with the advent of Xobni, Outlook will also have this ability.
  3. A Built-In Social Network. Just as it is easier to remember who sent you a message than it is to remember the subject line of a particular email, it's much easier to recall relationships between people than it is to remember a name. For example, one of our investor's names is Rob. I can never remember the name of Rob's assistant (sorry, Carly!). For this reason, we designed Xobni to analyze emails and automatically create a network of relationships around each contact. Now when I pull up Rob's name, Carly's name appears on his list of related people, and I can call or email her with the click of a button.

Research vs. Reality

If you take a look at the research that has been done to improve the usability and usefulness of email clients, you'll find that a lot of the work was performed at Microsoft Research. But these ideas haven’t yet made it into Outlook. It's difficult to change Outlook because the improvements have to be compatible with all of the previous versions of the software. Meanwhile, rebels like us are free to build the next generation of email clients, making them faster, smarter, and easier to use.

Be sure to check back next Monday for Part II of this series, where I’ll tell you about a big mistake we made early on: building the wrong product.

Further Reading

  • In my thesis, I review significant research into improving the UI and smartness of email. Chapter 2 gives you more insight into email overload, and Chapter 3 lists a lot of work done in this area.

  • For more background on interesting email-related research ideas, read my earlier blog entry, "How Researchers are Reinventing the Mail Client".