Tuesday, February 02, 2010

How to Replace IMAP

IMAP is a complex and difficult protocol. Its original design, RFC 1730, dates back to 1994. Here's a sample IMAP session that shows you all of IMAP's quirks: It's a stateful protocol where you login and select a mailbox to operate on, it has an unconventional parenthesis-heavy representation for structured data, and fetching messages is a multi-step operation.

Design Outline

One of these days, I'll write up a spec for the mail access protocol of my dreams. I'm planning to call it the "reimagined Mail Access Protocol" (short: reMAP) which is handy because reMail already owns the domain name.
I would aim for a RESTful design with the following properties:
  1. All communication over HTTP / HTTPS: Pure TCP connections are great, but for transferring large amounts of email, HTTP is the way to go. Problems like security, parallel downloads, persistent connections, caching, compression, download continuation via ranges, and so on have already been solved. There is no reason to solve them again.

  2. Stateless: There's no reason to introduce state like IMAP does with its selected mailboxes. All you need is a HTTP session cookie used for authentication purposes. Session cookies also allow for things like OAuth. OAuth would let third parties get your permission to access your email without having to give them your username and password.

  3. JSON and UTF-8: All data that's ever sent to or received from the server would be in JSON format. JSON is much more human-readable than XML. UTF-8 would be the only encoding allowed, since it is able to represent any character in the Unicode standard.

  4. Conversations as first-order objects: Gmail, the iPhone SMS app, and Facebook's messaging system have shown the value of viewing messages not individually but in the context of conversations. In reMAP, the server would be responsible for grouping together messages. While you could still access individual emails, the first-order unit of data would be a conversation.

  5. Labels, not folders: Labels are much for flexible than folders. Each conversation should have multiple labels, and the labels would be included when you request the message, rather than having to scan all folders for the message via IMAP.

  6. Stable and unique IDs: IMAP has a UID for each message, but it changes the moment you move the message into a different folder. An IMAP server can also declare all UIDs to be invalid at any moment throughout the session. No more! reMAP would have stable and unique IDs for all conversations, emails, and attachments.

  7. The beginning of the end for MIME: Yes, you could still get the MIME representation of each message that is sent. But MIME is a messy and complex beast. Instead of requesting the MIME-encoded message parts, you could just ask the server to give you the message as represented in plain text or HTML. Attachments can be downloaded in separate HTTP calls.

  8. Push built in: The two prevailing methods for implementing push email are the IMAP IDLE command (not widely available in IMAP servers) and Microsoft's ActiveSync, which requires developers to purchase a license from Microsoft. In reMap, clients could just call an HTTP endpoint on the server which returns as soon as new messages are arrive.

  9. Full-text index on the server: reMAP servers would need to maintain a full-text index of the contents of all messages. There's no reason clients should be required to download and index everything in order to do an exhaustive full-text search of your email.

Enormous Potential

I believe that one of the reasons for the lack of innovation in the email space is the lack of a simple yet powerful email access protocol. Every developer that wants to try something with email needs to first jump through the hoops of IMAP and MIME, or worse, the Outlook Object Model and MAPI. A new protocol like reMAP would lift this burden off their shoulders. We've seen what open, simple standards can do for innovation with Twitter's and Flickr's API. Now imagine unleashing the same sort of creativity to the vast ocean of data that is email. Let me know what you think. My goal with this post was to encourage a discussion about this topic, and your comments are much appreciated. If you liked this, consider following me on Twitter where I often tweet about email-related topics!

49 comments:

Sachin Rekhi said...

Definitely agree with you Gabor! Would love to see innovation in the IMAP protocol. It is very outdated and horrendous by today's API standards.

I think more likely though we'll see the major mail providers provide APIs on top of their own clients, like Yahoo, AOL, Gmail have all announced and Outlook provides to a certain extent.

Sachin

Michael said...

Seems as interestings as everything that starts with an "re"!!!! Let me know if you need beta or even alpha testers.

Voyagerfan5761 said...

Ever since I started dabbling just the user side of IMAP (configuring and using clients on desktop and PDA platforms) I have thought that IMAP is cumbersome. I for one would very much like to see reMAP developed and adopted by some providers, especially Gmail, as Google already uses most of the concepts introduced as part of the reMAP protocol.

Gabor said...

Sachin, Voyagerfan - I totally agree that the major mail providers will likely be the agents of change on this. Microsoft Exchange and Yahoo Mail both already have their respective XML-based protocols, but they're not widely supported or offered (for example, Yahoo heavily limits the power of their SOAP protocol for standard accounts). I'm hoping that we'll see these changes happen sooner rather than later. Gabor

shazow said...

For push, you probably want http://www.webhooks.org/

Gabor said...

Thanks for the suggestion shazow - I'll check it out!

Joshua Baer said...

I love it! I was just commenting to another email startup yesterday (Gtriage) about how badly we need OAuth for IMAP, but you're taking it to a whole new level.

I would love to contribute to this (time and code). Should we start a group to discuss?

Rob Sayers said...

It's interested to read this as I had nearly the same idea not long ago. I've implemented a half working prototype in Ruby using webrick and ferret for message indexing. I have it reading a Maildir and it seems to work fine.

I did however go with folders + tagging, although I don't know that the folders give you anything tagging doesnt.

If I find the time I'll clean it up a little and post some code.

Gabor said...

Joshua - wanna meet up at SXSW to discuss?

Aaron Scruggs said...
This comment has been removed by the author.
David Rusenko said...

What are you waiting for? I'd love to see a sample server implementation :)

Jon Scott Stevens said...

I'd suggest Hessian over JSON. Might as well be efficient in the wire protocol and hessian has a debug mode for when you need to actually read what is going in/out. Hessian also supports RPC which is a nice feature in this case.

http://hessian.caucho.com/

Anonymous said...

I don't see how 9. will work when emails are encrypted.

Gabor said...

Aaron: Totally agree, that's point 2 :-)

Gabor said...

Jon: Thanks for the pointer, will check it out

Andrew Benton said...

totally agree with david. you have the beginnings of the documentation already. if you put out a basic working server i bet people would get pretty excited.

Matt Brezina said...

You could really accelerate innovation with a change like this. hard thing to know is if it is possible for a small company to pull this off, or if it would require a more community-based approach like OpenID. Eventually they found a friend in Google in their fight against Facebook.

shazow said...

Please stick with JSON, or maybe Protobuffers/Thrift if JSON is too inefficient (I don't suspect it is with HTTP gzip).

Also, another thought:

Bonus points if you make the protocol distributed/p2p in nature, and allowing to post emails directly into the mailserver without having to go through SMTP.

For example, if I run a server implementing this protocol, and my friend also runs it, they should be able to talk to each other without going through a middleman.

Anonymous said...

What are you waiting for ?
I think you have the reason and the means to start a specification and a reference implementation.

Anonymous said...

This would be great in the short/medium term.

In the long term Google Wave / the Wave Protocol might replace email completely.

Gaël said...

I must say that I agree with most of your points, and I have for long as well thought of an IMAP replacement.

IMAP is the best standard we have available right now but it is both complex to get right as a server and as a client, mostly useless without a bunch of extensions that you're not sure of having on the server, and too much folder centered compared to today's mail volume and usage patterns.

I have a few ideas lying around about this, so if you get any further I'd be happy to contribute !

Jim said...

I think a key would be making the protocol bidirectional, rather than server->client only. That way you could use it both for mail submission and mail reception. That right there would make any MUA using the protocol 50% easier for the average user to set up. (Of course, whether "average users" actually use MUAs instead of webmail anymore is a different story...)

Admittedly you can push messages from the client back to the server with IMAP right now, but it's not normally used for this purpose, which is too bad.

Raph said...

Great article, i also think email is overly complicated and not adapted this web 2.0 era.You are right, the world really needs an easy way to share mailboxes and delegate access effectively, the complexity of today's CRM demonstrates this lack perfectly.

Such a platform would lead to a new wave of CRM and CMS tools based on the content of communications directly instead of having overly complex system that introduce an overhead to manage copies of the actual data beng exchanged.

My thoughts on implementing this:

WebMachine can be used to build the restful framework for push notifications, it is written in erlang which is a great match for this project due to its concurrency model.

The makers of WebMachine, Riak also make a decentralized, fault-tolerant, document oriented database that comes an http interface for json. In my opinion a document oriented database is definitely the way to go for storing email and related data in this context.

Erlang also has the ejabberd jabber server which could be extended to interact with Google Wave and offer a jabber interface.

It could also use the erlang-oauth combined with webmachine to build an interface to expose its email data

What do you think ?

Gabor said...

Matt: Totally agree on both your points. I think the key to doing this right might be to join up with a big email provider and introduce this change locally, then roll this out more widely.

Raph: Fantastic point about how every system needs to have its own copies of everything right now. I bet CMS'es and CRM systems could make a great leap forward if access protocols were easier.

Gael: Agreed on the need for extensions to make IMAP usable. Hits the nail on the head.

Adam MacBeth said...

Why restrict this to just current IMAP functionality? SMTP is a dinosaur too (though less complicated and broken). It's time to upgrade email with a single protocol covering the send AND receive side.

More generally, the internet needs a generic messaging protocol. XMPP is ok, but isn't broad enough. There's really very little difference between an email, IM, tweet, SMS, or Facebook message aside from usage conventions and historical limitations (size, immediacy). What you're describing could easily encompass all these existing messaging systems.

Gabor said...

Adam: Agree on the need to replace SMTP, but I believe that's going to be harder than switching out IMAP.

Replacing IMAP will only take replacing clients and servers. To replace SMTP, you'll need to migrate every single mail server out there.

Adam MacBeth said...

Right. I was referring to the usage of SMTP for sending from client to server.

Arjun Satish said...

This is definitely extremely needed. Email software is stuck in the '80s.

Its a big nightmare to create tools to sync emails. Its probably a good idea to use an XML based interface to store emails. That way, syncing your mails will be as easy as querying XML. Also, allowing easy integration with RSS (which opens up a lot of opportunities in itself).

j_king said...

I'd skip on the HTTP.

Stateful protocols make writing clients much easier. And as long as email is still hosted, why bother making things more difficult? I don't see how a stateful protocol makes managing email more difficult. It's a feature, IMO.

The rest however is somewhat agreeable except for JSON. It's still not standardized cannot be validated. Neither can email, but parenthesized structures are easily parseable -- just look at Lisp.

It'll be quite a task to revisit IMAP and see what one could do from a modern context. The best way is to experiment and share your findings. Best of luck!

Joshua Baer said...

Gabor, lets organize a time and we can host a meetup during SXSW. My office is walking distance from the Convention Center and we have a conference room big enough for 50 people.

mnot said...

You might want to ping Lisa Dusseault --

http://tools.ietf.org/id/draft-dusseault-httpmail-00.txt

Erik Terpstra said...

http://www.prescod.net/rest/restmail/

Dan Keen said...

Interesting post, but a few comments…

1) HTTP is over TCP, of course. Ignoring that, security is no different (IMAPS = HTTPS), persistent connections are poorly supported on HTTP, compression is already supported (IMAP COMPRESS extension), continuation via ranges is most definitely supported.

Parallel downloads is supported but requires multiple connections - but is very, very common. Caching... huh? I'm not sure how that relates to what you're asking. I'd imagine any HTTP email protocol would have an infinite cache life, which is no different than what you'd consider IMAP today.

2) State… do you have all your data on your device, or in the cloud? If you leave nothing on the device, state does become less important - but you'll always have some state on the device even if it's temporary. Managing differences between your current view and what has updated on the server is the whole point of having state.

3) UTF8, no argument. JSON? Ok… it's a manner of talking. That's like saying IMAP is ABNF. You have to define a grammar on top of it, and JSON wouldn't be it.

4) Already in IMAP. See the THREAD command, which takes many arguments for different methods of grouping messages into a conversation.

5) Again, already in IMAP, although it's a bit janky (only certain number of labels allowed).

6) MIME already gives you a stable ID with Message-ID, and good IMAP servers give you a method to search on it. IMAP generally will not invalidate the UIDVALIDITY unless the server itself has been reset (i.e. from backup). Exchange has a similar "reset state" mechanism, it's pretty much required.

7) Huh? You're saying "give me just the html or text". What about all the metadata? From? Subject? etc… where is that coming from now?

And how would you deal with more complex formats? You're reinventing the wheel here for little reason. MIME itself is surprisingly easy to grok. The worst part is all the encoding fun, but that goes away if we all just move to UTF8.

8) Push is built in. Use IMAP IDLE. Done. You can't complain about lack of support here - you're building something fresh, so whatever you make will have even less support. In the future there's also IMAP NOTIFY.

9) And.... again, this already exists in IMAP. Server implementations vary, of course, but many support full text searching.


IMAP isn't perfect by any means, but this list isn't a good reason to ditch it.

(disclosure: I work on a heavily used IMAP client)

Bron said...

Interesting comments Dan.

I work on a pretty commonly used IMAP server - and I'm sitting on the fence here.

The big pain I see (using offlineimap myself) is that it's:

a) a pain to merge data from two mailboxes without invalidating all the IDs
b) a pain to search or move messages between mailboxes.

Basically it's a very one-dimensional index.

Oh, and then there's:

c) you need to fetch ALL FLAGS every time you want to check if anything is changed. There are options (CONDSTORE - which is a bit dodgy across expunges) and QRESYNC which reduce this hurt, but they're not well supported yet.

Also, IMAP plus a random mix of extentions becomes a HUGE compatibility matrix, and you have to offer fallback from all the nice stuff to really shitty workarounds just in case the server doesn't support a feature that makes your life bearable.

So - I'd love to see a decent competitor to IMAP as a protocol. If nothing else because it would reduce the compatibility matrix. Alternatively, I'd be happy for an IMAP5 to be defined which included all the good extentions - so anything which advertised support for IMAP5 had to support them.

Along with a decent test suite a-la DAV's "litmus" this would allow us to move on and simplify a lot of both client AND server code.

MM said...

Adam: XMPP is not a bad idea actually. There are obvious limitations, but there clearly is an big advantage using such a generic protocol: it exists.

Yet, I clearly see that underlining pros of XMPP (such as versatility and presence management) would potentially lead to... re-inventing Wave.

randomwalter said...

I could not agree more, and in fact I've made a couple of attempts along the lines you suggest -- queries done via HTTP requests, replies contain JSON data. If you make sure messages are immutable (except for labels), you also gain benefits of HTTP caching.

I'd love to work on a Perl or Erlang implementation of this.

Gabor said...

@Dan Keen: Good thoughts.

One minor thing I want to point out: Unfortunately, the Message-ID header does not give you a good and stable ID since it's not defined on all messages. I know it's supposed to be there but a lot of senders simply don't follow the rules.

Gael said...

Interesting thoughts everywhere ;)

I agree with some others and my first choice of protocol wouldn't have been HTTP/REST/JSON, but I don't think whether it is JSON or IMAP/Lisp-like, or XML or binary is really relevant here.
What is however is the set of functionality that is accessible though it. Once the features are here, you can easily offer multiple interfaces to it as needed.

Having support for initial mail submission from clients sounds nice (simplifies setup and archiving sent mail), but then still use SMTP to transfert it to other servers as needed.

For having writen an IMAP4 server some while back, I agree with Bron and would welcome this as a potential good IMAP5, without the folder oriented biew but only labels (that can be seen as folders as well when needed).
IDLE, THREADS, SEARCH and UID do exist in IMAP4 but they operate on only one folder at time, requiring the client to open many connections for multi-folder operations; How many folders/labels do you have ? That's a huge limitation nowadays.

I wouldn't go for MIME's "Message-ID" either for identifying messages, since there are no guarantees on its unicity at all. A global mailbox UUID would be much better.

With all this, having partial IMAP4 support over the new protocol seems mandatory for easy transition, and as such keeping the original MIME-structure of messages around would be needed.

Hope that helps ;)

Anonymous said...

how about some data integrity too? Im sick of losing my email to these idiotic services. I like how Git uses hashcodes to keep data safer... corruptions and data loss are limited by this pattern. Id love to see stuff like this make into my email which happens to be damn important to keep alive!

Russ Nelson said...

Yes, IMAP is a turd (three encapsulation schemes???), but you're not the first person to propose alternatives.

http://en.wikipedia.org/wiki/Internet_Mail_2000

waseem said...

Hi Gabor,

In a couple of weeks we are launching our API or Inbox2 which does more or less what you describe here. I will be writing documentation in a few weeks on this. Its basically an http/xml/json based email push (using http long polling) protocol. Authentication is provided by oauth-wrap.

I'll ping you in due time would love to hear your thoughts on it.

Cheers,

Waseem

David Volgyes said...

Stable and unique IDs: SHA-512 vagy hasonló alapú hash? Így elég könnyű lenne két különböző szerver adatait merge-elni, soha nem lenne két azonos ID, ... S a szerver oldali adatintegritást is könnyű így ellenőrizni. (S mellesleg: ha bármi másra szeretnéd kiterjeszteni a szerver funkciót (wave, ...) , az SHA512 akkor is garantálja, hogy az új elemek nem fognak ütközni a régiekkel.)

Ehhez hozzájöhetne az, hogy egy levél formátuma bármely két szerveren legyen pontosan azonos, s egy letöltött levelet a kliens visszahelyezhessen a szerverre. Ezzel kényelmesen megoldható lenne a backup is, illetve az, hogy a levelek egyik szolgáltatótól a másikhoz áthelyezhetőek legyenek.

Push mail: jó lenne értesítés új levélről és külön a letöltési lehetőség. Tehát ne mindenképpen töltődjön le a levél, csak ha a user/kliens engedélyezi/kéri. Gondolj nagy méretű adatokra, mail bomber problémákra, ...

Keresés szerveren: valami szép SQL-szerű keresőnyelv lenne kényelmes, így alkalmasan nagybonyolultságú lekérdezések is végrehajthatóak, s elég nagy fejlesztői szabadságot ad a kliens készítőinek.

Protocol: Az OK, hogy http felett menjen. Következő rétegnek én ajánlanám a Google protocol buffer-jét. Ezzel viszonylag kényelmesen megoldható az, hogy a különböző verziójú kliensek és szerverek kommunikáljanak egymással.
(Nyilván elsőre nem lehet tökéleteset alkotni, s ez meghagyja a bővítés lehetőségét.)

Joel West said...

Gabor,

There's a wonderful organization called IETF that would love to hear your proposal. Perhaps now that you're working for a large cash-rich Silicon Valley company, you can use your 20% time to draft a proposal.

Joel

axel said...

The single most important design flaw of IMAP is the omission of first class conversations. This makes IMAP broken beyond repair.
Instead of having mailboxes containing messages it should be mailboxes containing threads/conversations containing messages. The THREADS extension to IMAP just doesn't cut it at all. A conversation has to be (potentially) included in more than one mailbox. No way to do that with IMAP properly - e.g. having read status right.

NotZed said...

The basics of MIME aren't that complicated, only the I18N stuff (and the awful pgp multipart stuff), which could probably be fixed. And forcing everything to use UTF8 might sound like a good idea in a country that doesn't need it anyway (i.e. one that speaks english), but it may not work for the rest of the world. The basic rules of rfc 822 envelopes and headers are even simpler.

JSON looks like the wrong approach - mail clients already need to deal with rfc822 headers and mime-extensions, and json just looks like a simplified-yet-slightly-different version of exactly the same thing. All you would do is confuse people and then they'd start sending out even more broken email messages than they do already, hard I know as that is to imagine.

MIME will have to stay around for email because HTML cannot be used to encapsulate multi-part content into a single package.


"Pure TCP connections are great, but for transferring large amounts of email, HTTP is the way to go. Problems like security, parallel downloads, persistent connections, caching, compression, download continuation via ranges, and so on have already been solved."

Haha, very funny. The only had to be solved because HTTP is such a horrid beast that turned into something way beyond it's initial imaginings. IMAP hasn't needed to solve most of those problems because they were already part of the protocol. e.g. caching is easier because uid's can be static if the server wants them to be, all connections are persistent anyway, multiple connections are possible, partial down-loads are part of the protocol (by mime-part or byte-range), even pipe-lining of requests is built-in (hides latency without overloading the server) etc. etc. IMAP was designed in an era when slow and unreliable connections were the norm, so it copes with them reasonably well.

And finally your example IMAP session ... well it would be interesting to see just how huge that would be in any implementation of a HTTP-based restful service. Including all of individual http re-connections that might be required, all of the headers passed in each state-less request, and so on and so forth. I think you'll find it will put your idea into a bit more perspective.

Just because *you* don't understand the output doesn't mean it doesn't make a lot of sense. Although the '12 FETCH FULL' command and result for example is the sort of stuff nobody actually uses - since servers never implement it properly.

e.g. just translate this single command into the equivalent restful service including all connections/auth and so on:

C: a004 fetch 12 body[header]
S: * 12 FETCH (BODY[HEADER] {342}
--body--
S: )
S: a004 OK FETCH completed

And note that the above text is the entirety of the protocol exchange required by IMAP.

Anyway - TBH I think the whole point is moot. IMAP will die a slow natural death simply because the major software vendor in the world hasn't blessed it. Most remote clients in use don't use IMAP because they use some proprietary version of the same, or even embed the client (e.g. gmail). Something like your idea might be another step along that path, but there's probably no point aiming to explicitly 'replace imap' with another hashed up protocol that missed things you didn't realise were needed.

(and i can tell you from experience, a lot of IMAP doesn't make sense until you finally grok it in its entirety, then it is actually a thing of surprising beauty and simplicity with only a few warts that are easily ignored).

Paul Koan said...

SMTP is fine for sending emails, but yes it doesn't need to come from the client.

I think both dovecot and courier support sending emails out via SMTP, all you need to do is put the email into the Outbox folder.

reMAP should do the same thing.

Oleg said...

Gabor, I am working with email systems and I am very interested in implementing your ideas about replacing IMAP.
Could we discuss this by email?

Thanks, Oleg

brousky said...

I remember reading and wholeheartedly agreeing with this post when it was published.

Since then we launched a Gmail contextual gadget to email organize attachments and had to build on API over IMAP to be able to do what we wanted (see dokdok.com and dokdok.com/email-api).

Would love to talk about it with you.

Unknown said...

You might be interested in checking out heliotrope - a server that does most of what you suggest in your post. The only client so far is turnsole.

They are working now, written in ruby using rack. The protocol has started to be documented.

They are written by the William Morgan, who previously wrote the mail client sup. The lists for discussing heliotrope/turnsole continue to sup-talk and sup-devel, together with the issues on github.