Email as a datasource for applications
4 likes3,034 views
The document discusses the role of email as a data source for applications, covering technologies such as IMAP and email protocols. It highlights common challenges faced by app developers when integrating email functionalities and explains how to navigate email protocols effectively. Additionally, it lists potential pitfalls regarding message identification and handling of attachments in email communication.
![1. Connect to the IMAP server and authenticate
>"openssl"s_client"-crlf"-connect"imap.gmail.com:993
["a"few"lines"of"SSL"and"server"info"]
*"OK"Gimap"ready"for"requests"from"123.14.12.20"zw8i38638oab.180
a001"LOGIN"username"password
*"CAPABILITY"IMAP4rev1"UNSELECT"IDLE"NAMESPACE"QUOTA"ID"XLIST"CHILDREN"X-
GM-EXT-1"UIDPLUS"COMPRESS=DEFLATE
a001"OK"username"authenticated"(Success)](https://image.slidesharecdn.com/lsrcpresentation-120810134937-phpapp01/85/Email-as-a-datasource-for-applications-32-320.jpg)
![4. SELECT a mailbox
a003"SELECT""Drive/Dev"
*"FLAGS"(Answered"Flagged"Draft"Deleted"Seen)
*"OK"[PERMANENTFLAGS"(Deleted"Seen"*)]"Limited
*"OK"[UIDVALIDITY"614213447]"UIDs"valid
*"OK"[UIDNEXT"1042]"Predicted"next"UID
*"84"EXISTS
*"3"RECENT
a003"OK"[READ-WRITE]"Drive/Dev"selected."(Success)](https://image.slidesharecdn.com/lsrcpresentation-120810134937-phpapp01/85/Email-as-a-datasource-for-applications-34-320.jpg)
![4. FETCH messages
a013"FETCH"80:81"(FLAGS"BODY[HEADER.FIELDS"(DATE"FROM"SUBJECT)])
*"80"FETCH"(FLAGS"(Seen)"BODY[HEADER.FIELDS"(DATE"FROM"SUBJECT)]"{101}
Date:"Mon,"26"Jul"2012"14:05:16"-0400
From:"Dominik"Gehl"<dominik@gmail.com>
Subject:"test
)
*"81"FETCH"(FLAGS"(Seen)"BODY[HEADER.FIELDS"(DATE"FROM"SUBJECT)]"{115}
From:"Dominik"Gehl"<dominik@context.io>
Subject:"Payment"required"error
Date:"Tue,"27"Mar"2012"09:28:01"-0400
)
a013"OK"Success](https://image.slidesharecdn.com/lsrcpresentation-120810134937-phpapp01/85/Email-as-a-datasource-for-applications-35-320.jpg)
![A simple message
Delivered-To:"sysadmin@context.io
Return-Path:"<2012050639be@bounces.amazon.com>
Received:"by"10.229.135.136"with"SMTP"id"n8mr410292qct.135.1336583200550;
""""""""Wed,"09"May"2012"10:06:40"-0700"(PDT)
Received:"from"smtp-out.amazon.com"(smtp-out.amazon.com."[72.21.212.39])
""""""""by"mx.google.com"with"ESMTP"id"b2si1383913qcd.195.2012.05.06.40;
""""""""Wed,"09"May"2012"10:06:40"-0700"(PDT)
Date:"Wed,"9"May"2012"17:06:39"+0000"(UTC)
From:"Amazon"EC2"Notification"<no-reply-aws@amazon.com>
To:""Sys"Admin""<sys@context.io>
Cc:""Alerts""<alerts@context.io>
Message-ID:"<urn.correios.msg.2012050639be@1336583199032.us-1.amazon.com>
Subject:"Notice:"Amazon"EC2"Instance"scheduled"for"retirement
MIME-Version:"1.0
Content-Type:"text/plain;"charset=UTF-8
Content-Transfer-Encoding:"7bit
Hello,"...](https://image.slidesharecdn.com/lsrcpresentation-120810134937-phpapp01/85/Email-as-a-datasource-for-applications-50-320.jpg)
Email as a datasource for applications
- 1. Email as a datasource for apps Bruno Morency bruno@context.io @brunomorency
- 2. • Overview of the technologies that make email What this • How your apps can fit in that picture presentation will be about • An intro to IMAP and message bodies with common pitfalls. • Overview of Context.IO
- 3. “The reports of my death were greatly exaggerated” - Email
- 4. 2.9 billion in 2010
- 5. 3.8 billion by 2014 180B messages/day 340M tweets/day
- 6. group collaboration task management document collaboration customer support app notification Email is a communications system project management client relationship applicant tracking photo sharing bug tracking Nigerian extortion
- 7. Overview of protocols and standards or “which acronym does what”
- 8. Protocol for SMTP transmission of emails across the internet
- 10. • Message transport, nothing to do with content • Defines the envelope (sender and recipients) • Does not define the message headers • Chain from client to recipient’s server
- 11. DKIM Standards for sender signatures and prevent SPF sender spoofing
- 13. • Complement spam filters • Opens the message and checks headers to decide if it will deliver it to the inbox • As a receiver, it’s one more way to block spam. • As a sender, it’s a tool you must master to avoid ending up in the spam folder • Email deliverability is an industry by itself
- 14. Protocol to allow a IMAP client to access and manipulate emails on a receiving server.
- 16. • All messages and their folder organization are on the server • Clients poll to know about with new messages that arrive or actions made through other clients • While it doesn’t send messages, clients usually store sent messages through it
- 17. Protocol to allow a POP client to retrieve emails from a receiving server.
- 19. • The server only serves as a temporary buffer for received messages • Classification and message state is purely a client-side concept • Many clients can access the same account but can’t coordinate anything
- 20. RFC-822 Standards defining MIME headers and the actual body of the message Multipart
- 21. Where does your app fit in there?
- 22. Typical 1. Send emails to users things apps 2. Receive emails from users want to do 3. Access emails users send and with email receive.
- 23. group collaboration task management document collaboration customer support app notification Email is a communications system project management client relationship applicant tracking photo sharing bug tracking Nigerian extortion
- 31. Me: “App Developer, meet IMAP. IMAP, meet App Developer.” IMAP: “I don’t give a sh*t about you, App Developer. Go away!”
- 32. 1. Connect to the IMAP server and authenticate >"openssl"s_client"-crlf"-connect"imap.gmail.com:993 ["a"few"lines"of"SSL"and"server"info"] *"OK"Gimap"ready"for"requests"from"123.14.12.20"zw8i38638oab.180 a001"LOGIN"username"password *"CAPABILITY"IMAP4rev1"UNSELECT"IDLE"NAMESPACE"QUOTA"ID"XLIST"CHILDREN"X- GM-EXT-1"UIDPLUS"COMPRESS=DEFLATE a001"OK"username"authenticated"(Success)
- 34. 4. SELECT a mailbox a003"SELECT""Drive/Dev" *"FLAGS"(Answered"Flagged"Draft"Deleted"Seen) *"OK"[PERMANENTFLAGS"(Deleted"Seen"*)]"Limited *"OK"[UIDVALIDITY"614213447]"UIDs"valid *"OK"[UIDNEXT"1042]"Predicted"next"UID *"84"EXISTS *"3"RECENT a003"OK"[READ-WRITE]"Drive/Dev"selected."(Success)
- 36. 4. FLAG a message as read a015"STORE"81"+FLAGS"(Seen) *"81"FETCH"(FLAGS"()) a015"OK"Success
- 37. 4. CLOSE the mailbox and LOGOUT the account a023"CLOSE a023"OK"Returned"to"authenticated"state."(Success) a024"LOGOUT *"BYE"LOGOUT"Requested a024"OK"LOGOUT"completed."(Success)
- 38. That didn’t seem so bad!
- 39. • There is no persistent primary key you can rely on to retrieve a Pitfall #1: message Identifying • Message Sequence Number messages • Unique Identifier
- 40. • Ascending and contiguous sequence. If the mailbox says Sequence 11 exist, you can fetch messages with seq. nb. 1 to 11 Number • They can (and will) be reassigned during a session.
- 41. • 32-bit value uniquely identifying a message within a mailbox. • Ascending but not necessarily Unique incremental nor contiguous. Identifier • If you move a message to (aka UID) another mailbox, it will get a new UID in that new mailbox • Changes if the mailbox UIDVALIDITY changes
- 42. • Only the INBOX mailbox has a special meaning. Pitfall #2: • Everything else has the Special-use meaning the client wants it to have (which may not be in folders (or English) lack thereof) • Gmail has XLIST which add mailbox attributes (Inbox, Sent, Starred, ...)
- 43. Pitfall #3: • Anything that searches or fetches messages is done No data until within the context of a mailbox you select a • Can’t get account-wide list of mailbox messages
- 44. • It's an extension that isn't widely Pitfall #4: available and even then, restricted to a single mailbox Threads • X-GM-THREAD-ID to the rescue
- 45. • You need to get and parse the body structure Pitfall #5: Attachment? • As far as IMAP is concerned, an attachment is the same thing as any other MIME part
- 46. • Setting the Deleted flag marks the message for deletion but it’s Pitfall #6: still there Deleting • EXPUNGE will remove all messages messages with Deleted flag from the currently selected mailbox
- 47. • Purging client side message list is a PITA. Pitfall #7: • Server won't tell you which Keeping up messages were deleted, you just have to figure out some with deleted have been and find which one messages were. • It's the same if you want to keep track of Seen flag.
- 49. The joys of parsing email messages Yé! I fetched a message! Now what do I do?
- 51. A message with an attachment MIME-Version:"1.0 Content-Type:"multipart/mixed;"boundary=_MYBOUNDARY_ --_MYBOUNDARY_ Content-Type:"text/plain This"is"the"body"of"the"message. --_MYBOUNDARY_ Content-Type:"image/jpeg;"name="IMG_713.jpg" Content-Disposition:"attachment;"filename="IMG_713.jpg";"size=6379099; Content-Transfer-Encoding:"base64 /9j/4AAQSkZJRgABAgAAZABkAAD/7AARRHVja3kAAQAEAAAAZA+4AJkFkb2JlAGTAAAAAAQMA AwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMD8IAEQgAegG1AwERAI RAQMRAfEASMAAQACAwEBAQEBAAAAAAAAAAAHCAUGCQQDAgEKAQEAAgIDAQAAAAAAAAAAAAAAB gcFCAEDBAIQAAEEAgEBBgQGAQUAAAAAAAUCAwQGAQcAEjBQERMUFRBgFhcgQHAhNAhBMSIjJD URAAIC== --_MYBOUNDARY_--
- 52. A message with alternative parts MIME-Version:"1.0 Content-Type:"multipart/alternative;"boundary=_MYBOUNDARY_ --_MYBOUNDARY_ Content-Type:"text/plain;"charset="us-ascii" Content-Transfer-Encoding:"quoted-printable Hello!"Here’s"a"message"with"*rich*"text --_MYBOUNDARY_ Content-Type:"text/html;"charset="us-ascii" Content-Transfer-Encoding:"quoted-printable <html><body>Hello!"Here’s"a"message"with"<b>rich</b>"text</body></html> --_MYBOUNDARY_--
- 53. Pitfall #1: • Great to track messages but spec says it's optional. Message-ID is optional ... and it’s not always there.
- 54. • Refers to Message-ID of other emails Pitfall #2: In-Reply-To • Very useful to rebuild threads References ... until an Outlook user jumps in and replaces it with their own Thread. Topic and Thread.Index headers
- 55. Pitfall #3: • Content-Disposition tells you Attachments attachment or inline. Should signature image be considered are what you as a file attachment? decide them • TNEF attachments to be
- 57. webhooks threads contacts messages files
- 58. Demo of Context.IO console