ryan kennedy’s blog

I have a hack day tradition. It’s not something cheesy like wearing my favorite shirt or having a certain type of food for breakfast. My new tradition is having my demos bomb.

The tradition started back in September 2006. I was chugging through my slides and made it to the point where it was time to show some running code. Right at that moment our Chief Product Officer, Ash Patel (pictured above), walked in. There’s nothing quite like going from “hey, awesome…an exec is here to see my talk” to “*$&! my demo just bombed in front of Ash.”

Today, the tradition continued. Dan managed to make it through his half of our talk with no issues. I took the podium to take up the second half. I was flowing through the simple slides I had put together and was heading right into the online demos I was going to do. I brought up the site, clicked a link…nothing. Checked another site…nothing. Someone in the crowd yelled out that connectivity was down. Crap. Deep down, I knew what happened. Ash had arrived…

So I started calling out, “Ash is here, isn’t he?” I did it jokingly, I hadn’t actually seen him. Sure enough, 30 seconds later, Ash was standing at the back of the crowd, waving with a smile. Now…I don’t want to say he’s a jinx, but it is becoming an awful coincidence that he’s around during my most spectacular bombs.

Photo courtesy of davglass.

I am mentally and physically exhausted right now. I woke up early this morning (8am) after a long night (out until 3am), headed up to Alexandra Palace with a few other Yahoo!’s and got to work with volunteers from Yahoo! UK and the BBC. We helped out for about 5 hours and had to leave to get to a prior engagement (pub) while the remaining volunteers stuck around and finished the job.

After spending some time at the pub relaxing with folks, we headed back up to the palace to inspect the handy work. Fucking…brilliant. The place looks amazing. After the Sunnyvale hack day, I figured it was going to be hard to top that experience. I was totally wrong. This hack day is going to be yet another one of those events in my life that leaves a permanent mark. I’d say this one will be difficult to top, but I said that last time and now look what they’ve done.

Of course, this doesn’t happen without the amazing people who are working on the event. I feel overwhelmed getting stuff ready for my talk…but I just have to worry about covering my share of an hour time slot. These people have been busting their asses for months and now, with the event just around the corner, they’re up against it to pull off what is sure to go down as one of the most epic events ever to take place at Ally Pally. If you’re at the event and you see a volunteer…thank them. You owe those people for the experience of a lifetime.

Now, if you’ll excuse me, I have my own hackathon getting underway. I still don’t have my demo app together for my talk tomorrow, I should figure out my gameplan for my portion of the talk and I need to transfer source code to 100 USB thumb drives. If I’m lucky, there will be some sleep somewhere in the cards for me before I have to leave at 8:45am to get up to the palace.

Update: I shot a very short video of the interior of the palace. As I said…fucking…brilliant.

Over at the Hack Day Blog, the speakers for the London Open Hack Day were announced. It’s quite a way down the list, but my name is actually on there (the yuiblog guys put me way up at the top last year…hmph).

I’m starting to get excited, it’s only a little more than a week until I leave for London. I still need to get some stuff together. I got my cell phone unlocked the other day after a gentle reminder from Jeremy Zawodny. I still need to find myself a SIM card, but Jeremy’s linked a bunch of URLs to have a look at. I also need to get a power adapter for my laptop and a voltage adapter (I assume) for my digital camera charger. Same song and dance I had to do before I left for India, minus the shots and pills.

I’m also a little nervous. I’m totally unprepared for my talk at this point, although I’ve stood in front of people and talked about the Yahoo! Mail web service so many times now that I can probably give an impromptu talk without much difficulty. That being said, I was hoping to bring some demos and code samples and I haven’t scraped together as many as I would like. Guess it’ll be a busy weekend for me.

The hackday.org site has some new badges put together for the London Open Hack Day next month.

This man El Guapo, he’s not just famous, he’s IN-famous.

I caught this on Chad’s link blog. It’s a video done by webpronews.com. The video is about the release of the Yahoo! Mail Web Service. While Chad was fixated on the old photo they have of him…I’m more interested in the screenshot they took. Why…because it’s a screenshot of my technically sweet hack (Yahoo! Mail Search - my technically sweet hack…see it 41 seconds into the clip)!

After more than two years of development, the Yahoo! Mail Web Service team finally had our offsite. We headed up to San Francisco and took a three hour tour (go ahead…we cracked Gilligan jokes too) on the Adventure Cat catamaran. Aside from a lack of sunscreen, it was a great time (I’m burnt…forehead to chin, pass the aloe). We started out from Pier 39 and wound around the south end of Alcatraz. After that, we headed north and sailed between Angel Island and Tiburon. When we finished there, we swung by and buzzed the tower at Sausalito. We were going to just sail by the Golden Gate Bridge, but we convinced the captain to take us under the bridge. For the most part, the trip had really calm waters and almost no wind. That all changed when we headed out to the bridge. The wind kicked up and the water became much choppier. This was actually my favorite part of the trip. Some of my best pictures came from that period of the trip.

It was a good time. If you ever get a chance, I totally recommend boating in the San Francisco Bay. Do a little research ahead of time, though, and make sure you get a day with good weather. Friday was gorgeous and made the experience so much better. Nothing like being out of the office on a nice, sunny day in the bay.

I mentioned before that there’s an opening to be my intern. We’re also looking for interns who want to work on Yahoo! Mail Beta. If you’re interested, shoot me an email (rckenned AT yahoo.com) with your resume and I’ll deliver it personally to the engineering manager.

It’s a great team to work with and they’ve got an awesome product. Really, you couldn’t ask for a better summer project to work on.

I’ve been waiting for a while to announce that I’ll be traveling to London in June for the next Yahoo! Open Hack Day!

The last Open Hack Day was epic. I had way too much fun giving my talk, helping out developers and getting my hands dirty. This Open Hack Day promises to be twice as epic simply because of the location (London, which I’ve always wanted to visit) and the setting (Alexandra Palace). I’ve been asked, once again, to come and give a talk about the Yahoo! Mail Web Service. Last time I gave an overview of the API, which was nice since the web service was entirely new at the time. But now that it’s officially released, I think I’m going to come loaded up with some sample applications that show the power of the platform.

I’m also heading to London early. I’m thinking about working a day or two in the Yahoo! London office and then taking a couple of days before Open Hack Day to check out London since I’ve never been. If you have recommendations of places to check out and things to do, lay them on me. If you’re in London and want to hook up, let me know. If you’re thinking about coming to Open Hack Day and need a spare programmer, I’d love to make my services available (especially if you want to hack on Yahoo! Mail!!!).

Anyway, it’s going to be awesome, so get your arse there. The sign up form is at the usual place.

Flickr image by Dan Harrod.

Now that the req is finally approved, I can start begging asking for someone to come intern with Yahoo!. The position is focused on helping out with product evangelism for Yahoo! Mail. I do a lot of mining of the Internet for people talking about Yahoo! Mail. I’m hoping to get someone who can help me build some cool tools to mine that data and bring the useful bits to the top. If you’re interested in interacting with users, well…I’ve got 250 million of those and there’s nothing they love better than knowing they’ve got the ear of someone within the company.

It won’t all be evangelism, either. I’m hoping whoever takes the position will be interested enough in mail to hack on some applications that use the Yahoo! Mail Web Service. Really, the position has the opportunity to be as boring or exciting as you want it to be. We can make stuff up as we go along. Anyway, if you’re interested or know someone who is, shoot me mail (rckenned AT yahoo.com) and we’ll get some interviews set up.

Update: Fixed the URL to the job description. The old link stopped working. The new one points to Hot Jobs.

Yeah, we’ve been busy bees. First the unlimited storage announcement. Then the web service announcement. Today we’re announcing an update to Yahoo! Mail Beta.

I broke the ymailupdates.com post down into three parts, mostly because it worked with the title (yeah, the “Bigger” part of the post is forced, but “better…FASTER” just didn’t seem to cut it). Anyway, the “Better” part of the update includes a ton of updates that were made in response to user feedback. We’ve really taken to heart the things we hear about the most in terms of switching costs from classic to Yahoo! Mail Beta. Checkboxes were one of those things. Some users either don’t want to Shift-click or just aren’t comfortable with it.

On the “Faster” end, we really have put a very focused effort into making it faster. Weekly meetings with the frontend folks, us web services folks and a couple of really good performance people have had major payoffs. We’ve nailed down several areas that were creating performance bottlenecks. While we didn’t get them all, we’ve made some serious headway.

One performance measure in particular, componentization, has yielded some good results. Componentization, in this case, means breaking up the app so it’s no longer monolithic and loaded entirely at launch. Now several components are loaded on demand as they’re needed. It was interesting to hear last week, while I was at ETech, that Zimbra announced a similar strategy in their application.

When I went to Bangalore (post 1, post 2, post 3 and post 4) last year, I was asked to give a brown bag talk about the Yahoo! Mail Web Service. After giving the talk, a handful of people approached to talk in more depth about certain features of the web service.

One person in particular, Gopal, was already building something with the web service. I had been in contact with him previously over email and had heard about his application. He invited me down to his desk to have a look and to help debug a few things. What I saw caused my jaw to drop. Gopal had built a XUL interface to Yahoo! Mail using the web service for communication with the backend. It was fast, it was pretty and he had managed to pull off client-side message threading.

If you need to see the benefit of open services in action, look no further. Gopal showed precisely what individuals can do when the platforms are exposed to the outside world (or, in this case, just exposed outside of the mail team). For more information, check out Gopal’s post on his project…Tapestry.

Well, now that we’ve released the Yahoo! Mail Web Service, I suppose we have some explaining to do. In particular, I’d love to talk from time to time about the choices we made, why we made them, things we’ve learned and so on. For this first installment, I figured I’d do a discussion of our two endpoints: SOAP and JSON-RPC.

SOAP
I’m sure most of you will have your opinions of SOAP. Be that as it may, some people actually like it. You can’t deny that certain languages make it dead easy to consume SOAP web services. You feed it your WSDL and it poops out a wad of code for you to call the service as though it were a local call. If only it were that easy. In the end, you may end up dealing with some interoperability problems. During Open Hack Day, we made v1 of the web service available to users. When we released on Wednesday, we were at v1.1. Why? Because we discovered an interoperability issue in the WSDL that made it basically impossible to consume the WSDL using Visual Studio .NET or Axis2. In fixing it, the XML generated by the service was tweaked just enough that it was no longer compatible with the v1 output.

So, while SOAP is supposed to make your life easy, it’s not perfect.

JSON-RPC
We added SOAP first because we had to get Oddpost up and running on the Yahoo! Mail backend. Oddpost ran using SOAP, so we figured the fastest way to get them up and running was to come as close as possible to duplicating the existing service. Once we got rolling however, we realized that it wouldn’t be difficult to add new endpoint types as long as they were close enough in behavior to SOAP that we could map things easily.

JSON-RPC isn’t all that widely used, but it’s semantics are similar enough to SOAP that it was viable. JSON-RPC is a JSON object consisting of three properties: a method name, a list of parameters and an (optional) ID. Maps pretty well with the WSDL. In addition, browsers love to consume it. A single method call (to either eval() or a strict JSON parser) and you can turn the string into a native JavaScript Object. Handy. No more futzing with the cumbersome DOM.

But beyond simplicity, JSON-RPC has another benefit. A benefit that becomes much more evident when you have to scale to a quarter of a billion users: efficiency. It may seem clear to some that parsing/generating JSON is more efficient than XML, but here’s some numbers that make things a little clearer.

JSON-RPC always beats SOAP, in the case of ListFolders (a very frequently called method) it beats SOAP pretty badly. Even in the other cases where the payloads are larger and SOAP overcomes the penalty of having to transmit an envelope and namespaces, JSON-RPC still beats it by a decent margin (especially if you’re looking at things from the context of the server where we’re serving requests for the quarter of a billion users mentioned earlier).

Say each user makes one request in a given day to each of the 3 methods mentioned above. The total combined difference between JSON-RPC and SOAP for a single user is only 3,374 bytes. Extend that out to a quarter of a billion users and you’re now talking 785 gigabytes. That’s close to a terabyte of data transfer saved doing nothing more than choosing a more efficient object encoding. Of course, it’s not entirely accurate because I haven’t accounted for HTTP compression, but you get the idea.

Set aside the network cost and also consider the parsing cost. In most cases you’re going to hold those bytes in memory while you’re parsing them or you’ll buffer them in memory after rendering them. More bytes in the object representation means more bytes in local memory that you’ll suck down just in handling the serialized objects.

I have a lot more I want to say about the Yahoo! Mail Web Service release, but I’m dead tired already and I want to make sure I do this first.

First thanks have to go to my former boss, David Koblas. He rescued me from a fate worse than death (boredom) at EarthLink and brought me to Yahoo! to work on the Yahoo! Mail Web Service. Without this opportunity, it’s unlikely that an open mail web service would have ever happened.

To the team of engineers who worked on and with the Yahoo! Mail Web Service, thanks. This includes people on my immediate team and those on other teams who use the web service, providing feedback and bug reports. There are way too many people to name here and I’ll likely forget to include someone, so I’ll just leave it like this. If you wrote a line of code in the web service or wrote a line of code that talked to the web service, thanks.

To the product people, you could have easily gone the other direction and kept Yahoo! Mail a closed system. Instead you embraced the idea of an open platform and encouraged us wholeheartedly. Thanks.

To the Yahoo! Developer Network, you’ve all been amazing in the last month helping us out with sample code, documentation, screencasts and setting up the developer site. It’s difficult to think of any group within Yahoo! that has been more supportive in getting this web service out into the world. Thanks.

To the hackers who stayed up until the wee hours with me at Open Hack Day, I had a great time watching and helping you build awesome applications on the pre-release version of our web service, forging the way for future Yahoo! Mail hackers. Thanks.

Lastly, to my wonderful, understanding and infinitely patient wife…Lisa. I’ve poured so much of the last 2+ years into this project, sometimes at the expense of our relationship. When this project took me to the brink of physical collapse in the last 5 months, you were there to take care of me. You’ve been incredibly understanding, knowing how much this project has meant to me and your encouragement when things were tough kept me going. Thanks and I love you.

Chad alluded to the imminent release of the Yahoo! Mail Web Service during his keynote talk at ETech today. I believe he said, “really, really, really, soon…hint, hint.” It’s true, the web service is officially released. I have a lot more to say about this, but I’m scrambling to get blog posts and email sent out so I’ll hold off until later tonight.

Until then, feel free to check out the documentation and check out the handful of apps we have in the gallery. In addition, there ought to be some screencasts linked from the YDN blog post. I’m in one of them discussing the mail web service. The other one is with Leah Culver, who helped build the Flickr Postcard hack.

Other coverage:

• GigaOM: Coming soon, Yahoo Mail API, turns it into a platform
• Chad Dickerson - Yahoo! Mail Web Service released!
• Jeremy Zawodny - Yahoo! Mail API and Unlimited Storage
• CNET News.com - Yahoo opens up e-mail APIs to outsiders
• ProgrammableWeb.com - Yahoo Mail API Launches
• Pulse 2.0 - Yahoo! Mail’s Unlimited Storage and API
• Dave Winer -
  Yahoo mail API
• Matt McAlister - The Yahoo! Mail Web Service screencasts
• WebProNews - Yahoo Mail Develops An Open API

Update: Added some more to the coverage list.

Update 2: Some more coverage.

• Alex Barnett - APIs, APIs APIs!!!!! (And Learn As You Play)
• Read/Write Web - Yahoo Mail API Launched
• LinuxWorld.com.au - Yahoo opens up Web mail APIs

Yeah, we went there. The throwdown. Unlimited mail storage. For free. Now, if you need me, I’ll be offering my services showing Gmail users at ETech how to sign up for Yahoo! Mail accounts.

I’ll be attending the O’Reilly ETech Conference next week in San Diego. Along with checking out some of the sessions, I’ll also be helping out at the Yahoo! Developer Network booth. I’ll be doing the 10:15-11:30am sessions on both Tuesday and Wednesday. Swing on by and introduce yourself (or say hello if we’ve already met) if you’re there.

I’m looking forward to it. I had to back out of the Boston Mashup Camp in January because I was having some health issues. On average, I haven’t been very healthy since early November (shingles, food poisoning, sinusitis, …). But I’m feeling pretty good now, so I’m excited to be hitting the road again.

I’m a little bummed that I’ll be missing our internal Hack Day on Monday/Tuesday. I missed the December Hack Day because I had shingles. The Hack Day before that I took my Technically Sweet Award, so I was looking forward to participating again. If I can make some time this weekend, I might do a little hack and then submit a screencast for people to check out while I’m in San Diego.

Anyway, like I said…if you’re down at ETech, feel free to look me up. Drop me a comment here or shoot me mail (rckenned AT yahoo.com). Also, if there are any talks at ETech that you highly recommend, feel free to make suggestions. I’m just starting to look over the talks now that I know my schedule a little better. I’ll be catching Chad’s talk for sure (and so should you), but outside of that I’m up for anything.

Download Squad is no stranger when it comes to covering Yahoo! Mail, especially Yahoo! Mail Beta. They’ve done it again, this time posting 11 tips and tricks for using Yahoo! Mail Beta.

I’ve actually been meaning to do a similar post on the Yahoo! Mail Updates blog, so now I’ve got to come up with something else. I actually have a couple of other tips and tricks up my sleeves, so maybe I’ll still do a post to cover those. I was thinking about doing those as a screencast, so maybe I’ll include the 11 Download Squad tips and tricks when I do that.

For any of my readers (all 211 of you, according to Feed Burner) who know Shawn Grunberger, you may not know that he went in yesterday for heart surgery. He has a blog that chronicles everything going on.

Anyway, during a meeting yesterday, we got the call that Shawn came out of surgery great and that they didn’t have to replace his aortic valve. Such awesome news, more can be found here.

For those of my readers who use Yahoo! Mail Beta, realize that every time you use it you touch some piece of code Shawn wrote. Shawn is by far one of the coolest people I’ve ever had the opportunity to work with and I’m very thankful that he came through surgery successfully.

I just posted over on the Yahoo! Mail updates blog regarding the new Messenger integration in Yahoo! Mail Beta. This was pretty cool because I did my first screencast for the blog. I had fun, although trying to come up with a scenario that doesn’t sound totally contrived is difficult (as evidenced by my performance). I’m looking forward to doing more screencasts, though. Using a trial of Camtasia I found out why it’s $300: the video editing capabilities. It’s not the most full-featured video editor, but it’s pretty damn simple to use. So I’ll be requesting that the company pony up the $300 so I can do more screencasts (I would have paid anyway, otherwise I’d feel bad about using a trial for a commercial screencast).

Anyway, I’m embedding the screencast below.

Coverage:

• CNET
• PCWorld.com
• Yahoo! Mail France - Damnit, their blog is both spiffier and updated more frequently than the one I run. This cannot stand.
• Download Squad - I love those guys!
• Press release

I saw Pipes a while back during our internal hack lunches. Ever have one of those jaw dropping moments crossed with one of those “duh” moments? That’s basically what it was for me. A slick, intuitive interface for manipulating RSS in a way that leaves every UNIX geek out there saying, “that’s what we’ve been talking about for decades!”

Ed Ho has some details on his blog. That links to a longer O’Reilly post that goes into some depth on Pipes.

Play with it, it’s awesome. And check out Uncle Hulka en Francais.