As I said before, I was invited to be on a panel at Velocity Conference. I was delighted to go. I had never been to San Francisco. I have been to Portland and Santa Clara several times. The panel was great. It was the Brian and photo sharing sites show. Seriously, it was me (dealnews.com), John Allspaw of Flickr, Don MacAskill of SmugMug and Farhan Mashraqi of Fotolog. Oh, there was also Shayan Zadeh of Zoosk, a social dating network and Michael Halligan, a consultant from BitPusher. We all had similar ideas. I told my Yahoo story. I told everyone that they should denormalize (or optimize as Farhan prefered) their data to improve performance. Others agreed. I have written about my methods for denormalizing normalized data before. (See pushed cache) Fun was had by all.
I mentioned John Allspaw above. He gave a talk on his own as well. It was good. The slides are on SlideShare. He and I see eye to eye on a lot of things. One thing he says in there that may shock a lot of people is to test using produciton. I agree fully. We could have never been sure our infastructure was ready last year without testing the production servers.
I also learned about Varnish at the conference. It is a super fast reverse proxy. It uses the virtual memory systems of recent kernels to store its cache. The OS worries about moving things from memory to disk based on usage. The claim is that the OSes are better at this than any programmer could do (without copying them of course). It is fast. The developers are proud. And by proud I mean cocky. I have been playing with it. As you know, I have my own little caching proxy solution. Varnish is much faster, as I expected. However, storing cache in memcached is very attractive to me. Varnish can’t do that. It would likely slow it down a great deal. MemProxy does do that. Also, because MemProxy is written in PHP and my application layer is PHP, I can do things at the proxy layer to inspect the request and take action. Works well for my use. But, if you are using squid or mod_cache or something, you may want to give Varnish a look.
There was a good bit of information about the client side of performance. There were folks from Microsoft there talking about IE8. It looks like IE8 will catch up with the other browsers in a lot of ways. Yahoo talked about image optimization. Good stuff in there. I use Fireworks and it does a pretty good job of making small images. I am looking more into combining images and making image maps that use CSS. We use a CDN, but fewer connections is better for users.
There was also a lot of great debate. SANs rock! SANs suck! Rails Scales! Rails Sucks! The Cloud is awesome! The Cloud is a lie! (lots of cloud)
I had dinner both nights with guys from Six Apart. Good conversations were had. I don’t know if I am a big vegan fan though. I mean, the food was good, but it all kinda tasted the same. Perhaps I ordered poorly. At dinner on Tuesday I met a guy going to work for Twitter soon. He is an engineer that hopefully will be another step toward getting them back to 100% again. Lets keep our fingers crossed.
They did announce that the conference would be held again next year. I am definitely going back. Probably two of us from dealnews will go. OSCON is fun. MySQL conference is too. But, more and more, capacity planning and scaling is what I do. And this conference is all about those topics.