Ramblings of a web guy

So, day two was the cool keynote day.  Day one keynotes were from Tim O’Reilly (not that he is not cool) and the vendors sponsors.  The Intel building blocks stuff was neat, but most of it was vendor stuff IMO.

Today we had the “cool thing to here and see, but I proabably won’t use it” keynote.  It was The Processing Development Environment.  It was really cool.  You can read more about it at processing.org.

The next keynote was hard for me to follow.  There were no slides he stood behind the podium the whole time.  Gnat  seemed to love it as he all told us in IRC.  You can read the guys blog at overcomingbias.com.  It was basically about overcoming the biases you have…. I think.

Interestingly, (speaking of bias) the next keynote was from Microsoft.  Coincidence?  According to the speaker, MS (or at least this guy) is really trying to make some Open Source stuff.  Time will tell.  Also, they are “working” with the OSI to get their licensing approved as Open Source licenses.  As somone in IRC said, its a win/win from them.  If they don’t get approved, they can just blame the OSI for being inflexible.  Nate kind of put him on the spot about patents after his talk.  He handled it well and kind of rode the fence.

The last keynote was, for me, the pay off keynote.  Its the one I will remember from this year the most.  It was about branding.  The poor guy did not have his slides due to technical issues and still did a great job.  You can read Steve’s blog at steve-yegge.blogspot.com.  Maybe he will post the slides.

I attended a couple of good sessions today.  One was about caching, mostly with APC.  But, if you stripped down the APC stuff and just took some of his concepts, you could apply some of it to lots of caching methods.  The talk was given by Gopal Vijayaraghavan of Yahoo! I don’t have a URL for the site where his slides may be.  If I find it, I will post it.

Another one was about legacy PHP code.  I didn’t agree with 100% of what he was saying, but if you are in the boat he described, anything is better than where you are.  The guys site is clintonrnixon.net.  Hopefully he will put of the slides and maybe a blog post about it.

The last talk that I want to tell you about was from Amy Hoy.   She gave the “When Interface Design Attacks!” again this year. Just like last year, it was brilliant.  There were new topics like web 2.0.  I was happy to see that the Phorum 5.2 template I have been working on (emerald) already included many of her recommendations.  I guess she rubbed off on me last year.  Amy has started her own consulting company.  If we need a usability and/or interface design help again (bleh, the last one was less than exciting) I will push for using her for sure.  Check out her site (linked above) for more stuff from her.

The day (and conference really) ended with parties.  We went to the Sourceforge Open Source Awards party.  phpBB won best tool for communication.  Gag me with a chicken bone.  I guess it has a large install base.  But, MySpace has lots of users too.  That does not mean its not a black eye on the internet.  Ok, MySpace is worse than phpBB for sure.  But, c’mon, I write Phorum.  I am biased (see above keynote =).  It was a popularity contest and I guess there are more kiddies to vote for them than say Pidgin which is what I voted for.  With all the trouble they have had with their name, I wonder if “Gaim” would have gotten more votes.  (see other keynote on branding =).  The phpBB team may need to see the branding keynote from this morning.  It talked about how it takes a generation to change perception about a brand.  Most people I talked to here have a negative reaction to the phpBB brand.

The rest of the night we just hung out at the party hosted by Jive Software. We use OpenFire from those guys.  I am not a big Java user on the server.  Its just one more different thing to admin in a company that is 99% GNU C apps on the servers.  But, Openfire does a damn good job with XMPP.

In closing, O’Reilly Open Source Convention was great.  I got some great ideas of stuff we should be doing.  I got confirmation of things that we are already doing.  And most important, IMO, we got to share with others how we solve problems.  As  Gopal said in his caching talk, sometimes is better to stop doing stuff and tell others what you are doing (paraphrase).

Day one is complete.  Portland is great as always.  Its really day 1 1/2 since we got in at 1PM yesterday.  That allowed us to go to the MySQL/Zend party last night.  Great party by those guys.  Touched based with old friends and made some new ones.

I kind of session hopped today.  Of note, I attended Andi Gutmans PHP Security talk which really had little to do with PHP.  Like Larry Wall’s onion metaphor, Andi presented an onion metaphor for security.  I stopped in for a while on the SOLR talk.  It looks neat.  I like that it is a REST interface to Lucene.  If we were not using Sphinx already I might take a longer look.  But, we like Sphinx and, SOLR and Lucene are Java.  Not that there is anything wrong with that, we just don’t use Java a lot, so its just one more thing that would be out of the norm.  I admit I spent a good bit of time in what is being called the “hallway track” working on some code.  Work does not stop just because you are at a conference.

I got to hang out with Jay Pipes of the MySQL Community team a good bit.  We talked about the MySQL forums (which or course runs Phorum) and how they want to improve them.  They would like to see tagging, user and post rating and some other things.  Some good things will come out of that.  Hopefully they have some of the tagging stuff done already at MySQL Forge and can contribute that code to Phorum, saving us time.

I hosted the Caching for fun and profit BoF.  It was not packed, but it was a good time.  The MySQL BoF was at the same time, so we lost some folks to that I am sure.  They had beer and pizza.  Brad Fitzpatrick did come by and contribute.  Thanks Brad.  It was mostly the same stuff you get on the memcached mailing list.  “How do we expire lots of cache at once?”  Questions about different clients.  Stuff like that.  It kind of turned into a memcached BoF, but I tried to share the dealnews experience with the attendees including our MySQL Cluster pushed caching.

I have met many readers of both dealnews and this blog (hi to you) while here.  Glad to know that both my professional work and my personal work are of use to folks.  The demographic at this conference is dead on for dealnews.  Maybe I can get them to sponsor it next year.  That would be cool.

I say every year that I want to present “next year”.  Something always keeps me from doing it.  Usually its just not having time to prep for it.  By the time I think about it, the call for papers has passed.  I really want to get it done this time.  We shall see I suppose.

We went to the Sun party tonight.  It was a good time.  There was beer that was free as in beer.  More hanging with friends and talking about all kinds of stuff.  Now, all you Slashdotters sit down.  I saw people from the PostgreSQL and MySQL teams drinking beer and having fun together.  OMGWTFBBQ!!!1!!  See, the people that really matter in those projects don’t bicker and fight about which is better.  They just drink beer and have a good time together.

Anyhow, I will blog more after day 2.  There won’t be a day 3 as I have to catch an 11:30 flight back home.  That is usually how it goes.  Not sure why they book anything on Friday really.  Even O’Reilly has its “after party” on Thursday night.  Its late, and I need sleep.

So, I had written a while back: “We currently have a DataMemory of 4GB and IndexMemory of 2GB. Based on the crude methods we have to monitor it, I think we are at about 40% capacity.” Boy, I was wrong.

After that post, I started looking at this more in detail because we were considering buying more RAM “just in case”. I figured out how to use the super secret command “all dump 1000″. The command is not documented in the MySQL documentation that I could find. I did find it in the NDB API documentation before writing this post however. Not sure why I could not find it before.

For those that still don’t know how to use it, simply type “all dump 1000″ from your management console. Then check your cluster log files on the management server. You will see something like this:

2007-07-20 17:56:47 [MgmSrvr] INFO — Node 2: Data usage is 46%(90672 32K pages of total 196608)
2007-07-20 17:56:47 [MgmSrvr] INFO — Node 2: Index usage is 1%(2006 8K pages of total 131104)
2007-07-20 17:56:47 [MgmSrvr] INFO — Node 3: Data usage is 46%(90724 32K pages of total 196608)
2007-07-20 17:56:47 [MgmSrvr] INFO — Node 3: Index usage is 1%(2039 8K pages of total 131104)
2007-07-20 17:56:48 [MgmSrvr] INFO — Node 4: Data usage is 43%(86153 32K pages of total 196608)
2007-07-20 17:56:48 [MgmSrvr] INFO — Node 4: Index usage is 1%(2016 8K pages of total 131104)
2007-07-20 17:56:48 [MgmSrvr] INFO — Node 5: Data usage is 46%(90672 32K pages of total 196608)
2007-07-20 17:56:48 [MgmSrvr] INFO — Node 5: Index usage is 1%(2007 8K pages of total 131104)

Anyhow, I ran that and low and behold I saw that we were at about 93% capacity. As you can see above, we have made some changes. This got me to really digging as to what the difference could have been. As far as I can tell, our use of TEXT fields in our data was causing the issue. We have several fields in our data structure that hold data larger than 256 bytes. However, they hardly ever are more than 600 bytes. Based on what I read, it seems that the data after the first 256 bytes would be stored in 2k chunks. So, our litte ~300 byte extras were being stuck into 2k chunks. That caused a huge amount of wasted space.

So, the first thing I realized was that we could use 6GB DataMemory instead of 4GB. The machines have 8GB of RAM in them. We were using only about 200MB of IndexMemory. That leaves well over 1GB of the system.

The second thing I did was carefully analyze our data. I went back to school and started counting bytes. Its easy to get lazy as a developer these days. We just use a data type we know will work and it really is no big deal. But, not in this case. I changed int to mediumint and some to even small and tiny. I realized that there were two TEXT fields I could eliminate altogether. I looked at the varchar fields and made them only as long as they need to be. In the end, we ended up at the numbers you see above, about 46% capacity or 2.76GB. We were at 3.72GB before. The removal of the two TEXT fields may have had more to do with the improvement than anything.

This was all with MySQL 5.0. 5.1 will bring better storage of the large data. For example, all data over 256 bytes, for a row, will be stored together in 2k chunks rather than one 2k chunk per column. That will likely save us 1GB of DataMemory. The other feature of 5.1 is Disk Data Tables. We are currently testing those and I will blog about them when I have more information. Early numbers look good though. We have just set our largest table (the one with the TEXT fields) as a disk data table and our DataMemory is down to just 280MB. Yeah, that is an MB. Our COO is a bit concerned with the performance hit we may see with data on disk.

Oh, one more pointer about all dump 1000. For a quick way to grab my current usage, I use this command from my prompt (on a Mac):
ssh user@mgmserver "ndb_mgm -e 'all dump 1000' && tail -n 1000 /var/lib/mysql-cluster/ndb_1_cluster.log | fgrep usage | tail -n 8";;
Use at your own risk. It may make your server catch fire. =)

DGTV has a great picture of a putt that is in question.

I am looking at the rules and to me, it looks like its in.   From 803.13 Holing Out:

B. Disc Entrapment Devices: In order to hole out, the thrower must release the disc and it must come to rest supported by the chains or within one of the entrapment sections. This includes a disc wedged into or hanging from the lower entrapment section but excludes a disc resting on top of, or hanging outside of, the upper entrapment section. The disc must also remain within the chains or entrapment sections until removed.

There is no mention about the disc resting on the basket and another object.  My guess is that the author never imagined the basket being this close to a rock.

So, we are not officially looking for PHP/MySQL developers.  However, we could start at any moment.  So, I want to be ready.  There is a really good chance that we will hire someone by the end of the year if not sooner.

So, if you want to work with an up and coming company that uses PHP, MySQL, Gentoo Linux and other open source technology, you can go ahead and send us a resume.  Just email it to me at brianm@dealnews.com.  You will have to confirm your emails.  Spam on that account reach huge numbers years ago, so I am not going to bother with cute notation for my email address.

This job would most likely require moving to Huntsville, AL.  Before you make an Alabama joke, Huntsville has been recognized as a great place for business and technology.  In 2006 it was named one of the 10 greenest cities in America.  You can read more about Huntsville on the very informative Wikipedia page.

Wanna talk to us in person?  The dealnews dev team will be at the O’Reilly Open Source Convention next week.  If you would like to meet with us in person, feel free to fire me an email.  Did I mention that we have been to OS Con every year for the past 4 years?  Nice perk.  Granted, the “hiring brass” won’t be at the conference, but the team has influence over that stuff.

About dealnews.com, Inc.
dealnews.com, Inc. operates a network of web sites that guide consumers to the best deals on consumer electronics, computers, men’s apparel, and more. Its flagship site, dealnews.com, was ranked 24th in PC World’s list of Top 100 Products of 2006. Relying on its team of expert editors, reader intelligence, and proprietary research tools, dealnews.com showcases over 100 of the top deals, coupons, and sales every day. Founded in 1997, dealnews is headquartered in Huntsville, Alabama, with editorial offices in New York City.

After my posts claiming I had no idea why I should want PHP 5.2 so badly, I realized what it was today.  Its the DateTime functions.  Supporting daylight savings time in Phorum has always sucked.  I admit it.  This will make it a breeze.So there, Phorum 5.2 will require PHP 5.2.

I think my earlier post has been misunderstood.  I am in no way advocating keeping support for PHP4 by the PHP team alive.  I voted +1 to kill support for it on the mailing list. I warned the Phorum users about it months ago.  I did the same for MySQL 4.1.

My main concern is with the tone of the GoPHP5 movement.  As I said in a comment on the previous post, I guess I never realized that a project like Drupal would be a slave to a particular PHP version.  Maybe that is what this is all about, finally freeing these projects from the bondage of legacy PHP versions. Because we don’t use OOP, we have had the luxury of coding for the current PHP versions and having the code work with a large range of PHP versions.  Phorum 5.1 has been in bug fix mode for months now too.  So, we have not had to worry about PHP changes in a while.  We don’t code for PHP4.  We code for PHP and whatever works works.  The minimum of 4.3.0 (now changed to fit the new definition of “requirements” that the community has adopted) on our home page means that it works, not that we code for that version.  I never considered ensuring compatibility with a 6 year old version of PHP.

If all that GoPHP5 means is  that we won’t pander to the PHP4 whiners, then sure sign us up.  I will change the front page right now as a matter of fact (done).  I have no interest in testing Phorum in PHP4.  If version 5.2 works in PHP4, I guess those lazy slackers get lucky.  If not, tough.  Does that qualify for “Make an announcement on your site that by February 5, 2008 you will accept PHP 5.2 features into your codebase and will no longer provide support for lesser PHP versions?”  Heck, I will accept 5.2 features now.  I am just not sure what PHP 5.2 feature I am supposed to be that excited for.  I guess input filter is ok.  Not sure what it does really.  All the stuff I have read make it sound like a really complicated way to escape HTML in GET and POST data.  Maybe I am reading the wrong stuff.  Does someone have a good example of filter that explains why I need to use it intead of urlencode and/or htmlspecialchars?

And what does  “stop supporting all lesser PHP versions by February 5th, 2008” mean?  If someone comes to our support forums and is using PHP 5.1 do I have to refuse him support?  That seems a little rude.  How about “stop ensuring compatibility with less PHP versions”.  That sounds better IMO.

Now, killing register_globals, magic_quotes, safe_mode and adding mysqlnd (I know, its going into 5.2 now) is something to get excited about.  Hmm, so maybe I should post a notice that I am accepting PHP6 code into Phorum!  Yeah, GoPHP6.org.  Darn!  The domain is taken by a squatter.  No one thought to grab it now?  It will be useful in about 2 years.  Maybe I should grab up 7 thru 10 now. =)

Death to PHP!  Long live PHP!

I have been excited about the MySQL native drive (mysqlnd) since I first heard about it.  They are looking for testers now.  The sites I could test on are not really that high traffic.  So, not sure I would be much help to them.

I am not sure how I feel about this bit of news in that post:

“While mysqlnd was originally designed for PHP 6, Andrey Hristov has now finished the backport of mysqlnd to PHP 5 and added support for ext/mysql.“

On one hand, it means I can play with it sooner.  But, on the other, it gives slackers another reason not to upgrade to PHP6 and mysqli.  Its a shame.  I really like mysqli.  I would really like to see an end of life for ext/mysql.  But, this just keeps it going.  Even more scary is this line:

“Given that 60 % of the PHP user base is still on PHP 4, some heavy PHP users would like to see the backporting done all the way to PHP 4. We are evaluating that, but have no immediate plans for such a backport.“

Ewww.  I bet the GoPHP5 folks are not sleeping well on that bit of information.  They have no plans, but the fact that its in this post at all means someone has discussed it.  Scary stuff.  Please, folks working on mysqlnd, realize that the reason people are using PHP4 is that they don’t like change.  So, what makes you think they would change the MySQL driver in their PHP4 builds.

So, some PHP projects have banded together to put the final death nell into PHP4.  They call it GO PHP5. The Phorum team received an invitation to join them. We didn’t officially join the list.  With Phorum, we have always written our code to do the job that needs to be done.  I have never really looked at the docs for a language and thought “oh, where can I use that in my code”.  That seems to be the direction this is going.  From time to time we have had to bump up the minimum versions of PHP and MySQL that are required for Phorum.  Those have been done because we had a problem that could not be dealt with in some older version.

Now, what the Phorum team is doing with Phorum 5.2 is writing it with PHP 5.2 and MySQL 5.0.  We have done zero testing on PHP 4.  We don’t really plan on it.  We are not however going to purposely break PHP4 which is the feeling I get from the GO PHP5 camp.  Is there a good chance that we will use some function that is not available in PHP4?  Yeah, probably.  Will I care when I use it?  No.  But, am I going to go find a function and use it just so my application is PHP5 only?  No.  If we finish it and it works fine in PHP4, I am not going to lie to my users and tell them it won’t.

Maybe its OOP that is driving this.  Phorum made a decision when we started 5.0 that we would not use OOP in the core.  We did it for speed reasons.  Speed was most important to us.  PHP OOP creates (it seems better now than then) overhead that we don’t want or need.

Maybe I took it all wrong.  I don’t know. I don’t mean to offend.  But, I also don’t want to be on the fictional list of projects that are not “PHP5 ready” because we are not publicly throwing PHP4 into the ether.

Last year at O’Reilly’s Open Source Conference I hosted a BoF (Birds of a Feather) about memcached. It was a popular event. So, this year, I decided to broaden the scope to caching in general. Its titled Caching for fun and profit. It will be Wednesday, July 25 from 8:30-9:30pm in Room E141.

Anything goes. We can talk about memcached, Tugela, basic file caching… whatever.

More and more web sites are finding that they need to uses caching to increase their performance. There are those of us that have solved some problems. Others that are new to these techniques have a lot of questions. This BoF is an opportunity for web developers to share their ideas on caching.

Specifically, the dealnews.com dev team can talk about the 3 main types of caching we use and where each is applicable.