Great customer service

Those of you following my build on Facebook know that I'm building a fEarful 12/6, which uses the Eminence 3012LF speaker. After assembling, I noticed that the speaker had a bunch of distortion, and found a hairline crease in the cone.

Called Eminence, got an RMA number, and shipped off my speaker with crossed fingers.

They called today, and he said that the speaker looks fine, sounds fine, and the crease I saw was probably a casting mark or something. Uh-oh, I'm thinking.

"But we'll get a new speaker right out to you."

Nice. Read more...

Swine Flu - Remarkable?

I was listening to NPR this morning, and they had a doctor (epidemiologist? immunologist? something like that) on to talk about Swine Flu (H1N1).

Now, there is a lot of coverage on Swine Flu lately, and I thought some of the things that the doctor said were very interesting.

1. That Swine Flu and the Seasonal Flu have the same symptoms.
2. That Swine Flu and the Seasonal Flu are treated using the same methods (fever reducer and fluids).

Now, you might forgive me for thinking at this point: what's the difference? If swine flu is no more dangerous, and is treated the same way, why the hullabaloo?

It turns out that the core difference is that the flu season has started early this year.

Interestingly, we had a relatively dry, cool summer. The areas that are being hardest hit (the south) had the driest summers. The flu spreads best in cold, dry environments. If you have been told that the flu has nothing to do with the weather, you've been misinformed.

Now, I'm not trying to imply causality here, but there's been so much misinformation, e.g. "57% of those tested showed positive for swine flu", that I thought I'd offer a possible explanation.

By the way, that statistic above? You probably didn't catch that 57% was the number of people who:

1. Came to the hospital with flu-like symptoms
2. Were tested (and they "selectively test"!). Selective testing means that they had severe symptoms and another compounding medical condition, like asthma.

You'll forgive me, then, for maybe thinking that the press is playing fast and loose with the numbers. And I love that the CDC publishes a map that shows "how geographically dispersed" the flu is. NO INDICATION OF SEVERITY. So a couple of cases, spread around == pandemic. Nuts.

Attribution: photo from Patrick Long Read more...

Remove me from your mailing list.

If you are sending out newsletters, here's some advice:

Make it very easy to unsubscribe.

Here's why:

1. It's just common decency.
2. Users will not appreciate your lock in tactics, and it will tarnish your brand.
3. Since it's hard to unsubscribe, they'll just mark you as spam.

If you require me to log in to unsubscribe, I'm marking you as spam.
If there is more than 1-2 steps to unsubscribe, I'm marking you as spam.

Ideally, put a link at the bottom of the email that says "unsubscribe". When clicked, it optionally asks, "are you sure?", and then unsubscribes the user. No fuss, no muss.

Image from Sarah G Read more...

Android is actually "Getting There"

I finally have all the applications I need for the G1 to be truly useable, and at least give it parity with the Blackberry or iphone.

The main things:

Push (Exchange) email, in the form of a $25 download called Touchdown. This app is a bit wonky from a usability standpoint, but has consistently gotten better as the developer has released improved versions.

While I feel that this should be DEFAULT FUNCTIONALITY (are you listening Google?!), I can say it's worth the price, and I haven't really had any major problems with it once it was configured (configuration is a bit of a pain).

PPTP VPN: Thanks to Cyanogen for backporting this to CyanogenMod (if you're not running this, you're missing out on a better experience). This is slated to be included in the Donut release, but CyanogenMod has it now, and it works (although I haven't been able to get it working on WiFi).

Those were the two biggest things that were just *missing* from the Android stack, and I can actually recommend the G1 for business users. The PPTP thing makes me really happy, especially since the PPP daemon has probably supported it since the launch, but there was no way to properly drive it.

Photo by Max.
Read more...

MegaCoaster

My son has been spending all his "computer time" lately on a site called PlayCrafter: basically a flash gaming site, but also allows you to create your own games. The result is below:



I'm obviously biased, but he made the "hottest games" list. I'm trying to use this as an encouragement to delve deeper into the programming aspects of game development (wish me luck!).
Read more...

Hazards of working on the 13th floor

I'm not superstitious, I promise.

I work on the 13th floor, and there's something hinky with the elevator. Car #3.

At least half the time I walk near the elevator shaft, Car #3 is either open, or opening.

Walk out of the bathroom? Ding! and the doors are opening. Like it hears you coming. (I'm not superstitious, but not opposed to anthropomorphicization...)

Here's the best part. When you get on the elevator, the 7th floor lights up. On its own. As the doors close.

So, Unlucky 13, Lucky 7, and then the lobby.
Read more...

Goodbye CompuServe

Back in the heady days of the 90s, I worked at CompuServe, and I was there for the AOL purchase.

Since CompuServe is now officially dead, I thought I'd write up some trivia that I remember. Some of this stuff should go in a textbook on how to get complacent and kill yourself as a business.

While I was there, CompuServe was owned by H&R Block. That might seem like an odd match; it was. The overly conservative management team there was not ready to deal with the changes required to adapt to the Internet era. Instead, they came out with Wow!. We used to laugh and say that the "tc-chk" noise that it made when you clicked on a "pog" was really coming from a "User Click Interaction Specialist II" (yes, there really were titles like that) sitting at a microphone in Columbus, Ohio.

CompuServe had some long-term investment in DEC 10 technology. So much so, that when DEC (Digital Equipment Corporation) decided to discontinue them, CompuServe started making their own-- they had a hardware team that managed to shoehorn a DEC 10 into a pair of VME boards with a mess of VLSI chips. These were used for at least half of everything that made up CompuServe at the time: all the Terminal Access Nodes (TANs), the "chatrooms" (which had exactly 40 channels-- it was designed to mimic a CB radio), etc.

The TANs were what sat in various closets/server rooms around the country, and had modems hanging off them; they handled all the local dialup for which CompuServe charged so handily. They also handled another function: Credit Cards.

One of the years that I was there, I remember some senior manager coming out from Columbus, and sharing some of the following facts (and, I'll admit that things have gotten a little hazy in the intervening dozen years):

• 1 billion credit card transactions were handled on the CompuServe network.
• The revenue per employee was something like $180,000
• CompuServe had been logging a 20% CAGR for something like 20 years. Note to the wise: I've seen lots of companies tank after making a statement similar to this.
  

Some additional bits that I remember:

• The entire (okay, maybe not entire) network ran on X.25. This made for amusing things like dialing into a TAN, and the tan giving you a PPP (Point-to-Point Protocol) connection, but this really worked by giving you a direct connection to a server in Columbus that actually hosted your PPP connection. The connection from the TAN to Columbus was over X.25. So you had an X.25 connection that PPP rode on top of.
• CompuServe really did have a good network, and I suspect, without looking, that CompuServe Network Services is still around in one form or another. The only really weird bit was the X.25 stuff.
  

Where did CompuServe go wrong? I think that the biggest hint should be in the DEC 10 hardware. They started to go wrong when the decision to stay with the old hardware, even when it meant that they had to do their own hardware engineering. Forget that the Internet was going into exponential growth: they ignored Moore's Law.

I'm assuming that they did this because they had a comfort level with rocking the 36-bit processor. In fact, when the question was asked, CompuServe management responded with some vague platitudes about the benefits of a 36-bit platform. I don't suppose it was ever really evaluated properly whether making hardware was part of the core business.

The conversion pain should have happened in the mid 80s, when there was a chance. Instead, systems calcified around a dead architecture. When someone finally saw the handwriting on the wall, the possibility of change was gone.

CompuServe, I believe, was a victim of their own success; there was a prevalent attitude of imperviousness (20% CAGR for 20 years!), fueled by the fact that they were printing cash. This was then compounded by the fact that they were owned by a financial services firm, where the dominant paradigm seems to be to leave something that is making money alone (current financial crisis, anyone?).

Even though it was clear that this had to happen eventually, I'm still sad to see CompuServe go. I worked with many great, smart people there, and learned a tremendous amount about internet scale engineering.
Read more...

Internet scale databases

After going to the Amazon Web Services event at Mariner Stadium, I've been doing quite a bit of ruminating on building highly scalable web sites.

I suppose that part of this is my roots in webmastering (back when that was a singular title!).

So, I've decided to do some analysis of various technologies that contribute to building highly scalable web sites. Today's analysis is on databases (and whether you should even use one!).

The default: use a relational database.

There are a whole raft of issues with this, not the least of which is write performance. Eventually, a single database cannot be scaled (no matter how much money you throw at it) to support the write volumes required by high volume web applications. There are a few solutions:

Database sharding:

This can take many forms, but there are several basic versions:

Table partitioning: moving tables to separate databases
Range partitioning: e.g. putting user number 1-100,000 on server a, 100,001-200,000 on server b, etc.
Directory partitioning: put a directory in front of the database, so you have to ask two questions (where is this user info stored, and ask for the data from that server) to retrieve data.

More in-depth analysis here.

Of these, I think that the last one is quite interesting, especially if architected in from the beginning. It would be hard to retrofit, but the biggest advantage is that repartitioning is quite feasible.

Caching:

Memcached has become de rigeur in building high scalability applications like web sites. Its ability to speed up websites has been well documented, and I think it is a fantastic solution for not only improving read performance on databases, but the ability to cache the results of expensive operations.

I do have one nit, however, and that is what happens when a memcached server fails: there is no redundancy (which is acceptable, and can be planned for), and the decision about where data should go is dependent on the number of machine instances that are running. It would be better for this to be able to scale up and down easily.

Google's Bigtable, Hypertable and Amazon SimpleDB:

The only one of these that is available as open source appears to be Hypertable. The idea here is that you forgo the whole relational model for a simple data model (think giant spreadsheet), with much more flexibility in what you can stuff into a column, e.g. columns can store multiple attributes. Underneath this simple exterior is a highly distributed datastore (designed to run on many machines) that is designed to handle huge throughput.

And now for the crazy idea:

If, e.g., you only needed to store customer records, and each of those records were identified by a natural key (email address), then you might not need a database at all (or, at least for this purpose).

What if you combined Google's Protocol Buffers with something like Danga Interactive's MogileFS. Every object is stored as a protocol buffer for fast serialization/deserialization, and MogileFS takes care of storing the serialized file. It handles replication, managing redundancy, etc. This could also work on Amazon's Simple Storage Service (S3).

I realize that this does not handle things like iteration, but what if that isn't in the use case? Alternately, what if the architecture is something like Memcached -> Protobuf + MogileFS -> Sharded DB? You could have the best of all worlds: fast access through Memcached, if that fails, retrieve the record from MogileFS (and push into Memcached), and if you need iteration, or an alternate lookup, you still have the relational DB in a scalable format.
Read more...

Why do I have physical infrastructure?

I've been off on one of my ADD fueled research binges lately-- investigating Cloud Computing. I looked at several offerings:

1. Nimbus
2. Amazon EC2
3. Google AppEngine

And here are the conclusions that I came to:

Nimbus:
A nice layer on top of Xen to manage VM instances on a set of machines. This is really targeted at scientific computing, and does have a bit of a learning curve. If you have a pile of hardware that is being underutilized, some technical know-how and elbow grease (a.k.a. 'Round Tuits'), this is a nice looking solution to run a mess of VMs on. Especially if there are many images, but they don't necessarily run all the time (e.g. testing machines, support/demo environments). Turning VMs on and off is fairly straight forward. It offers an emulation of the Amazon EC2 WSDL interfaces, which is nice.

Amazon EC2:
This is an online cloud computing platform. Upload your VM(s), and use the Web Services interface to activate machines. Pay per hour, or per year. I'd have a hard time justifying capital expenditure (on server hardware) in a start up with this service available. The user can activate as many VMs as they need to service demand. The amount of CPU and disk available is quite elastic (hence the name, Elastic Compute Cloud).

Google AppEngine:
This is an interesting service-- it basically offers a stripped Python (and now Java) environment. Applications need to be specifically ported to the environment, but it does utilize (require) a number of best practices so that your application should be somewhat "forced" to be scalable. Probably a good option for "green field" web 2.0 applications, but if you have code already, might not be the best choice. Integration seems like it might be a little problematic (e.g. no MySQL/Postgres/SQL Server or C Library support).

Summary
Of these, Amazon EC2 is the one that fascinates me. We regularly build VMs that mimic customer environments, and these might only get used for a couple dozens of hours per year. EC2 would be a great solution.

We also have regular requirements to do performance testing under specific circumstances. We can allocate 12 hours on a fairly beefy machine with very low cost, get our testing done, and have results for under $20-ish. That is amazing. Read more...

Zawinski's Law of Software Envelopment

Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.

—Jamie Zawinski, Jargon file entry

My friend Robert and I were talking the other day about Twitter, and he related that he's been seeing people using it like they would use email. This is no surprise; every new "social technology" that comes along ends up being treated like email. Instant messaging did it, etc.

So I'm coining Young's Corrolary to Zawinski's law:

Every social networking software or web application eventually implements an email analogue, or its users will use some function of the network as email.

Of course, you could just use email, but that wouldn't be "Web 2.0".
Read more...