Paul Kedrosky asks why Alexa's Web Search Platform is a big deal. He wants to know why this is any bigger/better than what they had back in October.

Simply put, AWSP allows you to do computation on Alexa's computers. Those computers have the internet itself (> 100TB) in local storage. Back in October, Alexa's AWIS (which Paul referred to) only allowed you to access the data, and not do computation.

Before: Similar to Google or Yahoo search API - you can do a search programmatically.

After: Now you can write a C function that processes every page of a search result - and run it on Amazon's / Alexa's boxes.

See the difference? Amazon is creating an open San Diego super-computer center where anyone can schedule jobs.

Key concepts:

• Grid Computing Framework: Any programmer has access to a supercomputer environment similar to what an employee of Yahoo or Google would have.
• Internet in local storage: Alexa's crawls the whole internet incrementally. Your program can now have fast access to every document - and all the metadata (like links, file types, etc.)
• Web Services: If you build an application, you can publish access to it through an web service.

I think it's a big deal. We'll see how many people can meet the pretty high technical skill standards to start using it. I also think that even though access to the data, storage, compute time and bandwidth is all priced a $1 per unit, it could be a bit more expensive to do stuff than people might think.

¶ 12/13/2005