I just got out of the panel on commodity hardware and did not get a chance to participate so here's my take on it. The panel started with an opening question: google, amazon and the likes run at a huge scale on commodity hardware, yet enterprise vendors still push customized hardware and expensive at that. To me the answer is pretty obvious: enterprise hardware is being for the most part sold to people who don't know how to architect and design software on a commoditized stack. Let's be honest, look at most "enterprise" hardware/software literature: it's just noise and a waste of both the writer's and the reader's time. And by stack I mean from the server, all the way up to the application code. If you constrain yourself to buy servers that cost no more than $5k, buying high-end database software makes little sense. Rather you recognize that low-end compute is how you get economies of scale and you apply the same reasoning to your networking gear, storage systems, database software, load balancing software, etc. Google, from its earlier papers, seems to be the first to have understood that, rejecting the usual marketing garbage from large vendors. And for that we should be grateful.

Two observations from talking and listening to people during the Hadoop summit; firstly hadoop is used quite often to process clickstream data -- in all fairness I missed the talk about hadoop used for genomics. Secondly and a corollary of the first, sub-second queries in hive or pig are not quite there yet. Since a hive query translate into maps and reductions their scheduling determines in addition to the sheer volume of data is going to determine response time. Undoubtedly pre-computing aggregates is a natural way to go much like what is done for data warehouses. Where these aggregated should be stored for consumption is a problem that could to hybrid solutions. Process data with hadoop and export then to postgres or infobright to enjoy a more mature (but less scalable) run-time environment. Get multi-terabyte daily processing and sub-second analytics and all that open source. If you've done something like that, I'd be interested to know before I embark on a route where others have failed before.

A few key points: 2 ops people, automatic scaling, 1000s of cores on EC2, PBs of storage on S3. http://mysqlconf.blip.tv/file/2037101

Cute.

Differential Power Analysis is a neat way to cryptanalyze smart cards and that triggered an interesting counter-measure: keeping power consumption constant regardless of the computation performed. Moving to a bigger scale and assuming low cloud compute costs, one could hide sensitive data processing in one VM by running ninety-nine others with slightly different data, whose results will be discarded silently.

Yesterday at CloudCamp, a few of us discussed methods to store and use sensitive data on a public cloud, where you presumably do not have strong assurances that your data are for your eyes only. To keep structured data (e.g. relational data) a pattern emerged among participants assuming your data have an easy key:

1. Store all non-identifiable data in the cloud, keyed by an arbitrary identifier.
2. Keep actual identifiable data on-premises with the mapping to that arbitrary identifier.
3. Let the client device resolve the mapping locally.

For instance suppose you store transactions, this would require at a minimum the scrambling of the transaction details such as item name and party name. That offers mitigation against simple analysis of the data, using statistical methods to derive information, which can be acceptable. Of course everyonen remembers the AOL search term fiasco, where people could be identified based on the search terms. Which is why this scheme should work best if the data are highly structured.

Berkeley's Reliable, Adaptable, Distributed Systems Lab has produced a nice synthesis of the current technological underpinning of cloud computing, in a paper called "Above the clouds". StorageMojo and Perspectives have done a fine exegesis of the paper so I thought I'd focus on a claim that has caught my attention. The claim That claim is the fundamental premise of perceived, unlimited compute and storage elasticity: "The illusion of infinite computing resources available on demand, thereby eliminating the need for Cloud Computing users to plan far ahead for provisioning". I would argue that this illusion has to be dispelled for a stable and long-term development of the clouds. Being in its infancy demand is still fairly limited (Amazon claims 400,000 registered AWS customers) so as a customer I can operate on the assumption that any individual demand does not significantly affect supply. To make sure that individual demand does not rock the boat, a provider such as Amazon rations the amount of resources available  to any customer, so that fluctuation in demand can be absorbed by their resource allocator. To go beyond that limit requires to enter a longer discussion with the provider, so that it can ensure that its resource allocator will handle the peaks and troughs and demand. This is a classic production/price control scheme. Amazon Reserved instances Recently Amazon has introduced the concept of “reserved instances”; a one-time payment per instance opens allows for lower per-hour charges. Presumably that one-time payment, while still profitable, allows the provider to better predict future sustained demand. Firstly it is only worth getting a reserved instance if you plan to use it:

• more than 193 straight days for a year or,
• more than 99 days per year for 3 years.

193 straight days is really 12 hours a day every day of the year if your business is cluttered around a few timezones. A three-year contract gives you 660 days of free run-time compared to signing up for 3 consecutive years. It’s a clever move. Let the customer pre-pay all or part of the fully-burdened, marginal cost of an instance, yet retain a variable part so that it does not turn into a all-you-can-eat feast that would devour margins. Also consider that in the Amazon Web Services (tm) Customer Agreement, “reserved instances” are really about “reserved instance pricing”, not about Amazon reserving capacity to serve these instances, their Service Level Agreement notwithstanding. This should allow Amazon to achieve at least 3 objectives:

1. better plan for capacity with at least part of the marginal cost of instances, pre-paid,
2. give corporate customers a greater perception of control and security, however tenuous it is in reality,
3. differentiate corporate customers (usage patterns, utilization) and tweak the resource allocator to that purpose.

As a side note I have not found a clause restricting the reselling of reserved instances though margins would be low and eaten away by the payment and billing system used by the reseller (e.g. Amazon FPS). Resource allocation For the provider this is an interesting and crucial problem to solve, that of properly oversubscribing their actual physical resources (physical cpus, physical drives, physical network pipes). To oversubscribe physical resources through virtualization at a large scale is after all, what the cloud is about. It's also the name of the game in traditional banking: loan 9 times more money than you have in trusted assets. So as a provider, using the circulated figure of a "best-case" 30% compute utilization, with 100 physical compute units I should be able to lease at most 300 units. This is of course simplistic and there is a wealth of literature on the topic of oversubscribing commodities, be it money, phone minutes, bandwidth or plane seats. All providers being for-profit enterprises we can foresee that they are going to drive their oversubscription to the limit and the winners are likely going to be driven by 2 factors:

• the marginal cost of physical resources (compute, storage, network) and management overhead for each,
• the sophistication of the resource allocator.

The first factor, the marginal cost of physical resources, is, as the authors of the paper point out, varies with the inverse of scale. The bigger you get, the cheaper it becomes to operate your data centers, the current figures documented call for a 5x-7x reduction in cost when you reach internet scale. Clearly this is not going to leave a lot of breathing room for the mom-and-pop operations. The second factor is by and large, independent of scale. How high you can drive your data center utilization as a cloud computing provider, without running out of resources, is going to make a significant difference once you've reached internet scale. The crystal ball Since this market is still young and because the barrier of entry to reach a profitable zone I suspect that the smaller players that do not offer a lot of added value on top of computing infrastructure leasing will have to drive their resource allocator very aggressively; too aggressively, which means that they’ll end up losing customers because in the end oversubscription will reach unsustainable levels. Once the small operators are out of the picture, the 3 major players will either cartelize the market or contend over price by deploying a better resource allocator. In the former case we would be looking at a situation analogous to the oil market or the U.S. ISP market; the illusion of competition shrouding an illicit price fixing agreement. In the latter case, assuming a greater shift on cloud computing, the best resource allocator will have to buffer demand spikes and maintain a high utilization. Make switching from one provider to another swift, public and reversible, and you have a public market where prices are not driven by supply and demand (who in 2009 can still believe that?) but also by trust and perception. Without more transparency to let the public get a sense of inventories (e.g. through a clearinghouse), who’s to say we won’t see another bubble?

When building a global network, you start building out knobs (usually implemented as routing policies): cost, packet loss, latency, maintenance, diversity, isolation, "special" [Really funny analogy between anycast and toilets, caching and water supply] After having developed routing policies, you start looking into anycast. One of the first services to be anycast is DNS. Anycast scaling: vip, ecmp Anycast considerations: how to monitor services? how to control users? how to handle transient network events?

Viewable on scribd, from the mysql 2008 conference.

[slideshare id=41197&doc=scaling-twitter-12758&w=425] A bit that caught my attention is the mention of ejabberd. I've always been a big fan of erlang for 3 main reasons: a message-passing interface, its hot code upgrades and its making time an explicit and operational part of the language.