Realtime Automation

The 2011 IDC Digital Universe Study notes that over the next ten years, data volumes will increase by a factor of 50, while the number of IT professionals will increase by only a factor of 1.5.

With data streaming in 50x faster, and streaming in continuously, it's clear that it will be way too much data, and coming in way too fast for humans to handle, even with the most sophisticated dashboards and other tools.

The only way forward in handling this exponential growth of big data is to automate decision making. We need to move to a situation where humans define the decision making rules, and machines execute them in realtime. Of course, this is already underway in several areas.

On Wall Street, algorithmic trading or "robo-trading" has become the normal way in which trades are executed. It is the only way in which the data rates in high frequency trading can be handled. Complex trading models and rules are set up and then executed in realtime by machines.

Similarly, in the world of information and search, realtime automation is now the norm. In the early days of the web, companies such as Yahoo would employ armies of cataloguers to organize and classify information for presentation on websites. Today, the leading information and search company, Google, operates an almost entirely automated infrastructure. Realtime decisions about what to show a particular user at a particular time are taken algorithmically, on a massive scale.

Machines And Rules

The shift from "people and dashboards" to "machines and rules" is inevitable in any area of business, once the volume and velocity of data passes the threshold at which humans can no longer cope. The explosion of big data everywhere means that in the next few years we will see realtime automation becoming an essential technology in every area of business, the web, and infrastructure.

Business, web and infrastructure automation is all about latency. To operate automatically, we need data analytics solutions that can guarantee a maximum latency of a few seconds, e.g. a maximum of three seconds from data to decision to action. Traditional data analytics tools such as data warehouses and Hadoop batch analytics have latencies that are measured in hours - a level that is 1000x too slow for automation.

Cloudscale's HRule architecture has been designed to deliver the high throughput and low latency required for realtime automation.

With HRule, users can concurrently track and analyze millions or billions of individual entities in realtime - customers, mobile devices, payments, social media, market data, sensors,... Rules fire and trigger actions whenever important patterns are detected and/or important metrics move outside pre-set limits. Each automation is based on a simple set of rules. Thousands of rules and automatic actions can be running together, continuously in realtime. Business users (“HRulers”) can add, refine, or remove rules easily at any time, using simple drag-and-drop interfaces.