Comments for O'Reilly Radar

Comment on The world beyond batch: Streaming 101 by Tyler Akidau

Tyler Akidau — Sat, 08 Aug 2015 21:54:00 +0000

Yes, in general with these systems, exactly-once notions apply strictly within the system itself. For true side effects, you either need to make sure they are idempotent (and thus can be applied multiple times), or are orchestrated in an exactly-once manner via something like a two-phase commit protocol.

Comment on Four short links: 7 August 2015 by Justin Schulwitz

Justin Schulwitz — Sat, 08 Aug 2015 04:41:00 +0000

You should feature this great resource: http://algorithmiccomplexity.com/

Comment on The world beyond batch: Streaming 101 by sandesh hegde

sandesh hegde — Fri, 07 Aug 2015 18:34:00 +0000

Please take look at the recently open sourced streaming system

https://github.com/DataTorrent/apex
https://www.datatorrent.com/project-apex/

Comment on The world beyond batch: Streaming 101 by Roland Kuhn

Roland Kuhn — Fri, 07 Aug 2015 07:13:00 +0000

Hi Tyler, concerning exactly-once processing I take it that the assumption is that user code cannot perform arbitrary actions, every effect that it can have must be controlled by the execution engine. If that is violated then it seems to me (from the 2013 paper) that Millwheel provides at-least-once (since it executes first and commits afterwards), so it should be possible to detect duplicate processing by observing uncontrolled side-effects of the user code (like a println statement). This is a rather fine point, but I’m raising it since you’re harping so much on strong consistency ;-)

Comment on The world beyond batch: Streaming 101 by Tyler Akidau

Tyler Akidau — Thu, 06 Aug 2015 22:27:00 +0000

I’m not sure I fully understand what you’re disagreeing with. If you’re trying to point out that there are use cases where one might want to make the tradeoff of having lower latency results at the cost of having only seen a portion of the relevant input data (and thus producing partial, speculative results): yes, that’s a very valid use case. But only by using a streaming system that provides strong consistency can those partial results be refined over time into correct results as the rest of your data arrive. We’ll be looking at speculative results (and talking about these sorts of tradeoffs) in much more detail in Part 2.

Comment on The world beyond batch: Streaming 101 by Marc-Philippe HUGET

Marc-Philippe HUGET — Thu, 06 Aug 2015 19:43:00 +0000

Hello Tyler,

Really interesting article on streaming architecture. I disagree on one point though: the comment on consistency for streaming architecture. Examples on which I tested with Storm do not necessarily imply approximate algorithms. Now, I agree, it depends greatly on event time and processing time and the difference between we accept to consider. If we want near real-time results, it is important to consider a subset of all the data. What is your opinion?

mph

Comment on Our world is full of bad UX, and it’s costing us dearly by Alex

Alex — Sun, 02 Aug 2015 02:08:00 +0000

it’s like saying cars are awful, because more than a million people die from traffic accidents every year.

Comment on The race to build ‘‘big data machines’’ in financial investing by Ron Bodkin

Ron Bodkin — Fri, 31 Jul 2015 18:23:00 +0000

I think there’s a lot of opportunity in blending machine intelligence with humans, as in Freestyle chess. This is eloquently argued in The Second Machine Age by Brynjolfsson and McAfee (chapter 12). This is especially enticing for investment, as long as you can avoid human intervention adding bias and defeating discipline…

Comment on Four short links: 27 July 2015 by TrevorLyman

TrevorLyman — Fri, 31 Jul 2015 16:49:00 +0000

There is a great legal review of the case of Georgia vs. Malamud here:

http://libertyupward.com/editorial-legal-review-the-state-of-georgia-vs-carl-malamud-and-your-right-to-access-government-documents/

Comment on How an enterprise begins its big data journey by Ilya Geller

Ilya Geller — Fri, 31 Jul 2015 14:25:00 +0000

Be careful: SQL.
SQL, Structured Query Language obtains and uses patterns from EXTERNAL queries, and statistics on how often they are used; neither the queries, nor patterns, nor statistics have anything to do with data itself.
SQL depicts the data from outside, and no portrait can create an ideal representation of its object.
I, however, discovered and patented how to structure any data without SQL, the queries – INTERNALLY: Language has its own INTERNAL parsing, indexing and statistics and can be structured INTERNALLY. (For more details please browse on my name ‘Ilya Geller’.)
For instance, there are two sentences:
a) ‘Sam!’
b) ‘A loud ringing of one of the bells was followed by the appearance of a smart chambermaid in the upper sleeping gallery, who, after tapping at one of the doors, and receiving a request from within, called over the balustrades -‘Sam!’.’
Evidently, that the ‘Sam’ has different importance into both sentences, in regard to extra information in both. This distinction is reflected as the phrases, which contain ‘Sam’, weights: the first has 1, the second – 0.08; the greater weight signifies stronger emotional ‘acuteness’; where the weight refers to the frequency that a phrase occurs in relation to other phrases.

SQL cannot produce the above statistics – SQL Industry Standard is obsolete and out of business.