Apache Beam


Apache Beam is an open source unified programming model to define and execute data processing pipelines, including ETL, batch and stream processing. Beam Pipelines are defined using one of the provided SDKs and executed in one of the Beam’s supported runners including Apache Flink, Apache Samza, Apache Spark, and Google Cloud Dataflow.

History

Apache Beam is one implementation of the Dataflow model paper. The Dataflow model is based on previous work on distributed processing abstractions at Google, in particular on FlumeJava and Millwheel.
Google released an open SDK implementation of the Dataflow model in 2014 and an environment to execute Dataflows locally as well as in the Google Cloud Platform service.
In 2016 Google donated the core SDK as well as the implementation of a local runner, and a set of IOs to access Google Cloud Platform data services to the Apache Software Foundation. Other companies and members of the community have contributed runners for existing distributed execution platforms, as well as new IOs to integrate the Beam Runners with existing Databases, Key-Value stores and Message systems. Additionally new DSLs have been proposed to support specific domain needs on top of the Beam Model.

Timeline