Pipes and Filters

The Pipes and Filters pattern extends the Pipes and Filters pattern [POSA96, Shaw95, SG96] with aspects of functional parallelism. Each parallel component performs a different step of the computation, following a precise order of operations on ordered data that is passed from one computation stage to another as a flow through the structure.


A Non-programming Example

Imagine the assembly line of an automobile factory. There are lots of tasks to manufacture a car, but consider only some very general tasks: 1) frame and power chain installation, 2) bolting the body on the frame, 3) mount the engine, and 4) put on the seats and the wheels. Based on this description of tasks, an assembly line is simply a pipeline of tasks, in which the procedure to manufacture a car requires four tasks that can be overlapped in time: while seats and wheels are put on for a car, the engine is being mounted on another, etc. Ideally, consider that each task is designed so that each completes in a cycle of time. After four cycles, seats and wheels are installed for the first car, the engine is mounted on the second car, the body of the third car is bolted, and the chassis of the fourth car is built. However, from the fifth cycle onwards, the manufacture of a new car requires only a cycle of time to be completed, exploiting the parallelism inherent in the ordered tasks.

A Programming Example

In image processing, rendering is a jargon word that has come to mean "the collection of operations necessary to project a view of an object or a scene onto a view surface". The best way to break down the overall rendering process is to consider each object as a different coordinate space, in which independent operations can be carried out simultaneously [Watt93]. The input to a polygonal renderer is a list of polygons and the output is a colour for each pixel on the screen. To build up, for instance, a 3D scene, four general tasks are to be performed per object (Figure 1).

Figure 1. A 3D rendering pipeline.

As in the non-programming example, each one of the tasks can be overlapped in time. In common applications for the film and video industry, the time required to render scene usually can be decreased. The rendering of a special effect scene of 10 seconds using a standard resolution of 2048´1536 pixels takes up to 130 hours of processing time on a single high-end Macintosh or PC platform. Using a parallel approach, with a 16 node CYCORE (a Parsytec parallel machine) this process is reduced to 10.5 hours [HPCN98].


The application of a series of ordered but independent computations is required, perhaps as a series of time-step operations, on ordered data. Conceptually, a single data object is transformed. If the computations were carried out serially, the output data set of the first operation would serve as input to the operations during the next step, whose output would in turn serve as input to the subsequent step-operations. Generally, performance as execution time is the feature of interest.


According to the problem description and considering granularity and load balance as other important forces for parallel software design [Fos94, CT92] the following forces should be considered for a parallel version of the Pipes and Filters pattern:


Parallelism is represented by overlapping operations through time. The operations produce data output that depend on preceding operations on its data input, as incrementally ordered steps. Data from different steps are used to generate change of the input over time. The first set of components begins to compute as soon as the first data are available, during the first time-step. When its computation is finished, the result data is passed to another set of components in the second time-step, following the order of the algorithm. Then, while this computation takes place on the data, the first set of components is free to accept more new data. The results from the second time-step components can also be passed forward, to be operated on by a set of components in a third-step, while now the first time-step can accept more new data, and the second time-step operates on the second group of data, and so forth [POSA96, CG88, Shaw95, Pan96].


This pattern is called Pipes and Filters since data is passed as a flow from one computation stage to another along a pipeline of different processing elements. The key feature is that data results are passed just one way through the structure. The complete parallel execution incrementally builds up, when data becomes available at each stage. Different components simultaneously exist and process during the execution time (Figure 2).

Figure 2. Pipes and Filters pattern.



Due to the parallel execution of the components of the pattern, the following typical scenario is proposed to describe its basic run-time behaviour. As all filters and pipes are active simultaneously, they accept data, operate on it in the case of filters, and send it to the next step. Pipes synchronise the activity between filters. This approach is based on the dynamic behaviour exposed by the Pipes and Filters pattern in [POSA96].

Figure 3. Scenario of Pipes and filters pattern.

In this scenario (figure 3), the following steps are followed:


The implementation process is based on the four stages mentioned above in the Context in general and Implementation in general sections.




Known uses

Related patterns

The Pipes and Filters pattern for parallel programming is presented as an extension of the original Pipes and Filters pattern [POSA96] and Pipes and Filters architectural style [Shaw95, SG96]. Other patterns that share the similar ordered transformation approach can be found in [PLoP94]; especially consider the Pipe and Filters pattern and the Streams pattern. Another pattern that can be consulted for implementation issues using C++ is the Pipeline Design Pattern [VBT95].

Contact Information

Jorge Luis Ortega Arjona.

E-mail jortega-arjona@acm.org