Member-only story
The Best Approach for a Pipeline Architecture in .NET
Create a clean pipeline with minimal code required
In my most recent project, I faced the fairly simple challenge of synchronizing data from Azure Blob Storage with an on-premises File Storage. While synchronizing, however, there should be several filterings and transformations applied to the metadata of the blobs or files, e.g. filter by specific locations, transform the path of the file to a blob route etc…
After sketching the data flows on my whiteboard, I quickly realized, that the filterings and transformations are just simple chained operations, like in a pipeline. I decided, to use the pipes and filter pattern as my application logic part of the architecture, but what is the best approach of implementing a pipeline in .NET?
Requirements
What we want to accomplish, is to break down several tasks into a sequence of logical steps. Additionally, our goal is to follow the Single Responsibility Principle for each separate step. A step should focus on just one thing. If we need another thing, we just add another step in between. Also very desireable would be, to have the pipes being injectable via Dependency-Injection, while also remaining configureable.
To sum up, we want to accomplish the following requirements:
- A method, to quickly set up pipes in a logical order.
- Pipes may be constructed with dependency injection.
- Pipes should be configureable.
- Pipes should take a stream of events as input and output another stream of events. The event-type may change hereby.
Approach
During my research, I found many approaches of realizing a pipeline architecture in .NET, however all of them were too overcomplicated and not flexible enough, in my opinion. That’s why I decided to reinvent this architecture style for .NET, making it much more clean and intuitive using Reactive Extensions.
Based on my experience, the best approach for handling streams of events are the Reactive Extensions. Reactive Extensions is a library, that enables you to handle asynchronous data streams with LINQ-like operators. This is achieved using…
