Scaling Your Data Pipeline: How to Process 100TB+ Pipelines in Real-Time