Architecture
A serverless data pipeline architecture diagram with Kinesis ingestion, Lambda transformation functions, S3 data lake storage zones (raw and curated), Glue catalog registration, and Athena query engine feeding QuickSight dashboards and SageMaker ML models. This template models a complete serverless ETL pipeline from data ingestion through transformation, cataloging, and analytics without managing any infrastructure. Ideal for data teams building cost-effective analytics platforms.
Full FlowZap Code
Ingestion { # Data Ingestion
n1: rectangle label:"Kinesis Data Stream"
n2: rectangle label:"API Gateway Endpoint"
n3: rectangle label:"IoT Core MQTT"
n4: rectangle label:"Kinesis Firehose"
n1.handle(right) -> n4.handle(left) [label="Stream"]
n2.handle(right) -> n4.handle(left) [label="Batch"]
n3.handle(right) -> n4.handle(left) [label="Device Data"]
n4.handle(bottom) -> Transform.n5.handle(top) [label="Buffer + Deliver"]
}
Transform { # Serverless Transform
n5: rectangle label:"Lambda Transform Function"
n6: rectangle label:"Validate Schema"
n7: diamond label:"Valid Record?"
n8: rectangle label:"Enrich with Reference Data"
n9: rectangle label:"Route to Error Bucket"
n10: rectangle label:"Partition by Date"
n5.handle(right) -> n6.handle(left)
n6.handle(right) -> n7.handle(left)
n7.handle(right) -> n8.handle(left) [label="Valid"]
n7.handle(bottom) -> n9.handle(top) [label="Invalid"]
n8.handle(right) -> n10.handle(left) [label="Enriched"]
n10.handle(bottom) -> Storage.n11.handle(top) [label="Partitioned"]
}
Storage { # Data Lake Storage
n11: rectangle label:"S3 Raw Zone"
n12: rectangle label:"S3 Curated Zone"
n13: rectangle label:"Glue Catalog"
n14: rectangle label:"Athena Query Engine"
n11.handle(right) -> n12.handle(left) [label="Glue ETL Job"]
n12.handle(right) -> n13.handle(left) [label="Register Schema"]
n13.handle(right) -> n14.handle(left) [label="Queryable"]
}
Analytics { # Analytics Layer
n15: rectangle label:"QuickSight Dashboard"
n16: rectangle label:"SageMaker ML Model"
n17: rectangle label:"Redshift Data Warehouse"
n14.handle(bottom) -> Analytics.n15.handle(top) [label="Visualize"]
n14.handle(bottom) -> Analytics.n16.handle(top) [label="Train Model"]
n14.handle(bottom) -> Analytics.n17.handle(top) [label="COPY INTO"]
}
Why This Workflow?
Traditional ETL pipelines require provisioning Spark clusters, managing job schedulers, and paying for always-on infrastructure. A serverless data pipeline uses managed services for every stage—ingestion, transformation, cataloging, and querying—eliminating infrastructure management while maintaining enterprise-grade data processing capabilities.
How It Works
- Step 1: Data streams from Kinesis, API Gateway, and IoT Core are buffered by Kinesis Firehose.
- Step 2: Lambda transform functions validate schemas, enrich records, and filter invalid data.
- Step 3: Transformed data lands in S3 raw zone, then is processed by Glue ETL into the curated zone.
- Step 4: The Glue Data Catalog registers schemas for all datasets.
- Step 5: Athena provides serverless SQL queries over the cataloged data.
- Step 6: QuickSight dashboards and SageMaker models consume the curated data for analytics and ML.
Alternatives
Self-managed Spark/Hadoop clusters offer more control but require significant DevOps investment. Snowflake or Databricks provide managed alternatives with different pricing models. This template shows the AWS-native serverless data pipeline.
Key Facts
| Template Name | Serverless Data Pipeline Architecture |
| Category | Architecture |
| Steps | 6 workflow steps |
| Format | FlowZap Code (.fz file) |
Related templates
Architecture
A microservices API gateway architecture diagram showing request routing, JWT authentication, rate limiting, service discovery, and response aggregation across distributed backend services. This template models the entry point for all client traffic in a microservices ecosystem, enforcing security policies before requests reach internal services. Ideal for platform engineers designing scalable API infrastructure with centralized cross-cutting concerns.
Architecture
A service mesh architecture diagram with Istio or Linkerd sidecar proxies handling mTLS encryption, traffic policies, circuit breaking, and distributed tracing across microservices. This template visualizes how a service mesh abstracts networking concerns away from application code, enabling zero-trust communication between services. Essential for teams adopting service mesh infrastructure to improve observability and security.
Architecture
A database-per-service architecture diagram where each microservice owns its dedicated data store, with event-driven synchronization via Kafka for cross-service data consistency. This template demonstrates the core microservices data isolation principle, showing how PostgreSQL and MongoDB coexist in a polyglot persistence strategy. Critical for architects enforcing service autonomy while maintaining eventual consistency.
Architecture
A microservices decomposition architecture diagram organized by business capabilities: Identity, Product Catalog, Pricing, and Order Fulfillment, each with independent data stores and APIs. This template shows how to break a monolith into services aligned with business domains, using a Backend-for-Frontend (BFF) pattern for client-specific aggregation. Useful for architects planning domain-driven microservice boundaries.
Architecture
A strangler fig migration architecture diagram showing the incremental replacement of a legacy monolith with new microservices, using a routing layer to split traffic between old and new systems. This template models the proven migration strategy where new features are built as microservices while legacy endpoints are gradually retired. Essential for teams modernizing legacy systems without risky big-bang rewrites.
Architecture
A service discovery architecture diagram with Consul or Eureka registry, client-side load balancing, health check heartbeats, and automatic instance registration and deregistration. This template visualizes how microservices dynamically locate each other without hardcoded endpoints, enabling elastic scaling and self-healing infrastructure. Key for platform teams building resilient service-to-service communication.