Mikołaj Sędek’s Post

🚀 Reviving Data Engineering Skills: Event‑Driven ETL with S3, Lambda, and Databricks Implemented a production‑ready pattern: S3 -> Lambda -> Databricks Job -> S3 (processed). New files under "databricks-event-trigger/" automatically kick off a parameterized notebook run and land clean Parquet for downstream consumers. What’s included: ✅  S3 ObjectCreated -> Lambda (bucket/prefix guard) -> Databricks Jobs API (run‑now) ✅  Secrets Manager for the PAT; no third‑party deps in Lambda ✅ Notebook reads input_s3_url, writes Parquet to processed/, with logs in CloudWatch and Databricks Job Runs Why this approach: • Scales to zero when idle; fast to trigger and simple to operate • Auditable and replayable (the S3 path becomes the run context) • Cost‑disciplined with minimal moving parts S3 events vs “just use streaming” (for file drops): 💡 Operational simplicity: native S3 → Lambda, no brokers/partitions/consumer groups. Configure in minutes, maintain with minimal overhead. 💡 Cost model: pay‑per‑invoke with near‑zero idle cost. Perfect for bursty arrivals; avoids always‑on spend typical of streaming stacks. 💡 Precision filtering: prefix/suffix at the source (e.g., “databricks-event-trigger/”, “.csv”) cuts noise, prevents accidental invokes, and boosts reliability with simple, auditable guards. #DataEngineering #Databricks #AWS #S3 #Lambda #ETL #EventDriven #CostOptimization #Analytics

#databricks_basics_10 🚀 Understanding Databricks Cluster Configuration: Single‑Node vs Multi‑Node Choosing the right Databricks cluster mode can significantly influence cost efficiency, performance, and developer productivity. Whether you're running quick experiments or processing massive datasets, knowing when to use single‑node or multi‑node clusters makes all the difference. 🧩 Single‑Node Clusters — Lightweight & Cost‑Friendly Single‑node clusters run on a single VM containing both the driver and worker functions. They are ideal for: 🔺 Developing and testing code 🔺 Learning the Databricks environment 🔺 Running small‑scale jobs or local‑like workloads Their simplicity makes them extremely cost‑efficient, although they naturally come with limited processing power. ⚡ Multi‑Node Clusters — Power for Big Data Multi‑node clusters distribute processing across multiple machines, providing: 🔺 High scalability 🔺 Faster parallel computation 🔺 Support for complex algorithms and large data volumes They're designed for production‑grade ETL, machine learning pipelines, and analytics at scale. 🔍 When to Choose Which? ✔️ Use Single‑Node for development, quick experimentation, or CI pipelines that don’t require distributed compute. ✔️ Use Multi‑Node when performance, scale, or parallel execution is critical to your workload. 🛠️ Bonus Tip: Configuration Matters Databricks recommends tuning cluster settings (node type, autoscaling, runtime versions, etc.) as part of an overall optimization strategy. Proper configuration can significantly reduce cost while boosting performance for both cluster types. If you're building scalable data platforms or optimizing ETL workloads, understanding these modes helps ensure you’re not over‑ or under‑provisioning your compute resources. Stay tuned to learn more about databricks basics :) #databricks v4c.ai #data

From Kafka → Raw S3 → Spark → Analytics Gold The Real Architecture Behind Modern Data Platforms One of the biggest misunderstandings in Big Data is assuming that Kafka produces clean, analytics-ready data. It doesn’t. In real production environments, the workflow looks like this: ⸻ 🔶 1) Kafka Connect → Raw Zone (S3 / Data Lake) Kafka ingests massive streams of events, but the data is always raw: • Missing values • Duplicates • Wrong types • Corrupted or incomplete messages • No schema enforcement This is expected — Kafka is a transport and buffering system, not a data cleaning tool. ⸻ 🟦 2) Spark ETL → Cleaned Zone (Structured, Reliable Data) Spark takes the raw files and performs the heavy lifting: • Data cleaning • Type enforcement • Deduplication • Normalization • Parsing + validation • Transformations • Schema alignment This creates a unified, high-quality dataset for downstream use. ⸻ 🟩 3) Gold Layer → Analytics, BI, AI/ML Models After Spark processes the data, we finally get: • KPIs • Aggregations • Customer metrics • ML-ready datasets • Parquet/Delta optimized storage • High-performance query layers This is the layer consumed by dashboards, machine learning pipelines, and business applications. ⸻ 💡 Key Insight Raw data is never clean. Kafka moves data → Spark transforms it → the business gains value. This separation of responsibilities is the foundation of every modern data lakehouse. If you’d like, I can share a full end-to-end tutorial (Kafka → Spark Streaming → Delta Lake → Dashboard). ⸻ 🏷 #BigData #Kafka #Spark #PySpark #DataEngineering #ETL #Streaming #DataLake #S3 #AWS #DataArchitecture #Lakehouse #Analytics #MachineLearning

Excited to share my latest article: Implementing SCD in Spark Declarative Pipelines https://lnkd.in/eG9h4UFy In this piece, I explore how Databricks has simplified the implementation of Slowly Changing Dimensions (SCD) with declarative pipelines. What used to require complex MERGE logic and manual handling of timestamps can now be expressed in just a few lines of code — making pipelines more reliable, maintainable, and streaming‑native. #Databricks #ApacheSpark #DataEngineering #ETL #CDC #StreamingData #BigData #DeclarativePipelines #SCD #Analytics This approach is a big step forward for data engineers working with CDC flows, ETL, and real‑time analytics.

🚀 𝗦𝘁𝗼𝗽 𝗣𝗮𝘆𝗶𝗻𝗴 𝗳𝗼𝗿 𝗜𝗱𝗹𝗲 𝗖𝗹𝘂𝘀𝘁𝗲𝗿𝘀: 𝗛𝗼𝘄 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 𝗔𝘂𝘁𝗼𝘀𝗰𝗮𝗹𝗶𝗻𝗴 𝗔𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝗪𝗼𝗿𝗸𝘀 (𝗳𝗿𝗼𝗺 𝗮 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿’𝘀 𝗣𝗢𝗩) One of the most misunderstood features in Databricks is Autoscaling. Many teams enable it, but very few truly optimize around it. As a Data Engineer, here’s how I explain Autoscaling to teams building production-grade pipelines 👇 🔹 What is Autoscaling in Databricks? Autoscaling dynamically adds or removes worker nodes in a Databricks cluster based on the workload demand. Instead of running a fixed-size cluster: Databricks scales up when there’s more parallel work Scales down when tasks finish or executors go idle This directly impacts: ✅ Cost optimization ✅ Job performance ✅ Pipeline reliability 🔹 How Autoscaling Works (Behind the Scenes) Databricks monitors: Number of pending Spark tasks Executor utilization Stage parallelism Based on this: New worker nodes are provisioned automatically Idle workers are terminated after a cool-down period You define: Minimum workers Maximum workers Databricks handles the rest. 🔹 Real-World Example (ETL Pipeline) 📌 Scenario: A daily ETL job processes 10 TB of data from ADLS Gen2 using Spark. Without Autoscaling Fixed cluster: 16 workers During peak joins → underpowered → slow stages During lightweight transformations → wasted resources 💸 With Autoscaling Min workers: 4 Max workers: 20 What happens: 1️⃣ Heavy joins & shuffles → cluster scales up to 18–20 workers 2️⃣ Aggregations finish → workers scale down to 6 3️⃣ Final write → cluster stabilizes at minimum ➡️ Result: Faster job completion Lower overall compute cost No manual tuning per workload 🔹 Autoscaling + Job Clusters = Best Practice For production pipelines: ✔ Use Job Clusters instead of All-Purpose clusters ✔ Enable Autoscaling with realistic min/max values ✔ Combine with Photon for even better performance Autoscaling shines when workloads are: Spiky Data-volume dependent Running multiple stages with different resource needs 🔹 Common Mistakes I See ❌ Setting min workers too high (kills cost savings) ❌ Setting max workers too low (no performance benefit) ❌ Expecting autoscaling to fix bad Spark code 👉 Autoscaling optimizes resources, not poor partitioning or skewed joins. 🔹 Final Thought Autoscaling is not just a checkbox — it’s a design decision. When used correctly, it turns Databricks into a self-optimizing data platform for modern data engineering workloads. 💬 How are you configuring autoscaling in your Databricks pipelines? Let’s discuss what works (and what doesn’t). 𝗖𝗼𝗺𝗽𝗹𝗲𝘁𝗲 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗣𝗿𝗲𝗽𝗮𝗿𝗮𝘁𝗶𝗼𝗻 𝗚𝘂𝗶𝗱𝗲 𝗳𝗼𝗿 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗻𝗲𝗿𝘀 & 𝗗𝗮𝘁𝗮 𝗔𝗻𝗮𝗹𝘆𝘀𝘁𝘀. 👉 𝗚𝗿𝗮𝗯 𝗶𝘁 𝗵𝗲𝗿𝗲(𝟭𝟬𝟬% 𝗥𝗲𝗳𝘂𝗻𝗱): https://lnkd.in/giHUMNx9 #Databricks #DataEngineering #ApacheSpark #BigData #AzureDatabricks #CloudData #ETL #Lakehouse

🚨 Problems every data team hits at scale — and how Databricks fixes them ❌ Pain #1: “Our data lake exists… but nobody trusts the data” Broken pipelines, partial writes, silent schema changes Engineers spend hours reprocessing data instead of building features Solution: ✅ Databricks + Delta Lake brings ACID transactions, schema enforcement, and time travel Result: fewer incidents, faster recovery, and measurably lower operational toil ❌ Pain #2: “Performance tuning = bigger clusters = higher bills” Slow queries → teams scale compute blindly Costs grow faster than data volume Solution: ✅ Photon engine + optimized Databricks Runtime delivers 2×–8× faster execution Same workload, fewer nodes → 30–60% lower compute cost in real deployments ❌ Pain #3: “Batch, streaming, and ML all live on different stacks” Duplicate logic Inconsistent data across systems High integration overhead Solution: ✅ Databricks unifies batch + streaming + BI + ML on one Lakehouse One engine, one data copy, one security model ❌ Pain #4: “Upgrading Spark feels risky and expensive” Platform upgrades usually mean regressions and rewrites Teams stay stuck on old versions Solution: ✅ Databricks absorbs latest Apache Spark 4.x improvements into managed runtimes Performance + SQL + Python improvements with minimal or zero code change ❌ Pain #5: “Metadata and governance don’t scale with teams” Manual permissions No clear lineage Security slows down analytics Solution: ✅ Unity Catalog centralizes governance across workspaces Fine-grained access control Built-in lineage Faster compliance, safer self-service analytics ❌ Pain #6: “Our warehouse is fast… but rigid” Great for BI Poor for ML, streaming, and large-scale ETL Solution: ✅ Databricks SQL proves warehouse-grade performance TPC-DS 100TB benchmark ~2.2× faster at ~10% lower system cost Without sacrificing openness or flexibility ❌ Pain #7: “Tool sprawl is killing velocity” Separate tools for ingestion, processing, BI, ML High licensing + maintenance cost Solution: ✅ Databricks consolidates the stack Fewer tools Fewer failure points Faster onboarding and delivery 📊 Real-world cost math (why leadership cares) Annual analytics compute spend: $300K Runtime + engine optimizations save ~40% $120K/year saved — before accounting for reduced engineering effort 🔑 Takeaway Databricks doesn’t just make data faster — It makes data reliable, governable, and cheaper at scale. That’s why it’s not just a data platform — it’s an operating model for modern data teams. #Databricks #Lakehouse #DeltaLake #UnityCatalog #ApacheSpark #DataEngineering #AnalyticsEngineering #BigData #CloudCostOptimization #ModernDataStack #DataPlatform #AIInfrastructure

Why Data Pipelines Break in Production (And How to Prevent It) Building pipelines isn’t the hard part anymore. The real challenge is keeping them reliable, observable, and cost-efficient once they hit production. The failure patterns are consistent across data platforms, regardless of whether the stack is Kafka, Airflow, Spark, dbt, Snowflake, or Databricks. Below are the most common reasons data pipelines fail, and the engineering practices that help prevent those failures. Why Pipelines Fail 1. Ingestion without validation Data is loaded into lakes or warehouses without schema checks or quality rules, which creates downstream failures later. Solution: validate at the source using schema contracts, profiling, and automated data quality checks. 2. Writing transformations directly into production tables Direct writes lead to corrupted tables, inconsistent data, and difficult rollbacks. Solution: use layered architecture: Raw → Staging → Curated → Serving. 3. Technology decisions before problem decisions Teams select tools before understanding requirements like latency, frequency, or volume. Solution: architecture should follow the problem, not the other way around. 4. Lack of data observability Monitoring focuses on compute, not on data drift, volume anomalies, or freshness gaps. Solution: implement lineage, anomaly detection, and alerts as a first-class requirement. 5. No cost governance Cloud systems scale by design, so costs grow silently until invoicing exposes the issue. Solution: enforce optimization strategies like storage tiering, compression, partitioning, indexing, and auto-suspend compute. A Scalable Architecture Blueprint Raw Layer: Immutable ingestion with schema registry Staging Layer: Standardized schema, validation, and change tracking Curated Layer: Business logic, dimensional modeling or Data Vault Serving Layer: Analytics, BI, ML features, or real-time services Observability: Lineage, monitoring, quality metrics, and alerting integrated across all layers This approach reduces breakage, makes pipelines auditable, and allows scaling without constant redesign. Technology Examples (Use Case Driven) Batch processing: Airflow, dbt, Snowflake, BigQuery, Databricks Streaming: Kafka, Kinesis, Pub/Sub, Flink, Spark Streaming Data quality and governance: Great Expectations, Monte Carlo, DataHub, OpenLineage Final Note Reliable pipelines are not achieved by stacking more tools. They are achieved through design discipline, data contracts, observability, and a clear separation of responsibility between layers. What failure point do you see most in real-world data systems? Share your thoughts. It always leads to useful discussions. #DataEngineering #DataPipelines #ETL #c2c #ELT #Kafka #Airflow #Lakehouse #Snowflake #Spark #DataQuality #Observability #Architecture

🛢 𝗙𝗶𝗹𝗲 𝗙𝗼𝗿𝗺𝗮𝘁𝘀 𝗶𝗻 𝗕𝗶𝗴 𝗗𝗮𝘁𝗮: 𝗔 𝗗𝗲𝗰𝗶𝘀𝗶𝗼𝗻 𝗧𝗵𝗮𝘁 𝗜𝗺𝗽𝗮𝗰𝘁𝘀 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲, 𝗖𝗼𝘀𝘁, 𝗮𝗻𝗱 𝗦𝗰𝗮𝗹𝗲. . . When working with Big Data, file format is not just a storage choice — it’s an architectural decision. 𝐈𝐧 𝐁𝐢𝐠 𝐃𝐚𝐭𝐚,, the format you choose directly affects: Query performance Storage costs Data ingestion speed Scalability across distributed systems So what makes a file format “Big Data–ready”? 1️⃣ 𝐑𝐨𝐰-𝐁𝐚𝐬𝐞𝐝 𝐯𝐬 𝐂𝐨𝐥𝐮𝐦𝐧𝐚𝐫 𝐅𝐨𝐫𝐦𝐚𝐭𝐬 𝐑𝐨𝐰-𝐛𝐚𝐬𝐞𝐝 𝐟𝐨𝐫𝐦𝐚𝐭𝐬 (𝐂𝐒𝐕, 𝐉𝐒𝐎𝐍, 𝐀𝐯𝐫𝐨)store data row by row Best for write-heavy workloads and streaming ingestion 𝐂𝐨𝐥𝐮𝐦𝐧𝐚𝐫 𝐟𝐨𝐫𝐦𝐚𝐭𝐬 (𝐏𝐚𝐫𝐪𝐮𝐞𝐭, 𝐎𝐑𝐂) store data column by column Optimized for analytics, filtering, and aggregation This is why tools like Spark, Hive, and Presto perform significantly faster with columnar formats. 2️⃣𝐂𝐨𝐦𝐩𝐫𝐞𝐬𝐬𝐢𝐨𝐧 𝐚𝐧𝐝 𝐒𝐭𝐨𝐫𝐚𝐠𝐞 𝐄𝐟𝐟𝐢𝐜𝐢𝐞𝐧𝐜𝐲 Big Data means volume. Modern formats like Parquet and ORC support: Built-in compression Encoding techniques (dictionary, run-length encoding) Result: 1. Less storage usage 2. Faster I/O 3. Lower cloud costs 3️⃣ 𝐒𝐜𝐡𝐞𝐦𝐚 𝐄𝐯𝐨𝐥𝐮𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐌𝐞𝐭𝐚𝐝𝐚𝐭𝐚 In real-world pipelines, data changes over time. Formats such as Avro and Parquet support: Schema evolution Embedded metadata This makes pipelines more resilient and easier to maintain at scale. 4️⃣𝐂𝐨𝐦𝐦𝐨𝐧 𝐅𝐢𝐥𝐞 𝐅𝐨𝐫𝐦𝐚𝐭𝐬 𝐢𝐧 𝐁𝐢𝐠 𝐃𝐚𝐭𝐚 𝐂𝐒𝐕 – Simple, readable, but inefficient for large-scale analytics 𝐉𝐒𝐎𝐍  – Flexible, but verbose and slower at scale 𝐀𝐯𝐫𝐨 – Great for streaming and data serialization 𝐏𝐚𝐫𝐪𝐮𝐞𝐭 – Industry standard for analytics and data lakes 𝐎𝐑𝐂 – Highly optimized for Hive-based workloads 𝐅𝐢𝐧𝐚𝐥 𝐓𝐡𝐨𝐮𝐠𝐡𝐭 In Big Data, choosing the right file format can mean the difference between: 1. Queries running in seconds vs minutes 2. Manageable costs vs unexpected cloud bills 3. Scalable pipelines vs fragile systems #BigData #Data #FileFormats #Parquet #ORC #Avro #DataLake #ApacheSpark #Hive #CloudComputing

SCD in Databricks (Delta Lake) Type 1. Type 2. Type 3 Real SQL Examples + Architectur Why SCD Matters in Databricks Databricks = Lakehouse. Lakehouse = dimensions that evolve. SCDs allow: ✓ Historical reporting ✓ Auditability ✔ Time-based analytics Delta Lake makes SCDs easy + reliable Tools Used in Databricks You implement SCDs using: Delta Lake (ACID + Time Travel) MERGE INTO Auto Loader DLT (Delta Live Tables) Workflows Delta Lake = perfect for SCD patterns\ SCD Type 1 (Overwrite) Updates existing rows - no history. SQL in Databricks MERGE INTO dim_customer t USING updates u ON t.cust_id = u.cust_id WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *: Use when: typo fixes, non-historical attributes. SCD Type 2 (Full History) Tracks every version of the record. Databricks SQL MERGE INTO dim_customer t USING updates u ON t.cust_id = u.cust_id AND t.is_current = true WHEN MATCHED AND t.hash <> u.hash THEN UPDATE SET t.is_current = false, t.end_date = current_date() WHEN NOT MATCHED THEN INSERT (cols..., is_current, start_date) VALUES (..., true, current_date()); Delta advantages: Time Travel Schema evolution Consistent merges SCD Type 3 (Limited History) Stores current + previous values. Databricks SQL UPDATE dim_product SET prev_price = price, price = u.price FROM updates u WHERE dim_product.prod_id = u.prod_id; Lightweight alternative to Type 2. - How Databricks Handles Late Arriving Records Delta MERGE + window logic lets you handle: Out-of-order events Backdated updates Multi-row per key issues Use: ROW_NUMBER() OVER (...) to pick the correct version. Medallion Architecture + SCD SCD fits naturally in Databricks layers: Bronze → raw data Silver→ dedupe + schema fix + hashing Gold → SCD Type 1 or Type 2 dimension tables Pro tip: Compute hash in Silver to detect changes efficiently.

SCD in Databricks (Delta Lake) Type 1. Type 2. Type 3 Real SQL Examples + Architectur Why SCD Matters in Databricks Databricks = Lakehouse. Lakehouse = dimensions that evolve. SCDs allow: ✓ Historical reporting ✓ Auditability ✔ Time-based analytics Delta Lake makes SCDs easy + reliable Tools Used in Databricks You implement SCDs using: Delta Lake (ACID + Time Travel) MERGE INTO Auto Loader DLT (Delta Live Tables) Workflows Delta Lake = perfect for SCD patterns\ SCD Type 1 (Overwrite) Updates existing rows - no history. SQL in Databricks MERGE INTO dim_customer t USING updates u ON t.cust_id = u.cust_id WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *: Use when: typo fixes, non-historical attributes. SCD Type 2 (Full History) Tracks every version of the record. Databricks SQL MERGE INTO dim_customer t USING updates u ON t.cust_id = u.cust_id AND t.is_current = true WHEN MATCHED AND t.hash <> u.hash THEN UPDATE SET t.is_current = false, t.end_date = current_date() WHEN NOT MATCHED THEN INSERT (cols..., is_current, start_date) VALUES (..., true, current_date()); Delta advantages: Time Travel Schema evolution Consistent merges SCD Type 3 (Limited History) Stores current + previous values. Databricks SQL UPDATE dim_product SET prev_price = price, price = u.price FROM updates u WHERE dim_product.prod_id = u.prod_id; Lightweight alternative to Type 2. - How Databricks Handles Late Arriving Records Delta MERGE + window logic lets you handle: Out-of-order events Backdated updates Multi-row per key issues Use: ROW_NUMBER() OVER (...) to pick the correct version. Medallion Architecture + SCD SCD fits naturally in Databricks layers: Bronze → raw data Silver→ dedupe + schema fix + hashing Gold → SCD Type 1 or Type 2 dimension tables Pro tip: Compute hash in Silver to detect changes efficiently.