close

DEV Community

# dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
Processing High Frequency Solar Data Without HPC: Real Constraints and Design Decisions in MackSun
wilians_conde_6d4bbc5eed2 profile

Processing High Frequency Solar Data Without HPC: Real Constraints and Design Decisions in MackSun

#dataengineering #mongodb #systemdesign #bigdata
Comments
3 min read
Bypassing the "Pandas RAM Tax": Building a Zero-Copy CSV Extractor in C
nareshcn2 profile

Bypassing the "Pandas RAM Tax": Building a Zero-Copy CSV Extractor in C

#c #python #dataengineering #performance
Comments
2 min read
Financial Data Integration: A Practical Guide
andrew_tan_layline profile

Financial Data Integration: A Practical Guide

#architecture #dataengineering #systemdesign #tutorial
Comments
7 min read
Why Real-Time Data Integration Matters for Modern Applications
andrew_tan_layline profile

Why Real-Time Data Integration Matters for Modern Applications

#data #dataengineering #systemdesign #devops
Comments
5 min read
Why My S3 Backup Setup Broke: Buckets, “Folders”, and Scheduling Misconceptions
mohhddhassan profile

Why My S3 Backup Setup Broke: Buckets, “Folders”, and Scheduling Misconceptions

#aws #s3 #cloudstorage #dataengineering
Comments
3 min read
Apache Data Lakehouse Weekly: April 9–15, 2026
alexmercedcoder profile

Apache Data Lakehouse Weekly: April 9–15, 2026

#news #database #dataengineering #opensource
Comments
7 min read
Trying Out Snowflake's Adaptive Warehouse — Auto-Scaling Compute Without Manual Sizing
sagara profile

Trying Out Snowflake's Adaptive Warehouse — Auto-Scaling Compute Without Manual Sizing

#snowflake #dataengineering
Comments
10 min read
How I Built an End-to-End Data Engineering Pipeline for Hong Kong's Public Transport Network
mfarizala profile

How I Built an End-to-End Data Engineering Pipeline for Hong Kong's Public Transport Network

#dataengineering #datazoomcamp #googlecloud #bigquery
Comments
8 min read
Backpressure in document pipelines is an architecture problem first
cy_ong_591 profile

Backpressure in document pipelines is an architecture problem first

#architecture #dataengineering #sre #systemdesign
Comments
2 min read
Why mixed document packs make extraction pipelines harder to trust
cy_ong_591 profile

Why mixed document packs make extraction pipelines harder to trust

#dataengineering #machinelearning #productivity #systemdesign
Comments
2 min read
Debugging a Broken Metrics Pipeline: What Actually Went Wrong
mohhddhassan profile

Debugging a Broken Metrics Pipeline: What Actually Went Wrong

#dataengineering #clickhouse #vector #debugging
Image Image 2
Comments
3 min read
Why Cursor AI Won't Replace Data Engineers (And How to Actually Use It)
santosh_ronanki_9438d5944 profile

Why Cursor AI Won't Replace Data Engineers (And How to Actually Use It)

#dataengineering #ai #webdev
Image Image Image 3
Comments
2 min read
Are ClickHouse JOINs Slow? A 2026 PR-by-PR Analysis
manveer_chawla_64a7283d5a profile

Are ClickHouse JOINs Slow? A 2026 PR-by-PR Analysis

#data #analytics #dataengineering #database
Image 5
Comments
18 min read
Apache Airflow 2 vs 3: A Deep Technical Comparison for Data Engineers
de_clerke profile

Apache Airflow 2 vs 3: A Deep Technical Comparison for Data Engineers

#airflow #dataengineering #python #docker
Comments
15 min read
Why I Bypassed Pandas to Process 10M Records in 0.35s Using Raw C and SIMD
nareshcn2 profile

Why I Bypassed Pandas to Process 10M Records in 0.35s Using Raw C and SIMD

#c #performance #dataengineering #python
Comments
2 min read
👋 Sign in for the ability to sort posts by relevant, latest, or top.