Andrzej Ermel

We’re continuing our Data Point Prague 2026 Precon Series with a topic that every growing #BI and Fabric team eventually runs into: CI/CD for #PowerBI and #MicrosoftFabric is no longer optional. This is a full-day, hands-on workshop led by Dániel Gábor Patkós and Kevin Chant - focused on building professional DataOps support with version control, automation, and repeatable deployments. Not theory. Not DevOps buzzwords. A practical day on how CI/CD actually works in real Power BI and Fabric projects. You’ll go from: ▪️using Power BI project files (.PBIP) and #GitHub for version control ▪️setting up repositories, branching strategies, and team collaboration ▪️handling merge conflicts, rollbacks, and quality checks with Best Practice Analyzer ▪️to configuring source control and CI/CD in Fabric, including Fabric CLI and scripted deployments ▪️and finally implementing automated testing, approvals, and post-deployment housekeeping for enterprise-scale workflows This precon is about building confidence and discipline in how data solutions are developed, shipped, and maintained - not just getting things to “work”. By the end of the day, you’ll be able to design and run a CI/CD pipeline for Power BI and Fabric projects that scales with your team and your organisation. This is Precon #2 of 6 we’ll be announcing for Data Point Prague 2026. More coming very soon. Prague | May 28–29, 2026 Preconference workshops | Limited capacity | Registration link in the comments

11

1 komentarz

There are many ways to use foundation models on Databricks, and it may get confusing what option to choose and when. 1️⃣Databricks-hosted models. There are several ready-to-use foundation model APIs available within your workspace, including models like Meta Llama 4, Open AI GPT-5 (not available on Azure), Gemini 3 Pro, and many others. This option is great for experimentation, does not require any additional setup. However, it may not be sufficient if you have serious compliance requirements, or require high throughput and performance guarantees. 2️⃣ Provisined throughput endpoints. You can deploy an open-source model from the ai schema under system catalog or a model downloaded from Hugging Face (possibly fine tuned) logged using MLflow and registered in Unity Catalog. Note that only certain model families with certain configurations are supported. 3️⃣ External model. You can create a serving endpoint that accesses an external model hosted by OpenAI, Azure OpenAI, AWS Bedrock, Anthropic, and other Open AI API compatible providers, with authentication and governance handled within Databricks. Why choose Databricks model serving? ➡️ Unified interface: all options use Mosaic AI Model Serving, which follows the OpenAI API specification (so you can use OpenAI client), allowing to easily switch between models. Mosaic AI model serving supports three endpoint types: llm/v1/chat (corresponding to the OpenAI Chat Completions API), llm/v1/embeddings (corresponding to the OpenAI Embeddings API), and llm/v1/completions (following the legacy OpenAI Completions API). The responses API is only supported for Databricks-hosted OpenAI foundation models (except gpt-oss models). ➡️ AI Gateway can be enabled on an endpoint allowing for usage tracking, payload logging, and traffic management. AI Gateway can be applied to any Mosaic AI Model Serving endpoint, including foundation model APIs. You can also enable AI Guardrails to ensure PII data is deleted or masked and harmful content is blocked, for both model input and output.

88

4 komentarze

Tired of the tradeoff between performance and usability in your data stack? Databricks brings the power. Sigma brings the polish. Together, they deliver fast, flexible analytics your whole team can trust—and actually use! In our latest blog, we break down: ✔️ Why this combo works so well for modern data teams ✔️ How Sigma unlocks real-time insights on top of Databricks ✔️ What it means for governance, scale, and self-service 👀 Curious how to get the best of both worlds? At Brooklyn Data, we help teams architect and optimize modern stacks like this—so you can go from raw data to real impact, faster. 👉 Read the full post: https://lnkd.in/ehejQVGT

11

1 komentarz

Composable Data Architectures: Rethinking the Data Platform Traditional, monolithic data warehouses were built for stability - not speed. But today’s business environment demands agility, integration, and scale. That’s where composable data architectures: like data mesh and data fabric come in. They replace rigid systems with modular, plug-and-play components that can evolve as fast as the business. New data source? Add it. Need real-time insights? Extend it. Scaling operations? Do it without breaking what already works. It’s a shift from viewing data as a fixed infrastructure to treating it as a flexible ecosystem - built to adapt. The real question: Are organizations ready to build data platforms that move at business speed? #DataArchitecture #DataStrategy #DataMesh #DataFabric #ModernDataStack #FluidataInsights

11

💡 𝐏𝐨𝐰𝐞𝐫 𝐁𝐈 𝐓𝐢𝐩 𝐨𝐟 𝐭𝐡𝐞 𝐃𝐚𝐲 Did you ever have to whitelist Power BI in the 𝐟𝐢𝐫𝐞𝐰𝐚𝐥𝐥 of a 𝐝𝐚𝐭𝐚 𝐬𝐨𝐮𝐫𝐜𝐞, and you needed to know the 𝐈𝐏 𝐧𝐮𝐦𝐛𝐞𝐫 that you need to whitelist? On your machine, with 𝐏𝐨𝐰𝐞𝐫 𝐁𝐈 𝐃𝐞𝐬𝐤𝐭𝐨𝐩, there are many web pages like ipinfo https://lnkd.in/dKT9jyVh that return the public IP of your machine. But what if you are setting up a refresh from the Power BI cloud or via a gateway, and you cannot just use a web browser to check the IP? There is one API that returns your IP as a JSON response that is super easy to use in Power Query: 𝐢𝐩𝐢𝐟𝐲.𝐨𝐫𝐠 Now, if you run the refresh in the 𝐜𝐥𝐨𝐮𝐝, you can see in the report the IP number that is used by the cloud service. If you run this on a 𝐠𝐚𝐭𝐞𝐰𝐚𝐲, you can see under which IP number the gateway is seen. And, of course, it also works in Power BI Desktop and shows the public IP number of your machine.

24

2 komentarze

Maxteabag/sqlit: A user friendly TUI for SQL databases. Written in python. Supports SQL server, Mysql, PostreSQL and SQLite, Turso and more., https://lnkd.in/egv6rixE IA Summary: Tired of cumbersome GUI tools for quick SQL queries? Discover `sqlit`, a lightweight, Python-based terminal user interface designed for fast, intuitive interaction with SQL Server, PostgreSQL, MySQL, SQLite, and many other databases. #sqlit #sqlite #TUI #cli #sqlserver #mysql #mariadb #postgresql #turso

1

𝐘𝐨𝐮𝐫 𝐃𝐚𝐭𝐚 𝐉𝐮𝐬𝐭 𝐆𝐨𝐭 𝐒𝐦𝐚𝐫𝐭𝐞𝐫… 𝐁𝐮𝐭 𝐃𝐨 𝐘𝐨𝐮 𝐊𝐧𝐨𝐰 𝐇𝐨𝐰? Snowflake Cortex brings AI natively to your data - and that changes everything. No more exporting data. No more broken pipelines. No more waiting for insights. 𝐖𝐢𝐭𝐡 𝐒𝐧𝐨𝐰𝐟𝐥𝐚𝐤𝐞 𝐂𝐨𝐫𝐭𝐞𝐱, 𝐲𝐨𝐮 𝐜𝐚𝐧: ✅ Run AI directly on your data ✅ Automate intelligence inside your workflows ✅ Get faster, more accurate business decisions ✅ Keep security, governance, and performance intact 𝐒𝐦𝐚𝐫𝐭𝐞𝐫 𝐝𝐞𝐜𝐢𝐬𝐢𝐨𝐧𝐬 𝐝𝐨𝐧’𝐭 𝐬𝐭𝐚𝐫𝐭 𝐰𝐢𝐭𝐡 𝐦𝐨𝐫𝐞 𝐭𝐨𝐨𝐥𝐬 - 𝐭𝐡𝐞𝐲 𝐬𝐭𝐚𝐫𝐭 𝐰𝐢𝐭𝐡 𝐀𝐈 𝐰𝐡𝐞𝐫𝐞 𝐲𝐨𝐮𝐫 𝐝𝐚𝐭𝐚 𝐥𝐢𝐯𝐞𝐬. Ready to activate AI inside your Snowflake environment? 𝐁𝐥𝐮𝐄𝐧𝐭 helps you implement, optimize, and scale 𝐒𝐧𝐨𝐰𝐟𝐥𝐚𝐤𝐞 𝐂𝐨𝐫𝐭𝐞𝐱 for real business impact. #Snowflake #SnowflakeCortex #EnterpriseAI #DataAnalytics #CloudData #AIInnovation #BluEnt #CFO #Snowflakeintegration

6

❄️ What makes Snowflake the new standard in cloud data platforms? In the new episode of #Data_in_Action, Rostyslav Fedynyshyn sits down with Oleksandr Chakalov, Big Data Engineer and Data Competence Lead at N-iX, to break it down. 💡 In this episode, you’ll learn: • What make Snowflake a key choice for modern architectures • How Snowflake is evolving into an AI Data Cloud • What AI/ML workflows look like using Snowpark, Cortex AI, and more • How Snowflake delivers built-in governance, privacy, and security ⏯️ Watch the full conversation: https://lnkd.in/dktN4imy #Snowflake #AIDataCloud #DataEngineering #ML #Analytics

20

Every SQL Server ships with a setting that lets it eat all your RAM. It's called max server memory and the default is 2,147,483,647 MB. Essentially: "take everything." Most people never change it. I've seen production outages caused by exactly this. That's just one of five misconfigurations we find on almost every SQL Server we scan. The others: - Optimize for Ad Hoc Workloads disabled (plan cache bloated with thousands of single-use plans) - TempDB running on a single file (hello PAGELATCH waits) - Auto-Shrink enabled (silently fragmenting your indexes in a loop) - Cost Threshold for Parallelism still at 5 (parallelizing queries that would run faster on one core) The frustrating thing? All five take less than a minute to fix. One line of T-SQL each. But they go unnoticed for years because nothing alerts on them; they just quietly degrade performance. After 15+ years of reviewing SQL Servers, I'd estimate 90% of instances have at least two of these. Some have all five. Full write-up with the T-SQL fixes for each one: https://lnkd.in/eX_jMHc4 #SQLServer #DatabasePerformance #DBA

32

8 komentarzy

Many semantic layers are not starting as enterprise BI projects, they are showing up first inside a single chatbot. In some companies the layer sits at the center of the BI stack. In others it exists only to support one AI interface. That should get the attention of marketing and digital leaders. Many teams are trying to operationalize AI on top of fragmented marketing data, then wondering why the answers vary across dashboards, spreadsheets, and agents. The friction usually sits in the translation layer underneath. AI depends on shared definitions of things like customers, pipeline, CAC, influence, and attribution that remain consistent regardless of where someone accesses the data. The surge of interest in semantic web and ontology-driven systems reflects the same underlying need. Companies are investing in structure because interoperability, explainability, and auditability are becoming operational requirements. Without that structure, “self-serve” analytics tends to push inconsistency further downstream. Teams gain access to more data, but the number of competing metric definitions grows even faster. AI interfaces make that problem visible quickly. When a chatbot confidently returns the wrong number, everyone suddenly cares about definitions. Many organizations are finding it easier to start smaller than trying to solve everything at once. A semantic layer built for one use case still needs to produce metrics that match what people see in Power BI, Excel, and activation tools. If your AI roadmap is still mostly model selection and prompt experimentation, it may be worth stepping back for a moment. Where do your business definitions actually live, and who owns them when finance and marketing both need the same number? #AI #DataStrategy #SemanticLayer #MarketingAnalytics

30

ETL in Databricks: Which tool for what? A brilliant guide from Mariusz Kujawski about ETL in Databricks: https://lnkd.in/eT_4B887 1. Auto Loader: for incremental file ingestion. It handles schema evolution, archiving and Delta integration with minimal setup. 2. Custom Pipelines: for advanced use cases with custom logic, streaming ingestion, or complex data validation. SQL, Python, Scala. 3. Declarative Pipelines: standardised, reusable ingestion with built-in data validation and SCD2 handling. 4. Direct Access: for SaaS sources like Salesforce and ServiceNow. Simple, API-driven ingestion directly into streaming or materialised tables. Thank you for writing this Mariusz. They are called Lakeflow in Databricks: https://lnkd.in/edwk2jSF 1. Lakeflow Connect (#4 above) 2. Lakeflow Declarative Pipeline (#3 above). This was known as Delta Live Tables (DLT) 3. Lakeflow Jobs (orchestrator) 4. Spark Runtime (compute) So if you think about ETL in Databricks, think Lakeflow. That's the new name for the umbrella. What about Auto Loader? Is it gone? No it's still there. You do Auto Loader using Lakeflow Declarative Pipeline: https://lnkd.in/eHyGCnsa What about Custom Pipeline?Use Lakeflow Declarative Pipeline to create custom pipelines: https://lnkd.in/eFEiyKTu Keep learning! My Linkedin articles: https://lnkd.in/eRTNN6GP. My blog: https://lnkd.in/eDdTNzzW #Databricks #ETL

127

2 komentarze

Data modeling used to be simple. OLTP = 3NF OLAP = Dimensional Clear rules. Clear structure. Clear expectations. Then came the data lake. And everything got… flexible. (Which is a polite way of saying “chaotic.”) Now we’re in a new era: LLMs RAG Vectors Graphs And the question is back: 👉 Do we still model data? Here’s the answer: Yes. But not the way we used to. We don’t just model data anymore. We model: • Meaning (vectors) • Relationships (graphs) • Context (RAG) • Behavior (agents) The shift is this: From: Rigid schemas To: Layered intelligence 3NF didn’t go away. Dimensional didn’t go away. They just became… layers. The real evolution? We stopped modeling data structures. We started modeling understanding. #DataModeling #DataArchitecture #AgenticAI #RAG #LLM #Graph #Databricks #AILeadership

1

https://lnkd.in/d_P-Un5f Incremental processing of files from volumes / stream processed messages means no manual bookkeeping and faster / more efficient compute (your pipeline processes only new data and runs minimal merge updates on existing one, data reprocessing is minimised) ⚡️💰Because of that if you operate on Databricks Lakehouse an your ETL processing logic is based on incrementally added events or snapshots it makes a lot of sense to use Lakeflow Spark Declarative Pipelines to ingest, transform and finally model data using SCD1 / SCD2 (the latter supporting tracking history on selected columns to remove duplicates and limit history scope) using fully incremental logic. The framework supports SQL and PySpark APIs, static frame joins and incremental processing that can be combined with batch processing in a single pipeline if needed. Combined with Standard Serverless Compute (very good price / performance ratio), ability to write to custom schemas and catalog in UC, Workflows scheduling and alerting, Asset Bundles CI/CD deployment and a brand new WebUI editor - Lakeflow Pipelines have truly become extremely valuable part of Lakehouse ecosystem. PS. You can run incremental ingestion / processing both in batch and real time mode with zero code changes (of course if you need sub-second refresh you need to switch appropriate sources like Kinesis / Kafka). #Databricks #LakeflowPipelines #ETL #IncrementalProcessing #Spark #DataEngineering

28

Interesting recap about new skills and ways of working related to Data engineers Teams in the new AI era. Schema Drift, behaviour DQ, use the DAG as a part of agents brains, context Design and another part y f the new datatoolkit. https://lnkd.in/dreGtxTp

9

Enterprise Data Architect: From “Paper Architect” to “Field Architect” 🧭👷‍♂️ In an era where AI can generate a full Reference Architecture in seconds, do we still need Enterprise Data Architects (EDAs)? My answer: Yes — more than ever. But the role has fundamentally changed. 🔥 🛑 When “Best Practices” Are No Longer a Moat Today, AI can instantly draft: 🏗️ Data Streaming Architecture 🧩 Data Mesh operating model 🧰 Modern Data Stack blueprints Any senior engineer can access “gold standards” with a good prompt. But the real challenge isn’t drawing a beautiful diagram. The challenge is this: ➡️ Will that architecture survive in your organization’s context? Culture. Skills. Technical debt. Regulatory constraints. Delivery pressure. 🛠️ Enterprise Architect 2.0: From Theory to the Trenches In the journey of building Data Platform, I’ve learned the modern EDA needs to grow across 3 pillars: 1) Hands-on Implementation (Code, Don’t Just Draw) 💻 Move beyond diagrams — be able to build a POC. Don’t just recommend tools — test them on your real infra. Don’t only stare at “Ideal State” — look directly at technical debt and design the most viable path forward. 2) Context-Aware Decision Making (Trade-offs, Not Ideals) ⚖️ Every architecture is a trade-off: Data streaming is great — but do you have the capability to build & operate Spark pipelines reliably and cover billions of transaction per minutes? Data Mesh is inspiring — but is your culture ready for decentralization and true ownership? A best practice from Big Tech can fail inside a Vietnamese bank facing strict local regulatory and compliance expectations (e.g., Decision 2439). 🏦📜 ➡️ AI knows patterns. You know the exceptions. 3) Collaborative Maturity Builder (From Gatekeeper to Enabler) 🤝 Architecture is a journey, not a document. Shift from “Architect as Gatekeeper” 🚧 to “Architect as Enabler” ✅ Work side-by-side with DE / QA / Ops / Security to raise maturity step-by-step — not just hand over a blueprint and walk away. ⚠️ The Risk: When “Solutions” Become Blockers I’ve seen “perfect” architectures turn into operational nightmares: 🧱 Over-engineered governance that kills business agility 🕳️ Projects stalled because designs ignored team capability ❌ Architecture without implementation feasibility = a blocker, not a solution 💡 Bottom Line AI democratizes architectural knowledge — but it can’t replace contextual intelligence. An Enterprise Data Architect is more valuable than ever — if they can: ✅ Get hands dirty (POC + implementation) ✅ Understand business constraints + regulatory nuance ✅ Balance Ideal vs Practical AI can draw the map 🗺️ — but only an architect who understands the terrain can lead the way 🧗‍♂️. #DataArchitecture #EnterpriseArchitect #DataStrategy #DigitalTransformation #DataGovernance #Fintech #BankingTechnology #DataPlatform #Architecture #Leadership

65

13 komentarzy

I’ve seen this many times: if the answer to “Why are we moving to Databricks?” is “to modernize,” then the real transformation hasn’t started. Modernization isn’t a goal. It’s a by‑product of unlocking business value. The conversation should begin with: What revenue, cost, risk, or productivity benefits are we targeting? Without that, adopting a new platform is just accumulating technology. Once value is explicit, sponsors understand that while new use cases deliver visible benefits, there is also essential “invisible” work, e.g.: migrating logic, removing duplication, and retiring legacy patterns....to avoid mismatched numbers and parallel stacks. From experience, a practical approach is: - Bring all data - including transformed reporting tables - into one place to avoid the “system X vs system Y” issue. - Stop new work on legacy systems and govern exceptions. - Migrate and build new capabilities in parallel, with clear focus if the team is small. - Keep legacy‑derived tables in the new platform for backwards compatibility. - Educate continuously: analytics is an iceberg—value is visible, but most effort sits below the surface. If business value leads, modernization follows. If it doesn’t, you’re just adding another platform.

13

2 komentarze

This is great news and helps hugely as an option to manage data in an operational sense that is “not ready” to be prepped for analytics. This allows a fast OLTP style piece of infrastructure to exist and help process operational style data that needs single record type movement aligned with business process system of record data movement. Looking forward to how we weld this into our eco-system.

2

Welcome to Elitmind's Expert Corner ☕ - our new series where we explore the latest developments shaping the #data & #AI landscape. Today, our Data Platform & BI Lead, Adrian Kukiełka, shares his thoughts on three significant announcements from Databricks. 💡Key updates: 🔹 Lakebase - New OLTP Architecture: An open database architecture that separates storage from compute, designed for serverless scaling. This addresses a common need we've observed among clients working with PostgreSQL who have been seeking this type of solution. 🔹Agent Bricks - Simplified AI Agent Development: A platform that automates the technical complexities of building AI agents. Companies can focus on defining objectives in natural language while the system handles optimization. Adrian notes that this brings agentic workflows from AI research into practical data operations. 🔹Databricks Free Edition - Open Access to Platform. The complete Databricks suite is now available at no cost for learning and experimentation. This removes barriers for skill development and training across organizations. Adrian 's perspective: "These updates show Databricks is addressing real implementation challenges. From infrastructure gaps to skills development, they're making enterprise data and AI tools more accessible. The free edition particularly helps with the training bottleneck many of our clients face." 🎥 Live from the field: Today, our Elitmind team, Dorota Szepietowska, Andrzej Ermel, and Adrian Kukiełka, are attending the #DatabricksPartnerChampion Event, where they're diving deep into the latest innovations and gaining firsthand insights into their practical applications. What's your take on these developments? Share your thoughts in the comments below! #DataEngineering #AI #Databricks #DataPlatform #TechUpdates #DatabricksPartnerChampion

28

1 komentarz