Arthur De Roey

New blog post by Hannes Mühleisen and Mark Raasveldt: Changing Data with Confidence and ACID We explain the ACID properties of database transactions and illustrate each property with a concrete example in DuckDB. We show that transactions are also useful for data analysis workloads to guarantee the integrity and durability of data. Finally, we test DuckDB using the full TPC-H benchmark. https://lnkd.in/eshNT83m

96

1 Comment

𝐒𝐭𝐚𝐭𝐞 𝐨𝐟 𝐑𝐮𝐧𝐧𝐢𝐧𝐠 𝐋𝐚𝐫𝐠𝐞 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐌𝐨𝐝𝐞𝐥𝐬 (𝐋𝐋𝐌𝐬) 𝐨𝐧 𝐌𝐨𝐛𝐢𝐥𝐞 𝐃𝐞𝐯𝐢𝐜𝐞𝐬 This is where we are at with running LLMs on local devices...PowerInfer-2 by SJTU-IPADS Labs is a optimized inference framework designed specifically for smartphones. With TurboSparse-Mixtral-47B, it achieves an impressive speed of 11.68 tokens per second, which is up to 22 times faster than other state-of-the-art frameworks: 🔧 PowerInfer-2 uses heterogeneous computing, I/O-Compute pipelining, and TurboSparse with up to 97% sparsity. 📈 TurboSparse enables FFN sparsity of Mistral and Mixtral up to 90% and 97%. 🚀 Running Mixtral 8x7B at 11 tokens/s on a mobile phone. 🔐 Bringing cloud-scale models directly to your device without compromising privacy. Making AI accessible to everyone, everywhere. 🔓 PowerInfer-2 will be open-sourced in stages. Home: https://powerinfer.ai/v2/ Abs 1: https://lnkd.in/gKpricuW Abs 2: https://lnkd.in/gEtYnRnN Github: https://lnkd.in/gVTh8hGG HF: https://lnkd.in/gwNh6T2K #AI #GenAI #LLM #MachineLearning #LLM #MobileAI #MobileLLM #TurboSparse #Mistral #Mixtral #HeterogeneousComputing #IOComputePipelining #ModelSparsity #MobileAI #CloudScaleModels #Cloud #CloudComputing #PrivacyFirst #OpenSource #SmartphoneAI

1

Data Engineering: Working with semi-structured data types. In our previous post, we demonstrated how to parse comma-separated data into a structured file format within the Snowflake project that utilizes the medallion architectural framework. In this post, we’ll introduce another file format that you’ll frequently encounter in your data engineering projects: the JSON File Format. JSON (JavaScript Object Notation) is an open standard file format used for sharing data. It employs human-readable text to store and transmit data objects, which consist of attribute-value pairs and arrays. This file format is commonly used for transmitting data between web applications and servers, making it very popular and suitable for our discussion and demonstration. To explain in more detail, we have three common types of JSON: JSON objects, JSON nested objects, and JSON arrays. JSON objects: “User_Name” :{ “Fist_Name”: “Henry”, “Last_Name”: “Godson” } Nested JSON objects: “User_Informations”: { “Date”: “2024-05-21”, “Sports”: {“Football”: “Good”, “Swimming”: “Excellent” } “Name”: “Henry } JSON arrays: (JSON are written inside the square brackets). “Employees”: [ { “First_Name”: “Henry”, “Last_Name”: “Godson” }, { “First_Name”: “Endy”, “Last_Name”: “Junior” }, ] We also have JSON Array in the simple format as “Favourit_Sports”: [“Football”, “Baseball”]. While certain data migration tools, such as Azure Data Factory, possess built-in intelligence for transforming data from one format to another, most data engineering ETL projects necessitate data engineers to perform similar transformations using either SQL or Python. To transform a JSON array using the Medallion Architectural Framework, we have two major approaches: Copy to Bronze and Transform: In this approach, we copy the JSON file into the Bronze layer and then transform the data to a structured format before moving it into the Silver table. Direct Transformation to Bronze: The second approach involves transforming the data directly to a structured format and depositing it into the Bronze layer straight from the staging environment. This approach is recommended, and we’ll demonstrate how to achieve it in our upcoming post. Stay tuned!

3

What happens to data governance in the era of data mesh? Join Karin Håkansson, Kinda El Maarry, PhD, and Maarten Masschelein, for this very discussion on ➡ Data Governance in a Data Mesh ⬅ that will be moderated by Andrew Sharp, in partnership with Data Mesh Learning Community. 👀 Taking place on August 6 at 8:00 am PDT | 11:00 am EDT | 5:00 pm CEST, here's a taste of what there is to look forward to: 1️⃣ Strategies for maintaining high standards of data quality and security in a decentralized environment 2️⃣ Balancing autonomy and oversight to create scalable and efficient data governance models 3️⃣ How to achieve compliance without compromising on the flexibility and innovation that data mesh offers 4️⃣ Real-world examples of teams successfully implementing data mesh to drive their data strategy forward 📝 Sign up here: https://lnkd.in/dRmyh9Gj #datamesh #datagovernance #federatedgovernance #dataquality

14

🕹️ Our data engineering team often encounters #SSIS performance bottlenecks, ranging from inefficient queries to missing indexes. But one area that’s often overlooked? The impact of Data Flow Tasks and OLE DB Destination properties. In this blog, we highlight key settings—like AutoAdjustBufferSize, DefaultBufferMaxRows, and DefaultBufferSize—that are crucial for optimizing SSIS package performance. Looking for simple tips to boost your SSIS performance? Read the full article here: https://lnkd.in/eNi8HgcP

7

💡Case study in corporate BS ❓When does a corporation decide *not* to release their products into one of the largest markets in the world? 🤔We can safely assume, that the real reason behind Meta’s decision *not* to release their multimodal models in the EU is *the exact opposite* to what they are claiming. 🇪🇺EU regulatory environment in regards to AI is one of the most transparent ones, especially now, after the long-awaited and publicly discussed AI act was published and will be applied starting August 2nd. ⚠️And Meta (as everybody else) needs to comply with AI Act’s copyright and privacy provisions, but most likely can’t. 🧵Remember last year how we got Threads 6 months later than everybody else? It was for the similar reason(s) - at least a basic protection of *our privacy* was one of them. 🤷‍♂️That’s why a corporation decides *not* to release their products into one of the largest markets in the world. 🤝Let’s *not* fall into the corporate BS PR trap here, shall we?

17

𝗖𝗼𝗻𝗰𝘂𝗿𝗿𝗲𝗻𝗰𝘆 𝗰𝗼𝗻𝘁𝗿𝗼𝗹 𝗮𝗻𝗱 𝗹𝗼𝗴𝗴𝗶𝗻𝗴 𝗶𝗻 𝗗𝘂𝗰𝗸𝗗𝗕 Have you ever wondered what the .wal file is when working with DuckDB? Time to find out! This week, the co-creators of DuckDB, Hannes Mühleisen and Mark Raasveldt, wrote a blog post about the Multi-Version Concurrency Control and Write-Ahead-Logging schemes designed to ensure transactional guarantees: https://lnkd.in/dUHSfUDU

91

1 Comment

Feature Article: 3 Metadata Tasks that GenAI Can Automate Jake Athey, VP Go-To-Market and Sales for DAM and PIM at Acquia, explores the various ways in which DAM users can leverage generative artificial intelligence to automate common metadata workflows. https://lnkd.in/eFdGuHhv #DAM #digitalassetmanagement #AI #metadata

3

AtScale + Snowflake + Delphi Labs - Unlock the Potential of Self-Service Analytics – https://bit.ly/3UOjIBu The Future of Self-Service Analytics with Snowflake, Semantic Layers & LLMs - How to Deliver Self-Service Analytics on Snowflake Using a Semantic Layer and LLMs Join AtScale’s webinar on May 22nd to unlock next-level data analytics with Snowflake, AtScale, and Delphi Labs. Learn from industry experts how to integrate Snowflake's powerful platform with AtScale's Semantic Layer and leverage LLM’s for transformative insights. Register today! Wednesday, May 22, 2024 2:00 PM ET (11:00 AM PT) | 60 mins Featured Speakers • Dave Mariani, Founder & CTO, AtScale • Alex Ross, Senior Solutions Engineer, Snowflake • David Jayatillake, CEO, Delphi Labs • Michael Irvine, Co-Founder, Delphi Labs You’re invited! - https://bit.ly/3UOjIBu Learn how to integrate Snowflake's data platform with AtScale’s Semantic Layer and Large Language Models (LLMs) to enhance self-service analytics capabilities. You'll walk away with actionable insights to harness the full potential of this innovative approach. Don't miss this opportunity to enhance your data strategy and stay ahead in today's dynamic data landscape. In this webinar, we’ll cover: • The Foundation of Data Management: Learn from Alex Ross, Senior Solutions Engineer at Snowflake, about the fundamentals of an efficient data platform. • Introduction to Semantic Layers: Discover how a semantic layer can simplify data accessibility and enhance business insights with Dave Mariani, CTO at AtScale. • Power of LLMs in Analytics: Delphi Labs will present a live demonstration of how LLMs integrated with a semantic layer can revolutionize data interaction. Who should watch This webinar is designed for those eager to streamline their analytics processes and make data-driven decisions faster and more efficiently.

2

Is your data really bad - or is it just not fit for your purpose? Welcome to Databout #8, weekly data nibbles for non-data professionals. Subjective (or "fit-for-purpose") Data Quality issues arise when your data quality requirements are different - and more stringent - than the quality requirements of the process the data was originally meant to support. For example: -> missing attributes -> data filled in haphazardly, without much regard for correctness (the latter happens when the systems puts more requirements on data than the process that uses the data itself). So long the originating and consuming processes are happy, there is no real data quality issue. The data is perfectly fit for their purpose - just not for yours. Situations like this often result in unnecessary escalations. Here are some reasons why this happens - and how to prevent them. 1. Looking at metadata from a purely technical perspective. Often, a company's metadata set comes down to annotations of the datasets and Data Compliance classifications. This is not enough to provide the understanding of the purpose of data. How to prevent: -> Manage your metadata integrally. Make sure your data comes with the annotations relating it to processes that originate and consume it, as well as Data Quality requirements that make it fit to support these processes. 2. Managing data ownership at the object level. Defining data owners for the whole Customer or whole Product results in loss of necessary level of detail that would support the understanding of their different aspects. In addition, it creates a Data Governance organization that is unnecessarily complex, and introduces barriers to adoption of ownership by the business. After all, I would not want to be accountable for something I only understand a part of. Would you? How to prevent: -> replace Data Ownership with Knowledge Ownership. Each Business Expert will represent their knowledge - including the knowledge of the information they use and data it translates to - towards the rest of the organization. Each Expert can then fully own the data supporting their process - and the quality requirements and annotations thereof. Understanding the business purpose of data is important. Making this metadata available to data consumers will prevent Subjective Data Quality escalations. Yet, it will not remove the mis-understandings entirely. In the next episodes, we will dive into the topic of "good data that seems bad because it is misunderstood", exploring Semantic Data Quality . Stay tuned! #Databout #DataQuality #DataManagement #Hakoona Andreas Ingvar Õismaa

9

Wow, the next-generation voice of Chat GPT-4o sounds strikingly similar to Scarlett Johansson in the movie 'Her' (2013)—a must-see if you haven't watched it yet! Fasten your seatbelts for the new features of GPT-4o 🚀 * Desktop Version & UI Refresh: The launch of a desktop application for the CHBT platform, along with a refreshed user interface, aims to simplify and enhance user accessibility and experience. * Launch of GPT 4o: The new model is now available to all users, including free users, and promises faster performance, reduced latency, and more natural interactions through voice communication. * Enhanced Accessibility: Significant strides have been made to make the technology more accessible, including removing the signup barrier and introducing real-time conversational abilities and multi-modal interactions. * Interactive Capabilities: You can now interact in text, audio (with a voice assistant), and video. Check out some powerful demos on YouTube, such as real-time translation, AI talking to AI, meetings with GPT, and language learning. View the playlist here: https://lnkd.in/e5tQ59S4 * Benefits of Plus: Enjoy 5x higher limits and the earliest access to features like our new macOS desktop app and next-generation voice and video capabilities. These features will be rolled out iteratively in the coming weeks, ensuring users can leverage the full potential of the updates. #OpenAI #LLM #Multimodal #Microsoft #SpringUpdate #GPT4o

16

2 Comments

You know a Semantic Layer would be hugely valuable, but how do you actually build such a thing? Pro tip: Crafting a Semantic Layer is about building iterative velocity alongside accuracy, so that when your stakeholders ask about Revenue MoM grouped by Attribution Channel, you can answer instead of adding a ticket to the backlog. Start with these four steps: 1. Identify a Data Product that is impactful: Find something that is in heavy use and high value, but fairly narrow scope. Don’t start with a broad executive dashboard that shows metrics from across the company because you’re looking to optimize for migrating the smallest amount of modeling for the highest amount of impact that you can. For example, a good starting place would be a dashboard focused on Customer Acquisition Cost (CAC) that relies on a narrow set of metrics and underlying tables that are nonetheless critical for your company. 2. Catalog the models and their columns that service the Data Product, both in dbt and the BI tool, including rollups, metrics tables, and marts that support those. Pay special attention to aggregations as these will constitute metrics. 3. Melt the frozen rollups in your dbt project, as well as variations modeled in your BI tool, into Semantic Layer code. 4. Create a parallel version of your data product that points to Semantic Layer artifacts, audit, and then publish. Creating in parallel takes the pressure off, allowing you to fix any issues and publish gracefully. You’ll keep the existing Data Product as-is while swapping the clone to be supplied with data from the Semantic Layer. Dig deeper into the step-by-step process of how to ship a Semantic Layer in pieces at our link in the comments.

371

25 Comments

Data quality is paramount. Garbage in = garbage out. Before surfing on the wave of AI, re-think your data strategy and ensure it is robust. #AI #datastrategy #projectmanagement

2

Are you looking for a unified platform that is safe for developing and deploying large language models (LLMs) and machine learning (ML) models? The Snowflake platform efficiently processes data at scale using fine-tuned models or foundation models such as Snowflake Arctic, Meta Llama 3, Reka Core and more. Users can: ❄ Parallelly run LLMs and ML models next to your data ❄ Enable easy usage of AI by diverse teams ❄ Secure their data with industry-accepted compliance standards ❄ Ensure systems are robust and adaptable through role-based access control (RBAC) Build a document chatbot, extract value from both documents (unstructured) and tabular data (structured), convert data and models into interactive apps, and much more. Leverage the Snowflake platform to accelerate your business operations with a suite of features and services for data and AI. Learn more about the benefits of AI and ML in Snowflake here: https://lnkd.in/gEt3gsR3 At Novigo Solutions, we are committed to delivering top-tier technology services. Reach out to us and experience an innovative approach to tailoring business needs. Mohammed Jarood Arsalan Shakeel Nizar Abdulhadi Yogesh Patil Varun Nagarajan Ankita Avadhani Mohammed Abulhaija   #snowflake #ai #machinelearning #llm #datascience #dataengineering #novigosolutions

42

1 Comment

ABAC is a common use case on Databricks, but how do you automate it for scale? Predefined masking rules, combined with automated data classification and tagging, make this task way easier. David Levy Nahum and Aviv David's new blog goes through exactly how the process works in Satori: https://lnkd.in/de8VWCWU

14

🎧 Listen to our very own Johan Baltzar guesting on the Rittman Analytics podcast, discussing why self-service BI tools, in most cases, don't live up to their promise and why Figma is the perfect inspiration for changing the way companies work with data.

10

💡 Data is everywhere. But without the right overview, it can be challenging to make well-informed decisions based on it. The numbers are there, but truly understanding them at a glance can be a real challenge. That's where a dashboard comes in.   In our latest blog, Mats Hermanrud, Data Processing Manager at Norstat, explains how dashboards can make your data more comprehensible, increase its value, and simplify decision-making.   For a more in-depth look into our approach to dashboards for market research and insight, read the full blog here: https://lnkd.in/dU9jV2-A   Reliable data and intuitive dashboards can transform your decision-making. If you'd like more information about how our dashboards can bring your data to life, please get in touch with Mats Hermanrud.   Norstat: Data to trust – for decisions that matter ⭐ #marketresearch

85

1 Comment

🤔 What is analytics as code and why is it necessary in today’s analytics and BI landscape? In part one of our series "Don't Let Your Pipeline Die at BI: Understanding Analytics as Code", GoodData VP of Product Strategy speaks with industry expert, Donald Farmer. In this video, hear about why analytics has lagged behind in the collaboration revolution and how the analytics as code approach makes advanced analytics more accessible—even for non-coders. With topics like generative AI and self-service analytics, this conversation highlights the future of data collaboration. Check it out: https://hubs.ly/Q0300PSm0

4