From Information to Locations: How Skyscanner Optimizes Traveler Experiences with Databricks Unity Catalog

At Skyscanner, we’re greater than only a flight search engine. We’re a world chief in journey in serving greater than 110 million customers every month to plan and guide their journeys with confidence and ease. Working in over 30 languages, our platform connects vacationers with a variety of flights, inns, and automotive rental choices from over 1,200 journey companions throughout 180 nations.  

We use information and AI to boost the traveler expertise in addition to assist inside decision-making. For our vacationers, we use machine studying (ML) fashions to verify over 80 billion costs each day, rating and recommending inns, flights, and automotive leases, aiming to offer one of the best choices primarily based on journey time and prices. Databricks Information Intelligence Platform powers a few of these journey insights. On this weblog, we talk about our journey with Databricks and the way Unity Catalog helped us streamline our information administration and governance.

To be taught extra, attend the Information + AI Summit 2024 for our session titled  Skyscanner’s Journey of Enabling Sensible Information and AI Governance. 

Understanding Our Information Panorama and Challenges

Information has at all times been central to Skyscanner’s operations. Every single day, our platform handles 35 million searches, producing over 30 to 35 billion analytical occasions. The sheer quantity of knowledge—roughly 15 to twenty petabytes saved at any given time—poses vital challenges in information administration and utilization.  Our information is essential for each consumer-facing options and inside decision-making processes, making its efficient administration a high precedence for our engineering groups. This scale of knowledge operations presents a number of challenges:

• Quantity and Velocity: Dealing with billions of occasions generated day by day requires sturdy infrastructure and environment friendly information processing capabilities.
• Scalability and Efficiency Points: As Skyscanner grew, the information infrastructure struggled to maintain tempo with the growing demand. Our legacy techniques couldn’t scale effectively, resulting in delays in information processing and an lack of ability to deal with large-scale information workloads successfully.
• Complexity and Value: Earlier than transitioning to extra streamlined options, our information administration concerned a number of techniques, which frequently led to inefficiencies and elevated operational prices.
• Information Silos and Inconsistency: The disparate techniques led to information being siloed, which hindered information accessibility and high quality, affecting decision-making processes.
• Compliance and Safety Dangers: With information unfold throughout varied techniques, guaranteeing complete safety and compliance with worldwide information safety rules (like GDPR) was more and more difficult. This danger was compounded by the shortage of centralized management over information entry and processing.

Databricks: A Sport-Changer for Skyscanner

At Skyscanner, our dedication to leveraging cutting-edge know-how is clear in our strategic partnership with Databricks. Databricks has been instrumental in reworking our method to information administration, enabling us to streamline operations and improve the traveler expertise. 

All our information pipelines are constructed on high of the Databricks Information Intelligence Platform. we have established a sturdy information ingestion framework that captures information from a wide range of sources, incorporating each batch and real-time streams. We make the most of AWS Kinesis for streaming and Fivetran for batch information ingestion, guaranteeing that every one incoming information is collected effectively into our preliminary staging space, which we confer with because the ‘bronze layer’ of our medallion structure. This stage is essential because it handles the uncooked information collected from our numerous channels, together with direct interactions from our internet and cell platforms.

As soon as within the bronze layer, the information undergoes a sequence of transformations and enrichments to arrange it for deeper analytical duties. It then strikes to the ‘silver layer,’ the place it’s cleaned, consolidated, and structured, prepared for analytical consumption. On this part, Databricks’ highly effective Spark engine performs a vital position, enabling quick and scalable information transformations.

Advancing the information to the ‘gold layer,’ our information is optimized for consumption by varied enterprise items the place it’s modeled and aggregated into metrics that immediately assist decision-making throughout the corporate. We leverage MLflow, to handle the entire machine studying lifecycle. This consists of every thing from experimentation and reproducibility to the deployment of ML fashions, permitting us to trace experiments, package deal code into reproducible runs, and deploy fashions immediately into manufacturing seamlessly. Whereas we’re at present serving these fashions into manufacturing utilizing our personal model-serving structure, we’re within the strategy of evaluating Databricks’ model-serving capabilities which are a part of the Databricks Mosaic AI providing.

Past processing and machine studying, we make the most of Databricks for operational reporting and analytics. Databricks SQL permits our groups to carry out SQL queries immediately in opposition to our information lake, create dashboards, and execute complicated analytical operations at scale. Integration with BI instruments like Tableau Cloud enhances our capabilities, enabling us to visualise information and extract actionable insights effectively.

Our Migration Journey to Unity Catalog

Information governance is a important element of Skyscanner’s structure. It underpins our potential to handle information securely and effectively, guaranteeing that we will belief our information for making enterprise selections and sustaining compliance with international information safety rules, together with GDPR. As a subsidiary of an organization listed on NASDAQ, adhering to strict regulatory requirements such because the Sarbanes-Oxley Act is paramount for guaranteeing transparency and accountability in our operations. Databricks Unity Catalog, being constructed into the platform, helped us streamline these necessities.

Earlier than implementing Unity Catalog, we confronted a number of vital challenges

• Low Ranges of Information Possession: One of many extra vital challenges we confronted was the low degree of possession over datasets throughout the corporate. This typically led to accountability points, the place no particular crew or particular person was chargeable for the accuracy, privateness, and safety of explicit datasets.
• Lack of Centralized Oversight: Managing information throughout disparate techniques made it tough to implement constant information governance insurance policies. This lack of centralized management led to inefficiencies and elevated the danger of non-compliance with information rules akin to GDPR.
• Entry Management Difficulties: And not using a unified system, managing who had entry to what information was cumbersome and sometimes insecure. Dealing with IAM insurance policies was notably difficult, requiring substantial guide effort and being liable to errors. Guaranteeing the suitable degree of entry for varied groups concerned navigating complicated IAM roles, which frequently led to both overly permissive entry or overly restrictive practices, each of which may impede operational effectivity.
• Insufficient Information Lineage and Auditing: We lacked automated instruments for monitoring information lineage and auditing adjustments, that are important for troubleshooting and understanding the influence of knowledge modifications. In consequence, lineage graphs needed to be ready manually.

Recognizing these challenges, we developed a strategic method emigrate to Unity Catalog. Our technique included:

• Prioritizing Enterprise-Important Tables: We performed a complete evaluation of all information belongings to categorise them in keeping with their significance to enterprise operations, sensitivity, and compliance necessities. Though we had 30,000 tables in whole, our energetic tables numbered solely about 1,500, and of these, solely about 350 have been business-critical. That discovery was a recreation changer for us as this simplified our migration course of.
• Leveraging Automation: Initially, our groups manually migrated tables into Unity Catalog and tailored them to suit our area mannequin, which was a gradual and time-consuming course of. By leveraging Databricks’ automation instruments, we considerably accelerated the migration without having to rewrite our pipelines. To expedite the combination of all our information into Unity Catalog, we turned much less inflexible about adhering strictly to the Medallion structure, which requires all information to be categorised into bronze, silver, and gold layers. As an alternative, we adopted a extra versatile method: “We’ll meet you the place your information is.” This technique allowed us to make information seen within the Unity Catalog instantly, with the intention of aligning it with the bronze, silver, and gold definitions over time.

Enhancing information visibility and governance with Unity Catalog

Unity Catalog has develop into a pivotal ingredient in our information governance framework at Skyscanner. it now manages and governs a major quantity, roughly 15 to twenty petabytes, of our information. This information consists of every thing from uncooked information in our ‘bronze’ layer to processed information in our ‘silver’ and ‘gold’ layers, that are used extensively throughout varied enterprise capabilities for analytical and operational functions. 

The implementation of Unity Catalog has introduced substantial enhancements to our information administration and governance capabilities, yielding a number of key advantages:

• Enhanced Information Safety and Compliance: Unity Catalog has enabled us to centralize our information governance, offering sturdy safety features and streamlined compliance processes. This centralization lowered the complexities related to managing permissions throughout disparate techniques and helped be sure that solely licensed personnel had entry to delicate information and is essential for adhering to stringent information safety legal guidelines, together with GDPR. 
• Value Optimization: The streamlined information administration course of enabled by Unity Catalog has led to extra environment friendly use of our information storage and computing assets.
• Scalability and Future-Proofing: Unity Catalog has supplied a scalable structure that accommodates our rising information wants. As Skyscanner continues to broaden and evolve, Unity Catalog helps this development by enabling us to handle growing volumes of knowledge with out compromising on efficiency or safety.
• Enhanced Information Lineage:  With Unity Catalog, we have considerably enhanced our information lineage capabilities. This implies we now have a transparent and detailed view of the place our information originates, the way it’s processed alongside the way in which, and the place it finally ends up. This degree of transparency is essential not only for day-to-day operations but additionally for our compliance efforts, notably with GDPR. With the ability to hint your entire journey of our information helps us be sure that we’re dealing with it appropriately and staying compliant with all needed rules. It additionally simplifies the audit course of, as we will readily present detailed mappings of our information flows.
• Information Observability:  Constructing on our information in Unity Catalog, we now have built-in Monte Carlo to enhance information reliability throughout our energetic datasets. We’ve got launched a wholesome information framework in order that we will measure the adoption of knowledge governance throughout Skyscanner. 

Planning for the long run: Capitalizing on new alternatives 

As we glance forward, I feel the worth in generative AI will come from the distinctive, worthwhile information we now have at Skyscanner. There’s loads of potential, however a key step for us is ensuring we now have every thing, together with ML fashions, managed and ruled with Unity Catalog to capitalize on any alternatives.

At the moment we’re evaluating utilizing Databricks’ Mannequin Serving functionality. We’re taking a look at enabling Unity Catalog in a number of areas utilizing Delta Sharing to maneuver information between areas. We’re additionally occupied with utilizing this for exterior information sharing – we now have some information merchandise the place we share information with third occasion firms.

Sooner or later, we would like our information groups to give attention to issues distinctive to Skyscanner. Databricks does loads of the heavy lifting on the subject of mannequin serving and offers a superb framework for occupied with the AI journey—from immediate engineering to constructing your individual mannequin. We’ve got confidence in our potential to appreciate the alternatives we’re figuring out utilizing the Databricks ecosystem.

Study extra about Skyscanner’s journey on the Information + AI  2024 Summit by becoming a member of Michael’s session, Skyscanner’s Journey of Enabling Sensible Information and AI Governance.