Blog

Protecting sensitive data and preventing unintended data exposure is critical for businesses. However, many organizations lack the tools to stay on top of where sensitive data resides across their enterprise. It’s particularly concerning when sensitive data shows up in unexpected places – for example, in logs that  services generate, when customers inadvertently send it in a customer support chat, or when managing unstructured analytical workloads. This is where Automatic Data Loss Prevention (DLP) for BigQuery can help.

Data discovery and classification is often implemented as a manual, on-demand process, and as a result  happens less frequently than many organizations would like. With a large amount of data being created on the fly, a more modern, proactive approach is to build discovery and classification into existing data analytics tools. By making it automatic, you can ensure that a key way to surface risk happens continuously – an example of Google Cloud’s invisible security strategy. Automatic DLP is a fully-managed service that continuously scans data across your entire organization to give you general awareness of what data you have, and specific visibility into where sensitive data is stored and processed. This awareness is a critical first step in protecting and governing your data and acts as a key control to help improve your security, privacy, and compliance posture.

In October of last year, we announced the public preview for Automatic DLP for BigQuery. Since the announcement, our customers have already scanned and processed both structured and unstructured BigQuery data at multi-petabyte scale to identify where sensitive data resides and gain visibility into their data risk. That’s why we are happy to announce that Automatic DLP is now Generally Available. As part of the release we’ve also added several new features to make it even easier to understand your data and to make use of the insights in more Cloud workflows. These features include:

Premade Data Studio dashboards to give you more advanced summary, reporting, and investigation tools that you can customize to your business needs.

Finer grained controls to adjust frequency and conditions for when data is profiled or reprofiled, including the ability to enable certain subsets of your data to be scanned more frequently, less frequently, or skipped from profiling.

Automatic sync of DLP profiler insights and risk scores for each table into Chronicle, our Security Analytics platform. We aim to build synergy across our security portfolio, and with this integration we allow analysts using Chronicle to gain immediate insight into if the BQ data involved in a potential incident is of high value or not. This can significantly help to enhance threat detections, prioritizations, and security investigations. For example, if Chronicle detects several attacks, knowing if one is targeting highly sensitive data will help you prioritize, investigate, and remediate the most urgent threats first.

Managing data risk with data classification

Examples of sensitive data elements that typically need special attention are credit cards, medical information, Social Security numbers, government issued IDs, addresses, full names, and account credentials. Automatic DLP leverages machine learning and provides more than 150 predefined detectors to help discover, classify, and govern this sensitive data, allowing you to make sure the right protections are in place. 

Once you have visibility into your sensitive data, there are many options to help remediate issues or reduce your overall data risk. For example, you can use IAM to restrict access to datasets or tables or leverage BigQuery Policy Tags to set fine-grained access policies at the column level. Our Cloud DLP platform also provides a set of tools to run on-demand deep and exhaustive inspections of data or can help you obfuscate, mask, or tokenize data to reduce overall data risk. This capability is particularly important if you’re using data for analytics and machine learning, since that sensitive data must be handled appropriately to ensure your users’ privacy and compliance with privacy regulations.

How to get started

Automatic DLP can be turned on for your entire organization, selected organization folders, or individual projects. To learn more about these new capabilities or to get started today, open the Cloud DLP page in the Cloud Console and check out our documentation.

IT leaders pick different clouds for many reasons, but the rest of the company shouldn’t be left to navigate the complexity of those decisions.  For data analysts, that complexity is most immediately felt when navigating between data silos. Google Cloud has invested deeply in helping customers break down these barriers inherent in a disparate data stack.  Back in October 2021, we launched BigQuery Omni to help data analysts access and query data across the barriers of multi cloud environments. We are continuing to double down in cross-cloud analytics: a seamless approach to view, combine, and analyze data across-clouds with a single pane of glass.  

Earlier this year, one of BigQuery Omni’s early adopters, L’Oreal, discussed the merits of a cross-cloud analytics to maximize their data platform.  We know that enterprises need to analyze data without needing to move or copy any data.  We also know that enterprises sometimes need to move small amounts of data between clouds to leverage unique cloud capabilities.  A full cross-cloud analytics solution offers the best of both worlds: analyzing data where it is and flexibility to replicate data when necessary. 

Last week, we launched BigQuery Omni cross-cloud transfer to help customers with combining data across clouds.  From a single-pane-of-glass, data analysts, scientists, and engineers, can load data from AWS and Azure to BigQuery without any data pipelines. Because it is all managed in SQL, it is accessible among all levels of an organization.  We have designed this feature to provide three core benefits:

Usability: With one single-pane-of-glass, users tell BigQuery to filter and move data between clouds without any context-switching

Security: With a federated identity model, users don’t have to share or store credentials between cloud providers to access and copy their data

Latency: With data movement managed by BigQuery’s high-performance storage API, users can effortlessly move just the relevant data without having to wait for complex pipes

A core use case that we have heard from customers is to combine point of sales (PoS) data from AWS/Azure with Google Analytics data and create a consolidated purchase prediction model. Here’s a demo of that:

As you saw in the demo, a data analyst can drive end-to-end workflows across clouds. They can transform data using BigQuery Omni, they can load data using cross-cloud transfer, and they can train an ML model all in SQL. This empowers them to drive real business impact by providing the ability to:  

Improve training data by de-deuplicating users across datasets

Improve accuracy of marketing segmentation models

Improve Return on Ads Spend and save potentially millions for enterprise campaigns

But we’re not stopping there, we will continue to build upon this experience by providing more BigQuery native tools for our customers to assist with smart data movement.  Over time, our cross-cloud data movement will be built on pipeless pipelines:  A cross-cloud lakehouse without the fuss. 

Get involved with the preview and start participating in our development process by submitting this short form.

For 117 years, the U.S. Department of Agriculture’s Forest Service has been a steward of America’s forests, grasslands, and waterways. It directly manages 193 million acres and supports sustainable management on a total of 500 million acres of private, state, and tribal lands. Its impact reaches far beyond even that, offering its research and learning freely to the world.

At Google, we’re big admirers of the Forest Service’s mission. So we were thrilled to learn in 2011 that its scientists were using Google Earth Engine, our planetary-scale platform for Earth Science data and analysis, to aid its research, understanding, and effectiveness. In the years since, Google has worked with the Forest Service to meet its unique requirements for visual information about the planet. Using both historical and current data, the Forest Service built new products, workflows, and tools that help more effectively and sustainably manage our natural resources. The Forest Service also uses Earth Engine and Google Cloud to study the effects of climate change, forest fires, insects and disease, helping them create new insights and strategies.

Besides gaining newfound depths of insight, the Forest Service has also sped up its research dramatically, enabling everyone to do more. Using Google Cloud and Earth Engine, the Forest Service reduced the time it took to analyze 10 years worth of land-cover changes from three months to just one hour, using just 100 lines of code. The agency built new models for coping with change, then mapped these changes over time, in its Landscape Change Monitoring System (LCMS) project. 

Emergency responders can now work better on new threats that arise after wildfire, hurricanes, and other natural disasters. Forest health specialists can detect and monitor the impacts of invasive insects, diseases, and drought. More Forest Service personnel can use new tools and products within Earth Engine, thanks to numerous training and outreach sessions within the Forest Service.

Researchers elsewhere also benefited when the Forest Service created new toolkits, and posted them to GitHub for public use. For example, there’s geeViz, a repository of Google Earth Engine Python code modules useful for general data processing, analysis, and visualization. 

This is only the start. Recently, the Forest Service started using Google Cloud’s processing and analysis tools for projects like California’s Wildfire and Forest Resilience Action Plan. Forest Service researchers also use Google Cloud to better understand ecological conditions across landscapes in projects like Fuelcast, which provides actionable intelligence for rangeland managers, fire specialists, and growers, and the Scenario Investment Planning Platform for modeling local and national land management scenarios.

The Forest Service is a pioneer in building technology to help us better understand and care for our planet. With more frequent imaging, rich satellite data sets, and sophisticated database and computation systems, we can view and model the Earth as a large-scale dynamic system. 

We are honored and excited to respond to the unique set of requirements of the scientists, engineers, rangers, and firefighters of the USFS, and look forward to years of learning about — and better caring for — our most precious resources. 

*Image 1: The USDA Forest Service (USFS) Geospatial Technology and Applications Center (GTAC) uses science-based remote sensing methods to characterize vegetation and soil condition after wildland fire events. The results are used to facilitate emergency assessments to support hazard mitigation, to inform post-fire restoration planning, and to support the monitoring of national fire policy effectiveness. GTAC currently conducts these mapping efforts using long-established geospatial workflows. However, GTAC has adapted its post-fire mapping and assessment workflows to work within Google Earth Engine (GEE) to accommodate the needs of other users in the USFS. The spatially and temporally comprehensive coverage of moderate resolution multispectral data sources (e.g., Landsat, Sentinel 2) and analytical power provided by GEE allows users to create geospatial burn severity products quickly and easily. Box 1 shows a pre-fire Sentinel-2 false color composite image. Box 2 shows a post-fire Sentinel-2 false color composite image with the fire scar apparent in reddish brown. Box 3 shows a differenced Normalized Burn Ratio (dNBR) image showing the change between the pre- and post-fire images in Boxes 1 and 2. Box 4 shows a thresholded dNBR image of the burned area with four classes of burn severity (unburned to high severity), which is the final output delivered to forest managers.

*Image 2: Leveraging Google Earth Engine (GEE), the USDA Forest Service (USFS) Geospatial Technology and Applications Center (GTAC) and USFS Region 8, developed the Tree Structure Damage Impact Predictive (TreeS-DIP) modeling approach to predict wind damage to trees resulting from large hurricane events and produce spatial products across the landscape. TreeS-DIP results become available within 48 hours following landfall of a large storm event to allow allocation of ground resources to the field for strategic planning and management. Boxes 1 and 3 above show TreeS-DIP modeled outputs with varying data inputs and parameters. Box 2 shows changes in greenness (Normalized Burn Ratio; NBR) that was measured with GEE during the recovery from Hurricane Ida and is shown as a visual comparison to the rapidly available products from TreeS-DIP.

*Image 3: Severe drought conditions across the American West prompted concern about the health and status of pinyon-juniper woodlands, a vast and unique ecosystem. In a cooperative project between the USDA Forest Service (USFS) Geospatial Technology and Applications Center (GTAC) and Forest Health Protection (FHP), Google Earth Engine (GEE) was used to map pinyon pine and juniper mortality across 10 Western US States. The outputs are now being used to plan for future work including on-the-ground efforts, high-resolution imagery acquisitions, aerial surveys, in-depth mortality modeling, and planning for 2022 field season work.

Box 1 contains remote sensing change detection outputs (in white) generated with GEE, showing pinyon-juniper decline across the Southwestern US. Box 2 shows NAIP imagery from 2017 with, with box 3 showing NAIP imagery from 2021. NAIP imagery from these years shows trees changing from healthy and green in 2017 to brown and dying in 2021. In addition, box 2 and box 3 show change detection outputs from Box 1 for a location outside of Flagstaff, AZ converted to polygons (in white). The polygon in box 2 is displayed as a dashed line to serve as a reference, while the solid line in box 3 shows the measured change in 2021. Converting rasters to polygons allows the data to be easily used on tablet computers, as well as the ability to add information and photographs from field visits.

To compete in a fast moving, transforming, and increasingly digital world, every team, business, process and individual needs to level up the way they think about data. Which is why this year at our Data Cloud Summit 2022, we saw record turnout, both in volume and diversity of attendance. Our thanks go out to all the customers, partners, and data community who made it such a great success!

Did you miss out on the live sessions? Not to worry – all the content is now available on demand. 

Here are the five biggest areas to catch up on from Data Cloud Summit 2022:

#1: Product announcements to level up your data skills

Data is no longer solely the realm of the analyst. Every team, customer and partner needs to be able to interact with the data they need to achieve their goals. To help them do so, we announced 15 new products, capabilities and initiatives that help remove limits for our users. Here are some highlights:

BigLake allows companies to unify data warehouses and lakes to analyze data without worrying about the underlying storage format or system. 
Spanner change streams tracks Spanner inserts, updates, deletes, and streams the changes in real-time across the entire Spanner database so that users will always have access to the latest data. 
Cloud SQL Insights for MySQL helps developers quickly understand and resolve database performance issues for MySQL
Vertex AI Workbench delivers a single interface for data and ML systems. 

Connected Sheets for Looker and the ability to access Looker data models within Data Studio combine the best of both worlds of BI, giving you centralized, governed reporting where you need it, without inhibiting open-ended exploration and analysis.

More product news announced at Data Cloud Summit can be found here.

#2: Customers to learn from

Customers are at the heart of everything we do, and that was evident at the Data Cloud Summit. Wayfair, Walmart, Vodafone, ING Group, Forbes, Mayo Clinic, Deutsche Bank, Exabeam and PayPal all spoke about their use of Google’s Data Cloud to accelerate data-driven transformation. Check out some of their sessions to learn more:

Unify your data for limitless innovation, featuring Wayfair and Vodafone

Unlocking innovation with limitless data, featuring Exabeam

Spotlight: Database strategy and product roadmap, featuring Paypal

We also heard from you directly! Here are some great quotes from the post-event survey:

“This is the first time that I have been exposed to some of these products. I am a Google Analytics, Data Studio, Search Console, Ads and YouTube customer…so this is all very interesting to me. I’m excited to learn about BigQuery and try it out.”

“The speakers are very knowledgeable, but I appreciate the diversity in speakers at these cloud insights.”

“Great experience because of the content and the way that it is presented.”

“This is definitely useful to new Google Admin Managers like I am.”

“This was a great overview of everything new in such a short time!”

#3: Partners to deliver the best customer experiences

Our partner ecosystem is critical to delivering the best experience possible for our customers. With more than 700 partners powering their applications using Google Cloud, we are continuously investing in the ecosystem. At Data Cloud Summit, we announced a new Data Cloud Alliance, along with the founding partners Accenture, Confluent, Databricks, Dataiku, Deloitte, Elastic, Fivetran, MongoDB, Neo4j, Redis, and Starburst, to make data more portable and accessible across disparate business systems, platforms, and environments—with a goal of ensuring that access to data is never a barrier to digital transformation. In addition, we announced a new Database Migration Program to accelerate your move to managed database services. Many of these partners delivered sessions of their own at Data Cloud Summit 2022:

Accelerate Enterprise AI adoption by 25-100x, featuring C3 AI

Rise of the Data Lakehouse in Google Cloud, featuring Databricks

The Connected Consumer Experience in Healthcare and Retail, featuring Deloitte

Investigate and prevent application exploits with the Elasticsearch platform on Google Cloud

#4: Product demos to elevate your product knowledge

Experts from Google Cloud delivered demos giving a hands-on look at a few of the latest innovations in Google’s Data Cloud:

Cross-cloud analytics and visualization with BigQuery Omni and Looker, with Maire Newton and Vidya Shanmugam

Build interactive applications that delight customers with Google’s data cloud, with Leigha Jarett and Gabe Weiss

Build a data mesh on Google Cloud with Dataplex, with Prajakta Damie and Diptiman Raichaudhuri

Additional demos are available here on-demand.

#5: Resources to dig into

If you want to go even deeper than the Summit sessions themselves, we’ve put together a great list of resources and videos of on-demand contentto help you apply these innovations in your own organization. Here are some of the highlights:

Guide to Google Cloud Databases (PDF)

How does Pokémon Go scale to millions of requests? (video)

MLOps in BigQuery ML using Vertex AI (video)

Database Engineer Learning Path (course)

Machine Learning Engineer Learning Path (course)

BI and Analytics with Looker (course)

Thanks again for joining us at this year’s Data Cloud Summit. Join us again at Applied ML Summit June 9, 2022!

Without a central place to manage models, those responsible for operationalizing ML models have no way of knowing the overall status of trained models and data. This lack of manageability can impact the review and release process of models into production, which often requires offline reviews with many stakeholders. 

Earlier this week we announced Vertex AI Model Registry, a central repository to manage and govern the lifecycle of your ML models. Model Registry organizes your model artifacts by version, making it easy for data scientists to share models and application developers to deploy them. It’s designed to work with any type of model and deployment target, whether that’s through BigQuery, Vertex AI, custom deployments on GCP or even out of the cloud. 

In this blog, we’ll dive into how Model Registry works with BigQuery ML, showcasing the features that allow you to register, version, and easily deploy your BigQuery ML Models to Vertex AI: 

Registering BigQuery ML models with Vertex AI Model Registry

1. With Vertex AI Model Registry, you can now see and manage all your ML models (AutoML, custom-trained, and BigQuery ML) in the same place
2. You can register BigQuery ML models to Vertex AI Model Registry when creating your model using SQL

Model versioning with Vertex AI Model Registry

3. Model versioning is now available on Vertex AI Model Registry, including for BigQuery ML models

Easier deployment of BigQuery ML models to Vertex AI endpoints

4. From Vertex AI Model Registry, you can deploy BigQuery ML models to Vertex endpoints directly 

Let’s dive deeper into each of these new and exciting capabilities.

Registering models with Vertex AI Model Registry

View and manage all your ML models in the one place

You can now see all your ML models within Vertex AI Model Registry, making it easier for your organization to manage and deploy models. This includes models built with BigQuery ML, AutoML, and custom trained models.

Registering BigQuery ML models to Vertex AI Model Registry

Let’s go over some common questions you might have:

How do you register a BigQuery ML to Vertex AI Model Registry?

Using the CREATE MODEL syntax, now you can add in an optional model_registry=”vertex_ai” field to register the model to Model Registry when the model has finished training. You can also specify a Vertex AI model ID to register to, otherwise it will register it as a new model in Model Registry using the BigQuery ML model id. You can also specify any custom tags to help you label your model, such as “staging”, “production”.

Here’s an example of using CREATE MODEL with model_registry=’vertex_ai’:

Full documentation here: Managing models with Vertex AI | BigQuery ML | Google Cloud

Note: If you see an error indicating Access Denied: BigQuery BigQuery: Permission ‘aiplatform.models.upload’ denied on resource, you may first need to follow the instructions here to set the correct permissions. This is temporary. In a future release, you won’t need to explicitly set these permissions before registering BigQuery ML models with Vertex AI Model Registry. 

After training is complete, the BigQuery ML model (my_penguins_model) now shows up in Vertex AI Model Registry:

Clicking on the model lets me inspect the model with more details, including the model version and aliases:

You might have a few questions at this point:

Do all BigQuery ML models get automatically registered to Vertex AI Model Registry? 

No, BigQuery ML models do not get automatically registered to Model Registry unless the user wants them to. As data scientists iterate and experiment through different models, they might want to only register a subset of models to the Model Registry. So users of BigQuery ML can pick and choose which models they explicitly want to register to the Vertex AI Model Registry using model_registry=”vertex_ai” in the CREATE MODEL query. All models created using BigQuery ML will still be viewable within BigQuery, regardless of whether or not they have been registered to Vertex AI Model Registry.

Can any BigQuery ML model be registered to Vertex AI Model Registry?

Not all of them, currently. BigQuery ML has many supported model types, and built-in models as well as imported TensorFlow models can be registered to the Vertex AI Model Registry (full documentation). 

Can you delete BigQuery ML models directly from Vertex AI Model Registry?

Currently, no you cannot. The only way to delete BigQuery ML models is from BigQuery ML. If you delete a BigQuery ML model, it will automatically be removed from Vertex AI Model Registry. More information on deleting BigQuery ML models can be found in the documentation.

Model versioning with Vertex AI Model Registry

Model versioning is now available on Vertex AI Model Registry, including for BigQuery ML

Users can now keep track of model versions on Vertex AI Model Registry, including BigQuery ML models. Model versioning allows you to create multiple versions of the same model. With model versioning, you can organize your models in a way that helps you navigate and understand which changes had what effect on the models. With Vertex AI Model Registry you can view your models and all of their versions in a single view.

So when you register an initial BigQuery ML model to Model Registry, and then register a second version to the same model_id, you will see two versions on Model Registry.

For example, after training the initial model my_penguins_model, you can train another model version and register it to Vertex AI Model Registry, using the same vertex_ai_model_id, and adding any aliases you’d like:

Looking at the model details in the Vertex AI Model Registry allows me to see a new version of the model:

Easier deployment of BigQuery ML models to Vertex AI endpoints

Why might you consider deploying BigQuery ML models to a Vertex AI endpoint? Today, BigQuery ML is great for batch predictions on large datasets. However, BigQuery ML is unsuitable for situations requiring online predictions, which typically involve low-latency and high-query-per-second inference. In other situations, sometimes data scientists and ML engineers may prefer to use a REST endpoint to serve predictions, rather than use SQL queries for model inference. To solve for either scenario, users can now more easily deploy their BigQuery ML models to a Vertex AI endpoint.

Deploy BigQuery ML models to Vertex endpoints directly from Vertex AI Model Registry

Once a BigQuery ML model is registered on Vertex AI Model Registry, you can now easily deploy the model to an endpoint in just a few clicks from the Model Registry interface. 

You can select to “Deploy to endpoint”:

Then you can select a name and compute resources to use for your Vertex endpoint:

Make an online prediction request to the Vertex endpoint

With a BQML model successfully deployed to an endpoint, you can now make online prediction requests. You’ll need to make sure your prediction request is following the correct input format. Here’s an example of what a prediction request (with new test data) as a JSON file might look like:

prediction_request.json

Then, you can make an online prediction request (documentation):

Conclusion

With these new integrations between BigQuery ML and Vertex AI Model Registry, you will be able to keep track of your models, version your models, and deploy with greater ease than before. Happy modeling!

Want to learn more?

Learn more about Vertex AI Model Registry

Learn more about BigQuery ML with Vertex AI Model Registry

Learn more about BigQuery ML and try out a tutorial

Learn more about Vertex AI and deploying private endpoints or traffic splitting

Read about using BigQuery and BigQuery ML operators in a Vertex AI Pipeline

Special thanks to Abhinav Khushraj, Henry Tappen, Ivan Nardini, Shana Matthews, Sarah Dugan, Katie O’Leary for their contributions to this blogpost.

At Google Cloud, we believe that democratizing access to fresh, consistent, enterprise data is the key to driving digital and cultural transformation. This means empowering everyone to leverage rich, governed data sets and harness the power of data — without learning new tools or building specialized skills. This is why we are excited to provide broader access to Looker’s new integration with Google Sheets. Connected Sheets for Looker brings modeled, trusted data into the familiar spreadsheet interface, enabling users to work in a way that is comfortable and convenient. And now, we’re making this integration more broadly available to eligible Looker customers with an Enterprise Workspace license on an opt in basis.

Better Together: Looker + Google Sheets

Looker’s platform offers a unified semantic modeling layer that works within Looker and through Google Sheets to enable collaborative, ad-hoc analysis of cloud data sources. This means that you can centrally define the metrics needed to understand your business, and take advantage of the scale and freshness of data in the cloud, while still empowering users to access that governed, trusted data. And leveraging Looker’s authentication and access mechanisms means data availability doesn’t come at the expense of security.

Connected Sheets for Looker brings connectivity between Google Sheets and  the 60+ data sources available within Looker’s open ecosystem. This integration creates a live connection between Looker and Sheets, so that 1) your data is always up to date, and 2) access is secured based on the user exploring the data. Users can flexibly analyze consistent data from a single source of truth using pivot tables, charts, and formulas, and even integrate other data sources for deeper analysis.

Whether using the Looker UI or accessing data through Google Sheets, everyone throughout your organization can quickly derive consistent, governed insights with no specialized SQL skills required. Exploring data, obtaining valuable insights, and sharing those findings has never been easier.

Several customers have already experienced the benefits of using Connected Sheets for Looker. Mercado Libre, the largest online commerce and payments ecosystem in Latin America, is using this integration as part of their efforts to build an inclusive and impactful data culture across many locations. Jorge Vidaurre, Sr. Digital Analytics Engineer at Mercado Libre says, “All the use cases you can solve with Looker and different Google products like Firebase, BigQuery, Cloud SQL… and now Google Sheets. It blows my mind. We can offer this to someone that doesn’t know anything about data and they will have the same power as analysts and that’s awesome.”

Building a Data Culture

Because Looker’s semantic model standardizes common metrics and definitions for business users, data teams can remove silos and safely democratize access to governed and secure data. Connected Sheets for Looker creates a familiar data access point for users through the Google Sheets interface that has a low barrier for entry into the data ecosystem.  This means putting usable, reliable information into the hands of more people. 

Connected Sheets for Looker aims to make it as easy as possible for everyone in an organization to confidently use data in their day-to-day work. Data is most helpful when people have the ability to derive insights and take action that make it valuable. With Connected Sheets for Looker, more people in your organization can use data to make decisions that drive efficiency, innovation, and growth. This ability to easily access and operationalize data is what fuels digital and cultural transformation. Opt in to confirm your eligibility and start using Connected Sheets for Looker to help drive your own digital transformation.

At Mercado Libre, we are obsessed with unlocking the power and potential of data. One of our key cultural principles is to have a Beta Mindset. This means that we operate in a “state of beta”, constantly asking new questions of our data, experimenting with technologies and iterating our business operations in service of creating the best experiences for our customers.

To provide context about the footprint of our organization; Mercado Libre has 30,000+ employees, across six countries in LATAM, with multiple corporate and thousands of home offices. We serve over 65+ million customers and only last year processed 75+ million payments. Like many of our peers, we are learning to adapt to new challenges of scaling our organization, training programs and business to build the most inclusive and impactful company culture across many locations, with data at its core. 

This is the first of a three part blog series on continuous intelligence, cutting edge technology and data culture, in which we reveal some of our capabilities to win ecommerce and become the fintech leaders.

Continuous Intelligence – Building the Foundation

The foundation to our success is our continuous intelligence ecosystem. These are the systems and interfaces our analytics team has put in place to consistently serve many of the needs of our users at scale. 

According to Gartner: 

“Continuous intelligence is a design pattern in which real-time analytics are integrated into business operations, processing current and historical data to prescribe actions in response to business moments and other events.”

At Mercado Libre, while delivering analytics, we aim to build design patterns that allow us to programmatically consume data from Google BigQuery (BQ), our powerful cloud data warehouse. These design patterns are built using the universal semantic modeling layer of Looker. It is our goal that our continuous intelligence ecosystem both demonstrates the art of the possible with data and allows others to innovate and build upon existing work to meet their very specific needs at that moment in time.  

Real-time analytics at scale with Google BigQuery

We attempt every effort to make decisions based on data. To infuse our decision making with data, we have found that data needs to be timely, credible, and available for analysis, no matter the source. This includes streaming, collecting and presenting data from our whole ecosystem, which includes external data, our internal management systems, web traffic from products like Google Analytics and App Annie, warehouse and network logs, cloud usage and costs, and, of course, all of our APIs.

We chose Google BigQuery as the primary query engine within our ecosystem, due to its resiliency and reliability working with our growing volumes and number of data sources. With Google’s serverless, auto-scaling data warehouse, we are able to process hundreds of terabytes of raw data and run hundreds of thousands of queries every day without sacrificing performance or additional management overhead. This combination of speed and scale on-demand has helped accelerate our journey towards real-time analytics to support improved decision making with the freshest insights and data available. 

Bridging the technical gap between data and users with Looker

Having raw data available through powerful query engines is a fundamental component of our continuous intelligence ecosystem, but it isn’t enough on its own. Even with an army of analysts with SQL knowledge ready to query data, could result in inconsistent logic and ungoverned business rules.

To build trust in our continuous intelligence ecosystem at scale, we needed a data modeling and business intelligence tool that would enable us to consistently and programmatically consume data –  and could grow at the speed of our business. Enter Looker, a modern business intelligence and data platform. Powered by its robust universal semantic modeling layer and numerous modern integrations, it allows us to easily infuse data into workflows. 

At Looker’s core is a collaborative and version-controlled semantic model that’s created using LookML, a dependency language which is easy to learn and maintain for anyone familiar with SQL. We were able to create a single source of truth for our business by defining dimensions, metrics, and join logic centrally. For our analysts and business users, this defined logic is used to automatically generate consistent SQL statements on behalf of a user, which is where things start to become interesting: trustworthy self-service analytics at scale. 

BigQuery+Looker: Power to the People

This powerful data stack connects users to the power of cloud computing and analytics. Insights can be delivered and created in many different ways from viewing the classic dashboard, integrating with Slack, receiving email alerts, to embedding analytics. Data can also be accessed through other BI tools including spreadsheets which are a very popular tool that many are comfortable with. This is why we are particularly excited about Looker’s new Google Sheets integration. As alpha testers for this new integration, we have been able to provide even broader access to data through the spreadsheet interface that everyone is familiar with. Rather than asking people to learn yet another new tool, our users’ existing workflows aren’t dramatically changing, they’re constantly evolving!

Our Business and Customer Experience has Improved. 

Following are some  examples of outcomes from different points along the data lifecycle which are propelling our business forward and improving our customer’s experience: Once we were on BigQuery, we managed to reach 99% of the SLA availability required by processes that feed +30 near-real-time monitors we developed in Looker which are consumed by business, transport and operations teams to make key decisions. Along with BQ and the ease of use of Looker ML and Looker, we gained agility and speed in the creation and deployment of new dashboards, adapting to changes in our incredibly competitive industries.

And in creating value for customers, we are able to monitor (in near real-time) the delivery promise of our shipments and optimize scheduling based on the capacity of our aircrafts, providing reliability to our stakeholders.  

Building Upon The Foundation – A Culture of learning and innovation

As an analytics team, we can’t possibly respond and scale as fast as the company grows. Knowingly, we’ve developed our continuous intelligence ecosystem as a foundational layer that can help solve 80% of people’s data questions. The additional 20% still depends on our ability to empower people to leverage their own business knowledge and expertise with the tools we provide in order to innovate on top of what we have built. 

Ultimately, a true continuous intelligence ecosystem is bigger than technologies – it requires user empathy to understand how they want to use and consume data, and provide education.  To realize our longer term adoption and data literacy goals, we have built out comprehensive programs employees go through, to instill a beta mindset and how to use these tools to innovate and build on their own insights. 

As our user’s data skills evolve in parallel with our data technologies, our vision is to get to a point where people can build their own models, frameworks, decision frameworks on top of the continuous intelligence ecosystem that we’ve built. This continuous iteration and innovation is what fuels our data-driven culture. How we teach data literacy and culture will be the topic of our third blog installment. 

In our second blog installment, we will explore a Shipping use case of how our continuous intelligence ecosystem was leveraged and innovated upon by our users to deliver impactful outcomes.

Gartner IT Glossary, Continuous Intelligence, as on Mar 28, 2022.

GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.

Today, a typical enterprise uses over 100 SaaS applications while large organizations  commonly use over 400 apps. These applications contain valuable data on customers, suppliers, employees, products and more, offering the potential for valuable insights and powerful workflows. However, in the past, this data has often remained siloed and underutilized, even within an application. 

Customer demand is driving an unprecedented wave of innovation across the SaaS industry, with providers building intelligence into their applications through rich analytical and AI/ML capabilities, and enabling their customers’ data ecosystems through real time data sharing. The data cloud platform powering these applications is a critical factor in companies’ ability to innovate and optimize efficiency, while keeping its customer’s data secure. 

Today, over 700 tech companies including Zoominfo, Equifax, Exabeam, Bloomreach, and Quantum Metric power their products and businesses using Google’s data cloud. This week at the Data Cloud Summit, we announced the Built with BigQuery initiative which helps ISVs get started building applications using data and machine learning products. By providing dedicated access to technology, expertise and go to market programs, this initiative allows tech companies to accelerate, optimize and amplify their success.

“Enabling customers to gain superior insights and intelligence from data is core to the ZoomInfo strategy,” ZoomInfo CEO Henry Schuck says. “We are excited about the innovation Google Cloud is bringing to market and how it is creating a differentiated ecosystem that allows customers to gain insights from their data securely, at scale, and without having to move data around. Working with the Built with BigQuery initiative team enables us to rapidly gain deep insight into the opportunities available and accelerate our speed to market.”

Building intelligent SaaS applications with a unified data platform

Google’s data cloud provides a complete platform for building data-driven applications, from simplified data ingestion, processing and storage to powerful analytics, AI/ML and data sharing capabilities, all seamlessly integrated with Google Cloud’s open, secure, sustainable platform. With a huge partner ecosystem and support for multicloud, open source tools and APIs, we provide technology companies the portability and extensibility they need to avoid data lock-in. 

Through the Built with BigQuery initiative, we are helping tech companies to build the next-gen SaaS applications on Google’s data cloud with simplified access to technology, dedicated engineering support and joint go to market programs.

Exabeam, a next generation cybersecurity solution, is a great example. The company leverages Google’s data cloud to provide their customers with a “limitless-scale” cybersecurity solution.

“Built with Google’s data cloud, Exabeam’s limitless-scale cybersecurity platform helps enterprises respond to security threats faster and more accurately” said Sanjay Chaudhary, VP of Products at Exabeam. “We are able to ingest data from over 500 security vendors, convert unstructured data into security events, and create a common platform to store them in a cost effective way. The scale and power of Google’s data cloud enables our customers to search multi-year data and detect threats in seconds.” 

Foster innovation and unlock new business models through data sharing

Data becomes even more valuable when shared. According to research, the global data monetization market size is growing at a CAGR of 47.9% and is projected to reach $11.7 Billion by 2026. It’s a compelling proposition for SaaS companies as it enables them to deliver increased value for their customers while expanding their own partner ecosystem, increasing stickiness and unlocking new revenue streams. 

Google Cloud’s strategy for data sharing encompasses three key areas: secure data sharing in BigQuery, the ability to create private and public exchanges alongside commercial and public datasets in Analytics Hub, and a robust ecosystem of premier partner and Google data. 

Through the real-time data sharing capabilities of BigQuery, SaaS companies are enabling their customers to combine their data with the customer’s own proprietary data, and other third-party data sources, to derive 360-degree insights. Over 4,500 organizations share more than 250 Petabytes of data weekly in BigQuery, not accounting for intra-organizational data sharing*.

For example, manufacturers can get real-time visibility into their entire supply chain by combining datasets from ISVs, such as supply chain innovator, Blume Global, and data publishers.

“Our partnership with Google Cloud is helping us achieve our mission to build the next-generation supply chain operating system. Blume Maps, our digital twin of the supply chain world built with Google’s data cloud, allows our customers to generate accurate lead times, real-time shipment location and ETAs” said Blume Global CEO Pervinder Johar. “We apply the power of Google Cloud’s data and analytics capabilities to our growing database of over 1.5 million global data points to feed our lead time and dynamic ETA engine. We are also able to create data twins of unique logistics data and share with users around the globe.”

Google Cloud’s Analytics Hub is a fully-managed service built on BigQuery that allows organizations to efficiently and securely exchange valuable data and analytics assets across any organizational boundary.  With unique datasets that are always-synchronized and bi-directional sharing, you can create a rich and trusted data ecosystem between business units or partnerships–one in which everyone gains value immediately.

“As external data becomes more critical to organizations across industries, the need for a unified experience between data integration and analytics has never been more important. We are proud to be working with Google Cloud to power the launch of Analytics Hub, feeding hundreds of pre-engineered data pipelines from hundreds of external datasets,” said Dan Lynn, SVP Product at Crux. “The sharing capabilities that Analytics Hub delivers will significantly enhance the data mobility requirements of practitioners, and the Crux data integration platform stands ready to quickly integrate any external data source and deliver on behalf of Google Cloud and its clients.”

Innovate, optimize and amplify with Google Cloud

The Built with BigQuery initiative provides access to technology, expertise and go-to-market:. 

Access to Technology: Get started fast with a Google-funded, pre-configured sandbox. Access Cloud Credits to fund your, and your customer’s, POCs and apply for innovative pricing models designed for ISVs. 

Access to Expertise: Accelerate and optimize product design and architecture with access to designated experts in building data-driven SaaS applications from the ISV Center of Excellence, providing insight into key use cases, architectural patterns and best practices. Access experts from across Google to enable co-innovation with products such as Earth Engine and Google Marketing Platform.

Access to Go-to-market: Amplify your success with joint marketing programs to drive awareness, generate demand and increase adoption.

Get Started

Start building your applications on Google Cloud with $300 in free credits or apply for the Built with BigQuery initiative. 

*As of November, 2021, within a typical 7 day period in BigQuery, and not accounting for intra-organizational data sharing.

Now more than ever, organizations rely on real-time and accurate data analytics solutions to drive innovation and achieve operational excellence. With the help of AI and machine learning, companies combine data from various sources to derive actionable insights, forecast outcomes and make more informed decisions. Many organizations are also looking for new ways to monetize their data assets and provide users with secure access to data outside their ecosystem. 

However, with security threats and privacy regulations on the rise, companies find it difficult to securely share or exchange data with partners, customers, or other organizations. Traditional data sharing techniques such as batch ETL pipelines or FTP downloads are costly to maintain, do not scale well, and cause fragmented or stale data.

To help organizations overcome these data sharing challenges, we are excited today to announce the preview of Analytics Hub. This new fully-managed service enables you to efficiently and securely exchange valuable data and analytics assets across organizational boundaries. Analytics Hub is built on top of BigQuery, ​​Google’s petabyte-scale, serverless cloud data warehouse. BigQuery’s unique architecture enables you to share data at scale without making multiple copies of your data. Data is always live and can be consumed in real-time using the built-in streaming capabilities. With BigQuery, you can also leverage the built-in machine learning, geospatial, and advanced analytics capabilities or take advantage of the native business intelligence support with tools like Looker, Google Sheets, and Data Studio.

Data sharing is not new to BigQuery. In fact, we have had cross-organizational, in-place data sharing capabilities since 2010. As of November 2021, we see over 4,500 different organizations sharing over 250 petabytes of data per week in BigQuery. Analytics Hub makes data sharing even easier, enabling organizations to realize the full potential of their shared data.

Here is what some of our early adopters have to say:

Secure data exchanges

Analytics Hub makes it easy for organizations to govern and distribute their data centrally. As a data publisher, you can create secure data exchanges and publish listings that contain the datasets you want to deliver to your subscribers. By default, exchanges are completely private. Only the users or groups you provision can view or subscribe to the listings. You also have the ability to make your exchange public – providing all Google Cloud customers access to view or subscribe to your listings. You can easily create multiple exchanges to meet your data sharing needs. 

As a data subscriber, Analytics Hub provides a seamless experience for you to browse and search through all the listings across all exchanges that you have access to. Once you find the dataset of interest, you subscribe to the listing. This will create a read-only linked dataset within your project where you can query or perform analytics on the data. The linked dataset is not a copy of the data; it is just a symbolic link to the shared dataset and stays in sync with any changes made by the data publisher.

Bootstrapping our data ecosystem

Organizations have increasingly started to consume data from third-party sources and complement them with internal data to derive new insights. With Analytics Hub, we want to make it easy for you to discover and subscribe datasets to trusted valuable sources. On day one, you can subscribe to:

Public datasets: Easy access to hundreds of public datasets managed by Google that include data about weather and climate, cryptocurrency, healthcare and life sciences, and transportation. 

Google datasets: Unique, freely-available first-party datasets from Google. An example of this is the Google Trends dataset that allows users to measure interest in a particular topic or search term across Google Search, from around the United States, down to the city-level, and now also available for 50 international markets.

Commercial datasets: In partnership with Crux Informatics, we have onboarded commercial providers across finance, geospatial, retail, etc. who have brought all their data into BigQuery. We are excited to feature business data and insights from our recent strategic agreement with Dun & Bradstreet.

Next steps

Get started with Analytics Hub today by using this guide, starting a free trial with BigQuery, or contacting the Google Cloud sales team. Stay tuned for future feature updates such as usage metrics for publishers, parametrized datasets, privacy-safe queries, commercialization management, and much more.