Migrating your Oracle and SQL Server databases to Google Cloud

Migrating your Oracle and SQL Server databases to Google Cloud

For several decades, before the rise of cloud computing upended the way we think about databases and applications, Oracle and Microsoft SQL Server databases were a mainstay of business application architectures. But today, as you map out your cloud journey, you’re probably reevaluating your technology choices in light of the cloud’s vast possibilities and current industry trends.

In the database realm, these trends include a shift to open source technologies (especially to MySQL, PostgreSQL, and their derivatives), adoption of non-relational databases, and multi-cloud and hybrid-cloud strategies, and the need to support global, always-on applications. Each application may require a different cloud journey, whether it’s a quick lift-and-shift migration, a larger application modernization effort, or a complete transformation with a cloud-first database.

Google Cloud offers a suite of managed database services that support open source, third-party, and cloud-first database engines. At Next 2022, we published five new videos specifically for Oracle and SQL Server customers looking to either lift-and-shift to the cloud or fully free themselves from licensing and other restrictions. We hope you’ll find the videos useful in thinking through your options, whether you’re leaning towards a homogeneous migration (using the same database you have today) or a heterogeneous migration (switching to a different database engine).

Let’s dive into our five new videos.

#1 Running Oracle-based applications on Google Cloud

By Jagdeep Singh & Andy Colvin

Moving to the cloud may be difficult if your business depends on applications running on an Oracle Database. Some applications may have dependencies on Oracle for reasons such as compatibility, licensing, and management. Learn about several solutions from Google Cloud, including Bare Metal Solution for Oracle, a hardware solution certified and optimized for Oracle workloads, and solutions from cloud partners such as VMware and Equinix. See how you can run legacy workloads on Oracle while adopting modern cloud technologies for newer workloads.

#2 Running SQL Server-based applications on Google Cloud

By Isabella Lubin

Microsoft SQL Server remains a popular commercial database engine. Learn how to run SQL Server reliably and securely with Cloud SQL, a fully-managed database service for running MySQL, PostgreSQL and SQL Server workloads. In fact, Cloud SQL is trusted by some of the world’s largest enterprises with more than 90% of the top 100 Google Cloud customers using Cloud SQL. We’ll explore how to select the right database instance, how to migrate your database, how to work with standard SQL Server tools, and how to monitor your database and keep it up to date.

#3 Choosing a PostgreSQL database on Google Cloud

By Mohsin Imam

PostgreSQL is an industry-leading relational database widely admired for its permissive open source licensing, rich functionality, proven track record in the enterprise, and strong community of developers and tools. Google Cloud offers three fully-managed databases for PostgreSQL users: Cloud SQL, an easy-to-use fully-managed database service for open source PostgreSQL; AlloyDB, a PostgreSQL-compatible database service for applications that require an additional level of scalability, availability, and performance; and Cloud Spanner, a cloud-first database with unlimited global scale, 99.999% availability and a PostgreSQL interface. Learn which one is right for your application, how to migrate your database to the cloud, and how to get started.

#4 How to migrate and modernize your applications with Google Cloud databases

By Sandeep Brahmarouthu

Migrating your applications and databases to the cloud isn’t always easy. While simple workloads may just require a simple database lift-and-shift, custom enterprise applications may benefit from more complete modernization and transformation efforts. Learn about the managed database services available from Google Cloud, our approach to phased modernization, the database migration framework and programs that we offer, and how we can help you get started with a risk-free assessment.

#5 Getting started with Database Migration Service

By Shachar Guz & Inna Weiner

Migrating your databases to the cloud becomes very attractive as the cost of maintaining legacy databases increases. Google Cloud can help with your journey whether it’s a simple lift-and-shift, a database modernization to a modern, open source-based alternative, or a complete application transformation. Learn how Database Migration Service simplifies your migration with a serverless, secure platform that utilizes native replication for higher fidelity and greater reliability. See how database migration can be less complex, time-consuming and risky, and how to start your migration often in less than an hour.

We can’t wait to partner with you

Whichever path you take in your cloud journey, you’ll find that Google Cloud databases are scalable, reliable, secure and open. We’re looking forward to creating a new home for your Oracle- and SQL Server-based applications.

Start your journey with a Cloud SQL or Spanner free trial, and accelerate your move to Google Cloud with the Database Migration Program.

Related Article

What’s new in Google Cloud databases: More unified. More open. More intelligent.

Google Cloud databases deliver an integrated experience, support legacy migrations, leverage AI and ML and provide developers world class…

Read Article

Source : Data Analytics Read More

How The FA is moving the goal posts with a data cloud approach in Qatar

How The FA is moving the goal posts with a data cloud approach in Qatar

We’re moments away from the kick-off of another historic tournament. After the England men’s football team reached the Euro 2020 final in last year’s pandemic-delayed competition, there is genuine confidence in a successful run in Qatar.

The Football Association (The FA) is the governing body of association football in England, and has left no stone unturned in its preparations; they have increasingly looked to physical performance data as a way to help support players on the pitch. Maintaining accurate and insightful information on fitness, conditioning, and nutrition also helps ensure player welfare – something that gets more important with every fixture in a tournament environment.

The need for improved understanding of how players are faring was the reason The FA set up the Performance Insights strand of its Physical Performance, Medicine, and Nutrition department during lockdown in 2020. And they used Google Cloud to help them revolutionize the way they capture, store, and process information.

A single 90-minute squad training session can generate millions of rows of data. In football, things change so quickly that this data begins to lose relevance as soon as the players are back in the dressing room. That’s why The FA needed a solution which could turn raw data into valuable, easy-to-understand insights. This led the team to BigQuery, Google Cloud’s data warehouse solution.

BigQuery enables The FA’s Performance Insights team to automate previously labor-intensive tasks, and for all the information to be stored in a single, centralized platform for the first time. By collating different data sources across The FA’s squads, there can be greater clarity and fewer siloes – everyone is working towards the same goals. 

 A unique solution for a unique tournament

Access to insights is vital in any tournament situation, but this year there is a need for speed like never before. 

Unlike previous tournaments, Qatar will start in the middle of domestic league seasons throughout the world. Traditionally, international sides are able to meet up for nearly a month between the end of the league season and the start of the tournament – a critical time to work on all aspects of team preparation, including tactics and conditioning. By contrast, this year the England players will have less than a week to train together before the first kick-off.

BigQuery allows The FA’s data scientists to combine data on many aspects of a player’s physical performance captured during a training camp, from intensity to recovery. This can enable more useful conversations on the ground and can help create more individualized player management. And by using BigQuery’s user-defined customisable functions, the same data can be tweaked and tailored to fit the needs across departments. 

This customizability provides a foundation for a truly ‘interdisciplinary’ team in which doctors, strength and conditioners, physios, psychologists, and nutritionists have a common understanding of the support a player needs.

Every minute will count during such a compressed training window, so automation is key. While BigQuery is the core product The FA uses to store and manipulate data, it’s just one part of a suite of Google Cloud products and APIs that help them easily turn data into insights. 

In-game and training performance data, along with data pertaining to players’ sleep, nutrition, recovery, and mental health can be captured and fed through Python, which links straight into BigQuery using its Pub/Sub functionality. BigQuery’s native connectors then stream insights to visual dashboards that convey them in a meaningful, tangible format.  

Before leveraging the power of Google Cloud, this work could take several hours each day. Now, it can take a minute from data capture to the coaches having access to clear and actionable information. 

Predicting a bright future for the Beautiful Game

We won’t have long to wait to see how England will perform in Qatar. But the benefits of The FA’s cloud-enabled approach to data science will continue long after the final whistle has blown.

The short preparation window has posed challenges for The FA, but it has also given the organization a unique opportunity to discover how predictive analytics and machine learning on Google Cloud could further enhance its player performance strategy. 

The Physical Performance, Medicine, and Nutrition department has collected performance data from players throughout this year’s league season, taking into account fixture density and expected physical demand. They hope to use this to support the players’ physical preparation and recovery during the tournament based on individual physical performance profiles.

This ML work is still in the early stages. But the Performance Insights team is confident that by developing even closer relationships with Google Cloud and even greater familiarity with its technology, they will be able to unlock an even greater level of insight into player performance.

Learn more about how Google Cloud can turn raw data into actionable insights, fast.

Source : Data Analytics Read More

Built with BigQuery: How Connected-Stories leverages Google Data Cloud and AI/ML for creating personalized Ad Experiences

Built with BigQuery: How Connected-Stories leverages Google Data Cloud and AI/ML for creating personalized Ad Experiences

Editor’s note: The post is part of a series highlighting our awesome partners, and their solutions, that are Built with BigQuery

In the field of producing engaging video content such as ads, many marketers ignore the power of data to improve their creative efforts to meet the consumers’ need for personalized messages. The demand for creative tech to efficiently personalize is real as marketers need personalized video Ads to reach their audience with the right message at the right time. Data, Insights and Technology are the main ingredients to deliver this value while ensuring security and privacy requirements are met. The Connected-Stories team partnered with Google Cloud to build a platform for Ad personalization. Google Data Cloud and BigQuery are at the forefront to assimilate data, leverage ML models, create personalized ads, and capitalize on real-time intelligence as the core features of the Connected-Stories NEXT platform.

Connected-Stories NEXT is an end-to-end creative management platform to develop, serve, and optimize interactive video and display ads that scale across any channel. The platform ingests first-party data to create custom ML models, measure numerous third-party data points to help brands develop unique customer journeys and create videos that their data signals can drive. An intelligent feedback loop passes real-time data back, enabling brands to make data-driven and actionable video ads that take the brand’s campaigns to the next level.

The core use case of the NEXT platform revolves around collecting user’s interaction data and optimizing for precision and speed to create an actionable Ad experience that is personalized for each user. The platform processes complex data points to create interactive data visualizations that allow for accurate analysis. The platform uses Vertex AI to access managed tools, workflows, and infrastructure to build, deploy, and scale ML models that have improved the accuracy to identify segments for further analysis. 

The platform ingests 200M data events with peaks and valleys of activity. These events are processed to generate dashboards that enable users to visualize metrics based on filters in real-time. These dashboards have high performance requirements in terms of a responsive user interface under constantly changing data dimensions.

Google Cloud’s serverless stack coupled with limitless data cloud infrastructure has been the core to the NEXT platform’s data-driven innovation. The growing volume of data ingested, streamed and processed were scaled uniformly across the compute, storage and analytical layers of solution. A lean development team at Connected-Stories were able to focus all-in on the solution, while the serverless stack scaled, lowered attack service in terms of security and optimized the cost footprint through pay-as-you-go features. 

BigQuery has been the backbone to support the vast amounts of data spreading over multiple geos resulting in workloads running at petabyte scale. BigQuery’s fully managed serverless architecture, real-time streaming, built-in machine learning and rich business intelligence capabilities distinguishes itself from a cloud data warehouse. It is the foundation needed to approach data and serve users in an unlimited number of ways. For an application with zero tolerance for failure, given its fully managed nature, BigQuery handles replication, recovery, data distributed optimization and management. 

The platform’s requirements include the need for low maintenance, constantly ingesting and refreshing data and smart-tuning of aggregated data. These capabilities can be implemented by BigQuery’s materialized views feature. Materialized views are useful for precomputed views that regularly cache query results for better performance. These views possess the innate feature to read only the delta change from base tables and calculate the up-to-date aggregations. Materialized views impart faster outputs and consume fewer resources while reducing the cost footprint.

Some key considerations in using Google cloud and focusing on the Serverless stack include:  quick onboarding to development, prototyping in short sprints and ease of preparing data in a rapidly changing environment. Typical considerations around low code / no code include data transformation, aggregation and reduced deployment time. These considerations are fulfilled through  using serverless capabilities within Google Cloud such as PubSub, Cloud Storage, Cloud Run, Cloud Composer, Dataflow and BigQuery as described in the Architecture diagram below. The use of each of these components and services are described below.

Input/Ingest: At a high-level, microservices hosted in Cloud Run collect and aggregate incoming Ads events. 

Enrichment: The output of this stage is a Pub-Sub message enriched with more attributes based on a pre-configured campaign. 

Store: a Cloud Dataflow streaming job to create text files in Cloud Storage buckets. 

Trigger: Cloud Composer triggers the spark jobs based on text files to process and group them to produce desired output as one record per impression, a logical group of events. 

Deploy: Cloud Build is then used to automate all deployments. 

Thus far, all Google cloud managed services work together to ingest, store and trigger the orchestration, all of which are scalable based on configurations including autoscaling capabilities. 

Visualization: A visualization tool reads data from BigQuery to compute pre-aggregations required for each dashboard. 

Data Model Evolution considerations: Though the solution served the purpose of creating pre-aggregations, as the data model evolved by adding a column or creating a new table, it led to recreating pre-aggregations and querying the data again. Alternatively, creating aggregate tables as an extra output of current ETLs seemed like a viable option. However, this would increase the cost and complexity of jobs. A similar situation to reprocess or update aggregated tables would occur as data is updated. 

Precomputed views of data that is periodically cached are critical to reach the audience with the right message at the right time. 

Performance: In order to increase the performance of the platform, we need to have regularly precomputed views of the data, cached . 

Materialized Views: Consumers of these views needed faster response times, to consume fewer resources and output only the changes in comparison to a base table. BigQuery Materialized views were used to solve this very requirement. Materialized views have been highly leveraged to optimize the design resulting in lesser maintenance and access to fresh data with high performance with a relatively low technical investment in creating and maintaining SQL code. 

Dashboards: Application dashboards pointing to the Materialized views are highly performant and provide a view into fresh data. 

Custom Reports with Vertex AI Notebooks: Vertex AI notebooks directly read data from BigQuery to produce custom reports for a subset of customers. Vertex AI has been hugely beneficial to data analysts, where an environment with pre-installed libraries simplifies the readiness to use. Vertex AI Workbench notebooks are used to share these reports within the team allowing them to work always on the cloud without having the need to download data at any time. Besides, it increases the velocity to develop and test ML models faster.

The NEXT platform has yielded benefits such as customers having the ability to create unique consumer journeys powered by AI / ML personalization triggers, using first-party data and business intelligence tools to capitalize on real-time creative intelligence, which is a dashboard to measure campaign performance for cross-functional teams to analyze the impact of Ad content experience at a granular level. All of these while ensuring controlled access to data to enrich data without moving across clouds. The NEXT platform can keep up with increased demands for agility, scalability and reliability through the underlying usage of Google Cloud.

Partnering with Google, in the context of the Google Built with BigQuery program has surfaced the differentiated value in areas of creating interactive personalized Ads by using real-time data. In addition, by sharing this data across organizations as assets, ML models have fueled higher levels of innovation. Connected-Stories plan to deepen the penetration into the entire spectrum of services offered in the AI/ML area to enhance core functionality and provide newer capabilities to the platform. 

Click here to learn more about Connected-Stories NEXT Platform capabilities.

The Built with BigQuery Advantage for ISVs 

Through Built with BigQuery, launched in April ‘22 as part of Google Data Cloud Summit, Google is helping tech companies like Connected-Stories co-innovate in building  applications that leverage Google’s data cloud with simplified access to technology, helpful and dedicated engineering support, and joint go-to-market programs. Participating companies can:

Get started fast with a Google-funded, pre-configured sandbox. 

Accelerate product design and architecture through access to designated technical experts from the ISV Center of Excellence who can share insights from key use cases, architectural patterns, and best practices encountered in the field. 

Amplify success with joint marketing programs to drive awareness, generate demand, and increase adoption.

The Google Data Cloud spectrum of products and specifically BigQuery give ISVs the advantage of a powerful, highly scalable data warehouse that’s integrated with Google Cloud’s open, secure, sustainable platform. And with a huge and expanding partner ecosystem and support for multi-cloud, open source tools and APIs, Google provides technology companies the portability and extensibility they need to avoid data lock-in and exercise choice. 

We thank the Google Cloud and Connected-Stories team members who co-authored the blog: Connected-Stories: Luna Catini, Marketing Director, Google: Sujit Khasnis, Cloud Partner Engineering

Related Article

Built with BigQuery: How True Fit’s data journey unlocks partner growth

True Fit, a data-driven personalization platform built on Google Data Cloud to provide fit personalization for retailers by sharing curat…

Read Article

Source : Data Analytics Read More

What can you build with the new Google Cloud developer subscription?

What can you build with the new Google Cloud developer subscription?

To help you grow and build faster – and take advantage of the 123 product announcements from Next ‘22 – last month we launched theGoogle Cloud Skills Boost annual subscription with new Innovators Plus benefits. We’re already hearing rave reviews from subscribers from England to Indonesia, and want to share what others are learning and doing to help inspire your next wave of Google Cloud learning and creativity.

First, here’s a summary of what the Google Cloud Skills Boost annual subscription1 with Innovators Plus benefits includes;

Access to 700+ hands-on labs, skill badges, and courses

$500 Google Cloud credits

A Google Cloud certification exam voucher

Bonus $500 Google Cloud credits after the first certification earned each year

Live learning events led by Google Cloud experts

Quarterly technical briefings hosted by Google Cloud executives

Celebrating learning achievements

Subscribers get access to everything needed to prepare for a Google Cloud certification exam, which are among the top paying IT certifications in 20222. Subscribers also receive a certification exam voucher to redeem when booking the exam.

Jochen Kirstätter, a Google Developer Expert and Innovator Champion is using the subscription to prepare for his next Google Cloud Professional certification exam, and has found the labs and courses on Google Cloud Skills Boost have helped him feel ready to go get #GoogleCloudCertified 

“‘The only frontiers are in your mind’ – with the benefits of #InnovatorsPlus I can explore more services and practice real-life scenarios intensively for another Google Cloud Professional certification.”

Martin Coombes, a web developer from PageHub Design, is a new subscriber and has already become certified as a Cloud Digital Leader. That means he’s been able to unlock the bonus $500 of Google Cloud credit benefit to use on his next project. 

“For me, purchasing the annual subscription was a no brainer. The #InnovatorsPlus benefits more than pay back the investment and I’ve managed to get my first Google Cloud certification within a week using the amazing Google Cloud Skills Boost learning resources. I’m looking forward to further progressing my knowledge of Google Cloud products.”

Experimenting and building with $500 of Google Cloud credits 

We know how important it is to learn by doing. And isn’t hands-on more fun? Another great benefit of the annual subscription is $500 of Google Cloud credits every year you are a subscriber. And even better, once you complete a Google Cloud certification, you will unlock a bonus $500 of credits to help build your next project just like Martin and Jeff did. 

Rendy Junior, Head of Data at Ruangguru and a Google Cloud Innovator Champion, has already been able to apply the credits to an interesting data analysis project he’s working on. 

“I used the Google Cloud credits to explore new features and data technology in DataPlex. I tried features such as governance federation and data governance whilst data is located in multiple places, even in different clouds. I also tried DataPlex data cataloging; I ran a DLP (Data Loss Prevention) inspection and fed the tag where data is sensitive into the DataPlex catalog. The credits enable me to do real world hands-on testing which is definitely helpful towards preparing for certification too.”

Jeff Zemerick, recently discovered the subscription and has been able to achieve his Professional Cloud Database certification using the voucher and Google Cloud credits to prepare.  

“I was preparing for the Google Cloud Certified Professional Cloud Database exam and the exam voucher was almost worth it by itself. I used some of the $500 cloud credits to prepare for the exam by learning about some of the Google Cloud services where I felt I might need more hands-on experience. I will be using the rest of the credits and the additional $500 I received from passing the exam to help further the development of our software to identify and redact sensitive information in the Google Cloud environment. I’m looking forward to using the materials available in Google Cloud Skills Boost to continue growing my Google Cloud skills!”

Grow your cloud skills with live learning events 

Subscribers gain access to live learning events, where a Google Cloud trainer teaches popular topics in a virtual classroom environment. Live-learning events cover topics like BigQuery, Kubernetes, CloudRun, Cloud Storage, networking and security. We’ve set these up to go deep: mini live-learning courses consist of two highly efficient hours of interactive instruction, and gamified live learning events are three hours of challenges and fun. We’ve already had over 400 annual subscribers reserve a spot for upcoming live learning events. Seats are filling up fast for the November and December events, so claim yours before it’s too late. 

Shape the future of Google Cloud products through the quarterly technical briefings  

As a subscriber, you are invited to join quarterly technical briefings, getting insight into the latest product developments and new features, with the opportunity for subscribers to engage and shape future product development for Google Cloud. Coming up this quarter, get face time with Matt Thompson, Google Cloud’s Director of Developer Adoption, who will demonstrate some of the best replicable uses of Google Cloud he’s seen from leading developers. 

Start your subscription today 

Take charge of your cloud career today by visiting cloudskillsboost.google to get started with your annual subscription. Make sure to activate your Innovators Plus badge once you do and enjoy your new benefits. 

1. Subject to eligibility limitations. 
2. Based on responses from the Global Knowledge 2022 IT Skills and Salary Survey.

Source : Data Analytics Read More

BigQuery helps Soundtrack Your Brand hit the high notes without breaking a sweat

BigQuery helps Soundtrack Your Brand hit the high notes without breaking a sweat

Editor’s note: Soundtrack Your Brand is an award-winning streaming service with the world’s largest  licensed music catalog built just for businesses, backed by Spotify. Today, we hear how BigQuery has been a foundational component in helping them transform big data into music. 

Soundtrack Your Brand is a music company at its heart, but big data is our soul. Playing the right music at the right time has a huge influence on the emotions a brand inspires, the overall customer experience, and sales.  We have a catalog of over 58 million songs and their associated metadata from our music providers and a vast amount of user data that helps us deliver personalized recommendations, curate playlists and stations, and even generate listening schedules. As an example, through our Schedules feature our customers can set up what to play during the week.  Taking that one step further, we provide suggestions on what to use in different time slots and recommend entire schedules.

Using BigQuery, we built a data lake to empower our employees to access all this content and metadata in a structured way. Ensuring that our data is easily discoverable and accessible allows us to build any type of analytics or machine learning (ML) use case and run queries reliably and consistently across the complete data set. Today, our users are benefiting from this advanced analytics through the personalized recommendations we offer across our core features: Home, Search, Playlists, Stations, and Schedules.

Fine-tuning developer productivity

The biggest business value that comes from BigQuery is how much it speeds up our development capabilities and allows us to ship features faster. In the past 3 years, we have built more than 150 pipelines and more than 30 new APIs within our ML and data teams that total about 10 people. That is an impressive rate of a new pipeline every week and a new API every month.  With everything in BigQuery, it’s easy to simply write SQL and have it be orchestrated within a CI/CD toolchain to automate our data processing pipelines. An in-house tool built as a github template, in many ways very similar to Dataform, helps us build very complex ETL processes in minutes, significantly reducing the time spent on data wrangling. 

BigQuery acts as a cornerstone for our entire data ecosystem, a place to anchor all our data and be our single source of truth. This single source of truth has expanded the limits of what we can do with our data. Most of our pipelines start from a data lake, or end at a data lake, increasing re-usability of data and collaboration. For example, one of our interns built an entire churn prediction pipeline in a couple of days on top of existing tables that are produced daily. Nearly a year later, this pipeline is still running without failure largely due to its simplicity. The pipeline is BigQuery queries chained together into a BigQuery ML model running on a schedule withKubeflow Pipelines

Once we made BigQuery the anchor for our data operations, we discovered we could apply it to use cases that you might not expect, such as maintaining our configurations or supporting our content management system. For instance, we created a Google Sheet where our music experts are able to correct genre classification mistakes for songs by simply adding a row to a Google Sheet. Instead of hours or days to create a bespoke tool, we were able to set everything up in a few minutes. 

BigQuery’s ability to consume Excel spreadsheets allows business users who play key roles in improving our recommendations engine and curating our music, such as our content managers and DJs, to contribute to the data pipeline.

Another example is our use of BigQuery as an index for some of our large Cloud Storage buckets. By using cloud functions to subscribe to read/write events for a bucket, and writing those events to partitioned tables, our pipelines can easily and in a natural way quickly search and access files, such as downloading and processing the audio of new track releases. We also make use of Log Events when a table is added to a dataset to trigger pipelines that process data on demand, such as JSON/CSV files from some of our data providers that are newly imported into BQ. Being the place for all file integration and processing, BQ allows new data to be quickly available to our entire data ecosystem in a timely and cost effective manner while allowing for data retention, ETL, ACL and easy introspection.

BigQuery makes everything simple. We can make a quick partitioned table and run queries that use thousands of CPU hours to sift through a massive volume of data in seconds — and only pay a few dollars for the service. The result? Very quick, cost-effective ETL pipelines. 

In addition, centralizing all of our data in BigQuery makes it possible to easily establish connections between pipelines providing developers with a clear understanding of what specific type of data a pipeline will produce. If a developer wants a different outcome, she can copy the github template and change some settings to create a new, independent pipeline.

Another benefit is that developers don’t have to coordinate schedules or sync with each other’s pipelines: they just need to know that a table that is updated daily exists and can be relied on as a data source for an application. Each developer can progress their work independently without worrying about interfering with other developers’ use of the platform.

Making iteration our forte

Out of the box, BigQuery met and exceeded our performance expectations, but ML performance was the area that really took us by surprise. Suddenly, we found ourselves going through millions of rows in a few seconds, where the previous method might have taken an hour.  This performance boost ultimately led to us improving our artist clustering workload from more than 24 hours on a job running 100 CPU workers to 10 minutes on a BigQuery pipeline running inference queries in a loop until convergence.  This more than 140x performance improvement also came at 3% of the cost. 

Currently we have more than 100 Neural Network ML models being trained and run regularly in batch in BQML. This setup has become our favorite method for both fast prototyping and creating production ready models. Not only is it fast and easy to hypertune in BQML, but our benchmarks show comparable performance metrics to using our own Tensorflow code. We now use Tensorflow sparingly. Differences in input data can have an even greater impact on the experience of the end user than individual tweaks to the models. 

BigQuery’s performance makes it easy to iterate with the domain experts who help shape our recommendations engine or who are concerned about churn, as we are able to show them the outcome on our recommendations from changes to input data in real time. One of our favorite things to do is to build a Data Studio report that has the ML.predict query as part of its data source query. This report shows examples of good/bad predictions in the report along with bias/variance summaries and a series of drop-downs, thresholds and toggles to control the input features and the output threshold. We give that report to our team of domain experts to help manually tune the models, putting the model tuning right in the hands of the domain experts. Having humans in the loop has become trivial for our team. In addition to fast iteration, the BigQuery ML approach is also very low maintenance. You don’t need to write a lot of Python or Scala code or maintain and update multiple frameworks—everything can be written as SQL queries run against the data store.

Helping brands to beat the band—and the competition 

BigQuery has allowed us to establish a single source of truth for our company that our developers and domain experts can build on to create new and innovative applications that help our customers find the sound that fits their brand. 

Instead of cobbling together data from arbitrary sources, our developers now always start with a data set from BigQuery and build forward.  This guarantees the stability of our data pipeline and makes it possible to build outward into new applications with confidence. Moreover, the performance of BigQuery means domain experts can interact with the analytics and applications that developers create more easily and see the results of their recommended improvements to ML models or data inputs quickly. This rapid iteration drives better business results, keeps our developers and domain experts aligned, and ensures Soundtrack Your Brand keeps delivering sound that stands out from the crowd.

Related Article

How Telus Insights is using BigQuery to deliver on the potential of real-world big data

BigQuery’s impressive performance reduces processing time from months to hours and delivers on-demand real-world insights for Telus.

Read Article

Source : Data Analytics Read More

Accelerate innovation in life sciences with Google Cloud

Accelerate innovation in life sciences with Google Cloud

The last few years have underscored the importance of speed in bringing new drugs and medical devices to market, while ensuring safety and efficacy. Over this time, healthcare and life sciences organizations have transformed the way they research, develop, and deliver patient care by embracing agility and innovation. 

Now, the industry is set to reap the benefits of cloud technology and overcome the existing barriers to innovation.

Watch a 2-min overview of how Google Cloud helps life sciences accelerate innovation across the value chain.

What’s holding back innovation?

Costly clinical trials: The process of trialing and developing new drugs and devices is still long and costly, with more than 1 in 5 clinical trials failing due to a lack of funding.1 The high failure rate comes as no surprise when you consider the average clinical trial costs $19 million and takes 10-15 years (through all 3 phases) to be approved.2

Stringent security requirements: Pre-clinical R&D and clinical trials use large volumes of highly sensitive patient data – making the life sciences industry one of the top sectors targeted by hackers.3 On top of this, the FDA and other regulatory bodies have strict requirements for medical device cybersecurity. 

Unpredictable supply chains: Global supply chains are becoming increasingly complex and unpredictable. This can be brought on by anything from supply shortages, to geo-political events, and even bad weather. Making things worse is the lack of visibility into medical shipment disruptions – so when disaster strikes you’re often caught off guard.

Google Cloud for life sciences

At Alphabet, we’ve made significant investments in healthcare and life sciences, helping to tackle the world’s biggest healthcare problems, from chronic disease management, to precision medicine, to protein folding. 

Together with Google, you can transform your life sciences organization and deliver secure, data-driven innovation across the value chain. 

Accelerate clinical trials to deliver life-saving treatments faster and at less cost. Clinical trials require relevant and equitable patient cohorts that can produce clinically valid data. Solutions like DocAI can enable optimal patient matching for clinical trials, helping organizations optimize clinical trial selection and increase time to value.  How that patient data is collected is also important.  Collection in a physician’s office captures a snapshot of the participant’s data at one point in time and doesn’t necessarily account for daily lifestyle variables. Fitbit, used in more than 1,500 published studies–more than any other wearable device–can enrich clinical trial endpoints with new insights from longitudinal lifestyle data, which can help improve patient retention and compliance with study protocols. We have introduced Device Connect for Fitbit, which empowers healthcare and life sciences enterprises with accelerated analytics and insights to help people live healthier lives. We are able to empower organizations to improve clinical trials in key ways: 

Enable clinical trial managers to quickly create and launch mobile and web RWE collection mechanism for patient reported outcomes

Enable privacy controls with Cloud Healthcare Consent API and, as needed, remove PHI using Cloud Healthcare De-identification API 

Ingest RWE and data into BigQuery for analysis

Leverage Looker to enable quick visualization and powerful analysis of a study’s progress and results

Ensure security and privacy for a safe, coordinated, and compliant approach to digital transformation. Google Cloud offers customers a comprehensive set of services including pioneering capabilities such as BeyondCorp Enterprise for Zero Trust and VirusTotal for malicious content and software vulnerabilities; Chronicle’s security analytics and automation coupled with services such as Security Command Center to help organizations detect and protect themselves from cyber threats; as well as expertise from Google Cloud’s Cybersecurity Action Team. Google Cloud also recently acquired Mandiant, a leader in dynamic cyber defense, threat intelligence and incident response services.

Optimize supply chains and enhance your data to prepare for the unpredictable. With a digital supply chain platform, we can empower supply chain professionals to solve problems in real time including visibility and advanced analytics, alert-based event management, collaboration between teams and partners, and AI-driven optimization and simulation.

Ready to learn more? We’ll be taking a deep dive into each of the challenges outlined above in our life sciences video series. Stay tuned.

1. National Library of Medicine
2. How much does a clinical trial cost?
3. Life Sciences Industry Becomes Latest Arena in Hackers’ Digital Warfare

Related Article

How Google Cloud and Fitbit are building a better view of health for hospitals, with analytics and insights in the cloud

Exploratory research into detection and prevention of cardiovascular diseases using the latest wearable technology and AI-driven analytics.

Read Article

Source : Data Analytics Read More

Upskill for in-demand cloud roles with no-cost training on Coursera

Upskill for in-demand cloud roles with no-cost training on Coursera

Cloud technology has experienced accelerated adoption in recent years, with continued growth expected into 2023.1  This means that the need for organizations to attract and retain professionals with cloud skills continues to grow in parallel.2  

Keep your cloud career growing, at pace with digital transformation 

In partnership with Google Cloud, Coursera is offeringno-cost access to some of our most popular cloud training to help you hone your skills and stand out in the job-market. Whether you’re looking to enhance your technical competencies, advance your career, acquire more hands-on experience, or earn learning credentials to validate your knowledge, we have resources available to support your journey. 

Future-proof your career with select no-cost training and earn certificates

Claim one choice from a variety of popular Google Cloud Projects, Professional Certificates, Specializations and courses, available to claim until December 31st, 2022. 

The Google Cloud training included in this promotion spans a variety of roles, like machine learning engineering; data engineering; and cloud engineering, architecture and security. Training content is available for both technical and non-technical roles, from foundational to advanced knowledge and experience levels. The training descriptions include any prerequisite knowledge you should have before getting started.

The time requirements for completion also vary, so we’ve summarized it below to help you make your choice, and pick the level of commitment that is right for you. When you finish the training on Coursera, you will earn a certificate that you can share with your network on social media and your resume. 

Types of Google Cloud training available on Coursera 

Here is a rundown of the different types of training available on Coursera included in this offer, in order of time required to complete it:

Projects: Approximately 30-90 minute time commitment to complete

Learn new skills in an interactive environment by using software and tools in a cloud workspace with no download required.

Courses: Approximately 4-19 hour time commitment to complete

Courses typically include a series of introductory lessons, step-by-step hands-on exercises, Google knowledge resources, and knowledge checks. 

Specializations: Approximately 2-6 months time commitment to complete

Specializations are a series of courses that help you master a skill, and include a hands-on project. 

Professional Certificates: Approximately 1-9 months 

Professional Certificates include hands-on projects and courses, and upon completion you will earn a Professional Certificate. These can help you prepare for the relevant Google Cloud certification exam. 

Here is a look at some of our most popular training for in-demand cloud roles

Work through training at your own pace, and upskill for the role you’re in, or the one you’re looking to grow into. Popular training for in-demand roles include:

For those in non-technical roles, working closely with cloud technology 

Professional Certificate – Cloud Digital Leader 
This is a foundational level series of four courses designed to give you knowledge about cloud technology and data, and digital transformation. It helps increase confidence in contributing to cloud-related business initiatives and discussions. If you’re in a tech-adjacent role such as sales, HR or operations, you will benefit from this training. 

For Application Developers 

Specialization – Developing applications with Google Cloud
This Specialization is built for application developers who want to learn how to design, develop, and deploy applications that seamlessly integrate managed services from Google Cloud. It includes a variety of learning formats, including labs, presentations and demos. Labs can be completed in your preferred language: Node.js, Java, or Python. You’ll learn practical skills that are ready for immediate use in real IT environments.

For experienced ML and AI Engineers 

Professional Certificate – ML Engineer
Prepare for Google Cloud Certification with the Machine Learning Engineer Professional Certificate. This is an intermediate-level training recommended for participants who have data engineering or programming experience, and who want to learn how to apply machine learning in practice and to be successful in a machine learning role. There are 9 courses in this Professional Certificate, and completion time is about 7 months at the suggested pace of 5 hours per week. 

For beginners with Google Cloud in technical roles

Course – Google Cloud Fundamentals for AWS Professionals
This course introduces key concepts and terminology through a combination of videos and hands-on labs that can be completed in approximately 9 hours. You’ll learn about the components of the Google network infrastructure and differences between infrastructure as a service and platform as a service; how to organize projects and interact with Google Cloud; and jump into Google Cloud Compute Engine with a focus on virtual networking. 

For beginners in Data Engineering 

Project – Introduction to SQL for BigQuery and Cloud SQL
This is a self-paced lab that takes place in the Google Cloud console, giving you interactive practice running structured queries on BigQuery and Cloud SQL. This is a beginner level project that takes about an hour to complete.

As the year comes to a close, it’s a great time to prioritize growing your cloud skills. Check out our no-cost Google Cloud training offers on Coursera, available until December 31, 2022.

1. According to Forbes: The Top 5 Cloud Computing Trends in 2023
2. According to Forbes: From Crisis to Opportunity: Tackling the U.S. Cloud Skills Gap

Related Article

Build your cloud skills with no-cost access to Google Cloud training on Coursera

Advance your technical skills and boost your career by getting hands-on practice with Google Cloud projects.

Read Article

Source : Data Analytics Read More

How Telus Insights is using BigQuery to deliver on the potential of real-world big data

How Telus Insights is using BigQuery to deliver on the potential of real-world big data

Editor’s note: Today, we’re hearing from TELUS Insights about how Google BigQuery has helped them deliver on-demand, real-world insights to customers.

Collecting reliable, de-identifiable data on population movement patterns and markets has never been easy, particularly for industries that operate in the physical world like transit and traffic management, finance , public health, and emergency response. Unlike online businesses, these metrics might be collected  manually or limited by smaller sample sizes during a relatively short time. 

But imagine the positive impact this data could have if organizations had access to mass movement patterns and trends to solve complicated problems and mitigate pressing challenges such as traffic accidents, economic leakage, and more.

As one of Canada’s leading telecommunications providers, TELUS is in a unique position to provide powerful data insights about mass movement patterns. At TELUS, we recognize that the potential created by big data comes with a huge responsibility to our customers.  We have always been committed to respecting our customers’ privacy and safeguarding their personal information,  which is why we have implemented industry-leading Privacy by Design standards to ensure that their privacy is protected every step of the way. All the data used by TELUS Insights is fully de-identified, meaning it cannot be traced back to  an individual. It is also aggregated into large data pools, ensuring privacy is fully protected at all times.

BigQuery checked all our boxes for building TELUS Insights

TELUS Insights is the result of our vision to help businesses of all sizes and governments at all levels make smarter decisions based on real-world facts. Using industry-leading privacy standards, we can strongly de-identify our network mobility data and then aggregate it so no one can trace back data to any individual. 

We needed to build an architecture that would provide the performance necessary to run very complex queries, many of which were location-based and benefited from dedicated geospatial querying. TELUS is recognized as the fastest mobile operator and ranked first for network quality performance in Canada, and we wanted to deliver the same level of performance for our new data insights business.

We tested out a number of products, from data appliances to an on-premise data lake, but it was BigQuery, Google Cloud’s serverless, highly scalable, and fully managed enterprise data warehouse, that eventually came out ahead of the pack. Not only did BigQuery deliver fast performance that enabled us to easily and quickly analyze large amounts of data at infinity scale, it also offered support for geospatial queries, a key requirement for the TELUS Insights business. 

Originally, the model for TELUS Insights was consultative in nature: we would meet with customers to understand their requirements and our data science team would develop algorithms to provide the needed insights from the available data sets.

However, performance from our data warehouse proved challenging. It would take us six weeks of query runtime to extract insights from a month of data. To best serve our customers,  we began investigating the development of an API that, with simple inputs, would provide a consistent output so that customers could start using the data in a self-serve and secure manner. 

BigQuery proved itself able to meet our needs by combining high performance for complex queries, support for geospatial queries, and ease of implementing a customer-facing API.

High performance enabled new models of customer service

With support for ANSI SQL, our data scientists found the environment very easy to use.  

The performance boost was immediately apparent with project queries taking a fraction of the time compared to previous experiences – and that was before performing any optimization. 

BigQuery’s high performance was also one of the main reasons we were able to successfully launch an API that can be consumed directly and securely by our customers. Our customers were no longer limited on the size of their queries and would now get their data back in minutes. In the original consulting model, customers were dependent on our team and had little direct control over their queries, but BigQuery has allowed us to put the power of our data directly in our customers’ hands, while maintaining our commitment to privacy.

Using BigQuery to power our data platform means we also benefit from the entire ecosystem of Google Cloud services and solutions, opening up new doors and opportunities for us to deepen the value of our data through advanced analytics and AI-based techniques, such as machine learning. 

Cloud architecture enabled a quick pivot to meet COVID challenges

When the COVID-19 pandemic hit, we realized there was a huge value in de-identified and aggregated network mobility data for health authorities and academic researchers in helping reduce COVID-19 transmission without compromising the personal privacy of Canadians. 

As our TELUS Insights API was already in place, we were able to immediately shift focus and meet this public health need. Our API allowed us to provide supervised and guided access to government organizations and academic institutions to our de-identified and aggregated data, after which they were able to build their own algorithms, specific to the needs of epidemiology. BigQuery also enabled us to build federated access environments where we could safelist these organizations and, with appropriate supervision, allow them to securely access views they needed to build their reporting.

COVID-19 Use Case:  The image above shows de-identified and aggregated mass movement patterns in the City of Toronto into outlying regions in May 2020 when stay-at-home orders were issued by the City and residents started traveling to cottage country.  Public Health authorities were able to use this data to inform local hospitals of the surge in population in their surrounding geographic location and to attempt to provision extra capacity at nearby hospitals, including the provisioning of equipment such as much needed ventilators.

Our traditional Hadoop environments could never adapt to that changing set of requirements so quickly. With BigQuery, we were able to get the system up and running in under a month. That program, now called Data for Good, won both awards: the HPE International Association of Privacy Professionals’ Privacy Innovation of the Yearaward for 2020 and Social Impact & Communications and Service Providers Google Cloud Customer awardfor 2021. TELUS’ Data for Good program is supporting other areas of social good, in no small part because of the architectural benefits of having built on BigQuery and Google Cloud.

Ready to unleash the power of our data with Google Cloud

BigQuery is a key enabler of TELUS Insights, enabling us to shift from a slow, consultative approach to a more adaptive data-as-a-service model that makes our platform and valuable data more accessible to our customers. 

Moving to BigQuery led to major improvements in performance, reducing some of our initial queries from months of runtime to hours. Switching to a cloud-based solution with exceptionally high performance also made it easier for us to create an API to serve our commercial customers and enabled us to offer a key service, in a time of crisis, to the community with our Data for Good program

To learn more about TELUS Insights, or to book a consultation about our products and services, visit our website.

When we built our TELUS Insights platform, we worked with leading industry experts in de-identification. In addition, TELUS has taken a leadership role in de-identification and is a founding member of the Canadian Anonymization Network, whose mission is to help establish strong industry standards for de-identification. The TELUS de-identification methodology and, in fact, our whole Insights service, has been tested through re-identification attacks[1] [2] , stress-tested and, importantly, it has been Privacy by Design Certified. Privacy by Design certification was achieved in early 2017 for our Custom Studies product, and in early 2018 for our GeoIntelligence product.

Related Article

Unleashing the power of BigQuery to create personalized customer experiences

BigQuery’s high performance drives real-time, actionable decision-making that enables Wunderkind to bring large brands closer to their cu…

Read Article

Source : Data Analytics Read More

Access modeled data from Looker Studio, now in public preview

Access modeled data from Looker Studio, now in public preview

In April, we announced the private preview of our integration between Looker and Looker Studio (previously known as Data Studio). At Next in October, to further unify our business intelligence under the Looker umbrella, we announced that Data Studio has been renamed Looker Studio. The products are now both part of the Looker family with Looker Studio remaining free of charge. At Next we also announce that the integration between these two products is now available in public preview with additional functionality.

How does the integration work?

Customers using the Looker connector will have access to governed data from Looker within Looker Studio. The Looker connector for Looker Studio makes both self-serve and governed BI available to users in the same tool/environment. When connecting to Looker, Looker Studio customers are able to leverage its semantic data modelwhich enables complex data to be simplified for end users with a curated catalog of business data, pre-defined business metrics, and built-in transformations. This helps users make calculations and business logic consistent within a central model and promotes a single source of truth for their organization. 

Access to Looker-modeled datawithin Looker Studio reports allows people to use the same tool to create reports that rely on both ad-hoc and governed data. They can use LookML to create Looker data models by centrally defining and managing business rules and definitions in one Git, version-controlled data model.. 

Users can analyze and rapidly prototype ungoverned data (from spreadsheets, csv files, or other cloud sources) within Looker Studio and blend governed data from Looker with data available from over 800 data sources in Looker Studio to rapidly generate new insights. They can turn their Looker-governed data into informative, highly customizable dashboards and reports in Looker Studio and collaborate in real-time to build dashboards with teammates or people outside the company. 

What’s new in the public preview version?

We are excited that we are now able to offer this preview to a broader reach of customers, many of whom have already asked for access to the Looker connector for Looker Studio. Additionally, with this Public Preview, additional capabilities have been added to more fully represent the Looker model in Looker Studio:

We are providing support for field hierarchies in the Looker Studio data panel, to keep fields organized when working with large Explores. The data panel will now show a folder structure, and you will be able to see your fields organized in the usual ways – for Views, Group Labels, and Dimension Groups. 

We are providing greater visibility by exposing field descriptions in new ways to enable users to quickly check the description information specified in the Looker model. Field descriptions will be available within the data panel and within tables in the report.

Users will also see an option to “Open in Looker Studio” from Explores in Looker, enabling them to quickly create a Looker Studio report with a data source pointing back to that Explore.

And to ensure users are getting the most current data from the underlying data source, refreshing data in Looker Studio now also refreshes the data in the Looker cache. 

Specifically, for this public preview, we’ve implemented enhanced restrictions on Looker data sources in Looker Studio, so admins can rest easy about testing out the functionality:

We’ve disabled owner’s credentials for Looker data sources in Looker Studio, so each and every viewer needs to supply their own credentials including for shared reports.

We’re also currently disabling data download and email scheduling for these data sources in Looker Studio. We’re planning to integrate with these permissions in Looker in the near future.

Calculated fields are disabled, so end users cannot define their own custom metrics and dimensions in Looker Studio, and need to rely on the fields defined in the Looker Explore. 

How do I access the preview?

This integration encompasses the connector along with changes made to both Looker Studio and Looker to represent the Looker model and extend Looker governance in Looker Studio. There is much more to come as we continue our efforts to bring together a complete, unified platform balancing self-service and governed BI. We’re planning to continue adding functionality in Looker Studio to fully represent the Looker model, and want to ensure Looker admins have insight into API activity coming from Looker Studio – similar to the way they might use System Activity in Looker today. In extending governance, we want to expand the circle of trust from Looker to Looker Studio, and we’ll be looking for customers to help us plan the best way forward. 

This integration is compatible with Google Cloud hosted instances with Looker version 22.16 or higher. To get access, an admin of a Looker instance can submit the sign-up form providing an instance URL and specifying which organizational domain to enable. For more information on how to get started go to the Looker Studio Help Center.

For more information and demo, watch the Next ‘22 session ANA202: Bringing together a complete, unified BI platform with Looker and Data Studio and Keynote: ANA100: What’s new in Looker and Data Studio.

Source : Data Analytics Read More

Discover why leaders need to upskill teams in ML, AI and data

Discover why leaders need to upskill teams in ML, AI and data

Tech companies are ramping up the search for highly skilled data analytics, AI and ML professionals, with the race to AI accelerating the crunch.1 They are looking for cloud experts  who can successfully build, test, run, and manage complex tools and infrastructure, in roles such as data analysts, data engineers, data scientists, and ML engineers. This workforce takes vast amounts of data and  puts it to work solving top business challenges, including customer satisfaction, production quality and operational efficiency. 

Learn about the business impact of data analytics, ML and AI skills

Find out how Google Cloud ML, AI and data analytics training and certification can empower your team to positively impact operations in our latest IDC Business Value Paper, sponsored by Google. Key findings include:

69% improvement in staff competency levels

31% greater data accuracy in products developed

29% greater overall employee productivity

Download the latest IDC Business Value Paper, sponsored by Google, “The Business Value of Google Cloud ML, AI, Data Analytics Training and Certification.” (#US48988122, July 2022).

Google Cloud customers prioritize ML, AI and data training to meet strategic organizational needs

Our customers are seeing the importance and impact of data analytics, AI and ML training on their teams and business operations. 

The Home Depot (THD) upskilled staff on BigQuery to derive business insights and meet customer demand, with 92% reporting that training was valuable, and 75% confirming that they used knowledge from their Google Cloud training on a weekly basis.2

THD was challenged with upskilling IT staff to extract data from the cloud in support of efficient business operations. Additionally, they were working on a very short timeline (weeks as opposed to years) to train staff to enable a multi-year cloud migration completion. This included thousands of employees and a diverse range of topics. Find out how they successfully executed this major undertaking by developing a strategic approach to their training program in this blog.

LG CNS wanted to grow cloud skills internally to provide a high level of innovation and technical expertise for their customers. They enjoyed the flexibility and ability to tailor content to meet their objectives, and have another cohort planned.3

Looking to drive digital transformation and solution delivery, LG CNS partnered with Google Cloud to develop a program that included six weeks of ML training through the Advanced Solutions Lab (ASL). Read the blog to learn more about their experience.

Gain the latest data analytics, ML and AI skills on Google Cloud Skills Boost

Discover the latest Google Cloud training in data analytics, ML and AI on Google Cloud Skills Boost. Explore the role based learning paths available today which include hands-on labs and courses. Take a look at the Data Engineer, ML Engineer, Database Engineer and Data Analystlearning paths today for you and your team to get started on your upskilling journey. 

To learn about the impact ML, AI and data analytics training can have on your business, take a look at the IDC Business Value Paper, available for download now.

1. Tech looks to analytics skills to bolster its workforce
2. THD executed a robust survey directly with associates to gauge the business gains of the training program. Over the course of two years, more than 300 associates completed the training delivered by ROI Training.
3. Google Cloud Learning services’ early involvement in the organizational stages of this training process, and agile response to LG CNS’s requirements, ensured LG CNS could add the extra week of MLOps training to their program as soon as they began the initial ASL ML course.

Source : Data Analytics Read More