Building your first pipeline can be a daunting task for a developer. When there are so many tools that can get the job done, how should you go about getting started? For power, flexibility, and ease of use, we find that the combination of Apache Beam and Dataflow offers developers a variety of ways to create data processing pipelines that meet their unique needs. And to make it easier, we’ve built a sample e-commerce analytics application to provide a reference implementation that works for developers who are starting out with Apache Beam as well as those who are proficient but may have advanced problems they need to solve. In some cases the tasks can be accomplished in many valid ways; however, we’ll focus on what we think is the easiest path. 

This blog will cover some of the patterns used in the sample application, for example the initial processing of the clickstream data from server-side Google Tag Manager. The sample application can also be used to find examples of working with JSON and Apache Beam Schemas for both de-serialization and serialization within the pipeline. 

Note: while the application is only implemented in Java, many of the patterns in this post will also be useful for other SDK languages. 

About the sample application

Our sample application implements processing of clickstream data from Google Tag Manager and shows how to dynamically respond to customer actions by analyzing and responding to events as they happen. The application also uses BigQuery as a sink which is updated in real time, and stores, analyzes, and visualizes event data for strategic decision making. Like most real world enterprise applications, it interacts with other products. You’ll find we also use PubSub as a source and Cloud Bigtable as a sink. 

Clickstream processing using Dataflow

Imagine you’re the developer for a retail company that has an e-commerce site, a mobile application, and brick-and-mortar stores. For now, let’s focus on the e-commerce and mobile applications, which send clickstream events to a web tier and then forward this onto PubSub via Server-side Tagging (GTM). Server-side Tagging allows customers to use a single Google tag on their website to collect the superset of clickstream events needed to enable all of their measurement tools, then send this data to a customer-owned server, and finally fan it out to numerous destinations, such as Google Analytics and other third-parties, or other data storage and dissemination tools such as BigQuery or PubSub. (Learn how to get started with server-side GTM here.)

The architecture diagram below illustrates our integration with server-side Tagging:

The events are published to PubSub and in JSON format, such as the following example:

Now that we know where our clickstream data is coming from and where it’s going, we’ll need to accomplish two tasks to make it useful: 

Validate and correct the data

Create session windows to ensure activities are grouped correctly

First task: Validate and correct the data 

Just because data gets parsed doesn’t mean that the data types and schema were the right match for our business domain. They could have missing properties or inconsistencies, which is why a good data pipeline should always include validation and correction. We’ve modeled these as separate transforms in our retail application, which allows for clean data to simply pass through the validation step.

There are a few common approaches to data validation when it comes to the pipeline shape. We’ll illustrate some options, along with their pros and cons, and then show how we decide to do it within our retail application.

Let’s use an object Foo which has properties a,b for example: {Foo:{a:’bar’,b:’baz’}}. The business has validation logic that needs to check the values of a and b properties for correctness.

Approach A: Single transform

In this approach, the validation and correction code is all built into a single DoFn, where items that cannot be corrected are passed to a deadletter queue. 

Pros: This is a straightforward approach that will work very well for many cases. 

Cons: This Beam transform is specific to Foo and does not lend itself well to a transform library, which could be reused in different parts of the business. For example, property b might be something that is used in many objects within the retail organization, and this approach would potentially involve duplicate efforts down the line.

Approach B: Router validation transform and branch correction

In this approach, different properties (or groups of similar properties) are validated in one transform and then corrected in other transforms.

Pros: This approach allows you to integrate the transforms into a larger transform library. 

Cons: Compared to Approach A, this requires writing more boilerplate code. Moreover, there might also  be a performance disadvantage to this approach (although Dataflow fusion will make it fairly negligible). Another disadvantage to this approach is that if the element requires both property a and b to be corrected, then you’ll need a join operation downstream to bring back together all parts of the element. A join will require a shuffle operation (which is a redistribution of data based on keys), which, since it can be a relatively slow and expensive operation, is good to avoid if possible.

Approach C: Serial validation and correction transforms

In this approach, the element passes through a series of validation steps, followed by a series of correction steps. Using a multi-output transform, any element which has no errors is passed directly to the transformations downstream. If an element does have errors, then that element is sent on to correction transformations ( unless it is deemed not fixable, in which case it is sent to the deadletter sink). Thanks to Dataflow fusion, these elements can all be efficiently processed through all of these transformations.

While there are many more variations of this process (and a combination of approaches A and C can solve the majority of complex real-life scenarios), we’ll be using Approach C for our retail application, since it offers the most code to illustrate the process. Now that we know what we want to do with our element properties, let’s walk through an example that we’ll want to check and correct if needed:

Event DateTime

Every clickstream object will have an Event DateTime property attached to it. As this date is set client-side in our application, there are several issues that could arise that the business would like to check:

Date format: the string itself should be correct. Since this property is a string in the schema, it would not have failed, even if the date format is incorrect.

Date accuracy: the date should not be in the future. Given that this date is coming from the client-side, it could be incorrectly set. How to deal with incorrect date-times is a choice to be made by the business. For our purposes we’ll correct the time to be set to when the element was published to PubSub (the processing time), as opposed to when it was first seen by the pipeline (the event time).

Note that we’re not correcting the event datetime field itself, but rather the timestamp property of the clickstream event. This lets us keep the original data intact, while still ensuring that  any resulting analytics with this  data will use the correct time values.  

Items
Certain events, such as add_to_cart and purchase, both require the Items array to be populated with at least one item. These items normally have all the fields correctly populated, however, there have been situations where this was not done for some of the descriptive fields, such as item_name, item_brand, and item_category. If any of these are not populated, they need to be fixed. 

Implementation
A serial set of transforms can help check whether or not an element needs to be corrected. To do this, we’ll need a way to tag elements with problems. Adding a new property to the ClickStreamEvent element and all other elements that require validation is one way to go, however, this changes the element itself and will show up in any output down stream (e.g., if the element is passed to BigQuery). Instead, we decided to wrap the element in another schema, one which includes an array to tag elements with problems.

In the above code snippet, the errors field lets us add a list of problems to our clickstream, which is then checked for within the correction transforms. ValidateAndCorrectCSEvt transform is then used to compose all of the validation and correction transforms. A section of this is shown below. 

Note that we make use of Row.withSchema to create a new row element. Also note that PCollectionTuple is not able to infer the row schema, therefore we use withSchema to explicitly set the row schema.

Once the validation is complete, we’re ready to put the corrections into place:

For items which are not fixable they will follow the deadletter pattern and be sent to a sink for future correction, which is often a manual process.

Another modular pattern that could have been used is to create a DoFn in a way that  allows injecting multiple ‘verifiers’/’correctors’. The verifiers/correctors could be a library (i.e. not a PTransform) that is reused across the system. Only a failed corrector would trigger the object to the deadletter queue. 

With that step, we finish validation and correction, and can move on to creating session windows.

Second task: Creating Session windows

Session windows are a special type of data window, which, unlike fixed windows (which rely on discrete time intervals), make use of the data itself to determine when a window should be closed. From the Apache Beam documentation:

“A session window function defines windows that contain elements that are within a certain gap duration of another element.” 

To understand what this means, let’s imagine a customer logs into your company’s mobile shopping application in the morning before work and orders items to be delivered which will be used for the evening meal. Later on in that day, the customer logs in and browses some sports equipment to use over the weekend. Other windowing options (for example, a fixed window), might miss part of a session, or would require you to manually sort through the data to find the time gap and define these sessions. But with a session window, we can treat these two activities as two distinct sessions. As there is a long delay between the end of the morning activities and the start of the evening activities (the ‘gap’ duration), the session window will output the activities from the morning as distinct sessions and start a new session when the user begins the evening activities.

Note that we specify the value of the gap duration when we apply the window transform to our pipeline. While there’s no perfect value for this duration, in practice, something in the range of low single-digit hours works reasonably well.

We can use Apache Beam to create such session windows, and the code is fairly simple as you’ll see below (ClickStreamSessions). We’ll apply a session window to our validated and cleansed data, then do a group.by operation to locate all the different session events together. The grouping key for this will be the client_id which is provided by Tag manager.

The ITERABLE[ROW[ClickstreamEvent]] will hold all the events for that session and can be used to carry out further analysis work. For example, you might want to study what sequence of browsing events ultimately led to the purchase. This can then be found in the transform ClickStreamSessions within the retail application.

Conclusion

We covered a few core concepts and patterns around data validation and correction that are necessary for a healthy streaming pipeline by making use of clickstream data from Google Tag Manager. We also touched on how session windows can help separate the different journeys for a user and segment your data in meaningful ways. 

In our next post, we’ll explore data ingestion flows and how to use Apache Beam schemas to both represent the elements within a pipeline. We’ll also look at how to use schemas to write succinct, manageable pipelines. In the meantime, let us know how these approaches are helping you with your day-to-day workflows, and what else you’d like to learn about Dataflow pipelines and applications.

As customers grow their analytical workloads and footprint on BigQuery, their monitoring and management requirements evolve – they want to be able to manage their environments at scale, take action in context. They also desire capacity management capabilities to optimize their BigQuery environments. With our BigQuery Administrator Hub capabilities, customers can now better manage BigQuery at scale. Two key features of BigQuery Administrator Hub are Resource Charts and Slot Estimator which help administrators understand their BQ environments like never before.

Resource Charts empowers Administrators with a native out-of-the-box experience to monitor their slot usage, manage capacity based on historical consumption, troubleshoot job performance, and self diagnose queries, and take corrective action as needed. They provide visibility into key metrics such as slot consumption, job performance, concurrency, bytes processed and failed jobs.  Resource Charts are built and rendered usingINFORMATION_SCHEMA tables, enabling customers to understand data through these purpose-built dashboards or query the data directly to build their own dashboards and monitoring processes.

BigQuery customer Snap is an early adopter of Resource Charts; “Admin Resource Charts is a great tool that helps us understand how we’re consuming our slots and which workloads/queries are driving usage. It has provided us better visibility into our BigQuery environment”—Muthu Hariharasubramanian, Engineering Manager, BigData Infrastructure, Snap, Inc. 

Slot Estimator is an interactive capacity management tool that helps administrators estimate and optimize their BigQuery capacity based on performance. This tool helps customers make informed decisions on capacity planning based on historical usage. It also helps customers to estimate and optimize their capacity based on their workloads and performance. 

Paypal is a preview customer of Slot Estimator; “Slots Estimator is just amazing and a differentiator for BigQuery. Our trials with this feature have shown very good results in predicting slot requirements for critical analytical workloads.”—Bala Natarajan, Sr. Director of Data Infrastructure and Cloud Engineering, PayPal 

Let’s look at a day in the life of an BigQuery Administrator and see how these various features can help you: When you come in the morning and log into the BigQuery UI, you will see Administrator Hub as the central home to understand, manage, and monitor your queries, capacity, and BQ environments.

As you are monitoring your environment real time in Resource Charts, you see a decline in slot usage a couple of hours later and decide to investigate further.

You can look at the new Errors chart and see a sharp increase in permission denied and invalid errors. You can investigate these errors further using filters such as projects, reservations, users and job priorities to understand what changed since morning and fix them so you utilize the slots efficiently

Later in the day, a Data Analyst comes to you when they notice that their jobs have been running slower gradually over the week. Using Resource Charts, you see that slot utilization has hit the max capacity. On further drill down, you learn that ramping up a new workflow has caused a steady increase in slots and all the slots are fully utilized constantly.

You can now switch over to the Slot Estimator tab and see a similar full slot utilization view at 100% and how the slot usage has ramped up over the week. You can look at the reservation data and analyze how much you can improve performance by adding different amounts of slots. Once you decide to add slots, you can directly buy slots for a specific reservation in-context.

Resource Charts is generally available and Slot Estimator is available in preview for customers using Reservations. We hope these Administration capabilities help you better monitor and manage your BigQuery workloads at scale! 

We’re excited to announce that Datastream, Google Cloud’s serverless change data capture (CDC) and replication service, is now generally available. Datastream allows you to synchronize data across disparate databases, storage systems, and applications reliably and with minimal latency to support real-time analytics, database replication, and event-driven architectures. You can easily and seamlessly deliver change streams from Oracle and MySQL databases into Google Cloud services such as BigQuery, Cloud SQL, Google Cloud Storage and Cloud Spanner, saving time and resources and ensuring your data is accurate and up to date. Get started with Datastream today.

Since our public preview launch earlier this year, we’ve seen Datastream used across a variety of industries, by customers such as Chess.com, Cogeco, Schnuck Markets, and MuchBetter. This early adoption strengthens the message we’ve been hearing from customers about the demand for change data capture to provide replication and streaming capabilities for real-time analytics and business operations. 

MuchBetter is a multi-award-winning e-wallet app, providing a truly secure and enjoyable banking alternative for customers all over the world. Working with Google Cloud Premier Partner Datatonic, they’re leveraging Datastream to replicate real-time data from MySQL OLTP databases into a BigQuery data warehouse to power their analytics needs. According to Andrew McBrearty, Head of Technology at MuchBetter, “from MuchBetter’s point of view, leveraging Dataflow, BigQuery and Looker has unlocked additional insights from our ever-increasing operational data. Using Datastream in our solution ensured continued real-time capability – we now have trend analysis in place, improved efficiency across the business, and the ability to use our data to derive actionable insights and to make data-driven decisions. This means we can continue to grow and adapt at a pace our customers have come to expect from MuchBetter. And for the first time, the world of ML and AI is open to us.”

Getting to know Datastream

Google Cloud customers are choosing Datastream for real-time change data capture because of its differentiated approach:

Simple experience
Real-time replication of change data shouldn’t be complicated: database preparation documentation, secure connectivity setup, and stream validation should be built right into the flow. Datastream delivers on this experience, as MuchBetter discovered during their evaluation of the product. “Datastream’s ease-of-use and immediate availability (serverless) meant we could start our evaluation and immediately see results”, says Mark Venables, Principal Data Engineer at MuchBetter. “For us, this meant getting rid of the considerable pre-work needed to align proof of concept tests with third-party CDC suppliers.”

End-to-end solution
Building pipelines to replicate changes from your source database shouldn’t take up all of your team’s time. Use pre-built Dataflow templates to easily replicate data into BigQuery, Cloud Spanner or Cloud SQL. Out of the box, these Dataflow templates will automatically create the tables and update the data at the destination, taking care of any out-of-order or duplicate events, and providing error resolution capabilities. Leverage the templates’ flexibility to fine-tune Dataflow to fit your specific needs. “Google-managed Dataflow templates meant getting our pipelines up and running with minimal effort and fuss – this allowed more time to be spent on more complex pipeline development whilst tactically delivering solutions to our users,” says Venables.

Secure 
Datastream keeps your migrated data secure, supporting private connectivity between source and destination databases. “Establishing connectivity is often viewed as hard. Datastream surprised us with its ease of use & setup, even in more secure modes,” says Grzegorz Dlugolecki, Principal Cloud Architect at Chess.com, a leading online chess community and mobile application, hosting more than ten million chess games every day. “Datastream’s private connectivity configuration allowed us to easily create a private connection between our source and the destination, and ensure our data is safe and secure.”

High throughput, low latency
With Datastream’s serverless architecture, you don’t need to worry about provisioning, managing machines, or scaling up resources to meet fluctuations in data throughput. Datastream guarantees high performance – a single stream can process 10’s of MBs per second, while ensuring minimal latency. “We evaluated several market-leading ETL solutions”, says Dlugolecki,  “Datastream was the only tool able to successfully sync our complex, single-table datasets, doing this in weeks instead of years estimated by the other vendors.”

Getting started with Datastream

You can start streaming real-time changes from your Oracle and MySQL databases today using Datastream:

Navigate to the Datastream area of your Google Cloud console, under Big Data, and click Create Stream.

Choose the source database type, and see what actions you need to take to set up your source.

Create your source connection profile, which can later be used for additional streams.

Define how you want to connect your source.

Create and configure your destination connection profile.

Validate your stream and make sure the test was successful. Start the stream when you’re ready.

Once the stream is started, Datastream will backfill historical data and will continuously replicate new changes as they happen. 

Learn more and start using Datastream today

Datastream is now generally available for Oracle and MySQL sources. Datastream supports sources both on-premises and in the cloud, and captures historical data and changes into Cloud Storage. Integrations with Cloud Data Fusion and Cloud Dataflow (our data integration and stream processing products, respectively) replicate changes to other Google Cloud destinations, including: BigQuery, Cloud Spanner, and Cloud SQL.

For more information, head on over to the Datastream documentation, see our step-by-step Datastream + Dataflow to BigQuery tutorial, or start training with this Datastream Qwiklab.

One of the biggest challenges data executives have today is turning the immense amount of information that their organization, customers and partners — or rather their whole ecosystem — are creating into a competitive advantage. 

In my role here at Google Cloud, I specialize in everything data — from analytics, to business intelligence, data science and AI. 

My team’s role is split into 3 main activities:

Engagement with customers and partner community. About 70% of my time is spent with customers. And it’s where I’ve gathered all these insights that I’m going to share with you today.  

Product strategy and execution. This time is for strategizing and planning around all our new Cloud launches and products.

Go-To-Market globally. This is where we ask all the tough questions: How do we make it easier for our customers to onboard? And get the most out of our services? To transform and innovate? And then we solve for them.

It’s safe to say data-driven transformation is my bread and butter. And I want it to be yours too. My aim is to help people think about data in a new way — not something to be afraid of, but something to leverage and grow with. There are still lots of problems to be solved in our industry. But data is helping us unlock a world of opportunities. I talked with Stephanie Wong, Google Cloud’s Developer Advocate, and we’d like to share some key learnings that can help you plan for change and transform your business with your data.

What modern data architectures look like today

There’s a treasure trove of new technologies that are transforming the way companies do business at incredible speeds. I think of companies like Paypal, which migrated over 20 petabytes of data to serve its 3,000+ users, and Verizon Media, which ingested 200 terabytes of data daily and stored 100 petabytes in BigQuery. Even traditional retailers like Crate & Barrel are making strides in the cloud, doubling their return-on-ad-spend (ROAS) while only increasing investment by 20%.

But what do these companies all have in common? A modern approach to their data practices and platforms. And there are three attributes that I think all organizations should take into account: 

1. Embrace the old with the new.  Every single one of the most important brands on earth has legacy systems. They’ve developed leadership over decades and these systems (before the cloud came along) got them there.

2. Don’t discard what’s going to get you there (i.e multi-cloud). All modern architectures today are multi-cloud by default. According to Flexera, over 80% of businesses reported using a multi-cloud strategy this year and over 90%  have a hybrid  strategy in place.

3. Data is no longer a stagnant asset. Organizations that win with data think about it as part of an ‘ecosystem’ of opportunity, where insights arise from emerging data — whether it be from interconnected data networks or the data from their partners. And this is a trend organizations should keep their eye on. A study from Gartner predicts that by 2023, organizations that promote data sharing will outperform their peers on most business value metrics.

How to make the best hires for your data team

Leaders often say that their competitive advantage comes from their people, not just services or products. While most companies are now recognizing the importance of data and analytics, many still struggle to get the right people in place. 

The best way to look at how many data people to hire is to ask yourself, what percentage of my total employee base should they make up? I agree with Kirk Borne, Chief Scientist Officer at DataPrime Solutions, who says that your entire organization should be ‘data literate’. And when we say literate, we mean recognize, understand and talk data. 

One third of your company should be ‘data fluent’ — meaning able to analyze and present informed results with data. And finally, 10% of employees should be ‘data professionals’ that are paid to create value from data. That’s where all your chief scientists, data analysts, engineers and Business Intelligence specialists come into play.

The ideal data team structure of course depends on the type and size of the company. Furniture and home e-commerce company Wayfair for instance has approximately 3,000 engineers and data scientists — close to 18% of its total workforce. 

Who should own the data? 

There are a lot of questions around who data leaders should work for and who should own that data. It’s  tough to answer because there are so many choices. Should it be the CTO? Or the CFO, whose initiatives are around cost reduction? Or the CPO, who may focus on product analytics only? 

When asking customers at scale, it’s typically under the CFO or CTO. And while that makes sense, I think there’s something else we should be asking: How should data be approached so that companies are enabled to innovate with it?

A trend we’re hearing a lot more about is data mesh. This data ownership approach basically centralizes data and decentralizes analytics through ‘data neighborhoods.’ This allows business users and data scientists to access, analyze, and augment insights, but in a way that’s connected to the centralized strategy and abides by corporate rules and policies.

Data neighborhoods

Data: 2022 and beyond

Data analytics, data integration and data processing can be very complex, especially as we begin to modernize. So I’d like to leave you with a ‘gotcha’ moment — and that’s data sharing. 

You can’t expect to reap the benefits of data instantly. First you have to work with it, clean it up and analyze it. The real innovators are those looking at the wider picture — considering analytics solutions and sharing and combining datasets.

My advice for people who want to get started? Forget the notion of new and existing use cases and focus on business value from day one. How are you going to measure that? And how are you sharing that with leaders that are supporting your initiative? 

Data is constantly growing and trends are always shifting. So we need to stay on our toes. Data-driven transformation gives businesses real-time insights and prepares you for the unpredictable. So looking forward to 2022, I’d say use data to plan for change and plan for the unexpected.

A data cloud offers a comprehensive and proven approach to cloud — allowing you to increase agility, innovate faster, get value from your data and support business transformation. Google Cloud is uniquely positioned to help businesses get there. Learn how. 

Founded in 2009, Tokopedia is an ecommerce platform that enables millions of Indonesian to transact online. As the company grows, there is an urgent need to better understand customer’s behavior in order to improve the customer’s experience across the platform. Now, Tokopedia has more than 100 million Monthly Active Users and the demography and preferences of all these users are different. A way to meet their needs is through personalization. 

Normally, a user needs to browse through thousands of products in order to find the item they are looking for. By creating product recommendations that are relevant to each user, we shorten their search journey and hopefully increase conversion early on in the journey. In order to build personalization, the Data Engineering Team’s Customer Data Platform (CDP) helped to gain access to user’s attributes. These attributes developed by the Data Engineering team come in handy for different use cases across functions and teams.

Previously, two main challenges were observed:

The need for speed and answers caused an increase in data silos. As the needs for personalization increased across the company, different teams have been building their own personalization features. However, the limited time and the need to simplify communication across teams have resulted in the decision for each team to create their own data pipeline. This caused a few redundancies due to the development of similar data across different teams and these redundancies caused slower development time for new personalized feature, even though some of the attributes have been previously build in a different module.

Inconsistent data definitions. As each team created their own data pipeline, there are many cases where each team had a different definition of a user’s attributes. On several occasions, this caused misunderstandings during meetings and unsynchronized user journeys due to different teams applying different attribute values to the same user. For example, team A evaluated user_id 001 as a woman in their 20s. Meanwhile, team B, having a different set of attributes and definitions evaluated user_id 001 as a woman in their 30s. These differences in definition and attributes can lead to different conclusions and results, consequently giving different personalizations. As a result, customers might be facing inconsistent experience during their journey in Tokopedia and have a bad experience during their activity. Imagine that you’re being displayed by one set type of content that is related with college necessities and then in a different module you’re being given a a content that is related to mom and baby.

Currently, with CDP, different teams do not have to constantly rebuild the infrastructure. The same attributes will only need to be processed once, and can be used by different teams across the company. This optimizes the development time, cost, and effort. Another advantage of having CDP is the single definition of attributes across services and teams. Since different teams will be looking at the same attributes inside the CDP, this will reduce the chances of misunderstanding and strengthen synchronization between teams. This will give customers consistent experience across the Tokopedia platform and enable them to display relevant contents.

Moreover, there are several key factors required in building the CDP platform in Tokopedia. The journey is as follows:

1. Define and Make a List of Attributes
During this phase, we work with the Product and Analyst teams to define all of the user’s attributes required to build the CDP. Our product team interviewed several stakeholders to understand different perspectives regarding user attributes. As a result, an initial attributes list was made to include gender, age group, location, etc. This process is done repetitively in order to have the best understanding of the user’s attributes.

2. Platform Design
After doing comprehensive reviews, we decided to build our CDP platform using several GCP tech stacks.

Bigquery was chosen as the analytics backend of our CDP self-service. Meanwhile, Google Cloud BigTable was selected as the backend, where our services will interact to enable the personalization. In developing the storage for Big Table, the design of the scheme is very important. The frequency and categorization will affect how we design the column qualifier while the CDP attribute will affect how we design the row key.

We also opted to create a caching mechanism to reduce the load to big tables for similar read activity. We build the cache system using redis with certain Time to Live (TTL) to ensure an optimized performance. In addition, we also applied a Role Based Access Control (RBAC) mechanism on the CDP API to ensure access control of different services towards attributes in the CDP.

3. Monitoring and alerting
Another important point in building a CDP is developing the correct monitoring and alerting system to maintain stability on our platform. A soft and hard threshold on each metric is established and monitored. Once this threshold is reached, some alerts will be sent through the communication channel. Based on the current architecture, there are several parts in which we need to enable monitoring and alerting. 

Data Pipeline
One of the things that we will need to monitor is resource consumption during computation and data pipeline from data sources to the CDP storages, as we operate using Bigquery and Dataflow for Data Computation and Data Pipeline. In Bigquery, we need to monitor the slot utilization that is used to compute some data aggregation or manipulation to produce the attribute. 

Data Quality
When building the CDP, high quality data was important in order for it to be a trusted platform. Several metrics that are important in terms of data quality are Data Completeness, Data Validity, Data Anomaly and Data Consistency. Therefore, several monitoring needs to be enabled to ensure these metrics.

Storage and API Performance
Since CDP’s backend and API directly interact with several front facing features, we have to ensure the availability of the CDP service. Since we’re using Big Table as the backend, the monitoring of CPU, Latency and RPS is required. This metric, by default, is provided in the Bigtable monitoring.

4. Discoverability across company
Many users have been inquiring on how they can browse attributes that our CDP offers. Initially, we started out by documenting our attributes and sharing it to our stakeholders. However, as the number of the attributes increased, it became increasingly harder for people to go through our documentation. This pushed us to start integrating the CDP terminology into our Data Catalog. In this case, our Data Catalog plays an important role in enabling users to browse attributes in CDP, including the definition of each attribute and how they can retrieve the data.

5. Implementation and adoption of the platform 
Another key point for a successful CDP implementation is collaboration across teams on the front end services. There are several types of CDP implementation in Tokopedia: Personalization, Marketing Analytics, and Self Service Analytics.

Personalization
The most common usage of CDP would be in personalizing a user’s journey. One example of personalization is the search feature. The product team personalizes the user’s search result based on the user’s address, so that the user will be able to find products that are in proximity to their location. After discussing the definition of user address, we created a CDP API contract with the Search team, so the development can run in parallel. As a result, today our users are able to have a better user experience based on their location.

Marketing Analytics
When we started building the CDP platform, we discussed with the Marketing team on their existing use cases. One of their goals was to personalize and optimize marketing efforts, such as sending out notifications to the right user based on the user’s attributes to reduce unnecessary notification costs to unrelated users, and to enhance the overall user experience by avoiding spam notifications. Once we understood their needs, we looked at the ways in which CDP could cater to those needs. We discussed with the relevant team on how to integrate the segmentation engine and communication channel towards the CDP platform, the type of user attributes to use when sending marketing push/notifications, and how to integrate it with the segmentation engine and communication channel of the CDP platform.

Self-Service Analytics
CDP also often uses self-service analytics to enable quick insights on user demographics and behavior in certain segments. To build this self-serve analytics tool, our team consulted with the Product and Analyst teams to define the user demographics’ attributes that business/product users often select for insights. After understanding the attributes required, we discussed with the Business Intelligence team to enable the visualization for the end user. This allowed different teams to understand our users better and gain insights on how we can improve our platform.

CDP implementation has created a significant impact on different use cases and helped Tokopedia to be a more data-driven company. Through CDP, we are also able to strengthen one of our core DNA, which is Focus on Consumer. By sharing the CDP framework, we hope to bring value and help others to more easily create a thriving CDP platform.

Pub/Sub stores messages reliably at any scale. Publish a log entry or an event and you don’t have to worry about when it is processed. If your subscribers, event handlers, or consumers are not keeping up, we’ll hold the messages till they are ready. Bugs made it past your integration test? Not to worry: just seek back and replay. 

But, you had to worry a little: until today, you had up to a week to fix the code and process everything. And if you wanted to use historical data to compute some aggregate state, such as a search index, you had to use a separate storage system. In fact, we noticed that many of our users stored raw copies of all messages in GCS or BigQuery, just in case. This is reliable and inexpensive, but requires a separate reprocessing setup in case you actually need to look at older data. 

Starting today, Pub/Sub can store your messages in a topic for up to 31 days. This gives you more time to debug subscribers. This also gives you a longer time horizon of events to backtest streaming applications or initialize state in applications that compute state from an event log. 

Using the feature is simple. The interfaces and pricing are unchanged. You can just set a larger value for a topic’s message retention duration. For example, you can configure extended retention of an existing topic using the gCloud CLI: 

gcloud pubsub topics update myTopic –message-retention-duration 31d

Or use the settings in the Topic Details page in Cloud Console:

One limitation of this feature is that you cannot extend storage retention for an individual subscription beyond 7 days. This limits the control individual subscription owners have over storage. The limit comes with benefits: controlling storage costs is simpler and so is limiting access to older data across multiple applications.

We’d love to hear how you’ve used this feature or how it came short of your needs. Let us know by posting a message to the pubsub-discuss mailing list or creating a bug.