Make Google Analytics great again [discussion with Googlers]

The fact that traffic from Google Ads is counted as organic within the BigQuery export is not a bug; it was intentionally designed this way by Google’s product team. Therefore, we will need to continue to integrate Google Ads data in the GA4 export.

There won’t be a built-in attribution model within the BigQuery export.

They’ll investigate why some rollup properties only populate with data several days after collection.

There was a request to improve the GA4 course in Skillshop by integrating more general web analytics concepts, similar to what was done in the old course.

The idea of creating a kind of cookbook for custom reports within GA4 was discussed. I’m not entirely convinced about this idea, mainly because the request was made to replicate GA3 reports in GA4.

Then, a series of missing features in the platform’s UI were highlighted, including the ability to export report data in multiple formats, the option to copy table data with CTRL + C (not just right-click), and having the percentage weight of a cell compared to the total column, similar to how it was in GA3.

Automatize GA4 data workflow using Dataform and Terraform

My favourite session of the day was the one by Moritz Bauer, probably because I’ve recently started studying Dataform to understand how it works and how to use it.

He showed how he uses Dataform to create new views/table from GA4 raw data.

As you can see in the image above, Dataform can be used to prepare data for use as a source for dashboards or reporting in general.

He also shared a GitHub repository where he published the Terraform code he uses to automate the entire process.

All tables are automatically updated whenever Google loads GA4 data into the tables. From my perspective, this tool has tremendous potential and is still relatively unexplored.

I’m already conducting tests, and in a few months, I might publish something here explaining what I’m working on.

Bot traffic detection with Analytics Firewall

David Vallejo showcased Analytics Firewall, an application he’s developing with the goal of ensuring data collection is as accurate as possible.

In particular, he focused on how it attempts to determine whether GA4 hits are genuinely from a user or from bot traffic.

His concept is to have a system similar to anti-spam systems in emails.

The application performs a series of checks on IP addresses, geolocation, the number of hits sent, actions taken on the site, etc., to assess the likelihood that a user is a bot or not.

The score can then be used to tag hits or filter them out before sending them to the endpoint.

Git for data analysts

My colleague Claudio Ferrara spoke about Git and why its use is fundamental for a data analyst team.

The world of data is no longer a one-man band but more like an orchestra. That’s why it’s essential to have a system that allows everyone to share code and, most importantly, create variations without overwriting other people’s code.

Let’s make an experimentation dashboard

The Conductrics team then organized a brainstorming session to determine what should be displayed in a dashboard designed to showcase the success of an experimentation program.

One of the conclusions was that it’s impossible to measure the success of an A/B testing program with a single metric because there are various variables at play, and not all tests have a positive impact on revenue.

There was a discussion about the importance of learning from every test, even those that don’t lead to a winning outcome. For example, if the result of a test indicates that a particular feature should not be developed, the outcome might not be increased revenue, but rather resource savings that can be reinvested in other projects.

Another topic of discussion was the due diligence of ideas. Each idea should be carefully evaluated before implementation to avoid impulsive decisions based on someone’s immediate thoughts.

I attended other sessions as well, but I found these to be the most interesting. As always, it’s now time to roll up my sleeves and try out all the new things I learn and the new ideas I have.

The post MeasureCamp Stockholm 2023: my takeaways appeared first on Ruben Vezzoli.

My 6th Measurecamp was quite an adventure this time. My outbound flight was cancelled and I took 13 hours to reach London (compared to the usual 2 hours) because I had to make a stopover in Malaga. Fortunately, it was truly worth it.

Conductrics – Include Surveys & NPS in A/B Testing

I really liked the approach used by the folks at Conductrics in developing their platform. A/B testing doesn’t always mean simply identifying the variant that maximizes revenue. In fact, certain tests may have an indirect impact on revenue but still have a positive impact on the overall CX.

Here’s why the platform permits the integration of surveys, feedback, and NPS measurements within tests: it allows for a more comprehensive evaluation of the qualitative aspects of the tested variations.

A bunch of random GA4 tips & gotchas

Then I attended Julius Federovicious‘s speech, where he discussed some gotchas and workarounds he had created to make the most out of GA4. I’ll try to list below the ones that were most interesting to me.

He explained how to use the User Purchase Journey report and discussed why sometimes the report isn’t populated correctly. In such cases, he elaborated on why it’s necessary to recreate it within Explore.

As you can see in the video below, he explained how to monitor searches with zero results in GA4.

Furthermore, he explained why there are slight discrepancies in GA4 between the actual transaction value and the UI value.

GA4 takes all orders in EUR currency, converts them to USD, and then reconverts them to EUR within the UI, leading to minor discrepancies.

Additionally, he shared links to two GTM templates. One, developed by Eivind Savio, allows for the passing of promotion and item list name values across the different GA4 EEC events. This ensures proper attribution throughout the user journey when interacting with a promotion or item list.

The second template, created by Mohr Stade, enables the re-pushing of events in the data layer that had been pushed before the consent information was loaded.

How to better understand your visitors

Ton Wessling, on the other hand, delivered a speech/discussion where he aimed to define 10 tips for better understanding your users.

His main point was that the percentage of experiments yielding results is decreasing, so there is a need to focus more on generating new test ideas and prioritizing them effectively.

For instance, he recommended continuously monitoring and studying the tests that competitors are conducting. He also shared a Chrome extension that allows you to track which A/B tests a website is currently running (compatible with Optimizely).

He also talked about how he leverages AI, such as ChatGPT, to simplify his life. For example, he uses Scite Assistant to have scientific papers summarized, allowing him to concentrate more on the conclusions drawn by the authors.

He also feeds ChatGPT with competitor reviews to extract their strengths and weaknesses according to their customers.

I also liked the tips that came out of the discussion. In the end, to understand our customers better, there’s nothing like engaging with them, whether through surveys, listening to customer care, or examining what they and our competitors are saying about us. This allows us to identify new opportunities for optimization.

He also mentioned UsabilityHub. I haven’t had the time to explore it yet, but I definitely plan to do so.

How BBC Sport measure their content effectiveness

The talk given by Piano Analytics addressed a topic that has always been widely debated.

In fact, at the end of the speech, there were several considerations and discussions regarding the effectiveness and results of the method used.

In 2021, BBC Sport decided to help their journalists understand how to write more engaging content for users.

They took all their content and grouped it by the number of characters used. Once they identified the median (250 characters), they took data from various studies that indicate how fast we read and calculated that, on average, a user takes 15 seconds to read a BBC article.

They then began to collect events that were triggered after 5-10-15 seconds from the start of the user’s page reading.

They started monitoring how many users reached the various time thresholds for each article to understand which articles were actually being read in their entirety by users.

As seen in the chart below, they also calculated an estimated reading time next to each article (reading speed x number of characters in the article).

This was done because, naturally, they had articles longer than 250 characters, so analyzing only the first 15 seconds wouldn’t be sufficient to measure effectiveness.

In conclusion, they found, for example, that including fan impressions in articles appears to be a topic that greatly interests readers.

And it also seems that they have some very interesting trends regarding the time spent on the website.

How to audit 300 GTM Containers

Sébastien Monnier‘s speech was also fascinating. He utilized a workflow automation tool called N8N (which I haven’t tested yet, but I’ll definitely look into it) to download all his containers through GTM’s APIs and then load them into BigQuery.

By creating daily downloads, he examines what has changed and what isn’t working so that he can monitor numerous containers simultaneously using straightforward dashboards in Looker Studio.

As usual, a lot of interesting insights and talks. I can’t wait for this weekend’s Measurecamp in Stockholm.

The post MeasureCamp London 2023: my takeaways appeared first on Ruben Vezzoli.

As always, MeasureCamp far exceeded my expectations, not only in terms of the unconference content but also for the venue where it was held (the Allianz Arena!).

Web Analytics ≠ Google/Adobe Analytics: rethink your stack

For many years, Google Analytics and Adobe Analytics have been the leaders in the web analytics market.

However, it is important to remember that they are also big black boxes and not very flexible solutions, especially when it comes to collecting behavioural data for product analytics and AI purposes.

Nicolas explained how and why BurdaForward transitioned from Adobe Analytics (and other solutions) to Snowplow Analytics after a year of work.

Nicolas Ajaram talking about BurdaForward migration to SnowPlow

He discussed the various advantages of Snowplow, such as its flexibility, the ability to enrich data, and having control over the entire process.

Additionally, he highlighted the responsibilities that come with it, such as ensuring everything functions properly, managing infrastructure costs, implementing new customizations, and additional steps in the process.

Nicolas Ajaram talking about BurdaForward migration to SnowPlow

In this constantly changing world, I really liked the presentation made by Alexander Kirtzel about Walker.js.

In this ever-changing world, I thoroughly enjoyed Alexander Kirtzel’s presentation on Walker.js. According to the official documentation, “Walker.js is an open library to capture user events and send them to any analytics tool. It was created to enable data ownership, improve web data quality, and scale tracking implementation processes.“.

Alexander Kirtzel showing how Walker.js works

I appreciate the concept of a vendor-agnostic solution that prioritizes the definition and delivery of superior data to any analytics provider.

I’ll definitely look into it as I believe it holds the potential to be a future-proof solution.

Changing technology is easy, changing people is hard

The difficulty of adapting to new solutions and changing established habits (GA4 I’m looking at you) has been a prominent and widely discussed topic during different sessions.

For instance, Snowplow lacks a user interface, which prompted BurdaForward to invest efforts in implementing and customizing Apache Superset, a data visualization platform.

This was done to ensure a certain level of data democracy across the organization and assist data analysts in conducting their daily analyses seamlessly.

Nicolas Ajaram talking about BurdaForward migration to SnowPlow

Archit Goyal delivered an insightful presentation on the essential role of web analysts in building successful AI models.

A significant point of discussion revolved also around the lack of communication between web analysts and data scientists in many companies.

Web analysts often possess a deeper understanding of the data that should be incorporated into the models and have extensive expertise in the field. As you can see in the picture below, it has been observed that data scientists often do not frequently change the factors they utilize in their AI model development. This practice is problematic since it often overlooks valuable behavioural data, focusing solely on purchase data.

In conclusion, fostering better collaboration between web analysts and data scientists is crucial for leveraging their combined knowledge and optimizing AI model development.

Archit Goyal talking why Web Analysts should be always involved in AI projects

The era of “track everything” has come to an end

Last but not least, probably the hottest topic of the last few years. AdBlockers, Cookie Rejection, ITP,… are impacting the quantity and quality of the data you can collect.

Steen Rasmussen – “The Decision Economy”

The era where “everything” is trackable has come to an end, you need to focus more on defining business objectives and the specific decisions you aim to address.

Marketing-Mix Modeling, incrementality testing, and attribution are widely discussed and relevant instruments but it is crucial to complement these tools with real customer feedback.

Finding similarities among customer feedback, Marketing-Mix Modeling (MMM), and attribution can serve as evidence of specific behaviors occurring.

Simon Rumble – “Attribution is (still) bullshit”

During the event, Steen Rasmussen presented an interesting slide about the decay of the business value of data over time.

For instance, he highlighted the fact that everyone is trying to get all of the data out of GA3, despite the fact that much of it may not be utilized due to constantly changing customer behaviours and market conditions.

In summary, the ability to swiftly collect and utilize data is key if you want to maximize its business value.

Steen Rasmussen – “The Decision Economy”

The post Some takeaways from MeasureCamp Munich & Snowplow Meetup appeared first on Ruben Vezzoli.

A couple of days ago, I was analysing a GA4 property and I noticed that it was receiving a lot of traffic from many different referrals.

After some further analysis, I realised that many of those referrals were from the same website but with different subdomains or a port number at the end.

Just to give you an idea of what I mean you can see two examples below:

I had also an extreme case with hundreds of different values for the same referrer.

This is not good, especially in GA4, where it’s important to have a low cardinality (total number of unique values for a dimension) in order to avoid aggregated reports (the (other) row).

My solution

My goal was to consolidate some referrals for better reporting and lower the risk of data aggregation.

Luckily, page_referrer is one of the fields that you can set in a GA4 tag in order to override the default value.

I used a RegEx Table variable to mimic the Search and Replace filter function. I selected {{Referrer}} as my Input Variable and added some regular expressions and the desired outputs as you can see below.

In my case, I disabled the options Full Matches Only and Enable Capture Groups and Replace Functionality because I just wanted to match my regex anywhere in the value but it might not work in your case, so you should read very carefully the two information boxes.

Afterwards, I updated my GA4 Configuration Tag. I added page_referrer to the Fields to Set list and used my RegExp Table as a Value variable. 

This solution works with all the fields you can find on this list. For example, you could remove a specific directory from all the page URLs.

The post Search and Replace Filter in GA4 using GTM appeared first on Ruben Vezzoli.

It hasn’t been a straightforward process also because Piwik’s documentation is a bit outdated and that’s why I’m writing this post where I want to describe my own solution.

Disclaimer – If you want to fully understand this post, you have to know Consent Mode API and how they work. If you never used them, you should read the documentation I linked above.

How does Piwik’s Consent Banner work?

First of all, let’s analyze the Piwik’s consent banner (in this post, I won’t describe how to set up and implement Piwik Consent Manager). Every time a new user lands on the website a cookie ppms_privacy_YOUR_WEBSITE_ID is created. Inside the cookie, Piwik stores the user consent choices using 3 digits:

• -1 = the user has not yet given consent
• 0 = the user has not given consent
• 1 = the user has given consent

In order to use them, you have to create a variable which stores the information and then pass them to the Consent Mode API.

1. Store Consent Choices in a 1st party Cookie variable

In GTM, you have to create a new 1st-Party Cookie variable and paste your ppms_privacy_YOUR_WEBSITE_ID. Also, tick the option URI-decode cookie.

2. Read the cookie with Custom JavaScript variables

Google Consent Mode API doesn’t work with -1, 0, 1 but the words granted and denied. You have to create 3 custom JavaScript variables, one for each consent.

In the script below, parse the Cookie variable you created earlier, read the status of a specific consent (the example reads analytics consent) and returns granted or denied according to the number stored inside the cookie.

• -1 = denied
• 0 = denied
• 1 = granted

// Analytics Consent
function () {
var cookie = {{Piwik - Consent Cookie}};
if(!cookie){return 'denied'}
var json = JSON.parse(cookie)

if (json['consents']['analytics']['status'] == 1) {

 	return 'granted' 
  
	} else {

 	return 'denied'

	}
}

// Remarketing/Adv Consent
function () {
var cookie = {{Piwik - Consent Cookie}};
if(!cookie){return 'denied'}
var json = JSON.parse(cookie)

if (json['consents']['remarketing']['status'] == 1) {

 	return 'granted' 
  
	} else {

 	return 'denied'

	}
}

// A/B Testing and Personalization Consent
function () {
var cookie = {{Piwik - Consent Cookie}};
if(!cookie){return 'denied'}
var json = JSON.parse(cookie)

if (json['consents']['ab_testing_and_personalization']['status'] == 1) {

 	return 'granted' 
  
	} else {

 	return 'denied'

	}
}

3. Build a trigger that fires when the user makes/updates his consent choices

If you want to use Consent Mode, you have to be sure to update the consent choices every time the user changes his consent.

Luckily, you can use an event called stg.consentsWereSent that Piwik Consent Manager pushes to the dataLayer when the user makes his choice.

You have to create a Custom Event which will fires whenever the user interacts with the consent banner or the “Change Consent Options” button.

4. Update Consent Mode API using Simo Ahava’s template

At this point, you are ready to activate and set up Simo Ahava’s Consent Mode template. Go to the Template Gallery and search for Consent Mode (Google tags).

Then you should create two Consent Mode tags, one with the Default consent command that will be fired on every page. And a second tag with the Update consent command that is fired on the dataLayer event stg.consentsWereSent (when the user makes his choice).

Inside the tags, you also have to use the custom JavaScript we created earlier. They’ll pass the user choice to the Consent Mode API.

And that’s it! You are done with this tutorial.

If you want, you can also combine the custom JavaScript variables with the dataLayer event to fire tags right after the user gave consent to analytics or advertising, this is quite important if you don’t want to lose the first pageview

The post Piwik Consent Manager integration with GTM and Google Consent Mode appeared first on Ruben Vezzoli.

The URL Inspection API are a good way to access URL-level data outside of the Google Search Console interface. It provides different information like:

• date of the last crawl
• indexing status
• canonical tag
• mobile-friendliness
• structured data
• …

If you want to know more about the Inspection API I recommend you to read the official documentation.

In this post, I’m going to show you how to access the URL Inspection API using R. This tutorial is quite similar to the one I wrote about Google Indexing API in R.

How to generate Google URL Inspection API credentials

The code that I’m going to show you only work if you activate the Google Search Console API and generate a client id and client secret.

Open the Google API Console and search “Google Search Console API” in the APIs library and then you have to enable the API.

Then go to the Credentials section where you’ll be able to generate your credentials. Click on Create Credentials and choose the OAuth client ID option.

Select Desktop App and choose the name you want.

Now, you should have a popup on your screen showing your client id and client secret credentials.

The R code for the URL Inspection API

My R code uses as input a .csv file like the one you see below. I decided to use it because I know that many SEOs are not familiar with R and they prefer to work with Excel / CSV files.

But if you know how R works you can directly use a vector containing all the URLs you want to analyze.

Let’s not forget that the URL Inspection API has a limit of 2,000 queries per day!

Once I ran for the first time the R code, I realized that in the response of the API there are 3 different types of information:

• indexStatusResult – which contains the information about the last crawl, canonical tag, indexing status, sitemap,…
• mobileUsabilityResult – which contains the result of the mobile usability test
• richResultsResult – which returns all the structured data implemented on that url

In my case, I decided that I wanted as output only the first “sub-report” indexStatusResult.

The code below accepts in input a .csv file of URLs, if you want to use it you have to replace the 4 placeholders you find in the code:

• client id
• client secret
• siteURL – the name of your Google Search Console Property e.g. https://www.mydomain.com/
• languageCode – in which language you want the output

# LAST UPDATE: 2022-02-01
# Install & Load the packages

install.packages("googleAuthR")
install.packages("tidyverse")
install.packages("readr")

library("googleAuthR")
library("tidyverse")
library("readr")

# Set credentials and scope

clientId <- "PASTE HERE YOUR CLIENT ID"
clientSecret <- "PASTE HERE YOUR CLIENT SECRET" 
scope <- "https://www.googleapis.com/auth/webmasters"

options("googleAuthR.client_id" = clientId, 
        "googleAuthR.client_secret" = clientSecret, 
        "googleAuthR.scopes.selected" = scope,
        "googleAuthR.verbose" = 0 # Not mandatory - I just use it to debug the script
)

# Google API OAuth

gar_auth()

# List of URLs - Daily Limit of 2,000 queries per day 

urls <- read_csv("~/Desktop/Your-file-name.csv")
urls <- urls[ ,1]

# get_inspection_api function - you can use the function to send requestes to the inspection API using urls vector as an input
# It also GET the response from the API and stores it in a data frame

get_inspection_api <- function(page) {
  
  body <-  list(inspectionUrl = page,
                siteUrl = "PASTE HERE YOUR SEARCH CONSOLE PROPERTY NAME", #e.g. https://www.mydomain.com/ 
                languageCode = "it-IT" # CHANGE IT WITH YOUR LANGUAGE CODE
                )
  
  f <- gar_api_generator("https://searchconsole.googleapis.com/v1/urlInspection/index:inspect",
                         "POST")
  
  result <- f(the_body = body)
  result <- data.frame(result[[6]][[1]][[2]])
  
  return(result)
  
}

# The API responses are stored in a data frame

api_response <- map_dfr(.x = urls,
                        .f = get_inspection_api
                        )

# You can download the API response as .csv file

write.csv(api_response, "Your-file-name.csv")

And this is an example of the output you can get from the URL Inspection API with some of the columns.

The post How to use Google Search Console URL Inspection API with R appeared first on Ruben Vezzoli.

In particular, last year, Google released an update that had a huge impact on all the search advertisers. Since then, they are basically hiding part of the search queries that triggered a search campaign on Google.

As you can see in the screenshot below, they added at the end of the search terms report an extra row called “Other search terms” where they provide costs, clicks, conversions,… but not the text of the search queries.

This is a big issue for many advertisers because this “hidden group” can go up to 60-70% of all the search queries data and this information is very useful when you are optimizing a campaign and especially when you want to run an N-gram analysis.

In this post, I want to share an idea I had a couple of days ago and that can help you to still run an N-gram analysis on 100% of your search queries data.

Search queries report in Google Analytics

If you connected your Google Ads Account to Google Analytics, you should be able to see part of your Google Ads data in Google Analytics.

Especially, if you look at the search queries report you’ll notice that there are some search queries that generated sessions, conversions but that don’t have clicks and costs data.

If we want to run an N-gram analysis we have to use the Google Analytics data because it’s the only report where we still have all the search queries. The problem is that we need to know how many clicks and costs the hidden queries generated, otherwise, we would take into account all the conversions but not the costs generated by those queries.

How to estimate clicks and costs for the hidden search terms

If we look at the data we have, we know that for the known queries is not a problem we have everything we need.

But for the unknown queries, we only know how many sessions and conversions they generated. Clicks and Costs on a search term level are missing.

The only way I see to attribute costs and clicks to the hidden search queries is to take costs and clicks from the “Other search terms” row (either for a specific Ad Group or Keyword) and use the % of sessions that a specific search term generated as a “weight”.

I have chosen to use the sessions because it’s the closest metric to clicks.

In the example, I took as an example the hidden search terms for the keyword “google merchandise”. In total, the queries generated 32 sessions, 10,213 clicks and 253,049.46€ costs.

The search term “google merchandise home” generated 15,63% of the 32 sessions and that’s why it gets 1,596 clicks and 39,538.98€ costs.

Conclusion

I’m currently working on the R code that will automate the whole process and maybe I’ll share it later on.

I know that especially nowadays Google CPCs on search term level can vary a lot but we have to work with the data we have and I don’t see a way to attribute a CPC that is not an average but please let me know your opinion in the comments below.

The post How to run an N-Gram analysis even when Google Ads hides the data appeared first on Ruben Vezzoli.

In this post, I won’t talk about the differences between the old Google Analytics (Universal Analytics) and GA4 because it’s still a beta version and they are releasing every day new features. So, if you want to know all the new features of GA4 you should read this post.

Instead, I’ll try to cover all the basic aspects you should know if you want to start to analyze Google Analytics raw data that you can import (for free) in BigQuery.

This post is mainly for people with a basic knowledge of SQL and understanding of how a relational database works.

An overview of the Google BigQuery GUI

In order to get started, you have to connect Google Analytics and Google Big Query, here you can find the official documentation which will guide you through the setup process.

After the initial setup, you should be able to access the BigQuery interface. On the left there is a sidebar where you’ll find:

• Query history: where you can find all the queries you ran in the past
• Saved queries: if you don’t want to lose your queries, you can save them in the Saved queries section
• Scheduled queries: of course, you can also schedule a query and run it every day, week,…
• Resources: in this section, you’ll find all your projects. In each project, you’ll find one or more dataset and in each dataset, you’ll find one or more tables

While in the main section of the console you can find:

• SQL editor: where you’ll write the SQL queries. But you’ll also be able to save, schedule and format your code
• Output & dataset details: the last section of the interface contains the output area where you’ll be able to check the output of your queries but at the same time find more details about you tables (e.g. field names, dataset names, table size,…)

Google Analytics 4 export schema & first queries

As I said before, you can’t work with BigQuery if you don’t know a bit of SQL. So if you’ve never used it my advice is to read a guide about the basics of SQL.

In the BigQuery interface, you should also be able to see all your BigQuery projects with inside multiple datasets (one for each Google Analytics property you’ve connected).

Inside each dataset, you’ll find multiple tables,  that happens because of the Google Analytics export which split the events data by day, meaning that every single table contains only the data collected during a specific day.

And that’s why I want to show how to query multiple table at the same time.

Query multiple Google Analytics 4 tables

As you can see in the code below, you can query multiple tables in three different ways.

You can choose a static date range (e.g. 2020-10-01 / 2020-10-15), which is recommended if you are writing a query that you’ll only use once. Or, you can use a dynamic date range like “Last 30 days” which is quite often used for reports or scheduled queries.

You can also mix the first two methods and have a dynamic and static date range.

-- Option 1 - Static date range
SELECT *
FROM `ga-data-XXXXXX.analytics_XXXXXXXXX.events_*`
WHERE _TABLE_SUFFIX BETWEEN '20201101' AND '20201114';

-- Option 2 - Flexible date range (e.g. Last 30 days - Yesterday)
SELECT *
FROM `ga-data-XXXXXX.analytics_XXXXXXXXX.events_*`
WHERE _TABLE_SUFFIX BETWEEN FORMAT_DATE('%Y%m%d',DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))
      AND FORMAT_DATE('%Y%m%d',DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY));

-- Option 3 - A mix of static and flexible date range
SELECT *
FROM `ga-data-XXXXXX.analytics_XXXXXXXXX.events_*`
WHERE _TABLE_SUFFIX BETWEEN '20201001'
      AND FORMAT_DATE('%Y%m%d',DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY));

The last thing you should know about BigQuery before writing your first lines of SQL is how to deal with repeated records.

Handle nested fields / repeated records

Repeated records? What? Of course, I’ll try to quickly explain it with my own words, but, if you want to know everything about nested fields / repeated records, you should read this post written by the Firebase team.

If you are familiar with Excel or databases, you probably know that a table has rows and columns and inside each unit (cell) there is a single piece of information.

Well, Google Analytics data are organized in a slightly different way (because it’s more efficient). The structure of the Google Analytics tables looks like a JSON, which means that a column can contain a single piece of information but as well an array (a list) of values.

So you’ll have to know how to flatten a nested field in order to be able to query the table.

For example, the column event_params is a repeated record because inside it contains the value event_params.key, event_params.value,… .

And if you open the table preview, you’ll see something like that:

Let’s say that you want to filter the table to get only the rows where the page_referrer is “https://www.google.it”.

SELECT event_date, event_timestamp, event_name, page_referrer
FROM `ga-data-XXXXXX.analytics_XXXXXXXXX.events_*`
WHERE _TABLE_SUFFIX BETWEEN '20201101' AND '20201121' 
      AND event_name = 'first_visit'
      AND page_referrer = 'https://www.google.it';

If you run the query, you’ll get an error message because your WHERE condition is using page_referrer which is nested inside the column event_param.

If you want to use page_referrer you have to first flatten the nested field using the command CROSS JOIN UNNEST (or , UNNEST)which basically creates a new column

You have to add the , UNNEST in the FROM statement and specify the nested field you want to flatten.

SELECT event_date, event_timestamp, event_name, params
FROM `ga-data-XXXXXX.analytics_XXXXXXXXX.events_*`, UNNEST(event_params) AS params
WHERE _TABLE_SUFFIX BETWEEN '20201101' AND '20201121'
      AND event_name = 'first_visit';

And you’ll get this. All the params.key have their own row.

Then you can in the WHERE condition the filter for “https://www.google.it”

SELECT event_date, event_timestamp, event_name, params
FROM `ga-data-XXXXXX.analytics_XXXXXXXXX.events_*`, UNNEST(event_params) AS params
WHERE _TABLE_SUFFIX BETWEEN '20201101' AND '20201121'
      AND event_name = 'first_visit'
      AND params.value.string_value = "https://www.google.it/";

Finally, you get the output you needed.

How to create a scheduled query in BigQuery and connect it to Data Studio

Last but not least, in this post, I want to explain how to schedule a query in Google BigQuery.

Instead of manipulating your GA data in Data Studio, you can do that using an SQL query which automatically updates a table every hour, day,… .

For example, the query below counts the daily unique users of the last 30 days.

SELECT event_date, COUNT(DISTINCT user_pseudo_id) AS uniqueUsers
FROM `ga-data-XXXXXX.analytics_XXXXXXXXX.events_*`
WHERE _TABLE_SUFFIX BETWEEN FORMAT_DATE('%Y%m%d',DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)) 
      AND FORMAT_DATE('%Y%m%d',DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY)); 
GROUP BY event_date;

If you want to schedule it, you have to click on Schedule query.

You should give a name to the scheduled query and choose when and how often update it.

In my example, I decided to update the query daily and store the results in a new table called test_scheduled_query. The data are overwritten every day.

Then you have to open Google Data Studio and choose the BigQuery connector. Here, you’ll find the test_scheduled_query table which you can use to create a dashboard.

The post How to query Google Analytics 4 data in BigQuery appeared first on Ruben Vezzoli.

Generate the client Id and client secret credentials

Open the Google Developers Console, the platform where you can generate the credentials for every Google API (e.g. Google Analytics API, Google Search Console API, Indexing API,… ) and go to the APIs Library.

Find the Google Ads API and Enable the service. In the credentials section, you can generate the client ID and client secret that you need for the OAuth process.

Click on OAuth client ID and select Web application in order to generate the right credentials for RAdwords.

You have to save somewhere the credentials because you’ll need them in a later step.

Get a Developer Token for the Google Ads API

You also need a developer token which you can get in the Google Ads interface.

Open the Tools & settings menu and go to API Centre.

If you want to work with the Google Ads API in R you’ll need a developer token with at least a Basic Access level, the packages don’t work with a Test Access level.

In order to get a Basic Access token, you have to fill out the application form.

Connect the API and RAdwords using the doAuth() function

Open your R Ide and install RAdwords using the install.packages() function. In order to access the Google Ads API you have to use the doAuth() function which guides you through the OAuth2.0 authentication process.

# Install and Load the RAdwords package
install.packages("RAdwords")
library("RAdwords")

# Launch the OAuth process using doAuth
googleAuth <- doAuth()

Run the code above and follow the messages you’ll see in your console, they’ll ask you to enter the credentials you generated earlier:

1. The client ID from the Google API Console

Authentication process needs your Client ID from the Adwords API project for native apps.

Paste the Client ID here :=>
XXXXXXXXXXXXXXXXXXXXXXXXXX.apps.googleusercontent.com

2. The client secret from the Google API Console

Authentication process needs your Client secret from the Adwords API project.
Paste the Client secret here :=>
XXXXXXXXXXXXXXXXXXXXX

3. The developer token from Google Ads

Authentication process needs your Developer Token from the Adwords MCC.
Paste the Developer Token here :=>
XXXXXXXXXXXXXXXXXXXXXXXX

4. The client token will be generated after you login in your Google Account and granted access to your Google Ads account.

Authentication process needs your Client token in order to receive the access token from the API. 
Copy the Client token from your webbrowser and paste it here.
Paste the client token here :=>
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Write your first API Request

If you want to download the data from the Google Ads interface, you have to use getData(). Don’t forget to specify the arguments:

• clientCustomerId – you have to specify the Id of a Google Ads account. You can’t use the Id of an MCC (manager) account!
• google_auth – it should contain the credentials you enter with doAuth()
• statement – an AWQL statement where you specify dimensions, metrics, date range and report. Here you can find the list of all the available fields

# First of all you have to specify the data you want to get from the API
CampaignData <-statement(select=c("Week",
                          "CampaignName",
                          "Clicks",
                          "Impressions",
                          "AccountDescriptiveName",
                          "Cost",
                          "Conversions"),
                          report = "CAMPAIGN_PERFORMANCE_REPORT",
                          start = "2020-01-01",
                          end = "2020-07-31"
                 )
                          

# Send a request to the API using getData
# ATTENTION - The clientCustomerId must be an account ID and it can't be an MCC (manager) ID

myData <- getData(clientCustomerId = "XXX-XXX-XXXX",
                  google_auth = googleAuth,
                  statement = CampaignData,
                  transformation = T)

Get the same data for multiple accounts

As I said before, you can only use an account ID and not an MCC ID. This could be an issue if you want to get the data from multiple accounts at the same time.

But, you can use a vector and map_dfr() to run the same call over a list of account IDs, here you can find an example.

# Load the packages
library("tidyverse")
library("RAdwords")

# First of all you have to specify the data you want to get from the API
CampaignData <-statement(select=c("Week",
                                  "CampaignName",
                                  "Clicks",
                                  "Impressions",
                                  "AccountDescriptiveName",
                                  "Cost",
                                  "Conversions"),
                         report = "CAMPAIGN_PERFORMANCE_REPORT",
                         start = "2020-01-01",
                         end = "2020-07-31"
                         )

# Vector containing the Account IDs
accounts <- c("XXX-XXX-XXXX", "XXX-XXX-XXXX", "XXX-XXX-XXXX")

# The getAdsData function will get the data from the API
getAdsData <- function (i) {
  
  myData <- getData(clientCustomerId = i,
                    google_auth = googleAuth,
                    statement = CampaignData,
                    transformation = T)
  
  return(myData)

}

# map_dfr execute the getAdsData function over the accounts vector, the resulting data frames are merged into one
myAccountsData <- map_dfr(
                        .x = accounts,
                        .f = getAdsData)

The post RAdwords: how to access Google Ads API using R appeared first on Ruben Vezzoli.

“You can use the Indexing API to tell Google to update or remove pages from the Google index. The requests must specify the location of a web page. You can also get the status of notifications that you have sent to Google. Currently, the Indexing API can only be used to crawl pages with either job posting or livestream structured data.”

Many SEOs are using Indexing API also for non-job-related websites and that’s why I decided to build an R script to try out the API (and it worked).

I’m going to show you how the script works, but don’t forget that there is a free quota of 200 URLs sent per day!

Create the Indexing API Credentials

First of all, you have to generate the client id and client secret keys for the APIs.

Open the Google API Console and go to the API Library.

Open the Indexing API page and enable the API. Then go to the Credentials page and there you’ll find your credentials.

googleAuthR package & Indexing API options

I created the script using the googleAuthR package, which allows you to send requests to Google APIs.

The code takes in input, a character vector of maximum 200 URLs and returns in a data frame the response of the API (see the screenshot below).

You can use the script to update or delete pages. You just have to change the line 38:

• Use type = “URL_UPDATED” if you have to update the page
• Use type = “URL_DELETED” if you have to remove the page

# LAST UPDATE: 20-03-2021
# Install & Load the packages

install.packages("googleAuthR")
install.packages("tidyverse")
install.packages("readr")

library("googleAuthR")
library("tidyverse")
library("readr")

# Set credentials and scope

clientId <- "PASTE HERE YOUR CLIENT ID"
clientSecret <- "PASTE HERE YOUR CLIENT SECRET" 
scope <- "https://www.googleapis.com/auth/indexing"

options("googleAuthR.client_id" = clientId, 
        "googleAuthR.client_secret" = clientSecret, 
        "googleAuthR.scopes.selected" = scope,
        "googleAuthR.verbose" = 0 # Not mandatory - I just use it to debug the script
        )

# Google API OAuth

gar_auth()

# List of URLs - Daily Limit of 200 URLs 

urls <- read_csv("~/Desktop/Your-file-name.csv")
urls <- urls[ ,1]

# indexingApi function - you can use the function to send requestes to the indexing API using urls vector as an input
# It also GET the response from the API and stores it in a data frame

indexingApi <- function(page) {
    
    body <-  list(
                  url = page,
                  type = "URL_UPDATED")
    
    f <- gar_api_generator("https://indexing.googleapis.com/v3/urlNotifications:publish",
                           "POST")
    
    result <- f(the_body = body)
    result <- as.data.frame(result[[6]][[1]][[2]])
    
    return(result)
    
}

# The API responses are stored in a data frame

APIResponse <- map_dfr(
                      .x = urls,
                      .f = indexingApi)

# You can download the API responses as .csv file

write.csv(APIResponse, "Your-file-name.csv")

The post Send requests to the Google Indexing API using googleAuthR appeared first on Ruben Vezzoli.