generative AI to improve your analytics workflow: The pros and cons

In the past few months, we’ve heard a lot about how AI-generated marketing will change digital marketing. We work as consultants with brands to harness the technology for innovative marketing. We immediately delved into ChatGPT – the buzziest large language model-based bot on the market. We can see now how generative AI acts as an assistant, generating first drafts of code, and visualisations that our experts then refine into useful materials.

Our view is that the end-user must have clear expectations for the final product so that any AI generated materials can be edited or shaped. You should never trust generative AI to give you the correct answer to your questions.

ChatGPT only answered 12 out of 42 GA4 Questions correctly

We decided to test ChatGPT on a task that our consultants perform regularly – answering questions from clients about GA4. ChatGPT’s results weren’t that impressive. Out of 42 questions, it only gave 12 acceptable answers, which we sent to our clients. That is a 29% success rate.

Eight other answers (19%) were “semi correct.” They either misunderstood the question, and gave a different answer (although it was factually accurate), or contained a small amount misinformation within a correct response.

There are a href=”https://support.google.com/analytics/answer/13331684?hl=en#::text=What%20causes%20the%20(other)%20row,label%20them%20as%20(other).” There are standard rules in place to define this.

Dig deeper: Artificial Intelligence: A beginner’s guide

ChatGPT has its limitations and overconfidence

In some cases the answers were actively misleading. ChatGPT is not using training data after 2021. As a result, many of the most recent updates have not been factored in its answers.

ChatGPT, for example, couldn’t tell you when Universal Analytics would be deprecated. Google announced this in 2022. The bot in this case did not answer directly, but provided context by saying “…as far as I know, cut-off is 2021 …”

Some questions were answered incorrectly with an alarming amount of confidence. The bot, for example, told us that “GA4 is based on machine learning to track events. It can identify purchase events automatically based upon the data it gathers.”

GA4 has “enhanced measurements” that are automatically tracked. However, they are defined more by simple code in the metadata of a website than any statistical or machine-learning model. Moreover, the enhanced measurement does not include purchase events.

How can we use ChatGPT or other AI-based tools to generate chatbots?

ChatGPT is not a reliable source of information, as demonstrated by our GA4 test. It is still a useful assistant that can provide first drafts for analyses and code to an expert, reducing the time needed for tasks.

It can’t replace the knowledge of an analyst who understands what they want to see. ChatGPT can save time by producing analyses from sample data, without the need for heavy programming. You can get a good approximation of the data in seconds, and then instruct ChatGPT on how to manipulate or modify it.

ChatGPT was used to optimize and analyze the shopping baskets of a retailer. We were looking to understand the average basket size and determine how to best offer free shipping. This required a regular analysis of the distribution and margin of revenue, and an understanding over time of the variance.

ChatGPT was asked to examine how basket sizes changed over a 14-month period using the GA4 dataset. We suggested initial SQL queries to be used for further analysis in BigQuery, and also some options for data visualization for the insights found.

The options provided useful information for further exploration, even though they were not perfect. To finalize the output, our analyst adapted ChatGPT queries. The time it took for a senior analyst to work with junior support was reduced from three days to just one day.

Dig deep: 3 ways to make AI work in your favor

Automation of manual tasks can save time

It can also be used to automate manual tasks in a process. For example, quality assurance checks on a table of data or code. It is important to flag discrepancies and anomalies in any project.

ChatGPT can save a lot of time by validating a piece 500+ rows code that combines and processes multiple datasets. What would have normally taken someone two hours to review manually could be done in 30 minutes.

The quality of ChatGPT output depends on the parameters that you specify in your instructions. A task with very clear parameters, and no ambiguity (the numbers are either the same or not) is perfect for generative AI.

Consider generative AI as an assistant, not an expert

ChatGPT has made remarkable progress in the last few months. We can now request technical materials in conversational English for a wide range of tasks, including programming, communication, and visualization.

We’ve shown above that the outputs of these tools must be treated with expert judgement and care to make them useful. One good example is to increase efficiency in the way we build analyses for our daily work, or to speed up long and complex tasks that are normally done manually. We use our technical expertise to refine the outputs into materials that add value for our clients.

Although generative AI has the potential to revolutionize our digital workflows in many ways, as demonstrated by ChatGPT’s application, it is important to view its applications from a balanced point of view. Accuracy is limited, especially when it comes to recent updates and fine details.

As the technology develops, it will become easier to use AI as a tool that enhances our abilities and drives efficiencies in the work we do every day. I believe we should be focusing less on generative AI as a replacement for the expert, and more on its ability to improve productivity.

The original post How to use generative AI for your analytics workflow first appeared on MarTech.