Sponsored by
As promised — data project this week.
Last time we did exploratory data analysis. Today something different: A/B test analysis. This is one of the most common analyses you will do as a data analyst, and one of the most requested in interviews.
The question we are answering: did showing ads actually make people buy?
Before we proceed, a small ad. Your clicks on ads help cover subscription costs. Thank you
Join 2M+ Professionals Getting Ahead on AI
Keeping up with AI shouldn't feel like a second job.
But between the new tools, viral posts, and endless hot takes, most people spend hours every week trying to figure out what actually matters.
The Rundown AI fixes that.
It's a free newsletter that gives you the AI news, tools, and tutorials you actually need to know. All in just 5 minutes a day.
Over 2M professionals at companies like Apple, Google, and NASA already read it every morning to stay ahead.
Plus, if you complete the quiz after signing up, they'll recommend the best tools, guides, and courses for your specific job and needs.
What is A/B testing?
A/B testing is how companies make decisions without guessing.
You take two groups of users. Group A sees the current version (control). Group B sees the new version (experiment). You measure what happens. You let the data decide.
It is everywhere, you just don't notice it.
YouTube lets creators upload up to 3 different thumbnails for the same video. Your friend and you might see a completely different cover image when the video appears in your feed — YouTube is quietly testing which one makes more people click. Netflix does the same with trailers, adjusting what you see based on your location and watch history to maximise the chance you actually press play.
I do A/B testing with my own content too — changing topics, hooks, or the type of media (carousel vs image vs plain text) and analysing what resonates with my audience. Same principle, much smaller scale.
It sounds simple. The analysis is where it gets interesting.
The dataset
We are using the Marketing A/B Testing dataset from Kaggle.
Download it here: https://www.kaggle.com/datasets/faviovaz/marketing-ab-testing
588k rows. No missing values. A real marketing experiment — one group saw ads, another saw a public service announcement (essentially a neutral placeholder). Did the ads actually drive more conversions?
The columns:
user_id — unique identifier for each user
test_group — "ad" (saw the advertisement) or "psa" (saw the public service announcement)
converted — True if the person bought the product, False if not
total_ads — how many ads the person saw
most_ads_day — which day of the week they saw the most ads
most_ads_hour — which hour of the day they saw the most ads
Step 1 — Load the data into your database and explore it
If you are working in SQL, load the CSV into your database first.
Once it's in, start with the basics.
-- How many rows do we have?
SELECT COUNT(*) AS total_rows
FROM ab_test;
-- How are users split between the two groups?
SELECT
test_group,
COUNT(*) AS users,
ROUND(COUNT(*) * 100.0 / SUM(COUNT(*)) OVER (), 2) AS pct_of_total
FROM ab_test
GROUP BY test_group;You will immediately notice something important — the groups are not equal. Exactly 96% of users are in the ad group and only 4% saw the PSA. This is not a balanced experiment, and it matters for how you interpret results. Always check your group sizes before drawing any conclusions.
Step 2 — Calculate conversion rates
The core question: do people who see ads convert at a higher rate?
SELECT
test_group,
COUNT(*) AS total_users,
SUM(CASE WHEN converted = TRUE THEN 1 ELSE 0 END) AS converted_users,
ROUND(
SUM(CASE WHEN converted = TRUE THEN 1 ELSE 0 END) * 100.0 / COUNT(*),
2
) AS conversion_rate_pct
FROM ab_test
GROUP BY test_group;The ad group converts at 2.55% and the PSA group at 1.79%. That's a relative lift of 43% — a meaningful difference. But is it real, or could it have happened by chance?
That is the central question of A/B testing, and it's not optional to answer. You need a statistical significance test before making any business recommendation. We'll come back to this in Step 5.
Step 3 — Look at ad exposure
Not all users in the ad group saw the same number of ads. Does seeing more ads make you more likely to convert?
SELECT
test_group,
ROUND(AVG(total_ads), 1) AS avg_ads_seen,
MIN(total_ads) AS min_ads,
MAX(total_ads) AS max_ads,
PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY total_ads) AS median_ads
FROM ab_test
GROUP BY test_group;Both groups average 24.8 ads seen, with a median of 13. Then look at conversion rate by ad volume:
SELECT
CASE
WHEN total_ads BETWEEN 1 AND 10 THEN '1-10 ads'
WHEN total_ads BETWEEN 11 AND 50 THEN '11-50 ads'
WHEN total_ads BETWEEN 51 AND 100 THEN '51-100 ads'
ELSE '100+ ads'
END AS ads_bucket,
COUNT(*) AS users,
ROUND(
SUM(CASE WHEN converted = TRUE THEN 1 ELSE 0 END) * 100.0 / COUNT(*),
2
) AS conversion_rate_pct
FROM ab_test
WHERE test_group = 'ad'
GROUP BY ads_bucket
ORDER BY ads_bucket;Here is what the data actually shows:
Ads seen | Users | Conversion rate |
|---|---|---|
1-10 | 249,499 | 0.33% |
11-50 | 248,875 | 1.89% |
51-100 | 44,149 | 11.63% |
100+ | 22,054 | 17.14% |
Conversion rate increases consistently with ad exposure — there is no drop-off at high volumes. Users who saw 100 or more ads converted at 17.14%, compared to 0.33% for those who saw fewer than 10.
This raises an important analytical question: are high-frequency users converting because they saw more ads, or were they already high-intent buyers who happened to encounter more ads along the way? SQL can show you the correlation. It cannot establish the cause. That distinction matters when you present findings to a stakeholder.
Step 4 — Find the best day and hour to show ads
-- Which day of the week drives the most conversions?
SELECT
most_ads_day,
COUNT(*) AS users,
SUM(CASE WHEN converted = TRUE THEN 1 ELSE 0 END) AS conversions,
ROUND(
SUM(CASE WHEN converted = TRUE THEN 1 ELSE 0 END) * 100.0 / COUNT(*),
2
) AS conversion_rate_pct
FROM ab_test
WHERE test_group = 'ad'
GROUP BY most_ads_day
ORDER BY conversion_rate_pct DESC;
-- Which hour of the day drives the most conversions?
SELECT
most_ads_hour,
COUNT(*) AS users,
ROUND(
SUM(CASE WHEN converted = TRUE THEN 1 ELSE 0 END) * 100.0 / COUNT(*),
2
) AS conversion_rate_pct
FROM ab_test
WHERE test_group = 'ad'
GROUP BY most_ads_hour
ORDER BY conversion_rate_pct DESC
LIMIT 5;By day: Monday is the strongest at 3.32%, with Saturday the weakest at 2.13%. The spread across the week is 1.19 percentage points — noticeable but not dramatic.
By hour: The top hours are 16:00 (3.09%) and 20:00 (3.03%), followed closely by 15:00 and 21:00. Mid-to-late afternoon and early evening consistently outperform other time windows. That tells a marketing team exactly where to concentrate spend.
Step 5 — Test for statistical significance
This step is not optional. A 0.76 percentage point difference between two groups sounds meaningful, but with 588,000 users you need to confirm the result did not occur by chance before anyone acts on it.
SQL alone cannot do this. You need Python (or R, or at least Excel)
from scipy.stats import chi2_contingency
contingency = [
[14423, 550154], # ad: converted, not converted
[420, 23104] # psa: converted, not converted
]
chi2, p, dof, expected = chi2_contingency(contingency)
print(f"Chi-square: {chi2:.2f}, p-value: {p:.2e}")Running this on the actual data gives a chi-square of 54.01 and a p-value of 2e-13. The result is highly significant — the probability of seeing a difference this large by chance is essentially zero. The ad campaign worked.
The relative lift of 43% is also a number worth putting in front of a stakeholder. It is a much clearer way to communicate impact than the raw rate difference.
Step 6 — Summarise your findings
SELECT
test_group,
COUNT(*) AS total_users,
SUM(CASE WHEN converted = TRUE THEN 1 ELSE 0 END) AS conversions,
ROUND(
SUM(CASE WHEN converted = TRUE THEN 1 ELSE 0 END) * 100.0 / COUNT(*),
2
) AS conversion_rate_pct,
ROUND(AVG(total_ads), 1) AS avg_ads_seen
FROM ab_test
GROUP BY test_group;What would you tell a stakeholder?
The ad campaign drove a 43% relative lift in conversion rate compared to the control group (2.55% vs 1.79% across 588,000 users)
The difference is statistically significant (chi-square = 54, p < 0.001) — this is not noise
Conversion rate increases steadily with ad frequency, reaching 17% for users who saw 100 or more ads
Monday and mid-to-late afternoon (14:00-16:00) and early evening (20:00-21:00) are the highest-converting windows
Those are recommendations a business can act on.
How to turn this into a portfolio project
Don't just run the analysis and close the laptop. Write it up.
A good portfolio project has four parts:
Overview — one paragraph explaining what the dataset is, what time period it covers, and what business question you are answering. For this project: a marketing A/B test measuring whether ads drove more conversions than a neutral public service announcement, based on 588,000 users.
Approach — how you analysed it and why. What SQL queries did you write, what did you look for, what decisions did you make along the way. Mention that you checked group sizes first (and noticed the imbalance), that you segmented by ad frequency, that you looked at time patterns, and that you ran a chi-square test to validate the result.
Findings — your actual results in plain English. The ad group converted at 2.55% vs 1.79% for the PSA group — a 43% relative lift, confirmed as statistically significant. Conversion rate increases consistently with ad frequency, reaching 17% for heavy ad exposure. Monday and the 14:00-16:00 and 20:00-21:00 windows show the strongest performance.
Limitations — what you couldn't answer and why. The groups are heavily imbalanced (96% ads, 4% PSA), which means you should be cautious about direct comparison even though the chi-square test accounts for this mathematically. You don't know how users were assigned to each group or whether the assignment was truly random — if high-intent buyers were systematically shown more ads, the frequency correlation may not be causal. There is also no demographic data available.
You can publish this as a PDF, a Notion page, a GitHub README, or a Medium article. The format matters less than having all four parts present.
What should I write about next?
Keep pushing 💪,
Karina
Grab your freebies if you haven’t done already:
Data Playbook (CV template, Books on Data Analytics and Data Science, Examples of portfolio projects)
Need more help?
Just starting with Python? Wondering if programming is for you?
Master key data analysis tasks like cleaning, filtering, pivot and grouping data using Pandas, and learn how to present your insights visually with Matplotlib with ‘Data Analysis with Python’ masterclass.
Building your portfolio?
Grab the Complete EDA Portfolio Project — a full e-commerce analysis (ShopTrend 2024) with Python notebook, realistic dataset, portfolio template, and step-by-step workflow. See exactly how to structure professional portfolio projects.
Data Analyst & Data Scientist
