5 steps to split-test variations for your Shopping feed with FeedX

Most advertisers optimise their Google Shopping catalog based on instinct or best-guess logic. They update titles, rewrite descriptions, swap out images, and then watch overall campaign metrics for a few weeks hoping to find a signal in the noise. The problem with that method is that it is not a test. It is a change. Without a baseline cohort running alongside the variation, there is no reliable way to determine whether a performance shift came from the attribute update, a seasonal behavior change, a competitor budget movement, or any number of other variables.

FeedX solves this problem. Developed by Google Marketing Solutions and released as an open-source Python package, FeedX is a purpose-built A/B experimentation framework for Google Shopping listing data. It allows advertisers to run statistically reliable split tests on feed attributes, including titles, descriptions, and custom labels, and measure the direct impact on Shopping Ads performance using industry-standard statistical methods.

Understanding what AB testing actually means in a Shopping context is important before you implement the framework. A/B testing, sometimes called split testing, is a method of comparing two variations of the same attribute to determine which one performs better. The key difference between a catalog split test and other forms of advertising experiments is that it does not rely on cookies, user segments, or audience targeting to assign traffic. Instead, FeedX uses item-level data to assign each SKU in your listing data to either a control or treatment batch based on historical behavior. This is a more reliable method for ecommerce advertisers because it removes the user-level variation that makes audience-based tests difficult to analyze cleanly. The number of items you assign to each cohort matters: FeedX recommends a large enough catalog to detect small lifts, with at least 1,000 SKUs as the baseline for a study that can produce meaningful outputs.

A real-world example of FeedX in practice is documented in a Think with Google article covering Home24, where the framework was used alongside Google's generative AI tool FeedGen to measure a 29% increase in clicks and a 25% increase in impressions from optimised product titles. Those are not estimates. They are measured outcomes from a controlled study with a statistically validated baseline cohort running in parallel.

This guide covers exactly how FeedX works, what you need to set it up, and the five steps to execute your first reliable feed trial on Google Shopping.

What is FeedX and why does it matter for Shopping Ads performance?

Before running through the steps, it helps to understand what makes FeedX different from the informal testing most advertisers currently use.

Standard listing changes are applied globally. Every item gets the new title, every SKU gets the updated description. That means your original data is overwritten and there is no baseline left to compare against. You are looking at before-and-after performance across the same items, which is sensitive to every external variable that happened to change during the same window.

FeedX splits your catalog into two segments: a control cohort and a treatment cohort, rather than splitting users or traffic. The control cohort retains the original listing attributes throughout the study. The treatment cohort receives the optimised variation. Both segments serve Shopping Ads simultaneously under the same campaign settings, which means any performance difference between them is attributable to the attribute change itself rather than external factors.

FeedX uses two key statistical methods to make outputs as reliable as possible. The first is a crossover design. The second is CUPED, which stands for Controlled-experiment Using Pre-Experiment Data. CUPED adjusts item-level performance based on pre-experiment behavior, removing bias introduced by randomisation and improving the sensitivity of the study. The crossover design goes further. Halfway through the trial, the two cohorts swap, so the treatment segment reverts to original attributes and the control segment receives the optimisation. This approach doubles the statistical power of the study without requiring more items or a longer run time.

To ensure reliable outputs, the trial should include at least 1,000 items. The FeedX design notebook will alert you if the sample size is too small. For smaller catalogues with fewer than 1,000 SKUs, it is worth expanding the scope of what you are testing across an entire category rather than a single subcategory, or waiting until the catalogue grows before conducting a formal study.

What you need before you start

FeedX is a Python package available on GitHub that operates through Google Colab notebooks. You do not need to install anything locally. The entire workflow runs in the browser via Colab. However, you do need the following in place before opening the setup notebook.

A Google Merchant Center account with an active listing data source syncing to Google Ads. Your primary catalog should be healthy with no widespread disapprovals or policy violations before you start layering a study on top of it.

A Google BigQuery project connected to both your Google Ads account and your Merchant Center account via the BigQuery Data Transfer Service. FeedX requires historical performance information, typically four to eight weeks of click and impression datum at the item level, to design the trial and execute the CUPED statistical adjustment. Without BigQuery, the framework cannot access the item-level information it needs.

A supplemental listing file set up in Merchant Center. This is the source that will carry your treatment variation. The supplemental file overwrites specific attributes from the primary source for the treatment cohort only, leaving the control segment on the original data.

A clear hypothesis. Before you open the Colab notebook, decide exactly what you are comparing and why. A good study compares one change at a time. Testing a title update and an image change simultaneously makes it impossible to determine which variable drove the result. Common starting points include adding material and dimension information to titles, reordering title structure to lead with the most Search-relevant attribute, or adding high-intent modifiers based on Search query information from your Google Ads account.

The five steps to implement FeedX and start split-testing your Google Shopping catalog

Implementing this tool from Google will require technical expertise and a deep product catalog.

If you don't have an in-house developer to help you or your product feed is less than 1000 SKUs, we don't recommend going through this implementation. The rewards versus investment ratio won't be very interesting for you.

Step 1: design the trial

Open 01_experiment_designer.ipynb in Google Colab and save a copy to your Google Drive. This notebook contains all the logic for setting up the study before a single change is made to your listing data.

Input your BigQuery Project ID, your Merchant Center account ID, and the historical datum window you want to use for the baseline, typically four to eight weeks of prior performance. Execute the notebook cells. FeedX will pull item-level performance information from BigQuery, analyze historical metrics including clicks, impressions, and conversion behavior for every SKU in your catalog, and split the catalog into a control segment and a treatment segment.

The split is not random in the naive sense. FeedX pairs items mathematically to ensure both cohorts have balanced historical performance before the study begins. This pairing process eliminates the selection bias that undermines informal testing. The notebook outputs a table to BigQuery labelling every Product ID as either CONTROL or TREATMENT. Save this mapping. It is the reference point for every subsequent step.

The notebook will also flag whether your catalog is large enough to produce reliable outputs at a meaningful effect size. If you want to detect a 5% improvement in click-through rate, for example, you need more items than if you are only trying to detect a 20% improvement. FeedX can execute 1,000 simulations to validate that your study design will produce reliable outputs at your target effect size before you commit to conducting it live. Note the output carefully before moving to the next step. Starting a trial with an underpowered design is one of the most common reasons listing split tests produce inconclusive findings.

Step 2: prepare and apply the treatment variation

Before you prepare your treatment listing data, it helps to look closely at your existing titles and decide which type of change is most likely to increase CTR based on your Search query reports in Google Ads. A good practice is to compare your current top-performing titles against those with high impressions but low clicks, and use that information as the base for your variation. Third-party feed management software can help you bulk-generate optimised title variations at scale, but the core logic should come from your own account data. Identify the key attributes missing from your current listings, for example dimensions, materials, or compatibility information, and determine how to vary the title format in a way that better matches what your target audience is actually searching for. Come up with multiple options before selecting the final variation to test different approaches, and track your reasoning so you can refer back to it when you analyze the outcome.

Before finalising your title variation, it's worth reviewing what a well-structured Google Shopping feed should include at a base level. Our guide to the 7 key points of an optimized Google Shopping feed covers the core attributes that most advertisers underuse, which can help you identify the highest-value change to test first.

Export the list of Product IDs assigned to the TREATMENT cohort from BigQuery. These are the only items that will receive the optimised listing attributes. The control segment is not touched at any point during the study.

For the treatment cohort, create your optimised item data. The most impactful attribute to test in most Shopping listing sources is the title, since it directly determines which Search queries trigger your ads and influences CTR in the Shopping carousel. A well-structured title trial might compare a generic format such as "Outdoor Dining Set" against an attribute-rich variation such as "6-Piece FSC Wood Outdoor Dining Set, Weather-Resistant, Seats 6." The second format targets more specific Search queries, surfaces additional item information in the ad, and is more likely to match user intent at the point of comparison shopping.

Format the treatment item data as a supplemental listing file. This is a spreadsheet or structured data file containing only the Product IDs from the TREATMENT cohort and the specific attribute you are updating. Upload this supplemental file to Merchant Center and configure it to overwrite the relevant attribute from the primary source. Verify that treatment items Display the new title while control items retain the original. If both cohorts are showing the new attribute, the supplemental file mapping is incorrect and needs to be fixed before the study starts.

It is also good practice at this stage to label your treatment items using a custom label in the listing source. Assigning a custom label such as "feedx-treatment" to every TREATMENT Product ID makes it easy to segment performance by cohort inside Google Ads reporting and provides an additional layer of tracking alongside the BigQuery output.

Step 3: execute the study live on Google Shopping

Start the trial by confirming the supplemental listing file is active in Merchant Center and both segments are serving Shopping Ads within the same campaign. Do not create separate campaigns for the control and treatment cohorts. Both must serve under identical campaign settings, bidding strategy, and budget to ensure the only variable between them is the listing attribute being compared.

Let the study run for at least two to four weeks so the framework can accumulate sufficient traffic information for the statistical analysis to be meaningful. Do not alter campaign budgets, bidding strategies, or targeting during this window. Any change to how the campaign serves introduces a confounding variable that undermines the output.

If you are conducting a crossover design, which FeedX recommends for maximum statistical power, swap the cohorts at the halfway point. The treatment segment reverts to original attributes via the supplemental file, and the control segment receives the optimised variation. This crossover eliminates the effect of any item-level performance differences that exist between the two cohorts independently of the listing change, and produces a cleaner, more reliable outcome.

Note the start date, the midpoint swap date if applicable, and any external events such as promotions, seasonal peaks, or competitor activity that occur during the run. These are important context when interpreting the analysis output. A large uplift that coincides exactly with a sitewide promotion is harder to attribute solely to the listing change and should be flagged in the findings.

Step 4: analyze outputs with CUPED

Once the study has run for the full window, open 02_experiment_analyzer.ipynb in Google Colab. This notebook pulls the live performance information accumulated in BigQuery during the trial window and applies the CUPED statistical adjustment automatically.

CUPED works similarly to a difference-in-difference analysis. Rather than comparing raw performance during the study window, it analyzes the difference between each item's performance during the trial and the expected performance of that item based on its pre-experiment behavior. This removes any bias introduced by the randomisation process and significantly improves the sensitivity of the result. The key reason CUPED matters in a Shopping listing context is that item-level performance is naturally noisy. A SKU that happened to be featured in a Display campaign or shared on social during the study window will show inflated click numbers that have nothing to do with the title change. CUPED accounts for that pre-existing behavior and filters it out of the final output.

When you analyze the output from the analyzer notebook, the goal is to find a clear winner, not just a positive number. Look at the confidence interval closely. A result with a mean lift of 8% but a wide interval that ranges from negative 3% to positive 19% is not a reliable basis for a decision. A good result is one where the interval is narrow and close to the mean, meaning the outcome is consistent across the full range of items in the treatment cohort. FeedX is designed to serve this determination automatically, flagging results as significant or inconclusive based on the statistical thresholds built into the tool. The experience of reading a FeedX output for the first time can feel technical, but the framework is built to make the key finding easy to act on.

The notebook generates charts showing the statistical lift or drop across your target metrics, including CTR, conversion rate, and ROAS, for the treatment cohort relative to the control segment. Each result includes a confidence interval. A output is considered statistically significant when the confidence interval does not include zero, meaning the observed performance difference is unlikely to have occurred by chance.

If the result is positive and statistically significant, the optimised variation is the winner. If the output is negative or the confidence interval is wide and crosses zero, the original attribute performs as well or better and the change should not be rolled out. If the result is inconclusive, meaning a small positive lift with a wide interval, the trial may need to conduct longer or include more items to reach a reliable conclusion. Use the design notebook to recalculate the required sample size based on the effect size you observed, and use that as a guide for the next run.

Step 5: implement the winner and prepare the next trial

If the treatment variation produces a statistically significant positive result, remove the supplemental listing file override and apply the winning attribute directly to the primary source for all items. Once you determine a winner, implement the change, track the downstream impact on conversion rate and ROAS inside your Google Ads account, and prepare the next trial. Marketing teams that use this process consistently build a resource of validated listing changes that compound in value over time, because each winning variation improves the potential of every subsequent study by raising the performance baseline.

Update your catalog systematically. Do not apply the winning title format only to the items that were in the treatment cohort. The whole point of conducting a controlled study is to find a format that works across a category or item type. Once validated, apply it broadly. If your trial tested a specific title structure for garden furniture, roll that structure out to every garden furniture listing in the source, not just the 50% that served as the treatment cohort during the study.

The FeedX framework is designed to be repeatable. Once one trial concludes, the same process can be applied to a different attribute including descriptions, custom labels, and image selection, using the same BigQuery infrastructure and Colab notebooks. Each study builds your understanding of what actually moves performance in your specific catalog, rather than relying on general best-practice advice that may or may not apply to your item type or audience.

The key practice to carry forward is comparing one variable at a time. The control segment must always serve as a true baseline, unchanged and operating under identical campaign conditions, for the result to be trustworthy. Introducing multiple variations simultaneously or changing campaign settings mid-trial produces information that cannot reliably determine a winner.

Why reliable listing testing matters for Google Shopping now

Google Shopping listing quality now directly influences ad eligibility, auction competitiveness, and conversion efficiency, particularly in Performance Max campaigns where the platform's automation relies heavily on the item data you provide to match SKUs to high-intent queries. According to Google, people shop across the platform more than one billion times per day, and Google Lens now sees more than 20 billion visual searches per month with one in four carrying commercial intent. Structured, optimised listing data serves all of those surfaces simultaneously.

The listing source is no longer just a data upload requirement. It is the primary creative and targeting input for Shopping Ads, and the quality of that information determines how well the platform's systems can serve your items to the right audience at the right moment. For any advertiser conducting Shopping campaigns at meaningful scale, FeedX is the most reliable tool available for turning listing optimisation from a recurring task into a systematic performance lever. Execute the study, analyze the outcome, implement the winner, and repeat. That process, applied consistently across title formats, descriptions, and custom label structures, compounds over time into a catalog that performs measurably better than one managed on instinct alone.

Feed quality now extends beyond standard Shopping attributes. Google's AI-driven surfaces — including AI Overviews and Gemini — rely on a separate set of conversational attributes that most catalogs don't have at all. This breakdown of the 8 GMC attributes that help products appear in Google AI Overviews is worth reading before your next FeedX trial, since optimising for AI surfaces and optimising titles for Shopping clicks are increasingly part of the same feed strategy.

‍