The URL split testing feature was developed to address persistent challenges and align with Shoplift's product objectives:

• Consistent Feature Requests: Users frequently asked for testing capabilities beyond themes and templates.
• Trial Churn: High trial churn rates were attributed to the absence of URL testing as a core feature.
• Manual Workarounds: Customer success teams invested significant time implementing manual solutions for URL testing.
• Support Ticket Volume: There was a high volume of tickets related to "how to set up X."

This feature aimed to expand Shoplift’s testing capabilities, reduce friction for users, and enhance retention, paving the way for long-term growth.

Beta date: September 2024
Release date: December 2024

🤿 Discovery Phase

To guide development, we collaborated with customer success and sales teams, gathering insights into user needs and existing pain points. Key use cases identified included:

• Ad Campaign Testing: Testing the effectiveness of landing pages linked from Google and Facebook ads using URLs with UTM parameters.
• Price Testing: Redirecting users to pages with different prices for the same product.
• Product Image Testing: Comparing variations in product images to optimize conversions.
• Collection Page Testing: Testing product layouts and orders within collections.
• Landing page testing: Comparing homepage and landing page designs, including pages created with tools like Replo and supporting nuanced tests requiring custom tools.
• Validating Shopper Flow: Ensuring seamless redirection across multiple steps in the user journey.
• Product-Specific Redirects: Testing templates that showcase products differently.
• Audience Segmentation: Redirecting specific audiences to tailored pages and measuring their engagement.

These insights were pivotal in shaping the MVP and ensuring alignment with user needs.
On the other hand, it was a small scope and low risk feature, which created the perfect opportunity to finally build a beta program at Shoplift.

✅ Feature Priorities

The MVP was designed with simplicity and flexibility in mind:

• Maintain Shoplift’s advantage as an intuitive and easy to use A/B testing solution.

• Have a clear interface for setting up URL tests.
• Automated validation to avoid errors and streamline setup.
• Support for common test types, including price, product image, and collection layouts.
• Seamless integration into Shopify’s ecosystem, minimizing friction.

Beta Recruiting

Breakdown by relationship with Shoplift

🥂 Learnings

As we grew our user base and test creation kept evolving with their needs:

Metrics Review on Use Cases:
We were wrong about the primary use cases. Users were mainly testing their product page designs (PDPs) and conducting price tests, which was intersting to learn and apply for the future of the product (we were considering profit optimization features). The numbers:

• Ad Campaign Testing: Only 15% of users tested landing pages, suggesting a need for better user education on this use case.
• Price Testing: A significant 18% adoption validated its relevance.
• Product Image and Collection Page Testing: Moderate adoption rates indicated a demand for these functionalities but also highlighted areas for improvement.
• Audience Segmentation and Advanced Testing: Custom page builder tests and segmented audience experiments showed potential but were less frequently utilized.

Reflection on Original Goals:

• Churn Reduction: The feature addressed the "missing feature" cohort, reducing trial churn.
• Support Efficiency: Average ticket resolution time dropped from 62 hours to 35 hours during the beta, reflecting improved operational efficiency.
• High Adoption: Engagement and feedback indicated the feature met a real user need.

Challenges:

• Low engagement in post-test surveys due to timing around holidays.
• Manual onboarding and feedback collection were resource-intensive.

.

🥂 Opportunities

Best on the feedback we recived, we now have a plan how to further improve URL split testing and making it a truly seemless experiance.

1. Upgrade prompt / Better experience
When a user pastes a URL with a UTM/string, we strip it, but we can also auto-create an audience for him.

• Better experience for Advanced and Pro plans
• Upgrade opportunity for Core plans

2. Testing single products
Fits the most common use case and makes it easier for users to find and confirm the products users want to test

3. Profit optimization / profit reporting (research)
We've seen a lot of use cases of price and offer testing, and I believe we'll continue seeing more. We're starting to gather a good sense of what people are doing, so this is a great research opportunity.