SIMON HARMS FULL REPORT

FIELD REPORT 06 · PAYWALL TEST WALKTHROUGH · THREE BOARDS

Three questions
a paywall test
has to answer

One experiment, the onboarding paywall test on a Match Group dating app. One board for each question.

+13% first time paid subscribers,
net of churn
  1. 01 Did revenue move, or only the subscriber count? EXHIBIT A · THE READOUT
  2. 02 What did I ship, and who shipped the rest? EXHIBIT B · THE BUILD
  3. 03 How do I score the next test? EXHIBIT C · THE SCOREBOARD

01 NET REVENUE VERSUS SUBSCRIBER LIFT

One point apart at the screen. Seventeen a cycle later.

  1. 1

    Subscriber lift is the count at the screen. A read 110, B read 109. On that number you would ship either one.

  2. 2

    After the first cycle, A rose to 113 and B fell to 96. A kept 3% more of its subscribers than control did. B kept 12% fewer.

  3. 3

    Revenue follows the count only while plan mix holds. A sold the same plans at the same prices. B led with the shortest term, so its revenue fell further than its count.

Weekly billing opens that gap faster: four charges land inside 30 days. So the scoreboard grades on revenue directly. Board 03

● EXHIBIT A · THE READOUTINDEXED · CONTROL = 100
The readout by arm, indexed to control = 100 Slope chart with two reads. At the screen: control 100, Variant A 110, Variant B 109. After the first billing cycle, net of cancels and refunds: control 100, Variant A 113, Variant B 96. Net revenue: A moved with the count and shipped to 100 percent. B fell further than the count and was turned off at week 2. AT THE SCREEN AFTER THE FIRST CYCLE IN THE BANK gross subscribers net of cancels and refunds net subscription revenue VARIANT A value first 110 VARIANT B reordered plan ladder 109 CONTROL no onboarding paywall 100 3% MORE KEPT 12% FEWER KEPT 113 100 96 MOVED WITH THE COUNT same plans, same prices, plan mix inside its band SHIPPED TO 100% BASELINE FELL FURTHER THAN THE COUNT shortest term first, early cancels up OFF AT WEEK 2 A AND B · 1 POINT APART A AND B · 17 POINTS APART AT THE SCREEN AFTER 1 CYCLE A 110 B 109 CTRL 100 113 100 96 1 POINT APART 17 POINTS APART
  • A IN THE BANKMoved with the count. Same plans, same prices, plan mix inside its band. Shipped to 100%.
  • B IN THE BANKFell further than the count. Shortest term first, early cancels up. Off at week 2.
NET REVENUE=
SUBSCRIBERS

A up, 110

B up, 109

FIRST CYCLE SURVIVAL

A up, kept 3% more

B down, kept 12% fewer

PLAN VALUE

A held, same mix

B down, shorter terms

REFUNDS

A held

B off at week 2

Indexed figures. The raw subscriber and revenue numbers stayed with Match Group.

02 WHAT I SHIPPED, AND WHO SHIPPED THE REST

Product passed. I ran it myself.

  1. 1

    The screen: I made the first designs. In house designers finished them and developers shipped the code.

  2. 2

    The test: on web first, arms set at sign up and served by flag. The winner rolled out to iOS and Android.

  3. 3

    The wiring: four events I instrumented myself. Each one carried the arm and the acquisition source.

  4. 4

    The readout: I did it with an analyst then. Today I would do all of it myself with AI.

● EXHIBIT B · THE BUILDTHREE LAYERS · WHO DID WHAT
L1THE SCREEN
AND THE SWITCHFIRST DESIGNS: MEFINISH: DESIGNERSCODE: DEVELOPERS
  1. REGISTER
  2. FIRST LIKES + MATCHES the value moment
  3. FLAG set at sign up,
    sticky per account

  4. 15% + 15%PAYWALL

    A: same plans, same prices.
    B: shortest term first, price per week.

    70%STRAIGHT IN

    Control. The offer comes later, in product.

TESTED ON WEB FIRST iOS ANDROID The winner rolled out to both apps.

L2THE EVENT
SPINETRACKING: ME
  1. PAYWALL IMPRESSIONARMSOURCE
  2. PLAN SELECTEDARMSOURCE
  3. PURCHASEARMSOURCE
  4. FIRST CYCLE CANCEL OR REFUNDARMSOURCE
  5. ANALYTICSpaid and organic cohorts
L3THE READOUT
AND THE RULESRULES: MEREADOUT: ME + AN ANALYST
COHORTS

Every arm, paid and organic read apart.

NET OF CHURN

Subscribers left after the first cycle, reconciled to revenue.

RULES ON A CLOCK

Day 7 kill check. Weekly ceiling check.

THE CALL

WATCHKILLSHIP

THENMy designs, finished by in house designers. Code by developers. Readout with an analyst.

NOWAll of it myself, with AI: design, code, tracking and readout.

03 THE 30 DAY SCOREBOARD

One grade: net revenue per install at day 30.

  1. 1

    Built for a weekly plan with a 3 day trial. Four charges land in the first month.

  2. 2

    Trial starts are shown and never graded. The grade is cash, net of refunds and store fees.

  3. 3

    The limits are written before launch: exposure cap, loss ceiling, kill trigger.

  4. 4

    Paid and organic are read apart. Arms are substantially different experiences, tested against each other.

● EXHIBIT C · THE SCOREBOARDWEEKLY PLAN, 3 DAY TRIAL · PICK A READ DAY
ARM AND COHORT AT THE SCREEN THE FIRST MONTH THE GRADE, DAY 30 LIMITS
ARM COHORT TRIAL STARTS PAID AT DAY 3 RENEWALS KEPT REFUNDS NET REVENUE PER INSTALL VS CONTROL LOSS USED GUARDRAILS
CONTROLtoday’s paywallPAID
ORGANIC
ARM 1example: value first, then the askPAID
ORGANIC
ARM 2example: yearly plan on the first screenPAID
ORGANIC

GUARDRAILS: REFUND RATE · DAY 1 AND DAY 7 RETENTION · STORE RATING · SUPPORT CONTACTS

Day 30. All four charges have had their chance and refunds are in. Net revenue per install is final for the cohort. This is the only day a variant can ship.

CALLS ALLOWEDSHIPKILLWATCH

EXPOSURE CAP30%of new installs in test arms
LOSS CEILING2%of monthly subscription revenue, checked weekly
KILL TRIGGER10%below control at the first checkpoint

My defaults from the Match test. We set yours together, per test, before launch.

Answers in weeks, not quarters.

The full field report has the allocation, the five downside controls and the checkpoint log.

READ THE FIELD REPORT simon@twinsalesinc.com linkedin.com/in/simon-harms