How Garasu works

An experiment is one concluded test. A learning is a product claim, written in plain language, tied to the tests that support it.

Import an experiment

Call import_experiment with the project id and the test.

Send the hypothesis, start date, end date, every variant key, the control key, and raw results. Do not send processed results. Garasu computes lift, p-value, and power from the raw rows.

Each raw row is one metric on one variant. A rate metric needs users and conversions. A continuous metric needs users and mean, plus variance or sum_squares when you have them.

Skip drafts and tests that are still running. A second import of the same experiment returns an error instead of a second copy.

remove_experiment deletes that import, its links, and any learning left with no experiments. Confirm with the user first. It cannot be undone.

Create a learning

After the experiment is stored, Garasu starts a job that writes a learning. The learning is a principle about the product or the customer, not a recap of the test setup, and not advice about how to run the next test.

If the new write-up states the same principle as an older learning, Garasu attaches the experiment to the older learning.

sync_status is syncing while that job runs, then idle.

Strength of evidence

Each learning has strength_of_evidence from 0 to 1.

0 means the linked results do not support the claim. 1 means the linked results support the claim.

Garasu reads each result's post-hoc power, the chance that sample had of detecting the observed effect. A match with low power is weak support. A match with high power is strong support. A flat result with high power can support a claim that the change had no meaningful effect. An inconclusive result with low power supports little. The score ignores how important the idea feels.

The app shows the score as a percent.

list_learnings sorts by this score, highest first, unless you pass another order_by.