Our Testing Methodology

September 23, 2024

Rapid development of features without frequent user testing is like running blindfolded without ever being able to check if you’re moving in the right direction.

That’s why our rule N1 is:

1. Test early

We test as soon as we have figured out the product’s success formula, i.e., the concept.

It’s very tempting to start development if the team likes the concept or investors have backed it up. None of these conditions count as a validation though. It’s just wishful thinking.

The sooner your concept hits the market the better. The deeper you are in development when you start testing, the more likely it is to face these problems:

you have developed complexity, a lot of moving parts – it’s harder to isolate and identify what causes issues;
you have developed attachment to your “baby” – it’s harder and more expensive to make changes.

2. The first thing we test

The product concept should include one fundamental assumption developed in detail: What will drive adoption? Why will they start using exactly this product?

So the first thing we are trying to find out is:

Do they start using it at all?
Do they understand how to start using it?
Is there any early friction, and why?

3. The second thing we test

The product concept should include another fundamental assumption developed in detail: What will drive continuous use? Why will they keep using the product (especially critical for SaaS or other subscription based products)?

So we try to find out:

If and how do they start engaging?
What do they engage with, and how regularly?
Do they try to hack the product, and how?

We intentionally design tests to be hackable whenever possible. This is especially valuable in the context of Opportunity type 3: Discover.

4. Test without developing for as long as possible

Test design is a specialized skill and a major part of it is figuring out what is the simplest and cheapest but still reliable way to test something.

A lot of fundamentals can be tested with paper prints, a lot of interaction mechanics with screenshots, and a lot of flows with Figma prototypes.

“The PilotProduct team insisted on designing and implementing interactive tests on every step of the way, which not only saved time from useless arguing between opinionated people but also ensured that we develop only what is proven to work resulting in substantial budget savings.”

Vessela Gercheva

Program Director Educational Experiences

5. Best forms of Pilot Product testing

In our experience, in this order:

In-person (usability) testing is by far the most revealing form of testing. Catching users’ genuine reactions in the immediate context is invaluable. It has the best ROI – number of sessions for the valuable insights gained.
In-product open chat is the second best form, if in-person testing is not an option. It is better than support chat, because users will not share certain things with you, but will openly discuss with each other.
Interaction data when targeted, simplified and presented as custom conversion funnels, can be, in our experience, extremely valuable.
Session recordings is the last form of testing we will use. We even try to avoid it because it is subject of biases, unknown circumstances and wild interpretations. It can still be valuable in few occasions.

6. Connecting with the users

Developing empathy towards the users is a practical necessity. Quantitative data has proven time and time again to be insufficient aid in designing successful products, especially in the initial product development stage.

To output a product that connects with the users, you have to be one of them, like a method actor would do it —

Spend time among them in their physical or virtual world, do what they do, feel what they feel.

Even Zuckerberg who might give you a robot psychopath vibe now, back then, when developing Facebook, walked amongst the users. He understood what is important to them and supported that with the pilot product features he developed.

7. Essential skills for testing

Testing happens in three main stages:

I. Designing the test

To design a test you need:

design skills
and engineering skills
mixed with creativity
and experience with research in the context of product development.

This is already a broad set of skills, but it gets more complicated from here.

II. Carrying out the test

To carry out a test successfully you need:

listening skills to be able to listen without jumping to conclusions or projecting your own beliefs, feelings, or ambitions
self-control to be able to watch users struggle with the product without interfering but patiently observing how they go about a problem
developed instincts to quickly identify and jump on an opportunity to learn much more from a situation by asking a new question or modifying the test on the fly

III. Interpreting the results

To interpret the results you also need:

detachment from the idea so you are free to pivot in a direction you haven’t considered but test results show as promising. Also to be able to look at failures as opportunities to learn instead of taking things personally
bias control and self-awareness so your judgment doesn’t get swayed by wishful thinking, subconscious biases, or coping with triggered emotions
adequate cognitive modelling so judgements are based on what actually is going on with the users’ brains when they use the product
knowing your limits so you don’t try to analyse and conclude beyond the available data or current understanding

8. Finally - the most important thing

Methodology and skills are important, but at the end what matters most is your intent – it will ultimately shape your actions, your judgements, and the results you’re getting.

Before you start testing, ask yourself: