⚙️ Software Engineering

Testing Fundamentals

"Testing shows the presence, not the absence of bugs." — Edsger W. Dijkstra 1

Testing Pyramid (Revisited)

graph TD A[E2E Tests] --> B[Integration Tests] B --> C[Unit Tests]
Layer Purpose Tools Target Coverage
Property-based Discover edge cases automatically Hypothesis (Py), fast-check (JS), jqwik (Java) Core algorithms
Mutation Assess test suite quality Pitest, mutmut, stryker Critical paths
Contract Consumer-driven API verification Pact, Spring Cloud Contract Service boundaries
Unit Fast feedback on logic pytest, JUnit, Vitest >90% on domain logic
Integration Verify component wiring Testcontainers, WireMock Happy paths + failures
E2E Critical user journeys Playwright, Cypress <10 critical flows

Test Categories Deep Dive

1. Unit Tests — Fast, Isolated, Deterministic

# Good: Single behavior, no I/O, descriptive name
def test_calculate_discount_applies_tiered_rates():
    customer = Customer(tier="gold", years_active=3)
    order = Order(total=1000, items=5)

    discount = PricingEngine().calculate(customer, order)

    assert discount == 150  # 10% base + 5% loyalty

# Bad: Multiple assertions, external dependency, flaky
def test_user_registration():
    response = requests.post("/api/users", json={...})
    assert response.status_code == 201
    assert User.exists(email="...")
    assert sent_welcome_email()
// Good: Single behavior, no I/O, descriptive name
TEST(PricingEngineTest, AppliesTieredRatesForGoldCustomer) {
    Customer customer("gold", 3);
    Order order(1000, 5);

    Money discount = PricingEngine().calculate(customer, order);

    ASSERT_EQ(Money(150), discount);
}

// Bad: Multiple assertions, external dependency, flaky
TEST(UserRegistrationTest, BadExample) {
    auto response = Post("/api/users", json{...});
    ASSERT_EQ(201, response.status_code());
    ASSERT_TRUE(User::Exists("alice@example.com"));
    ASSERT_TRUE(EmailSent("welcome"));
}
// Good: Single behavior, no I/O, descriptive name
@Test
void testCalculateDiscountAppliesTieredRates() {
    Customer customer = new Customer("gold", 3);
    Order order = new Order(1000, 5);

    Money discount = pricingEngine.calculate(customer, order);

    assertEquals(150, discount.getCents());
}

// Bad: Multiple assertions, external dependency, flaky
@Test
void testUserRegistration() {
    var response = post("/api/users", json);
    assertEquals(201, response.getStatusCode());
    assertTrue(User.exists("alice@example.com"));
    assertTrue(Email.wasSent("welcome"));
}
// Good: Single behavior, no I/O, descriptive name
[Fact]
public void CalculateDiscount_AppliesTieredRates_ForGoldCustomer() {
    var customer = new Customer("gold", 3);
    var order = new Order(1000, 5);

    var discount = _pricingEngine.Calculate(customer, order);

    Assert.Equal(new Money(150), discount);
}

// Bad: Multiple assertions, external dependency, flaky
[Fact]
public void UserRegistration_BadExample() {
    var response = _client.Post("/api/users", json);
    Assert.Equal(201, response.StatusCode);
    Assert.True(User.Exists("alice@example.com"));
    Assert.True(Email.WasSent("welcome"));
}
Principle Practice
F.I.R.S.T. Fast, Independent, Repeatable, Self-validating, Timely
AAA Pattern Arrange → Act → Assert
One logical assert Multiple physical assertions OK if single concept
No I/O Mock filesystem, network, time, randomness

2. Property-Based Testing — Discover the Unknown

# Instead of hand-picked examples, generate thousands
from hypothesis import given, strategies as st

@given(st.lists(st.integers(min_value=0, max_value=100)))
def test_sort_is_idempotent(xs):
    assert sorted(xs) == sorted(sorted(xs))

@given(st.text(alphabet=st.characters(blacklist_categories=("Cc", "Cs"))))
def test_escaped_string_roundtrip(s):
    assert unescape(escape(s)) == s
#include <rapidcheck/rapidcheck.h>

void test_sort_is_idempotent() {
    rc::check("sort is idempotent", [](const std::vector<int>& xs) {
        auto sorted_once = xs;
        std::sort(sorted_once.begin(), sorted_once.end());
        auto sorted_twice = sorted_once;
        std::sort(sorted_twice.begin(), sorted_twice.end());
        RC_ASSERT(sorted_once == sorted_twice);
    });
}

#include <rapidcheck/rapidcheck.h>

void test_escaped_string_roundtrip() {
    rc::check("escape/unescape roundtrip", [](const std::string& s) {
        RC_ASSERT(unescape(escape(s)) == s);
    });
}
import net.jqwik.api.*;
import java.util.*;

class PropertyTests {
    @Property
    boolean sortIsIdempotent(@ForAll List<Integer> xs) {
        var sortedOnce = new ArrayList<>(xs);
        Collections.sort(sortedOnce);
        var sortedTwice = new ArrayList<>(sortedOnce);
        Collections.sort(sortedTwice);
        return sortedOnce.equals(sortedTwice);
    }

    @Property
    boolean escapeUnescapeRoundtrip(@ForAll String s) {
        return unescape(escape(s)).equals(s);
    }
}
// FsCheck for C#
using FsCheck;
using FsCheck.Xunit;

public class PropertyTests {
    [Property]
    public bool SortIsIdempotent(List<int> xs) {
        var sortedOnce = xs.OrderBy(x => x).ToList();
        var sortedTwice = sortedOnce.OrderBy(x => x).ToList();
        return sortedOnce.SequenceEqual(sortedTwice);
    }

    [Property]
    public bool EscapeUnescapeRoundtrip(string s) {
        return Unescape(Escape(s)) == s;
    }
}
Benefit Example
Finds edge cases Empty list, single element, duplicates, overflow
Documents invariants sorted(xs) == sorted(sorted(xs)) = idempotency
Shrinks failures Minimal counterexample for debugging

3. Mutation Testing — Test Your Tests

┌─────────────────────────────────────────────────────┐
│  Original Code           Mutant                    │
│  ────────────────────── ────────────────────────── │
│  if (x > 0)              if (x >= 0)    Killed?   │
│      return x                return x              │
│  else                    else                      │
│      return -x               return -x             │
└─────────────────────────────────────────────────────┘
Metric Target Action if Low
Mutation Score ≥80% (core) Add tests for surviving mutants
Kill Rate ≥90% Improve assertion specificity
Coverage ≠ Quality Mutation score > line coverage

4. Contract Testing — Consumer-Driven

# Consumer (frontend) defines expected contract
from pact import Consumer, Provider, Given, UponReceiving, WithState

pact = Consumer('OrderService').has_pact_with(Provider('PaymentService'))

pact.given('a valid payment request')
    .upon_receiving('a charge request')
    .with_request('POST', '/payment/charge', body={
        'amount_cents': 1999,
        'currency': 'USD',
        'source': 'tok_visa'
    })
    .will_respond_with(200, body={
        'transaction_id': 'txn_123',
        'status': 'succeeded'
    })

pact.verify()
// Pact C++ is limited
// Use pact-verifier CLI for verification
# pact-verifier --provider-base-url=http://localhost:8080 \
#   --pact-url=./pacts/order_service-payment_service.json
// Pact JVM
@ExtendWith(PactConsumerTestExt.class)
@PactTestFor(providerName = "PaymentService")
class PaymentContractTest {
    @Test
    @Pact(consumer = "OrderService", provider = "PaymentService")
    V2RequestResponsePact createPact(PactDslWithProvider builder) {
        return builder
            .given("a valid payment request")
            .uponReceiving("a charge request")
            .path("/payment/charge")
            .method("POST")
            .body(newJsonBody(o -> o
                .numberType("amount_cents", 1999)
                .stringType("currency", "USD")
                .stringType("source", "tok_visa")
            ))
            .willRespondWith()
            .status(200)
            .body(newJsonBody(o -> o
                .stringType("transaction_id", "txn_123")
                .stringType("status", "succeeded")
            ))
            .toPact(V2Pact.class);
    }
}
// PactNet
using PactNet;
using Xunit;

public class PaymentContractTests {
    [Fact]
    public void PaymentContract() {
        var config = new PactConfig { PactDir = "./pacts" };
        var pact = new PactVerifier(new PactConfig { PactDir = "./pacts" });

        var builder = new V2PactBuilder("OrderService", "PaymentService");
        builder
            .Given("a valid payment request")
            .UponReceiving("a charge request")
            .With(new ProviderServiceRequest {
                Method = HttpVerb.Post,
                Path = "/payment/charge",
                Body = new { amount_cents = 1999, currency = "USD", source = "tok_visa" }
            })
            .WillRespondWith(200, new {
                transaction_id = "txn_123",
                status = "succeeded"
            });
    }
}
Approach Pros Cons
Consumer-driven (Pact) Prevents breaking changes Requires broker
Provider-driven (OpenAPI) Single source of truth May miss consumer needs
Bi-directional Both sides validated More complex setup

5. Integration Testing — Real Dependencies

# Testcontainers: Real DB in container, auto-cleanup
@pytest.fixture
def postgres():
    with PostgresContainer("postgres:16") as pg:
        yield pg.get_connection_url()

def test_user_repository_saves_and_loads(postgres):
    repo = UserRepository(postgres)
    user = User(name="Alice", email="alice@example.com")

    repo.save(user)
    loaded = repo.find_by_email("alice@example.com")

    assert loaded.name == "Alice"
    assert loaded.id is not None
# Testcontainers C++ (using testcontainers-cpp or similar)
#include <testcontainers/postgres.hpp>

using namespace testcontainers;

auto postgres = PostgresContainer("postgres:16").start();
auto conn = postgres.get_connection_url();

// Or use testcontainers-cpp library
auto container = PostgresContainer()
    .withImageName("postgres:16")
    .withStartupTimeout(std::chrono::seconds(60))
    .start();
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;

@Testcontainers
class UserRepositoryTest {
    @Container
    static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>("postgres:16");

    @Test
    void testUserRepositorySavesAndLoads() {
        var repo = new UserRepository(postgres.getJdbcUrl());
        var user = new User("Alice", "alice@example.com");

        repo.save(user);
        var loaded = repo.findByEmail("alice@example.com");

        assertEquals("Alice", loaded.getName());
        assertNotNull(loaded.getId());
    }
}
using Testcontainers.PostgreSql;

public class UserRepositoryTests : IAsyncLifetime {
    private readonly PostgreSqlContainer _postgres = new PostgreSqlBuilder()
        .WithImage("postgres:16")
        .Build();

    public async Task InitializeAsync() => await _postgres.StartAsync();

    [Fact]
    public async Task TestUserRepositorySavesAndLoads() {
        var repo = new UserRepository(_postgres.GetConnectionString());
        var user = new User("Alice", "alice@example.com");

        await repo.SaveAsync(user);
        var loaded = await repo.FindByEmailAsync("alice@example.com");

        Assert.Equal("Alice", loaded.Name);
        Assert.NotNull(loaded.Id);
    }

    public async Task DisposeAsync() => await _postgres.DisposeAsync();
}
Anti-pattern Better Alternative
Shared test DB Testcontainers per test/class
Mocking repository Real DB + transaction rollback
Sleep/wait for async Polling with timeout + condition

Test Quality Metrics

Metric Good Warning Action
Execution time (unit) <100ms >1s Refactor slow tests
Flakiness rate 0% >1% Quarantine & fix
Mutation score >80% <60% Add missing assertions
Branch coverage >85% <70% Target uncovered branches
Test:code ratio 1:1 to 2:1 >3:1 Possible over-testing

Testing in CI/CD

Stage Tests Time Budget
Pre-commit Unit (changed files), lint <30s
PR Pipeline Full unit, contract, mutation (diff) <5min
Merge/Main All unit, integration, contract <15min
Nightly Full mutation, E2E, load, chaos Unlimited

Anti-Patterns to Avoid

Anti-pattern Symptom Fix
Test Pyramid Inversion Many E2E, few unit Push logic down; contract test boundaries
Testing Implementation Brittle tests on refactor Test behavior, not private methods
Over-mocking Tests pass, prod fails Use real dependencies for integration
Test-only Code Paths Coverage ≠ confidence Mutation testing reveals gaps
Ignoring Flaky Tests "It passes sometimes" Quarantine → root cause → fix/delete

Further Reading

Resource Focus
Effective Software Testing (Aniche, 2022) 2 Modern testing toolbox
xUnit Test Patterns (Meszaros, 2007) 3 Pattern catalog for test code
Growing Object-Oriented Software, Guided by Tests (Freeman & Pryce, 2009) 4 TDD at scale
The Little Book of Mutation Testing (Pizana, 2023) 5 Practical mutation testing
ISO/IEC/IEEE 29119 6 International testing standards

References


  1. Dijkstra, E.W. (1972). "The Humble Programmer." ACM Turing Award Lecture. https://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/EWD340.html 

  2. Aniche, M. (2022). Effective Software Testing: A Developer's Guide. Manning Publications. https://www.manning.com/books/effective-software-testing 

  3. Meszaros, G. (2007). xUnit Test Patterns: Refactoring Test Code. Addison-Wesley Professional. https://www.oreilly.com/library/view/xunit-test-patterns/9780131495050/ 

  4. Freeman, S. & Pryce, N. (2009). Growing Object-Oriented Software, Guided by Tests. Addison-Wesley Professional. https://www.growing-object-oriented-software.com/ 

  5. Pizana, N.J. (2023). The Little Book of Mutation Testing. Leanpub. https://leanpub.com/mutation-testing 

  6. ISO/IEC/IEEE. (2013). ISO/IEC/IEEE 29119:2013 — Software and systems engineering — Software testing. https://www.iso.org/standard/44889.html