Probability

Probability measures how likely an event is, from 0 (impossible) to 1 (certain). It is the mathematics of uncertainty behind sampling, hashing, networks, randomised algorithms and machine learning.

4.1 Basic definitions

An experiment has several possible outcomes; the sample space S is the set of all outcomes; an event is a subset of S. Events are mutually exclusive (cannot occur together), exhaustive (together cover S), independent (one does not affect the other's probability).

Classical definition: P (A) = (number of outcomes favourable to A ) / ( \mathrm{total} equally likely outcomes). Empirical (relative frequency): P (A) = \lim (times A occurred)/(trials).

Axioms: 0 \leq P (A) \leq 1 ;\ P (S) = 1; for mutually exclusive A ,\ B ,\ P (A \cup B) = P (A) + P (B). Complement: P (A^{c}) = 1 - P (A).

4.2 Two basic laws

Addition law: P (A \cup B) = P (A) + P (B) - P (A \cap B); for mutually exclusive events the last term is 0.

Multiplication law: P (A \cap B) = P (A) \cdot P ( B | A); for independent events P (A \cap B) = P (A) P (B).

4.3 Conditional probability and Bayes' theorem

Conditional probability of A given that B has occurred: P ( A | B ) = \frac{P (A \cap B)}{P} ( B), P (B) > 0. A and B are independent iff P ( A | B ) = P ( A).

Bayes' theorem reverses the conditioning: P ( A | B ) = P ( B | A ) \frac{P (A)}{P} ( B), with P (B) from the total probability law, P (B) = \sum P ( B | A_{i} ) P ( A_{i}).

4.4 Random variables and probability distributions

A random variable X assigns a number to each outcome. Discrete (counts) has a probability mass function p (x) with \sum p (x) = 1; continuous (measurements) has a density f (x) with total area 1, and probabilities are areas.

Expected value E (X) = \sum x \cdot p (x) (mean); variance Var(X) = E (X^{2}) - [E (X)]^{2}.

Binomial distribution

For n independent trials, each with success probability p: P (X = x) = C (n ,\ x) p^{x} (1 - p)^{n-x}, x = 0 \ldots n. Mean = n p, variance = n p q (q = 1 - p).

Poisson distribution

Counts of rare events in a fixed interval with average rate λ: P (X = x) = \frac{e^{- \lambda} \lambda^{x}}{x !}; mean = variance = λ. It approximates the binomial when n is large and p small (\lambda = n p).

Normal distribution

The bell curve, symmetric about the mean μ: f (x) = \left( \frac{1}{\sigma \sqrt{2 \pi}} \right) e^{- \frac{(x - \mu)^{2}}{(2 \sigma^{2})}}. Written X ~ N (\mu ,\ \sigma^{2}).

Standardise: Z = \frac{X - \mu}{\sigma} ~ N ( 0, 1), then use the standard normal table of \Phi (z) = P (Z \leq z). Empirical rule: P (\mu - \sigma < X < \mu + \sigma) = 0.6827 ,\ \pm 2 \sigma: 0.9545 ,\ \pm 3 \sigma: 0.9973.

The total area is 1, the curve is symmetric so \Phi (- z) = 1 - \Phi (z), and mean = median = mode.

z	0	0.5	1.0	1.5	2.0	2.5	3.0
\Phi (z)	0.5000	0.6915	0.8413	0.9332	0.9772	0.9938	0.9987

(a) Galton board (quincunx), Matemateca, University of São Paulo

(b) The 68-95-99.7 rule

4.6 Eight uses of probability and distributions

1. Risk and uncertainty in software. Probability turns "it may fail" into a number such as 0.4% per day. A team can then compare the cost of failure with the cost of prevention. Nepali digital-payment systems such as Fonepay plan their servers around such numbers.
2. Spam and fraud filters. Bayes' theorem updates the chance that a message is spam after seeing words such as "lottery". The same idea flags suspicious card or wallet transactions. As Figure 4.1 shows, a low base rate keeps many alarms false.
3. Reliability and packet loss. The binomial distribution gives the chance that exactly k of n packets are lost. Network engineers use it to size retransmission and buffer settings. A video call over a weak mobile link is exactly this problem.
4. Counting rare events. The Poisson distribution models failures, arrivals of calls or accidents per period. A bank branch in Butwal uses it to decide how many counters to open at peak hour. Too few counters make queues; too many waste salaries.
5. Measurement errors and natural variation. Heights, marks, weights and manufacturing sizes follow the normal curve. With only the mean and σ one can state what share lies above or below any value. Exam boards use this for grading on a curve.
6. Quality control limits. A factory fixes control limits at \mu \pm 3 \sigma because 99.73% of normal output falls inside them. A point outside signals that the process has shifted. This is the basis of Six Sigma programmes.
7. Insurance and finance. Insurers price a policy from the probability of a claim and its expected size. Banks estimate the chance a borrower defaults. Both rely on expected value E (X) and variance from this unit.
8. Machine learning and simulation. Classifiers output probabilities, and many algorithms assume normal errors. Monte Carlo simulation draws random numbers from these distributions to test designs before they are built. Games, weather models and network studies all use this.

(a) A rapid test cassette: one line or two

(b) 1,000 people tested twice: base rate matters

Formula and fact sheet

• P (A \cup B) = P (A) + P (B) - P (A \cap B) ;\ P (A \cap B) = P (A) P ( B | A ) ;\ P ( A | B ) = \frac{P (A \cap B)}{P} ( B).
• Bayes: P ( A | B ) = P ( B | A ) \frac{P (A)}{P} ( B).
• Binomial: C (n ,\ x) p^{x} q^{n-x}; mean n p ,\ v a r n p q. Poisson: \frac{e^{- \lambda} \lambda^{x}}{x}!; mean = v a r = \lambda.
• Normal: Z = \frac{X - \mu}{\sigma} ;\ \pm 1 \sigma 68.27%, \pm 2 \sigma 95.45%, \pm 3 \sigma 99.73%.

Chapter summary

• Probability follows the addition and multiplication laws; conditioning updates probabilities and Bayes' rule inverts them.
• Binomial models fixed trials, Poisson rare events, normal measurements, which are standardised to Z for table lookup.

Practice

1. Two coins are tossed. Find the probability of at least one head.
2. A bag has 5 red and 3 blue balls. Two drawn without replacement: P ( \mathrm{one} of each colour)?
3. P (A) = 0.6 ,\ P (B) = 0.5 ,\ P (A \cup B) = 0.8. Find P (A \cap B) and P ( A | B). Are A and B independent?
4. A fair coin is tossed 6 times. Find the probability of exactly 3 heads, and the mean and variance of the number of heads.
5. Calls arrive at 2 per minute (Poisson). P ( n o calls in a minute)?
6. X ~ N (50 ,\ 10^{2}). Find P (40 < X < 65).
7. Marks are N (60 ,\ 15^{2}). What fraction scores below 40? Above 90?
8. Explain independence v s mutual exclusivity.

Answers and worked solutions

1. 1 - P ( n o heads) = 1 - \frac{1}{4} = \frac{3}{4}.

2. P ( R then B ) + P ( B then R ) = \left( \frac{5}{8} \right) \left( \frac{3}{7} \right) + \left( \frac{3}{8} \right) \left( \frac{5}{7} \right) = \frac{15}{56} + \frac{15}{56} = \frac{15}{28}.

3. P (A \cap B) = 0.6 + 0.5 - 0.8 = 0.3; P ( A | B ) = 0.3 / 0.5 = 0.6 = P (A), so they are independent (0.6 \times 0.5 = 0.3).

4. \frac{C (6 ,\ 3)}{2^{6}} = \frac{20}{64} = 0.3125; mean = 3, variance = 6 \times 0.25 = 1.5.

5. e^{-2} = 0.1353.

6. z = - 1 and 1.5: 0.9332 - 0.1587 = 0.7745.

7. Below 40: z = - 1.33 ,\ \Phi = 0.0918 (9.2%). Above 90: z = 2 ,\ 1 - 0.9772 = 2.3%.

8. Independent: occurrence of one does not change the probability of the other (P (A \cap B) = P (A) P (B)). Mutually exclusive: cannot both occur (P (A \cap B) = 0). Events with nonzero probabilities cannot be both.

Model answers: medium (5 marks) and long (10 marks) questions

Q1 (5 marks). Machine A makes 60% of the chips in a factory and machine B makes 40%. 2% of A's chips and 5% of B's chips are defective. A chip chosen at random is defective. Find the probability that it came from A.

Introduction. This is a reversed-conditioning problem, so we use Bayes' theorem: P ( A | D ) = P ( D | A ) \frac{P (A)}{P} ( D), where P (D) comes from the law of total probability.

Given. P (A) = 0.60 ,\ P (B) = 0.40 ,\ P ( D | A ) = 0.02 ,\ P ( D | B ) = 0.05.

Total probability of a defect. P (D) = P ( D | A ) P (A) + P ( D | B ) P (B) = 0.02 \times 0.60 + 0.05 \times 0.40 = 0.012 + 0.020 = 0.032.

Bayes. P ( A | D ) = 0.012 / 0.032 = 0.375. Then P ( B | D ) = 0.020 / 0.032 = 0.625, and the two add up to 1 as they must.

Interpretation. Machine A produces more chips, but machine B is far worse, so a defective chip is more likely to be from B (62.5%) than A (37.5%).

Conclusion The prior share of production (60%) is updated to 37.5% after we learn the chip is defective; that is the meaning of a posterior probability.

Q2 (5 marks). The response time of a server is normally distributed with mean 120 ms and standard deviation 15 ms. Find (a) P (X > 150), (b) P (100 < X < 140), (c) the response time that 95% of requests beat.

Introduction. For X ~ N (\mu ,\ \sigma^{2}) we standardise with Z = \frac{X - \mu}{\sigma} and read \Phi (z) from the standard normal table (Figure 4.3).

\ast (a) Slow responses.* z = \frac{150 - 120}{15} = 2.0. P (X > 150) = 1 - \Phi (2.0) = 1 - 0.9772 = 0.0228, so about 2.3% of requests are slower than 150 ms.

\ast (b) Between 100 and 140 . \ast z_{1} = \frac{100 - 120}{15} = - 1.33 and z_{2} = + 1.33. P = \Phi (1.33) - \Phi (- 1.33) = 2 \Phi (1.33) - 1 \approx 2 \times 0.9088 - 1 ≈ 0.818 (more exact z = 1.3333 gives 0.8176). About 82% of requests fall in this band.

\ast (c) The 95th percentile.* We need z with \Phi (z) = 0.95, which is z = 1.645. x = 120 + 1.645 \times 15 = 144.7 ms. So 95% of requests finish within about 145 ms.

Conclusion. The tail beyond two standard deviations is small (2.3%), and an engineer would set the timeout alarm near 145 ms to catch only the slowest 5%.

Q3 (10 marks). Explain the binomial and Poisson distributions with their conditions, mean and variance. A batch has a 10% defect rate and 10 items are inspected. Find P (X = 2) and P (X \leq 1) using the binomial, then compare with the Poisson approximation.

Introduction. A random variable assigns a number to each outcome of an experiment. Two of the most useful discrete distributions are the binomial (counts of successes in fixed trials) and the Poisson (counts of rare events in an interval). Figure 4.2 shows both.

*Binomial distribution B (n ,\ p) . \ast

1. Conditions: a fixed number n of trials; each trial has two outcomes (success or failure); constant success probability p; trials independent.
2. Formula: P (X = x) = C (n ,\ x) p^{x} q^{n-x}, with q = 1 - p and x = 0 ,\ 1 ,\ \ldots ,\ n.
3. Mean = n p, variance = n p q.
4. Shape: symmetric if p = 0.5, skewed otherwise.

*Poisson distribution P (\lambda) . \ast

1. Conditions: events occur singly, at a constant average rate λ per interval, independently of each other.
2. Formula: P (X = x) = \frac{e^{- \lambda} \lambda^{x}}{x !}, with x = 0 ,\ 1 ,\ 2 ,\ \ldots
3. Mean = variance = λ.
4. Link: it approximates B (n ,\ p) when n is large and p is small, with \lambda = n p.

*Calculation (binomial, n = 10 ,\ p = 0.1 ) . \ast P (X = 2) = C (10 ,\ 2) (0.1)^{2} (0.9)^{8} = 45 \times 0.01 \times 0.4305 = 0.1937. P (X = 0) = 0.9^{10} = 0.3487 and P (X = 1) = 10 \times 0.1 \times 0.9^{9} = 0.3874, so P (X \leq 1) = 0.7361.

Mean = n p = 1.0 and variance = n p q = 0.9.

*Poisson approximation (\lambda = n p = 1) . \ast P (X = 2) = e^{-1} \times \frac{1^{2}}{2 !} = 0.1839. P (X = 0) = P (X = 1) = e^{-1} = 0.3679, so P (X \leq 1) = 0.7358.

Comparison of the two answers (Table).

Quantity	Binomial (exact)	Poisson (\lambda = 1)
P (X = 2)	0.1937	0.1839
P (X \leq 1)	0.7361	0.7358
Mean	1.0	1.0
Variance	0.9	1.0

Binomial versus Poisson (Table).

Point	Binomial	Poisson
Trials	fixed n	no fixed n, an interval
Possible values	0 to n	0, 1, 2, … without limit
Parameters	n and p	λ only
Mean and variance	n p and n p q	both λ
Typical use	defects in a sample, lost packets	calls per minute, failures per day

Conclusion. The Poisson answer is close to the exact binomial one (0.7358 against 0.7361 for P (X \leq 1)), so for rare events the simpler Poisson model is a good substitute. The small gap in P (X = 2) shows the approximation improves as n grows and p shrinks.