A/B testing

The term A/B is used to describe randomized experiments with two variants , A and B, one being the control and the other the variant. Another way to generally refer to A/B tests is with the term split test, although the latter method is applied when experiments with more than two variants are performed.

• Treatment Sometrhing to which a subject is exposed.
• Treatment group A group of subjects exposed to a specific treatment.
• Control group A group of subjects exposed to no treatment.
• Randomization The process of randomly assigning subjects to treatments.
• Subjects The items that are exposed to treatments
• Test statistic The metric used to measure the effect of the treatment.

Subjects are assigned to two (or more) groups that are treated exactly alike, except that the treatment under study differs from one group to another.

Ideally, subjects are assigned randomly to the groups

Hypothesis Tests

are ubiquitous in the traditional statistical analysis of published research. Their purpose is to help you learn whether random chance might be responsible for an observed effect

• Null Hypothesis The hypothesis that chance is to blame.
• Alternative hypothesis Counterpoint to the null.
• One-way test Hypothesis test that counts chance results only in one direction.
• Two-way test Hypothesis test that counts chance results in two directions. The hypothesis test assumes that the null hypothesis is true, creates a "null model", and tests whether the effect you observe is a reasonable outcome of that model

The null hypothesis

A null hypothesis is an assumption used to deny or affirm an event in relation to some parameter or parameters of a population or sample.

A null hypothesis is a logical construct embodying the notion that nothing special has happened, and any effect you observe is due to random chance

Alternative Hypothesis

Hypothesis tests by their nature involve not just a null hypothesis but also an offseting alternative hypothesis.

• Null="No difference between the means of group A and group B"; alternative = "A is different from B".
• Null="A<=A"; alternative ="A>B".

Taken togheter, the null and alternative hypotheses must account for all possibilities

Resampling

Resampling means to repeatedly sample values from observed data, with a general goal of assessing random variability in a statistic. It can also be used to asess and improve the accuracy of some machine-learning models

Drawind additional samples from an observed data set

there are two main types of resampling procedures: bootstrap and permutation tests.

Permutation test : also called randomization test,random permutation test, exact test The procedure of combining two or more samples together and randomly reallocating the observation to resamples..

• combine the results from the different groups into a single data set.
• Shuffle the combined data and then dandomly draw a resample of the same size as group A
• From the remaining data, randomly draw a resample of the same size as group B

Exhaustive and Bootstrap Permutation Test : In addition to the preceding random shuffling procedure, also called a random per‐ mutation test or a randomization test, there are two variants of the permutation test:

• Exhaustive permutation test: just randomly shuffling and dividing the data, we actually figure out all the possible ways it could be divided. This is practical only for relatively small sample sizes. This is also called Exact test
• A bootstrap permutation test: In this way the resampling procedure models not just the random element in the assigment of treatment to subject but also the random element in the section of subjects from a population. Permutation tests: Permutation tests are useful heuristic procedures for exploring the role of random variation. They are relatively easy to code, interpret, and explain, and they offer a use‐ ful detour around the formalism and “false determinism” of formula-based statistics, in which the precision of formula “answers” tends to imply unwarranted certainty. In a permutation test, are multiple samples are combined and then shuffled. The shuffled values are then divided into resamples, and the statistics of interest is calculated

Statistical Significance and p-Values

Statistical significance is how statisticians measure whether an experiment (or even a study of existing data) yields a result more extreme than what chance might produce. If the result is beyond the realm of chance variation, it is said to be statistically signicant

• p-value ; The p-value is the probability that results as extreme as the ovserved result might occur, given a null hypothesis model. given a chance model that embodies the null hypothesis, the p-value is the probability of obtaining results as unusual or extreme as the observed results .
• Alpha The alpha value is the threshold of "unusualness" in a null hypothesis chance model. The probability threshold of "unusualness" that chance results must surpass for actual outcomes to be deemed statistically significant
• Type 1 error Mistakenly concluding an effect is real (when it is due to chance)
• Type 2 error Mistakenly concluding an effect is due to chance (when it is real)
• Data science and p-values For a data scientist,a p-value iis a useful metric in sutitations where you want to know whether a model result that appears interesting and useful is within the range of normal chance variability

Significance test are used to determine whether an observed effect is within the range of chance variation for a null hypothesis model. the testing has been much more relevant for formal reporting of research than for data science

t-Test

A significance tests, depending on whether the data comprises count data or measured data, how many samples there are,and what's being measured. That is, a standardized version of common test statistics such as means

Multiple testing:

"Torture the data long enough, and it will confess."

For situations involving multiple statistical comparison, there are statistical adjustment procedures.

• Type 1 error Mistakenly concluding that an effect is statistically significant
• False discovery rate Across multiple test, the rate of making a Type 1 error.
• Alpha inflation The multiple testing phenomenon, in which alpha, the probability of making a Type 1 error, increases as you conduct more tests
• Adjustment of p-values Accounting for doing multiple tests on the same data
• Overfitting Fitting the noise
• alpha inflation The probability taht at least one predictor will (falsely) test significant is the flip side of this probability ,or 1-(Probability that all will be nonsigninicant)This issue is related to the problem of overfitting in data mining, or "fitting the model to the noise."

Degrees of freedom

The number of degrees of freedom is an input to many statistical tests. The degrees of freedom parameter, as applied to many probability distributions, affects the shape of the distribution

The number of degrees of freedom forms part of the calculation to standardize test statistics so they can be compared to reference distributions (t-distribution, F-distribution, etc.).And the concept of degrees of freedom lies behind the factoring of categorical vari‐ ables into n – 1 indicator or dummy variables when doing a regression (to avoid

ANOVA (Analisys of Variance)

The statistical procedure that tests for a statistically significant difference among the groups is called analysis of variance, or ANOVA

• Pairwise comparison A hypothesis test (e.g., of means) between two groups among multiple groups.
• Omnibus test A single hypothesis test of the overall variance among multiple group means.
• Decomposition of variance Separation of components contributing to an individual value (e.g., from the overall average, from a treatment mean, and from a residual error).
• F-statistic A standardized statistic that measures the extent to which differences among group means exceed what might be expected in a chance model. SS “Sum of squares,” referring to deviations from some average value.

Chi-Square Test

A measure of the extent to which some observed data departs from expectation The chi-square distribution is a good approximation of the shuffled resampling test jst described, except when counts are exremely low. In such cases, the resampling procedure will yield more accurate p-values. In fact,most statistical software has a procedure to actually enumerate all the possible rearrangements that can occur, tabulate their frequencies, and determine exactly how extreme the observed result is. The chi-square distribution is the reference distribution to which the observed calculated chi-square statistic must be compared

Multi arm bandit

An imaginary slot machine with multiple arms for the customer to choose from, each with different payoffs, here taken to be an analogy for a multitreatment experiment Traditional A/B tests envision a random sampling process, which can lead to excessive exposure to the inferior treatment. Multi-arm bandits, in contrast, after the sampling process to incorporate information learned during the experiment and reduce the frequency of the inferior treatment

Power and sample size:

• Power The probability of detecting a given effect size with a given sample size, That is the probability of detecting a specified effect size with specified sample characteristics (size and variability).
• Sample size The most common use of power calculations is to estimate how big sample you will need