Propensity Score Matching Generator

When to use the PSM?

You may use the propensity score analysis when you couldn't randomized the treatment. The propensity score matching help to reduce the effect of the confounding variables (covariates) by matching similar subjects between treatment group and the control group.

What is propensity score

Most often, propensity scores are estimated as the likelihood that a person would be assigned or self-select into a treatment condition. [1].
For example, the propensity score can be estimated using a logistic regression model.

Balance Estimation

To estimate balance, we compare the covariates between the treatment group and control group, before

1. Covariate.
2. Treatment mean - the average value of the covariate in the treatment groups.
3. Control mean - the average value of the covariate in the control groups.
4. Std Bias (SB) - standardized bias, the standardized mean difference.
5. Var Ratio - the ratio of the variance between the treatment group and the control group
6. Average D - the average difference of the empirical cumulative distribution function (eCDF) between the treatment group and the control group
7. D - the maximum difference of the empirical cumulative distribution function (eCDF) between the treatment group and the control group

Compares the distribution of each covariate between the treatment group and the control group.
A balanced propensity score does not imply balanced covariates (Austin, 2009), and vice versa.
For example, when using logistic regression to calculate the score, the score represents the probability of treatment based on the covariates.
Different combinations of covariates may lead to similar treatment probabilities (scores).
Since the data is matched using the score, it may result in non-balanced matching.

Standardized Mean Difference

There is no clear standard for the S_B value of a balanced covariate. However, |S_B| should be smaller than 0.2, or preferably smaller than 0.1 for important covariates.

Numerical Covariates

Calculates the standardized difference between the estimated mean of the treatment group and the estimated mean of the control group.

Categorical Covariates

For each value of the categorical covariate, calculates the standardized difference between the estimated proportion of the treatment group and the estimated proportion of the control group.

Variance Ratios Balance

The variance ratio is the ratio of the variance of a covariate in the treatment group to the variance of the same covariate in the control group.
For a well-balanced matching, we expect these variances to be similar, with the ratio close to 1.
The variance ratio should fall between 0.5 and 2 (Rubin, 2001).
$\mathrm{Var\; Ratio}=\frac{{{\sigma }^2}_T}{{{\sigma }^2}_C}$

Empirical Cumulative Distribution Function (eCDF)

Comparing the eCDF of the treatment group and the eCDF of the control group:
D - The maximum difference between the eCDF of the treatment group and the eCDF of the control group (the statistic of Kolmogorov-Smirnov test).
Average D - The average difference between the eCDF of the treatment group and the eCDF of the control group. For a well-balanced matching, we expect D, and Average D to be small.

Input Data Structure

1. ID - The first column contains the unique ID. The input data must include the ID column for data completeness, but the PSM process will not use it.
2. Covariates - The subsequent columns represent covariates, which can be either numerical or categorical variables. These covariates are independent variables that are not of primary interest.
3. Treatment - The second-to-last column represents the treatment. This is the key independent variable. For regular (two-group) matching, it should contain only values of 1 or 0 (1 indicates treatment, 0 indicates control). For three-group matching, it should contain values of 0, 1, or 2, representing the control group and two treatment groups respectively.
4. Output - The last column contains numerical data, representing the dependent variable.

How to use the PSM calculator

PSM calculator with optional Excel file input. Propensity score estimation is performed using logistic regression, and matching is done using the nearest neighbor method, with an optional caliper.

How to enter data?

• Enter raw data directly - usually you have the raw data.
  a. Enter the name of the group.
  b. Enter the raw data separated by 'comma', 'space', or 'enter'. (*you may copy only the data from excel).
• Enter raw data from excel

  Enter the header on the first row.

  1. Copy Paste
     
     • a. copy the raw data with the header from Excel or Google sheets, or any tool that separates data with tab and line feed. copy the entire block, include the header .
     • Paste the data in the input field.
  2. Import data from an Excel or CSV file.
     When you select an Excel file, the calculator will automatically load the first sheet and display it in the input field. You can choose either an Excel file (.xlsx or .xls) or a CSV file (.csv).
     To upload your file, use one of the following methods:
     1. Browse and select – Click the 'Browse' button and choose the file from your computer.
     2. Drag and drop – Drag your file and drop it into the 'Drop your .xlsx, .xls, or .csv file here!' area.
     Once the file is uploaded, the PSM calculator will display the data from the first sheet in the input field.
     Now, the 'Select sheet' dropdown will be populated with the names of your sheets, and you can choose any sheet.
  3. Filter Data
     When using the 'Enter data from Excel' option, you can filter the data by clicking the following icon above the header:
     You may select one or more values from the dropdown. Please note that the filter will include any value that contains the values you choose.

Assumptions

1. The treated subjects and the control subjects have a similar probability of receiving the treatment.
2. All subjects receive the same type and amount of treatment.
3. No general equilibrium effect - the control subjects don't get the treatment indirectly.
4. Sufficient overlap between the treated group and the control group.
5. Conditional independent - the outcomes (y) are independent of the treatment.
6. No Hidden Bias: All confounding variables are included in the model

Logistic regression parameters

1. Score Calculation:
   Logistic Regression - the columns Name-1, Name-2, etc. are covariates used to calculate the score using logistic regression.
   Existing score column - in this case, there is only one column, Name-1, which represents the score.
2. Learning Rate(α): The learning rate represents the size of the gradient step in each iteration. It controls how much the coefficients are adjusted during each iteration of the optimization process. A smaller alpha means smaller steps in gradient descent, which can lead to more precise convergence but might require more iterations and longer calculation time.
   Common alpha values typically range from 0.1 to 0.001 when using a constant learning rate (decay rate = 1).
   When using a decayed learning rate (decay rate < 1), you may start with a higher learning rate, typically ranging from 1 to 10.
3. Decay Rate:
   The decay rate refers to the reduction of the learning rate over time. A large learning rate allows faster training, while a small learning rate offers more accuracy by reducing the chance of overshooting the optimal point. Gradually decreasing the learning rate can combine the benefits of both approaches — it starts fast and becomes more accurate as it slows down toward the end of the optimization process. In each iteration:
   In any iteration in which the cost does not decrease, the algorithm reduces the learning rate as follows:
   Learning Rate = Learning Rate * Decay Rate.
4. Replacement:
   With replacement – when matching, the same control subject may be matched more than once (for ATT), or the same treated subject may be matched more than once (for ATC).
   Without replacement – each subject is matched only once.
   
5. Effect Type - Estimand (2 groups)::
   On Treated (ATT) – Average Treatment Effect on the Treated; for each treated subject, match the best control subject.
   On Controls (ATC) – Average Treatment Effect on the Controls; for each control subject, match the best treated subject.
   
6. Matching Order (3 groups):
   Sets the sequence in which the three groups are matched — the First, Second, and Third steps, shown as the Matching Step column in the results. Matching is done in two steps: the First group is matched to the Second, and the Second is then matched to the Third. This fixes the trio; the First–Third pair is not matched separately — its distance is only added into the total distance, and among the candidate trios the one with the smallest total distance is chosen.
   Relevant only to 3-group matching with replacement. Without replacement, the order is set automatically from the smallest group to the largest (small, medium, large).
7. Matches to check: (3 groups):
   Instead of keeping only the single closest match at each step, this is the number of nearest candidate matches considered in the First and Second matching steps. Every combination of these candidates forms a trio; we then compute the total distance of each trio and keep the one with the smallest total distance.
   A larger value explores more trios, which can improve match quality at the cost of longer calculation time.
8. Penalty(λ): This parameter controls the amount of regularization applied to the model. It is a shrinkage parameter that penalizes large coefficients to prevent overfitting. When lambda is set to zero, no regularization is applied, and the model behaves like ordinary least squares (OLS). As lambda increases, more penalty is applied, shrinking the coefficients towards zero. This helps in reducing model complexity and can improve generalization on unseen data
9. Maximum Iterations: On each iteration, the algorithm changes the coefficients in a direction that will increase the log-likelihood. A higher number of iterations leads to better results until it reaches the maximum log-likelihood. In this case, more iterations will not lead to a better result.
10. Maximum Run Time (Minutes): Limits the calculation time, even if the number of iterations does not reach the 'Maximum iterations' or the epsilon does not reach 0.
11. Epsilon: We calculate the cost every 100 iterations. Epsilon is the difference between the new cost and the previous cost.

Matching

1. Find the group with fewer subjects; let's assume it is the treatment group.
2. Sort both groups by score. If 'Score order' is 'Larger First', sort descending; if 'Smaller First', sort ascending.
3. For each treatment subject, starting from the first:
   a. Match the closest control subject: minimum |Treatment score - Control score|.
   b. If using a caliper, discard the treatment subject if it does not meet the caliper criterion.
   c. Remove the matched subject from the control group.

Options

1. Score order:
   Larger first - Sort the treatment scores in descending order and start matching from the highest treatment score.
   Smaller first - Sort the treatment scores in ascending order and start matching from the lowest treatment score.
2. Caliper Matching
   None - Use all treatment subjects; do not discard any treatment subjects.
   Apply Caliper Bandwidth - Discard a treatment subject if the distance to the nearest control subject is greater than the 'caliper distance'.
   For three-group matching the caliper applies to all three pairs of the triplet, not only to the two pairs that are matched in sequence. A subject whose every candidate triplet breaks the caliper on the first-to-third pair is reported as Unmatched rather than Exceed Caliper, because it was never discarded on its own nearest-neighbour distance.
   
3. Distance Measure
   Unstandardized - Uses the raw propensity score difference between subjects.
   Standardized - Uses the propensity score difference divided by the standard deviation of the propensity scores.
   
4. Caliper Distance - Used to calculate the threshold value in Caliper bandwidth.
5. Cost check frequency - the default is checking every 100 iterations. If the cost stays the same after 100 iterations, the algorithm will stop.
6. Logistic Regression - Displays the names of the coefficients or only x1, x2, x3 etc.
7. Matching Report (3-groups) - Controls which matching-related columns appear in the matching report for trio matching.
   Summary - Shows subject IDs and the total distance across all three pairs.
   Standard - Shows subject IDs, total distance, and the individual pairwise distances.
   Detailed - Shows subject IDs, all distances, and the individual propensity scores for each subject.
8. Matching Report Columns - Controls whether the input variables (covariates) are displayed alongside the matching results.
   All columns (incl. covariates) - Shows all original subject columns: covariates, outcome variable, propensity scores, and distances.
   Matching columns only - Shows only the essential matching columns: subject IDs, propensity scores, and distances.
9. Clean - clean the data automatically before running the PSM process.
   
10. Missing Data Values - define the data that will be counted as missing data, such as NA, "", or N/A.
    You may add more comma delimited values.
11. Clean Variables
    Numerical - remove subjects only if missing values are found in numerical variables.
    All - remove subjects if missing values are found in categorical variables or numerical variables.
    
12. Excel Pagination Display - Specifies the number of rows per tab. When you load a large Excel file, it will be displayed across multiple tabs.
13. Rounding - how to round the results?
    When a resulting value is larger than one, the tool rounds it, but when a resulting value is less than one the tool displays the significant figures.

Clean Data

If you choose to clean the data, data cleaning will occur automatically before running the PSM process.
If there are duplicate IDs, you will receive a warning, but the PSM process will not remove the record. However, the PSM process will remove records in the following cases:

1. Subjects with missing values as defined in the "Missing Data" field (e.g., "NA", "").
2. Treatment values that are not 0 or 1.
3. Outcome values that are not numerical.

Covariate Types

The calculator checks each covariate. If it finds even one non-numerical value, it defines the variable as categorical.
Please check the "Covariates" table to ensure that all categorical variables are intended to be categorical. If not, correct any non-numerical values in numerical variables.

Sample size

The sample size table gives the following for the control and treatment groups:

1. All - the number of subjects.
2. Matched - the number subjects that were matched to the other group.
3. Unmatched - the number of subjects that were not matched to the other group.
4. Exceed caliper - the number of subjects that were initially matched but were rejected because their distance from the other group exceeded the caliper threshold.