Question

Have you worked with dimensionality reduction before?

What methods did you use?

Why?

Reducing dimensionality of the data is done primarily for three reasons:

1. It reduces computation time for later models
2. It can increase performance (the curse of dimensionality)
3. It makes visualizations easier

PCA

The most common method for dimensionality reduction is (probably) Principal Component Analysis

Main idea is to create new variables:

• Which are orthogonal
• Each capture as much variance as possible

Creates linear ‘latent variables’

The method ‘intuitively’

• Find the direction of max variance in data
  • This is the first component
• Subtract the first component
• Find the direction of max variance in data
  • This is the second component
• Subtract, and create additional components until no more variance in data

Earlier components will explain more variance!

An illustration

Principal components in 2D

_{^{Source: Raschka & Mirjalili, 2022, ch. 6}}

The method ‘mathematically’

1. Standardize the features
2. Calculate the correlation matrix
3. Perform an eigenvector decomposition of the correlation matrix
4. Order eigenvectors by the size of the eigenvalues
5. Transform original data into \(K\)-dimensional space using \(K\) first eigenvectors

\(K\) is a hyperparameter that we choose - More on how to choose later

How to evaluate

• Downstream performance
  • Treat K as any other hyperparameter
• Computational limits
  • Use as many dimensions as feasible
• Explained variance plots
  • Stop when adding more components doesn’t add much information

Explained variance as a guide

Each principal components explains part of the variance in the original data

Introducing the scree plot

Scree plot

_{^{Source: Raschka & Mirjalili, 2022, ch. 5}}

Interpretation

As PCA is a linear transformation, we can look at how the variables are transformed

• Available in pca.components_

A way of interpreting the new variables

• Remember that variables are standardized

Non-linearity

If we want to incorporate non-linearity, one of two things are commonly done:

1. Do a polynomial tranformation of the input before doing PCA
2. Use a kernel (kernelPCA)

Question

Have you worked with clustering before?

What methods did you use?

Clustering

Find unknown groups within populations

• Observations within clusters are alike
• Observations between clusters are different

What constitutes different and alike?

A broad introduction

Different ways of formulating this, with three broad categories:

• Hierarchical clustering
• Prototype-based clustering
• Density-based clustering

Some methods fall into more than one category and there are other clustering methods (e.g. graph-based)

Hierarchical clustering

Clusters are based sequential merging or divison of clusters

• Agglomerative, start with n clusters and sequentially join clusters
• Divisive, start with one cluster and sequentially divide into n clusters
• Main hyperparameters: Distance measure (no. of clusters)
• Creates neat dendrograms
• No notion of noise, and all points will belong to a cluster
• An example: AgglomerativeClustering

Prototype-based clustering

A cluster is represented by a prototype (centroid)

• Main hyperparameter: Number of clusters
• Clusters will generally be convex (circular or ellipsoidal)
• No notion of noise, and all points will belong to a cluster
• An example: Kmeans

Density-based clustering

A cluster is represented by a dense area of points

• Main hyperparameters: What constitutes ‘dense’?
• Clusters can be any shape
• Distant points are labeled as outliers
• An example: DBSCAN

K-means

K-means aims to minimize the within-cluster sum-of-squares:

\[\sum_{i=0}^n \min_{\mu_j \in C}(||x_i - \mu_j||^2)\]

Where \(\mu_j\) are the centroids

This corresponds to an Euclidian distance, and thus we (often) standardscale

Iterative updating

This is done by:

• Initialize clusters
• Assigning points to nearest cluster
• Move centroids to mean values of the cluster samples

Repeating procedure until convergence

Illustrations are nice

The easiest way to get to know K-means is to look at the process

The following figures are from a cool tool made by brookslybrand

Initialize centroids

Centroids are randomly placed

Assign points

All points are assigned to the nearest centroid

Update centroids

Centroids are updated to the mean of assigned points

Assign points

All points are assigned to the nearest centroid after update

Update centroids

Centroids are updated to the mean of assigned points

Assign points

All points are assigned to the nearest centroid after update

Update centroids

At this point, we’ve converged

Assumptions

Violated assumptions

_{^{Source: sklearn examples, ‘Demonstration of k-means assumptions’}}

There are other methods

Methods implemented in sklearn

_{^{Source: sklearn User Guide, ‘Overview of clustering methods’}}

Evaluation

It is generally not easy to evaluate clustering methods, but methods exist

Choosing the right amount of cluster

Models optimize some sort of metric

• For K-means: Sum of squared distances

Example data

Clustered data

_{^{Source: Raschka & Mirjalili, 2022, ch. 11}}

K-means elbow plot

Sum of squared distances to nearest centroid as a function of cluster amount

_{^{Source: Raschka & Mirjalili, 2022, ch. 11}}

Convexity

Silhouette scores based on the idea that we want samples to be

• Near their cluster companions
• Far away from the nearest neighbor clusters

Works across models, but favors models that create convex clusters

Question

Have any of you used text as data in an analysis?

Was it done using qualitative methods or quantitative?

Aim today

Natural Language Processing (NLP) is a huge field

Aim today is to get you thinking about text as data

• How do I start working with text
• What do people do

Some terminology

A word is the basic unit of discrete data

• E.g. a word, but can also be parts of words

A document is a collection of words

• E.g. a sentence, paragraph or full documents

A corpus is a collection of documents

• I.e. a dataset

How to use text as data?

There are many different ways of using text as data

• There is no ‘correct’ way
• The way in which you use text should be guided by what you’re interested in

Words to values

The key in this method is a dictionary which includes:

• Words of interest
• Scores associated with these words

Using these dictionaries, we can

• Go through sentences and give each word a score
• Summarize these scores

How to get these dictionaries?

Dictionaries are generally either:

• General
  • Often a scale of an emotion (sentiment analysis)
  • More throughougly tested and verified
• Handcrafted for the purpose
  • Subject specific and often very niche
  • No to little testing

Sentiment analysis

One well known dictionary is VADER (Hutto & Gilbert, 2014), Valence Aware Dictionary and sEntiment Reasoner

• Encodes sentiment as both polarity and intensity

Verified through experiments and benchmarks

• A benefit over handcrafted dictionaries

Scores every sentence according to how positive or negative they are (-1 to 1), and the proportion of negative, neutral and positive words

One small problem

The difference

´VADER is not smart, handsome, nor funny.´ classified as positive

• Picks up on smart, handsome and funny
• Does not capture the negation

You can play around with it at this website, both in Danish and English

Use case: Racial animus in politics

Stephens-Davidowitz (2014) examines whether prejudice against blacks remains a potent factor within American politics (at the time)

People sometimes lie on surveys

• Could be a problem when asked about racial animus

A hancrafted dictionary

To examine this, they use Google searches up to the 2008 election in the US split by media market and examine the ‘racially charged search rate’, which is a fraction of the type

\[\frac{\text{Google searches including some word(s)}}{\text{All Google searches}}\]

Essentially counting

Question

What word(s) would you include in this dictionary?

Simplicity at its best

Stephens-Davidowitz (2014) uses just a single derogatory term

• Can be found in the appendix for the curious

Using this measure, they find that it is a robust negative predictor of Obama’s performance

• Racial animus causes about a 4%-point loss in the elections in 2008 and 2012

Downsides of dictionary based methods

Requires a dictionary

• Getting an apropriate one can be hard

Sometimes too simple

• Hard time with subtleties, i.e. difference between I hate this and I do not hate this

Question

How could one represent sentences numerically?

Counting words systematically

We can treat words as unique tokens

• Count how often each word appears in a sentence

• Much like categorical variables

High-dimensional and sparse

• Good that we know both how to regularize and reduce dimensionality!

An example

This allows us to go from sentences to a tabular format

• Use as covariates
• All the machine learning methods
  • Prediction, clustering, dimensionality reduction
• All the causal inference methods

A variant: tf-idf

A common variant is a term frequency times inverse document (tf-idf) matrix

• Here we normalize by how often the word appears in documents
• Intuition is that words which appear in only a few documents are more important for those documents than those who appear in many documents

A variant: n-grams

N-gram models are also another common variant

• Extend the simple unigram bag-of-words to consecutive words
  • e.g. Bigram (\(n=2\)) models would also include all pairs of words in order
  • 'Dogs like dogs' would become {'dogs':2, 'like':1, 'dogs like':1, 'like dogs':1}

You’re the experts

In the end, it’s text

• Read it, get a feel for it!
• Why are you interested in this text?

Room for subject-specific preprocessing

• What to to do with links? Hashtags? Emojis?
• Specific terms such as names
• regex is a nice tool for working with text
  • Can be daunting

Many further avenues

• Deep learning, e.g. transformers
  • Can do translation, prediction, generation and more
  • HuggingFace is a repository of (relatively) easily accesible models
• Topic modelling, e.g. Latent Dirichlet Allocation
  • Find topics in corpus and assign words and documents to topics
  • Focus on interpretability