Stemming and Lemmatization

Code

import nltk
from nltk.stem import PorterStemmer, SnowballStemmer, WordNetLemmatizer
nltk.download("wordnet", quiet=True)
nltk.download("omw-1.4", quiet=True)

words = ["running", "ran", "runs", "easily", "fairly", "studies"]

# Porter stemmer
porter = PorterStemmer()
print([porter.stem(w) for w in words])

# Snowball stemmer (more aggressive, supports languages)
snow = SnowballStemmer("english")
print([snow.stem(w) for w in words])

# Lemmatization produces real words using POS
lemma = WordNetLemmatizer()
print([lemma.lemmatize(w) for w in words])
print([lemma.lemmatize(w, pos="v") for w in words])  # as verbs

# spaCy lemmatization with context-aware POS
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("The mice were running quickly.")
print([t.lemma_.lower() for t in doc])

Explanation

Stemming chops affixes with simple rules and may produce non-words, while lemmatization uses a dictionary and part of speech to return real dictionary forms. Porter and Snowball stemmers differ in aggressiveness and language support. spaCy lemmatizes using the full sentence context, yielding more accurate canonical forms.

More NLP Snippets