Stopwords

Code

import nltk
from nltk.corpus import stopwords
nltk.download("stopwords", quiet=True)

en_stop = set(stopwords.words("english"))
text = "this is a simple test of removing stopwords"
filtered = [w for w in text.split() if w.lower() not in en_stop]
print(filtered)  # ['simple', 'test', 'removing', 'stopwords']

# Multi-language
print(len(stopwords.words("french")), len(stopwords.words("german")))

# spaCy stopwords
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("The quick brown fox jumps over the lazy dog")
spacy_filtered = [t.text for t in doc if not t.is_stop and not t.is_punct]
print(spacy_filtered)

# Extend with custom stopwords
custom = en_stop | {"said", "say", "says"}

Explanation

Stopwords are common words that add little semantic value and are typically removed before analysis. NLTK ships language-specific lists, while spaCy marks each token with is_stop. Extending the built-in list with domain-specific noise such as 'said' in news text keeps downstream features focused.

More NLP Snippets