NLP lib

NLTK (nltk)

• tokenize (text tokenization utilities)

  • word_tokenize (word-level tokenizer wrapper)
  • RegexpTokenizer (regex-based tokenizer)
  • TweetTokenizer (tokenizer tuned for social text)
  • PunktSentenceTokenizer (unsupervised sentence tokenizer)
  • TreebankWordTokenizer (Penn Treebank-style tokenizer)

• corpus (corpus access and corpus readers)

  • gutenberg (classic Gutenberg texts)
  • brown (Brown corpus reader)
  • wordnet (WordNet lexical database interface)
  • PlaintextCorpusReader (read text files as corpus)
  • LazyCorpusLoader (deferred corpus loading)

• stem (stemming algorithms)

  • PorterStemmer (Porter algorithm implementation)
  • SnowballStemmer (Snowball algorithm, multi-language)
  • LancasterStemmer (Lancaster stemming algorithm)

• lemmatize (lemmatization interfaces)

  • WordNetLemmatizer (lemma lookup using WordNet)

• tag (part-of-speech tagging)

  • pos_tag (convenience POS tagging)
  • UnigramTagger (frequency-based tagger)
  • BigramTagger (contextual n-gram tagger)
  • HiddenMarkovModelTagger (HMM-based tagger)
  • CRFTagger (third-party wrappers often used)

• parse (syntactic parsing)

  • RecursiveDescentParser (top-down parser)
  • ShiftReduceParser (shift-reduce parsing)
  • ChartParser (chart parsing for CFG/PCFG)
  • DependencyGrammar (dependency grammar utilities)
  • induce_pcfg (statistical grammar induction)

• chunk (phrase/chunking utilities)

  • RegexpParser (rule-based chunker)
  • conll2000 (chunking corpus/IO)

• sem (semantic processing)

  • logic (first-order logic expressions)
  • evaluate (semantic role utilities)

• classify (classification algorithms & wrappers)

  • NaiveBayesClassifier (naive Bayes implementation)
  • MaxentClassifier (maximum entropy / logistic)
  • DecisionTreeClassifier (decision tree implementation)

• probability (probability distributions & models)

  • FreqDist (frequency distributions)
  • LidstoneProbDist (smoothed distributions)

• collocations (collocation and n-gram utilities)

  • BigramAssocMeasures (association metrics for bigrams)
  • BigramCollocationFinder (finder for collocations)

• metrics (evaluation metrics)

  • precision, recall, f_measure (classification metrics)
  • bleu_score (BLEU machine-translation metric)

• data (corpus/tool data helpers)

  • find (locate corpus resources)
  • download (NLTK data downloader)

• io (input/output and serialization)

  • conll (CoNLL format readers/writers)
  • json2obj (JSON resource helpers)

• tools (utility toolboxes)

  • tokenize_demo, draw (visual/debug tools)

• misc models & wrappers (helper scaffolding)

  • ConditionalFreqDist, ConcordanceIndex (text analysis helpers)

spaCy (spacy)

• language (Language class / pipeline container)

  • Language (factory for pipeline and model I/O)
  • nlp.pipe (streaming processing over texts)

• tokens (core in-memory objects)

  • Doc (document container for token sequence)
  • Token (single token view with morphological attrs)
  • Span (slice of a Doc with labels/attributes)

• vocab & lexical (lexicon, string store, vectors)

  • Vocab (string-to-lexeme store, lexical attributes)
  • Lexeme (entry in Vocab with norms/orth)
  • vectors (KeyedVectors for dense embeddings)

• tokenizer (tokenization system & rules)

  • Tokenizer (rule+exception based tokenizer)
  • TokenizerExceptions (special-case tokenization rules)
  • Retokenizer (merge/split tokens safely)

• pipeline (core pipeline components)

  • tagger (assign POS/morph features to tokens)
  • parser (dependency parser producing heads/labels)
  • ner (named-entity recognizer producing spans)
  • lemmatizer (map token to lemma)
  • textcat (text classification component)
  • attribute_ruler (set token attributes deterministically)
  • entity_ruler (pattern-based entity injection)
  • morphologizer (statistical morphological analyzer)

• matcher (rule and pattern matching)

  • Matcher (token-pattern matcher with rules)
  • PhraseMatcher (fast exact phrase matching)
  • DependencyMatcher (match on dependency patterns)

• training (training orchestration & config)

  • Example (training example bundling gold/pred)
  • Scorer (evaluation and scoring utilities)
  • minibatch, compounding (batching helpers)
  • config system (pipeline hyperparameters and serializers)

• serialization & I/O (save/load and model packaging)

  • nlp.to_disk / from_disk (serialize pipeline and vocab)
  • nlp.to_bytes / from_bytes (binary serialization)
  • package_models (exportable model packages)

• vectors & similarity (vector handling + similarity helpers)

  • Doc.similarity, Token.vector (cosine similarity APIs)
  • KeyedVectors (loadable precomputed vectors)

• training utilities & optimizers (optimizer wrappers)

  • optimizers (Adam variants and schedules used by spacy)

• visualize (visual inspection)

  • displacy (dependency/NER visualizer)

• util (general utilities)

  • minibatch, compounding, normalize (helpers used in workflows)

• cli & project (command line and project scaffolding)

  • spacy train / evaluate / debug-data (CLI training tools)
  • project templates (reproducible pipelines)

• pipelines.extensions & factories (extension points)

  • Language.factory (register custom components)
  • Token.set_extension / Doc.set_extension (attach custom attrs)

Gensim (gensim)

• corpora (corpus representations and readers)

  • Dictionary (token-to-id mapping and filtering)
  • MmCorpus (Matrix Market corpus I/O)
  • WikiCorpus (streaming Wikipedia corpus reader)
  • LineSentence (iterate sentences from a file)
  • StreamingCorpus (iterator-based corpus interface)

• models (statistical and embedding models)

  • Word2Vec (predictive word embedding model)
  • keyedvectors (store and query trained vectors)
  • FastText (subword-aware embedding model)
  • Doc2Vec (document-level embeddings via paragraph vectors)
  • TfidfModel (TF–IDF weighting model)
  • LsiModel (latent semantic indexing)
  • LdaModel (Latent Dirichlet Allocation topic model)
  • HdpModel (hierarchical Dirichlet process topic model)
  • Phrases (collocation/phrase detection model)
  • Similarity (model wrappers for similarity indexing)

• similarities (indexing and similarity search)

  • Similarity (disk-backed similarity index)
  • SparseMatrixSimilarity (in-memory sparse similarity)
  • AnnoyIndexer / BM25Indexer (approximate / ranking wrappers)

• corpora.streaming & preprocessing (text preprocessing helpers)

  • simple_preprocess (tokenize+clean helper)
  • preprocess_string (pipeline of filters)
  • strip_tags, strip_punctuation, strip_numeric (filters)

• matutils (matrix & sparse utilities)

  • corpus2dense, sparse2full (convert between formats)

• utils (helper functions for workflows)

  • save, load (model persistence helpers)
  • callbacks (training callbacks for logging/saving)
  • tokenizer (smart tokenization helpers)

• downloader (fetch pretrained corpora/models)

  • api.load (download and load models/datasets)

• interfaces & wrappers (compatibility and I/O)

  • MmCorpus, SerializedCorpus, BleiCorpus (format adapters)

• evaluation & coherence (model evaluation helpers)

  • CoherenceModel (topic coherence metrics)
  • perplexity, log_perplexity (LDA evaluation helpers)

• streaming & memory optimizations (large-corpus strategies)

  • corpus streaming patterns (iterate + avoid full materialization)
  • save/load in incremental formats (KeyedVectors, mmap)

• training internals (algorithm control knobs)

  • update, alpha/beta/decay parameters (LDA/Word2Vec training controls)
  • epochs, chunk_size, workers (performance/training tradeoffs)

mediapipe → google best

Spacy

When we run nlp = spacy.load("en_core_web_sm"), it loads a multi-stage pipeline. Each stage is a separate “modality” trained to do one specific thing:

Component	What it does	How it works under the hood
Tokenizer	Splits text into tokens.	A non-statistical, rule-based system using a prefix/suffix/infix map.
AttributeRuler	Cleans up attributes.	Assigns rules for specific tokens (e.g., handling “don’t” as “do” + “not”).
Tok2Vec	Token-to-Vector.	A CNN (Convolutional Neural Network) that turns tokens into numerical tensors based on their neighbors.
Tagger	Part-of-Speech (POS).	Predicts if a word is a NOUN, VERB, ADJ, etc.
Parser	Dependency Parsing.	Determines the grammatical structure (e.g., which word is the Subject of which Verb).
NER	Named Entity Recognition.	Identifies PERSON, ORG, GPE, DATE, etc.
Lemmatizer	Root word extraction.	Reduces “running” to “run” using a lookup table and rules.

Resources

• https://spacy.pythonhumanities.com/intro.html (best)