Natural language processing

https://www.youtube.com/watch?v=kRv2ElPNAdY

NLP can be done in 3 way

Rule-based: regex + gazetteers → apply deterministic labels. No training. Fast inference. Hard to generalize.
- Rules: if token matches /^[A-Z][a-z]+$/ and previous token in {Mr., Ms.,Dr.} → PERSON
- Context-Free Grammars (CFG): production rules of form A → B C that generate parse trees. A parser (CYK, Earley) finds parse trees for input sentences.
- Symbolic rule engine: sequence of if/then rules or finite-state transducers that apply replacements or labels.
Classical ML (CRF): features per token → CRF optimizes conditional likelihood → Viterbi for inference. Data-efficient, interpretable.
Deep Learning (BiLSTM-CRF or Transformer): embeddings → BiLSTM (or transformer encoder) → feed into linear layer + CRF for structured output. Train end-to-end by backprop. Learns features from data and generalizes better with ample labeled data.

https://princeton-nlp.github.io/

https://web.stanford.edu/class/cs224n/

Best course

https://www.youtube.com/playlist?list=PLqC25OT8ZpD3WxQ0FwWMGPS_BcWdcKyZy
- Syllabus https://cmu-l3.github.io/anlp-spring2025/

https://huggingface.co/learn/nlp-course/chapter1/1

https://roadmap.sh/ai/roadmap/natural-language-processing-gb3zf

Linguistic Knowledge

Linguistic knowledge is everything humans implicitly know about how language works the rules, patterns, and structures that allow us to produce and understand sentences.

When NLP systems try to process human language, they’re essentially trying to simulate or model this human linguistic knowledge computationally.

Five Major Types of Linguistic Knowledge

Morphology (Bottom Layer)

What it is: The study of word structure how words are formed from smaller units called morphemes (roots, prefixes, suffixes).
Example:
- “Unhappiness” → “un-” (prefix, negation) + “happy” (root) + “-ness” (suffix, state/condition).
- Morphology helps NLP systems break down and understand words beyond their surface form.
In NLP tasks: Stemming, lemmatization, part-of-speech tagging.

2. Syntax

What it is: The arrangement of words into grammatically correct sentences the structure of language.
Example:
- Correct: “The cat sat on the mat.”
- Incorrect: “Cat mat the on sat.”
Why it matters: Syntax ensures words are in the right order and follow grammatical rules, which is crucial for parsing and understanding sentence structure.
In NLP tasks: Parsing, grammar correction, machine translation.

3. Semantics

What it is: The meaning of words and sentences. Goes beyond grammar to capture what the sentence means.
Example:
- “The cat sat on the mat” vs. “The dog sat on the mat” — syntax is the same, but semantics tells us that the animal is different.
Why it matters: Machines need to map words to meaning, handle synonyms, word sense disambiguation, etc.
In NLP tasks: Question answering, information retrieval, text classification.

4. Pragmatics (Top Layer)

What it is: The study of how language is used in context. It considers intention, background knowledge, and situation.
Example:
- If I say, “Can you open the window?” → Semantics: asking about ability. Pragmatics: it’s actually a polite request, not a question about capability.
Why it matters: Understanding sarcasm, politeness, implied meanings, and context is critical for real conversation.
In NLP tasks: Dialogue systems, chatbots, conversational AI.

Level	Linguistic Focus	Example NLP Tasks	Type of Understanding
Morphology	Word structure	Tokenization, Lemmatization	Sub-word level
Syntax	Sentence structure	POS tagging, Parsing	Grammar level
Semantics	Word/sentence meaning	Word embeddings, NER	Meaning level
Pragmatics	Context & intent	Dialogue systems, Coreference	Contextual level

Parsing in NLP

Constituency Parsing: Constituency Parsing builds parse trees that break down a sentence into its constituents, such as noun phrases and verb phrases. It displays a sentence’s hierarchical structure, demonstrating how words are arranged into bigger grammatical units.

Dependency Parsing: Dependency parsing depicts grammatical links between words by constructing a tree structure in which each word in the sentence is dependent on another. It is frequently used in tasks such as information extraction and machine translation because it focuses on word relationships such as subject-verb-object relations

concepts

Parts of Speech (POS)

Every word in a sentence belongs to a category based on its grammatical function.

Some key POS tags (in simplified Penn Treebank notation used in NLP):

Tag	Meaning	Example
NN	Noun (singular)	dog, car, idea
NNS	Noun (plural)	dogs, cars
NNP	Proper noun	John, India
DT	Determiner	a, an, the
VB	Verb (base)	eat, go
VBP	Verb (present plural)	run, play
VBZ	Verb (present 3rd-person singular)	eats, runs
JJ	Adjective	tall, happy
RB	Adverb	quickly, very

💡 In a sentence, each word is tagged with one POS label.
For example:

The cat eats fish.
→ DT NN VBZ NN

Constituents (Phrases)

A phrase is a group of words that acts as a single unit in the sentence.

Examples:

Noun Phrase (NP) — functions like a noun.
→ “The cat”, “A red car”, “John”
→ Often structure: (DT) + (JJ) + NN
Verb Phrase (VP) — contains a verb and possibly its object.
→ “eats fish”, “is sleeping”, “runs fast”
Prepositional Phrase (PP) — starts with a preposition.
→ “in the park”, “on the table”

Each phrase can be made of smaller phrases or words.
So parsing finds how words group into phrases.

Phrase Structure Grammar (PSG)

Now we describe the structure of sentences formally, using rules.

Example rules:

S → NP VP
NP → DT NN
VP → VBZ NP

Interpretation:

S → NP VP means:
A Sentence (S) is made of a Noun Phrase (NP) followed by a Verb Phrase (VP).
NP → DT NN means:
A Noun Phrase is made of a Determiner followed by a Noun.
VP → VBZ NP means:
A Verb Phrase is made of a Verb (third-person singular) followed by a Noun Phrase.

Context-Free Grammar (CFG)

A CFG is the formal system behind PSG. It’s called context-free because each rule applies independently of surrounding words (“context”).

Each rule has:

Left-hand side (LHS) — a single nonterminal symbol (like S, NP, VP)
Right-hand side (RHS) — a sequence of terminals (words) or nonterminals.

General form:

A → B C

means “A can be expanded into B followed by C”.

In NLP parsers:

Nonterminals = abstract syntactic categories (S, NP, VP)
Terminals = actual words (or POS tags) in the sentence

Parse Tree

A parse tree is a hierarchical tree showing how a sentence structure fits these rules.

Example sentence:

The cat eats fish.

Rules:

S → NP VP
NP → DT NN
VP → VBZ NN

Then the parse tree is:

        S
      /   \
    NP     VP
   / \    /  \
 DT  NN  VBZ  NN
 |    |   |    |
the  cat eats fish

This shows the syntactic structure of the sentence — what depends on what.

Different methods to perform POS tagging:

Rule-Based POS Tagging: This involves defining hand-crafted rules based on suffixes, prefixes, and word patterns. For example, a rule might state that words ending in “-ing” are usually verbs (VBG) or nouns (NN). While simple, it can be labor-intensive and struggle with ambiguity.
Statistical POS Tagging: These methods learn from large annotated datasets.
- Hidden Markov Models (HMMs): Model sequences of observations (words) and hidden states (POS tags). They learn the probability of a word given a tag, and the probability of a tag following another tag.
- Conditional Random Fields (CRFs): More advanced statistical models that consider the entire sequence of tags and words, often leading to better performance than HMMs.
Neural POS Tagging: Modern deep learning approaches, often using architectures like Recurrent Neural Networks (RNNs), LSTMs, or even Transformers (as mentioned in the “Deep Learning for NLP” section) to learn complex patterns and contexts for tagging. These typically achieve state-of-the-art performance.

Application of POS

Text understanding — Helps machines grasp who did what to whom (syntactic structure).
Information extraction — Identify entities like names, places, verbs of action, etc.
Speech recognition & synthesis — Helps disambiguate word meanings (“lead” noun vs. “lead” verb).
Sentiment analysis — Adjectives and adverbs often carry sentiment (“very good”, “terribly bad”).
Machine translation — Ensures correct grammatical alignment between languages.
Question answering / Chatbots — Understands intent and sentence structure.
Text summarization & keyword extraction — Focuses on nouns/verbs over filler words.

Hidden Markov model

HMM is a statistical model that assumes two things:

We can observe words, but the POS tags are hidden (we don’t know them yet).
→ That’s why it’s Hidden.
The POS tags form a sequence that follows a pattern (Markov property):
The tag of a word depends only on the previous tag not the entire history.

So, we use probabilities to find the most likely sequence of tags for the given sequence of words.

There are two important probabilities:

Transition probability:
Probability of one tag following another.
Example:
- P(NOUN | DET) = probability that after a Determiner, a Noun appears.
Emission probability:
Probability of a word appearing given a tag.
Example:
- P(cat | NOUN) = probability that the word “cat” appears when the tag is NOUN.

for more Markov Chain

Feature engineering

TF IDF
word2vec (https://jalammar.github.io/illustrated-word2vec/o)
fasttext
topic modeling

Tokenization

Tokenization can be done in a variety of ways, based on the task’s particular requirements and the language’s complexity. Here are a few typical methods:

Whitespace tokenization: In the most basic type of tokenization, text is divided using whitespace characters (space, tab, newline). It is assumed that whitespace is used to always separate words.
Punctuation-based tokenization: Punctuation symbols, such as commas, periods, and semicolons are also utilized as delimiters to divide tokens, in addition to whitespace. Sentences and phrases inside text can be distinguished more accurately with this method.
Rule-based tokenization: Tokens are identified using this method by applying a set of criteria (like regular expressions). Complex token structures like email addresses, dates, and contractions (such don’t as do and n’t) can be handled with its help.
Statistical tokenization: Sophisticated techniques tokenize text using statistical models or ML. These models work well even when managing irregular token boundaries since they have been trained on vast datasets to comprehend the subtleties of a language.

Token normalization

Stemming is the process of reducing words to their root or base form by removing suffixes, such as converting “running” to “run” and “happily” to “happi”.

Lemmatization is the process of reducing words to their base or dictionary form (lemma) by considering the context and morphological analysis, such as converting “running” to “run” and “better” to “good”.

Stop word: stopword removal aims to reduce noise in the text data and facilitate more effective and efficient analysis. Every word in the text is compared to a list of predetermined stopwords during the process, and any matches are removed. Consequently, the text is represented in a more succinct and meaningful way, which can enhance the focus on pertinent content and decrease computational complexity in NLP models, hence improving their performance

Term frequency: tatistic that measures how frequently a word appears in a text, providing important information about the significance of terms inside certain publications

The number of times a certain word appears in a document divided by the total number of words in that document yields the TF

Inverse Document Frequency A statistical metric used to assess a word’s significance within an entire corpus of texts, based on the principle that words appearing in fewer documents are more informative

total number of documents in the corpus divided by the number of documents containing the word is used to determine the IDF of a word

IDF is frequently combined with TF to produce the TF-IDF score, which is a crucial indicator of a word’s relative importance to a document within a collection

Example

Doc1: AI is the future of technology  
Doc2: AI and machine learning are the future  
Doc3: Technology advances with machine learning

$TF (t, d) = \frac{Number of times term t appears in document d}{Total number of words in document d}$ For “AI”:

Doc1 → 1/6 = 0.167
Doc2 → 1/7 = 0.143
Doc3 → 0

Inverse Document Frequency (IDF)

$I D F (t) = lo g_{10} (\frac{Total number of documents}{Number of documents containing the term t})$ For “AI”:
(N = 3), (n_t = 2) → (IDF = \log_{10}(3/2) = 0.176)

TF–IDF

TF - I D F = TF \times I D F

Doc	TF(“AI”)	IDF	TF–IDF
1	0.167	0.176	0.029
2	0.143	0.176	0.025
3	0	0.176	0

TF shows how frequent a word is in one document.
IDF reduces weight of common words.
TF–IDF highlights words that are frequent in one doc but rare across others.

intuition behind TF,TF-IDF

The core problem: How do we measure the “importance” of a word?

When we look at a document, not all words are equally informative.

Example:

“The cat sat on the mat.”

Now, if I show you 1,000 documents, and the word “the” appears in every single one does “the” tell you anything unique about any document?
→ No. It’s everywhere. It’s background noise.

But if the word “cat” only appears in 10 out of 1,000 documents,
and you find it in this one that tells you something specific about the topic.

So we want a number that captures this idea:

Word	Appears in many docs?	Appears many times in this doc?	Should score high?
the	✅ yes	✅ yes	❌ no
cat	❌ no	✅ yes	✅ yes
mat	❌ no	✅ yes	✅ yes

Term Frequency: A word that appears many times in a document probably matters more to that document.

If in one document the word “cat” occurs 10 times and “mat” once,
maybe it’s more about cats than mats.

“IDF” (Inverse Document Frequency)

If a word appears in every document, then it’s not discriminative.
It doesn’t help you tell which document is relevant.

So we want a penalty for common words.

We count:

How many documents contain this word? = document frequency (DF)

Then we invert it:

IDF = inverse of DF

→ The rarer the word across documents, the larger its IDF.

So the intuition:

Common words = appear everywhere = low IDF
Rare words = appear in few docs = high IDF

Why we use log?

We don’t want extremely rare words to dominate the score too much.

Without log:

If a word appears in 1 document out of 1,000 → IDF = 1000
If another appears in 2 → IDF = 500
Huge difference!

But in reality, both are “rare words.”
We want a gentler curve.This compresses big ratios keeps the scale human and stable.

Word Co-occurrence Matrices (Count-based Semantics)

“You shall know a word by the company it keeps.” — J.R. Firth (1957)

Instead of looking at single-word frequencies, we look at which words appear near each other.

For example, if “doctor” often appears near “hospital”, “patient”, “medicine”,
and “lawyer” near “court”, “judge”, “case” we can infer semantic similarity just from co-occurrence!

We build a word × context matrix, where:

Rows = target words
Columns = neighboring words
Values = how often they co-occur

Then apply dimensionality reduction (like SVD) → this gives us latent semantic analysis (LSA).

NLP pipeline technique

To obtain the intended analytical result, an NLP pipeline normally begins with raw text

input and processes it via several phases. The procedure consists of:

Preprocessing: Text should be cleaned and normalized to get rid of noise and irregularities.
Tokenization: It is the process of dividing a text into discrete words or units.
Stopword removal: Taking out frequently used terms with less semantic significance.
Lemmatization/stemming: Getting words down to their most basic or root form.
POS tagging: Classifying each word according to its grammatical category.
Named entity recognition (NER): Locating and categorizing the text’s named entities (people, places, and organizations).
Dependency parsing: Linking words together by examining a sentence’s grammatical structure

N-Gram

How it Works

Define ‘n’: Choose a positive integer ‘n’ to represent the length of the sequence.
Identify Sequences: Scan the text and extract all possible sequences of ‘n’ consecutive items.
Analyze Frequencies: The primary use of n-grams is to determine the frequency of these sequences within a dataset. This data is then used to build models.

An n-gram is just a sequence of n words (or tokens).

1-gram (unigram): one word at a time → ["I"], ["am"], ["going"]
2-gram (bigram): pairs of words → ["I am"], ["am going"], ["going to"]
3-gram (trigram): triples → ["I am going"], ["am going to"], ["going to the"]
n-gram (general): group of n consecutive words

So the text:
“I am going to the store”

Unigrams: I | am | going | to | the | store
Bigrams: I am | am going | going to | to the | the store
Trigrams: I am going | am going to | going to the | to the store

Use Case: Spell-Checker (Fixing “I going to school” → “I am going to school”)

Let’s try a bigram model (2-grams) for simplicity.

We’ll need probabilities learned from a large corpus (say, millions of English sentences).

From the sentence, extract bigrams:

I going
going to
to school

From real English data, we compute bigram probabilities:

P(going | I) = Count(“I going”) / Count(“I”)
P(am | I) = Count(“I am”) / Count(“I”)

In English text:

"I am" appears millions of times.
"I going" appears rarely (almost always incorrect).

So:

P(going | I) ≈ very small
P(am | I) ≈ very large

Now the system tries likely alternatives:

Replace "I going" with "I am going" → Probability increases.
Replace "I going" with "I was going" → Probability also reasonable.

It chooses the most probable correction based on n-gram statistics.

Embedding

Wrord2Vec (CBOW and Skip grammer)
Glove (Global vector for word representation)
ELMOC
BERT
Transformer based models (generative pretrained transformer and text to text transformer)

Word2vec

There are two main models in Word2Vec: Continuous Bag of Words (CBOW) and Skip-gram check out more here

Continuous Bag of Words (CBOW)

The CBOW model predicts the current word based on the context (surrounding words). For instance, in the sentence:

“The quick brown fox jumps over the lazy dog”

If we want to predict the word “fox” (target word), the context words might be “The”, “quick”, “brown”, “jumps”, “over”.

Example Process for CBOW:

Input Context Words: Let’s consider a simplified version where we use a window of size 2 (two words on each side of the target word).
- Context words for “fox” would be [“quick”, “brown”, “jumps”, “over”].
One-Hot Encoding: Each context word is converted into a one-hot vector. If our vocabulary consists of the words [“The”, “quick”, “brown”, “fox”, “jumps”, “over”, “the”, “lazy”, “dog”], the one-hot vector for “quick” might look like: “quick” → [0, 1, 0, 0, 0, 0, 0, 0, 0]
Hidden Layer: These one-hot vectors are fed into a neural network with a single hidden layer. The hidden layer typically has fewer neurons than the vocabulary size, capturing the dense representations.
Output Layer: The network then combines the hidden layer’s outputs to predict the target word. The output is a probability distribution over the vocabulary.
Training: The network is trained using backpropagation to minimize the prediction error, adjusting weights to improve accuracy.

Skip-gram

The Skip-gram model works oppositely to CBOW. It predicts context words from the target word. This model is particularly effective for larger datasets.

Example Process for Skip-gram:

Using the same sentence, “The quick brown fox jumps over the lazy dog”:

Target Word: Let’s choose “fox” as our target word.
Context Words: For “fox”, the context words could be [“quick”, “brown”, “jumps”, “over”] using a window size of 2.
One-Hot Encoding: Convert the target word “fox” into a one-hot vector.
- “fox” → [0, 0, 0, 1, 0, 0, 0, 0, 0]
Hidden Layer: This one-hot vector is fed into the hidden layer, producing a dense vector representation of the target word.
Output Layer: The dense vector is then used to predict the context words. The network outputs a probability distribution over the vocabulary for each context word.
Training: The network is trained by adjusting the weights to maximize the probability of the correct context words given the target word.

Cosine similarity is a metric used to measure how similar two vectors

Euclidean Distance: It measures the straight-line distance between two vectors in the multidimensional space

Manhattan Distance (L1 Distance): This calculates the sum of the absolute differences between corresponding elements of two vectors.

One-hot encoding is a technique used to represent categorical data as binary vectors. Each category is converted into a vector where one element is “hot” (1) and all other elements are “cold” (0). Example: Let say we take fruits and consider it only having 5 varites such as apple,orange,mango,grapes and banana the one hot encoding will be

“apple” → [1, 0, 0, 0, 0]
“orange” → [0, 1, 0, 0, 0]
“mango” → [0, 0, 1, 0, 0]
“grapes” → [0, 0, 0, 1, 0]
“banana” → [0, 0, 0, 0, 1]

The main disadvantage is If the number of unique categories is very large, the resulting one-hot encoded data can become very high-dimensional and sparse.

Stemming is the process of reducing words to their root or base form by removing suffixes, such as converting “running” to “run” and “happily” to “happi”.

lib that used for above nltk, spacy in python

Disadvantage of this apporach

the word wil lhave same vector even though it represent different meaning based ont the current context
- Bank → based on the context the word represent differnet mening to solve that only tranformer is introduced.

GloVe

Global Vectors for Word Representation.Unlike other word embedding methods like Word2Vec, which rely on local context information (e.g., predicting a word based on its neighbors), GloVe leverages global word co-occurrence statistics from a corpus to capture the meanings of words.

GloVe constructs a word-word co-occurrence matrix using the entire corpus. Each element of this matrix represents how often a pair of words appears together in a context window like below let say we have a “The cat is fluffy. The dog is fluffy. The cat and the dog are friends.”

Word	the	cat	is	fluffy	dog	and	are	friends
the	0	2	2	2	2	1	1	1
cat	2	0	1	1	1	1	0	0
is	2	1	0	2	1	0	0	0
fluffy	2	1	2	0	1	0	0	0
dog	2	1	1	1	0	1	0	0
and	1	1	0	0	1	0	1	1
are	1	0	0	0	0	1	0	1
friends	1	0	0	0	0	1	1	0

We create vectors (magic numbers) for each word. Here’s an example of what the vectors might look like after training

Word	Vector
the	[0.8, 0.6]
cat	[0.3, 0.7]
is	[0.5, 0.5]
fluffy	[0.3, 0.8]
dog	[0.4, 0.7]
and	[0.6, 0.4]
are	[0.6, 0.3]
friends	[0.7, 0.2]

Dimension

The dimensionality of word embedding refers to the number of dimensions in which the vector representation of a word is defined.
This is typically a fixed value determined while creating the word embedding.
The dimensionality of the word embedding represents the total number of features that are encoded in the vector representation.
Larger datasets can support higher-dimensional embeddings as they provide more training data to inform the model. As a rule of thumb, a dataset with less than 100,000 sentences may benefit from a lower-dimensional embedding

Byte Pair Encoding (BPE): A Tokenization Technique

BPE is a data compression algorithm that has been adapted for tokenization. It’s a bottom-up approach that starts by treating each character in the training data as a separate token. Then, it iteratively merges the most frequently occurring pairs of tokens into a single new token. This process continues until a predefined vocabulary size or a merging criteria is met.

Let’s illustrate with a simplified example. Suppose our training data contains the words “lower,” “lowest,” “newer,” and “newest.”

Initial Tokens: Start with individual characters as tokens: ‘l’, ‘o’, ‘w’, ‘e’, ‘r’, ‘s’, ‘t’, ‘n’, ‘ew’.
Merging: The pair ‘e’ and ‘r’ might occur most frequently, so they are merged into a new token ‘er’.

WordPiece

WordPiece tokenization is a subword tokenization algorithm developed by Google,

Massive Text Embedding Benchmark

MTEB, is a project developed by Hugging Face to address the challenge of evaluating and comparing the performance of different text embedding models

MTEB goes beyond a single metric and instead evaluates models across a wide spectrum of tasks1. This ensures a more holistic understanding of a model’s capabilities. Some of the tasks covered by MTEB include2:

Bitext Mining: Identifying pairs of sentences in different languages that convey the same meaning.

Classification: Assigning predefined categories to text snippets.

Clustering: Grouping text snippets based on their semantic similarity.

Pair Classification: Determining the relationship between two text snippets (e.g., paraphrase, contradiction).

Re-ranking: Ordering search results based on their relevance to a given query.

Retrieval: Finding the most relevant documents for a specific query (often used in semantic search).

Semantic Textual Similarity (STS): Measuring the degree of semantic overlap between two text snippets.

Summarization: Evaluating how well a short text summarizes a longer document.

All methods

Era	Method	Core Intuition
Statistical	BoW	Frequency = importance
Statistical	TF-IDF	Local frequency × global rarity
Count-based Semantics	Co-occurrence / LSA	Words in similar contexts share meaning
Neural	Word2Vec	Predict context → capture meaning geometry
Neural	GloVe	Global co-occurrence ratios define meaning
Neural	FastText	Subword units capture morphology
Neural	Doc2Vec	Whole documents can be embedded by predicting words
Deep Contextual	ELMo / BERT	Word meaning depends on context
Semantic	Sentence-BERT	Encode full sentence meaning for comparison

Text summarization

Methods

Extractive: summary entails picking out and concatenating the most crucial sentences or phrases from the text
Abstractive: summarization creates a fresh, condensed version of the text by paraphrasing and rephrasing it to convey its main ideas.

Resources

emdeding

https://arxiv.org/pdf/1411.2738 need ot read
https://ronxin.github.io/wevi/# word embedding visual inspector

Lib

https://fasttext.cc/

ROADMAP

learn concepts and theory

For practical python

https://www.youtube.com/watch?v=R-AG4-qZs1A&list=PLeo1K3hjS3uuvuAXhYjV2lMEShq2UYSwX

NLTK - TOkenizer (treebank) - stemmer (portstemmer) - lemmatizer (wordnet )

Project

build sentiment calssifer in old appoarch and in sequenital models
- SUmmarize text
- name entity
- building next workd prediction like keyboard
- Ad Click Prediction (do google)
- reply suggestion on mail

libe

https://vowpalwabbit.org/

Text classification

Naive Bayes i

Books

cookbook
- all wiht lib not from scrach concepts with python code lib

What next

tag of speach build model
margove model understand
n gram bag of words Smoothing Techniques
neural lstm etc
named entity recong
parsing

Problem → Data → Labels (if needed) → Preprocessing → Feature Representation → Model Training → Evaluation → Deployment → Monitoring

Essential Steps for NLP Problem Solving

Data Acquisition: Begin by gathering relevant and sufficient data with labels appropriate for your NLP task.
Data Preprocessing: Process the text by cleaning, tokenizing, lowercasing, removing stop words, and stemming or lemmatizing as needed.
Feature Extraction: Transform text into a suitable representation for modeling, such as using Bag-of-Words, TF-IDF, or embeddings.
Model Selection: Choose the right algorithm (e.g., Naive Bayes, SVM, neural networks) based on task complexity and available resources.
Model Training: Train your model on the processed data and use a separate validation or test set to evaluate performance on unseen data.
Evaluation: Assess model performance using metrics relevant to your target outcome, such as accuracy, precision, recall, or F1-score.
Error Analysis: Analyze types of errors to improve model robustness and handle edge cases better.
Deployment: Integrate the trained model into the desired production environment, and monitor for data drift or real-world issues.
Iterative Improvement: Refine preprocessing, retrain with new data, and update the model as requirements and data change

Feature engineering

need to use the feature that needed for the task example for sam detection we need to look for the words that are likely spam example (credit card,free spam) etc

Feature types

binary
categorical
numeric feature

Why we doing tokenization etc

we try to get all data to same dimension if we do lem and remove pucnation etc we can say all data will be come under same deminsion or precpective

NOTES NLP

Convolutional Neural Networks for Sentence Classification
Character-level Convolutional Networks for Text Classification
Bigram langauge model
log bilinear language model
art of eval in neural langauge models
vector space of models

Smoothing

In statistical NLP models, probabilities are often estimated by counting how frequently certain events (like word sequences or transitions) occur in a training dataset (corpus). The problem arises when a word or sequence of words that was not observed in the training data appears in a real-world application

If a specific sequence (e.g., “The dog barks”) has a count of zero in the training data, a simple probability calculation would assign it a probability of zero. This is problematic because:

Zero probability is an absolute value that is often unrealistic in language (most things are possible, just very rare).
It can break mathematical models. If a sequence with zero probability is part of a larger calculation (e.g., in the product rule), the entire sequence’s probability becomes zero, regardless of the other components

Smoothing is used to “adjust” the counts by taking a small amount of probability mass from the frequently observed events and re-distributing it to the unseen or low-frequency events . This ensures that even unseen events are assigned a non-zero, albeit very small, probability.

NOTE: , because multiplying probabilities across a sequence will then give zero sentence probability.

The goal is to produce more robust and reliable probability estimates that generalize better to new, unseen data

Common Smoothing Techniques

Laplace Smoothing (Add-One Smoothing): The simplest method. It adds a count of 1 (or a small fractional constant k) to every observed and unobserved event before recalculating probabilities. This is easy to implement but often overly aggressive in assigning too much probability to rare events

P (w_{i} ∣ w_{i - 1}) = \frac{count ( w _{i - 1} , w _{i} )}{count ( w _{i - 1} )}

So let say we have a word hello the probablity of world coming next was

p(hello/world) = count of hello and world together / count of world

Which will give probablity but let say if we going to get any unseen word then the probablity will be 0 and which make whole to 0 to avoid we add 1

Imagine we have a bigram model — probabilities of next word given a previous word.

Let’s say your vocabulary (V) = 3 words:
👉 ["eat", "run", "sleep"]

we can think of your training counts as a table:

Previous word	Next = “eat”	Next = “run”	Next = “sleep”	Total
I	2	1	0	3
you	0	2	1	3
they	1	0	0	1

This is all the raw data we’ve seen in the training corpus.

What the denominator means

When we compute ( P(w_i \mid w_{i-1}) ),
for a given previous word (say, "I"), we divide by total times “I” appeared:

$P (w_{i} ∣ " I ") = \frac{count ( " I " , w _{i} )}{count ( " I " )}$

Here:

(\text{count}(“I”) = 3)
(since we saw “I eat” twice and “I run” once)

So the probabilities (unsmoothed) are:

Next word	Count	Probability
eat	2	2/3
run	1	1/3
sleep	0	0

“sleep” never followed “I” → it gets probability 0.
That’s bad, because it makes any sentence with “I sleep” impossible.

Laplace says:

Pretend we saw each possible next word once more than we actually did.

So for "I", we now have:

Next word	Original count	+1 added	New count
eat	2	+1	3
run	1	+1	2
sleep	0	+1	1

Now, if we add up the new counts in this row: 3 + 2 + 1 = 6

Before smoothing, total was 3.
After smoothing, total is 3 + 3 = 6.

Where did that extra +3 come from?
Because we added +1 to each of the 3 possible next words in the vocabulary.

So the denominator must also increase by +V

If vocabulary size (V = 3), then total count after smoothing = $original total + V = 3 + 3 = 6$

That’s why denominator becomes $(count (w_{i - 1}) + V)$ .

Compute probabilities now

$P_{Laplace} (e a t ∣ I) = \frac{2 + 1}{3 + 3} = \frac{3}{6} = 0.5$ $P_{Laplace} (r u n ∣ I) = \frac{1 + 1}{3 + 3} = \frac{2}{6} = 0.33$ $P_{Laplace} (s l ee p ∣ I) = \frac{0 + 1}{3 + 3} = \frac{1}{6} = 0.17$

Now all three have nonzero probability. Sum = 1 (since (3+2+1 = 6)).

Every row (every “previous word”) gets +1 for each possible next word in the vocabulary.
Since there are V possible next words, the total count in that row increases by V.
So we adjust the denominator to stay normalized.

P_{Laplace} (w_{i} ∣ w_{i - 1}) = \frac{count ( w _{i - 1} , w _{i} ) + 1}{count ( w _{i - 1} ) + V}

Lidstone Smoothing: A generalization of Laplace smoothing where a fractional value k (usually between 0 and 1) is added instead of 1.

Good-Turing Smoothing: A more sophisticated method that re-estimates the frequency of items with frequency N based on the frequency of items with frequency N+1. It is good for estimating the probability of unseen events without knowing the exact total number of possibilities .

Kneser-Ney Smoothing: One of the most effective and widely used methods, particularly for modern NLP systems. It accounts for “how likely a word is to appear in an unfamiliar context” and works very well for higher-order language models (trigrams, etc.) .

Backoff and Interpolation: Techniques that combine probability estimates from different N-gram orders (e.g., using bigram probabilities if a trigram wasn’t seen, or linearly combining unigram, bigram, and trigram models)

sequential probabilistic models like:

HMMs (Hidden Markov Models)
MEMMs (Maximum Entropy Markov Models)
CRFs (Conditional Random Fields)

Resources

NLP course by dan jurafsky

Natural language processing

Table of Contents

Linguistic Knowledge

Morphology (Bottom Layer)

2. Syntax

3. Semantics

4. Pragmatics (Top Layer)

Parsing in NLP

concepts

Parts of Speech (POS)

Constituents (Phrases)

Phrase Structure Grammar (PSG)

Context-Free Grammar (CFG)

TAGS

Different methods to perform POS tagging:

Application of POS

Hidden Markov model

Feature engineering

Tokenization

Token normalization

intuition behind TF,TF-IDF

Word Co-occurrence Matrices (Count-based Semantics)

NLP pipeline technique

N-Gram

Embedding

Word2vec

Continuous Bag of Words (CBOW)

Example Process for CBOW:

Skip-gram

Example Process for Skip-gram:

GloVe

Dimension

Byte Pair Encoding (BPE): A Tokenization Technique

WordPiece

Massive Text Embedding Benchmark

All methods

Text summarization

Resources

Smoothing

Resources

Graph View