K.Boopathi
BlogNotesSecond BrainProjectsHire MeYour reading file0 pts

LLM DataSets

Jul 13, 20252 min read

DataSet For finetuning

Data Creation Techniques

  • Synthetic Data Generation: Synthetic data, data generated by LLMs, is commonly used for fine-tuning other LLMs.
    • Generating Prompts and Completions: LLMs can be prompted to generate prompts and completions, often in conjunction with initial context to enhance the quality and complexity of the generated data.
    • AI Feedback and Judging: LLMs can be used to judge and score the quality of generated data, providing feedback on aspects like instruction following, truthfulness, and helpfulness.
  • Data Set Formatting: Data needs to be formatted correctly for different fine-tuning techniques and algorithms.
    • Supervised Fine-tuning: Typically involves a question-and-answer format, where the data set provides input questions and corresponding desired responses.
    • Reinforcement Learning (RL) Algorithms: While there are various RL algorithms for LLM training, many utilize similar data set structures, making it easier to adapt existing data.
      • Direct Preference Optimization (DPO): DPO data sets consist of input prompts paired with chosen (preferred) and rejected responses.
      • K-shot Training (KTO): KTO data sets present model-generated responses along with a binary preference (thumbs up or thumbs down).
      • Spin and Oro: Spin is an iterative approach that starts with a small data set and synthetically generates new responses to expand it. Oro uses the same format as DPO but skips the initial supervised fine-tuning step.

Data Set Quality Improvement

  • Data Duplication: While data duplication can be useful for training, it’s important to control the amount of duplication, as more data is not always better.
    • Deduplication Pipelines: Tools like Hugging Face’s data library can be used to implement deduplication pipelines.
    • Deduplication Techniques: Deduplication can be achieved through various techniques, including rule-based filtering based on metadata, topic-wise deduplication, and using embedding similarity to identify and remove redundant examples.
  • Rule-based Data Cleaning: Simple, human-understandable rules, such as regular expressions (regex), can be used to filter out unwanted patterns in the data, improving data quality.
  • Advanced Techniques: More advanced methods like topic modeling, zero-shot classifiers, and LLM-based judging can further enhance data quality.
  • Human Annotation: Human annotation, while potentially expensive, remains an important component of data set quality improvement.

Graph View