Haystack docs home page

PreProcessor

Use the PreProcessor to normalize white spaces, get rid of headers and footers, clean empty lines in your Documents, or split them into smaller pieces.

Splitting is generally recommended for long Documents as it makes the Retriever's job easier and speeds up Question Answering. For suggestions on how best to split your documents, see Optimization.

Position in a PipelineAs early in an indexing Pipeline as possible but after File Converters and Crawlers
InputDocuments
OutputDocuments
ClassesPreProcessor

Tutorial: Check out our preprocessing tutorial if you'd like to start working with code examples or our advanced indexing tutorial for ideas on what can be done at indexing time.

Usage

Initialization:

from haystack.nodes import PreProcessor


processor = PreProcessor(
    clean_empty_lines=True,
    clean_whitespace=True,
    clean_header_footer=True,
    split_by="word",
    split_length=200,
    split_respect_sentence_boundary=True,
    split_overlap=0
)
ArgumentTypeDescription
clean_empty_linesboolWhether to normalize 3 or more consecutive empty lines to be just a two empty lines.
clean_whitespaceboolWhether to remove any whitespace at the beginning or end of each line in the text.
clean_header_footerboolWhether to remove any long header or footer texts that are repeated on each page.
split_bystringDetermines what unit the document is split by. Choose from 'word', 'sentence' or 'passage'.
split_lengthintSets a maximum number of 'word', 'sentence' or 'passage' units per output document
split_respect_sentence_boundaryboolEnsures that document boundaries do not fall in the middle of sentences
split_overlapintSets the amount of overlap between two adjacent documents after a split. Setting this to a positive number essentially enables the sliding window approach.

To run the processor by itself:

doc = converter.convert(file_path=file, meta=None)
docs = processor.process(doc)

Document Format

When you are not using an indexing Pipeline, the PreProcessor can take either Document objects (recommended) as input or plain dictionaries. To learn more about the Document class see Documents, Answers and Labels.

# Option 1: Native Haystack Documents
docs = [
    Document(
        content='DOCUMENT_TEXT_HERE',
        meta={'name': DOCUMENT_NAME, ...}
        ...
    ), ...
]


# Option 2: Plain dict
docs = [
    {
        'content': 'DOCUMENT_TEXT_HERE',
        'meta': {'name': DOCUMENT_NAME, ...}
    }, ...
]