Contao Utils Bundle

Tokenizer
in package

Splits text into normalized and optionally stemmed words.

Shared by the indexer and the query side, so stored vectors and search terms are always normalized the same way.

Table of Contents

Methods

foldAccents()  : string
Lowercases a word and strips accents: ä → a, é → e, ñ → n, å → a. Letters without a decomposition into base letter and accent are transliterated: ß → ss, æ → ae, ø → o, œ → oe, ð → d, þ → th, ł → l, đ → d, ı → i.
parseQuery()  : array<int, array<int, string>>
Parses a user search query into phrases of tokens.
parseQueryWords()  : array<int, array<int, string>>
Parses a user search query into the words of each phrase, in the order of the query.
tokenize()  : array<int, string>
Tokenize free text into normalized, optionally stemmed words, preserving their order.
parseAlternatives()  : array<int, array<int, array{0: bool, 1: non-empty-list}>>
Splits a query into its alternatives and each alternative into quoted and unquoted segments of tokens. Segments and alternatives without tokens are dropped.
splitPhrases()  : array<int, string>
Splits by commas outside of double quotes. Quotes are kept to mark phrases. A quote inside quotes can be escaped as "" or \" and is replaced by a space, as quotes are not searchable anyway.

Methods

foldAccents()

Lowercases a word and strips accents: ä → a, é → e, ñ → n, å → a. Letters without a decomposition into base letter and accent are transliterated: ß → ss, æ → ae, ø → o, œ → oe, ð → d, þ → th, ł → l, đ → d, ı → i.

public static foldAccents(string $word) : string
Parameters
$word : string
Return values
string

parseQuery()

Parses a user search query into phrases of tokens.

public static parseQuery(string $query, string $locale[, StemmerInterface|null $stemmer = null ][, NormalizerInterface|null $normalizer = null ]) : array<int, array<int, string>>

Commas separate phrases (OR), the terms of a phrase are combined with AND. Words wrapped in double quotes stay a single term, so their order matters. Quoted and unquoted words can be mixed, an unclosed quote is ignored.

Example: Tokenizer::parseQuery('Altstadt Führungen, Museum "am Abend"', 'de', new SnowballStemmer()) returns [['altstadt', 'fuhrung'], ['museum', 'am abend']].

Parameters
$query : string
$locale : string

The language of the query, passed to the normalizer and the stemmer

$stemmer : StemmerInterface|null = null
$normalizer : NormalizerInterface|null = null
Return values
array<int, array<int, string>>

parseQueryWords()

Parses a user search query into the words of each phrase, in the order of the query.

public static parseQueryWords(string $query, string $locale[, StemmerInterface|null $stemmer = null ][, NormalizerInterface|null $normalizer = null ]) : array<int, array<int, string>>

Unlike parseQuery(), repeated words are kept and quoted phrases are split into their words, e.g. for scoring how closely a record matches the query. The phrases correspond to the ones of parseQuery().

Example: Tokenizer::parseQueryWords('Haus an Haus, "am See"', 'de') returns [['haus', 'an', 'haus'], ['am', 'see']].

Parameters
$query : string
$locale : string

The language of the query, passed to the normalizer and the stemmer

$stemmer : StemmerInterface|null = null
$normalizer : NormalizerInterface|null = null
Return values
array<int, array<int, string>>

tokenize()

Tokenize free text into normalized, optionally stemmed words, preserving their order.

public static tokenize(string $text, string $locale[, StemmerInterface|null $stemmer = null ][, NormalizerInterface|null $normalizer = null ]) : array<int, string>

Words are lowercased, normalized, stemmed and then stripped of accents, as normalizers and stemmers rely on accents (e.g. German umlauts).

Example: Tokenizer::tokenize('Häuser in München', 'de', new SnowballStemmer()) === ['haus', 'in', 'munch']

Parameters
$text : string
$locale : string

The language of the text, passed to the normalizer and the stemmer

$stemmer : StemmerInterface|null = null
$normalizer : NormalizerInterface|null = null
Return values
array<int, string>

parseAlternatives()

Splits a query into its alternatives and each alternative into quoted and unquoted segments of tokens. Segments and alternatives without tokens are dropped.

private static parseAlternatives(string $query, string $locale, StemmerInterface|null $stemmer, NormalizerInterface|null $normalizer) : array<int, array<int, array{0: bool, 1: non-empty-list}>>
Parameters
$query : string
$locale : string
$stemmer : StemmerInterface|null
$normalizer : NormalizerInterface|null
Return values
array<int, array<int, array{0: bool, 1: non-empty-list}>> —

Per alternative the segments as [quoted, tokens]

splitPhrases()

Splits by commas outside of double quotes. Quotes are kept to mark phrases. A quote inside quotes can be escaped as "" or \" and is replaced by a space, as quotes are not searchable anyway.

private static splitPhrases(string $input) : array<int, string>
Parameters
$input : string
Return values
array<int, string> 

        

        
    




    

    

                                

                

                                
    

        On this page

        
    


                

                            

            

    

        

            
Search results