Tokenization may refer to:
*
Tokenization (lexical analysis)
In computer science, lexical analysis, lexing or tokenization is the process of converting a sequence of characters (such as in a computer program or web page) into a sequence of ''lexical tokens'' (strings with an assigned and thus identified m ...
in language processing
*
Tokenization (data security)
Tokenization, when applied to data security, is the process of substituting a sensitive data element with a non-sensitive equivalent, referred to as a token, that has no intrinsic or exploitable meaning or value. The token is a reference (i.e. ...
in the field of data security
*
Word segmentation
Text segmentation is the process of dividing written text into meaningful units, such as words, sentences, or topics. The term applies both to mental processes used by humans when reading text, and to artificial processes implemented in comput ...
*
Tokenism
Tokenism is the practice of making only a perfunctory or symbolic effort to be inclusive to members of minority groups, especially by recruiting people from underrepresented groups in order to give the appearance of racial or gender equality wit ...
of minorities
{{disambiguation