Perl Compatible Regular Expressions (PCRE) is a
library
A library is a collection of materials, books or media that are accessible for use and not just for display purposes. A library provides physical (hard copies) or digital access (soft copies) materials, and may be a physical location or a vir ...
written in
C, which implements a
regular expression
A regular expression (shortened as regex or regexp; sometimes referred to as rational expression) is a sequence of characters that specifies a search pattern in text. Usually such patterns are used by string-searching algorithms for "find" ...
engine, inspired by the capabilities of the
Perl programming language.
Philip Hazel
Philip Hazel is a computer programmer best known for writing the Exim mail transport agent in 1995 and the PCRE regular expression library in 1997. He was employed by the University of Cambridge Computing Service until he retired at the end o ...
started writing PCRE in summer 1997.
PCRE's syntax is much more powerful and flexible than either of the
POSIX regular expression flavors (BRE, ERE)
and than that of many other regular-expression libraries.
While PCRE originally aimed at feature-equivalence with Perl, the two implementations are not fully equivalent. During the PCRE 7.x and Perl 5.9.x phase, the two projects have coordinated development, with features being ported between them in both directions.
In 2015 a fork of PCRE was released with a revised programming interface (API). The original software, now called PCRE1 (the 1.xx–8.xx series), has had bugs mended, but no further development. , it is considered obsolete, and the current 8.45 release is likely to be the last. The new PCRE2 code (the 10.xx series) has had a number of extensions and coding improvements and is where development takes place.
A number of prominent
open-source programs, such as the
Apache and
Nginx
Nginx (pronounced "engine x" ) is a web server that can also be used as a reverse proxy, load balancer, mail proxy and HTTP cache. The software was created by Igor Sysoev and publicly released in 2004. Nginx is free and open-source software ...
HTTP servers, and the
PHP
PHP is a general-purpose scripting language geared toward web development. It was originally created by Danish-Canadian programmer Rasmus Lerdorf in 1993 and released in 1995. The PHP reference implementation is now produced by The PHP Group. ...
and
R scripting languages, incorporate the PCRE library; proprietary software can do likewise, as the library is BSD-licensed. As of Perl 5.10, PCRE is also available as a replacement for Perl's default regular-expression engine through the
re::engine::PCRE module.
The library can be built on Unix, Windows, and several other environments. PCRE2 is distributed with a POSIX C wrapper,
several test programs, and the utility program
pcre2grep built in tandem with the library.
Features
Just-in-time compiler
In computing, just-in-time (JIT) compilation (also dynamic translation or run-time compilations) is a way of executing computer code that involves compilation during execution of a program (at run time) rather than before execution. This may cons ...
support
This optional feature is available if enabled when the PCRE2 library is built. Large performance benefits are possible when (for example) the calling program utilizes the feature with compatible patterns that are executed repeatedly. The just-in-time compiler support was written by Zoltan Herczeg and is not addressed in the POSIX wrapper.
Flexible memory management
The use of the system stack for backtracking can be problematic in PCRE1, which is why this feature of the implementation was changed in PCRE2. The heap is now used for this purpose, and the total amount can be limited. The problem of
stack overflow
In software, a stack overflow occurs if the call stack pointer exceeds the stack bound. The call stack may consist of a limited amount of address space, often determined at the start of the program. The size of the call stack depends on many facto ...
, which came up regularly with PCRE1, is no longer an issue with PCRE2 from release 10.30 (2017).
Consistent escaping rules
Like Perl, PCRE2 has consistent escaping rules: any non-alpha-numeric character may be escaped to mean its literal value by prefixing a
\ (backslash) before the character. Any alpha-numeric character preceded by a backslash typically gives it a special meaning. In the case where the sequence has not been defined to be special, an error occurs. This is different to Perl, which gives an error only if it is in warning mode (PCRE2 does not have a warning mode). In basic POSIX regular expressions, sometimes backslashes escaped non-alpha-numerics (e.g.
\.), and sometimes they introduced a special feature (e.g.
\(\)).
Extended character classes
Single-letter
character classes are supported in addition to the longer POSIX names. For example,
\d matches any digit exactly as
:digit: would in POSIX regular expressions.
Minimal matching (a.k.a. "ungreedy")
A
? may be placed after any repetition quantifier to indicate that the shortest match should be used. The default is to attempt the
longest match first and backtrack through shorter matches: e.g.
a.*?b would match first "ab" in "ababab", where
a.*b would match the entire string.
If the
U flag is set, then quantifiers are ungreedy (lazy) by default, while
? makes them greedy.
Unicode character properties
Unicode
Unicode, formally The Unicode Standard,The formal version reference is is an information technology standard for the consistent encoding, representation, and handling of text expressed in most of the world's writing systems. The standard, wh ...
defines several properties for each character. Patterns in PCRE2 can match these properties: e.g.
\p.*?\p would match a string beginning with any "opening punctuation" and ending with any "close punctuation" such as
bc/syntaxhighlight>. Matching of certain "normal" metacharacters can be driven by Unicode properties when the compile option PCRE2_UCP is set. The option can be set for a pattern by including (*UCP) at the start of pattern. The option alters behavior of the following metacharacters: \B, \b, \D, \d, \S, \s, \W, \w, and some of the POSIX character classes. For example, the set of characters matched by \w (word characters) is expanded to include letters and accented letters as defined by Unicode properties. Such matching is slower than the normal (ASCII
ASCII ( ), abbreviated from American Standard Code for Information Interchange, is a character encoding standard for electronic communication. ASCII codes represent text in computers, telecommunications equipment, and other devices. Because of ...
-only) non-UCP alternative. Note that the UCP option requires the library to have been built to include Unicode support (this is the default for PCRE2). Very early versions of PCRE1 supported only ASCII code. Later, UTF-8 support was added. Support for UTF-16 was added in version 8.30, and support for UTF-32 in version 8.32. PCRE2 has always supported all three UTF encodings.
Multiline matching
^ and $ can match at the beginning and end of a string only, or at the start and end of each "line" within the string, depending on what options are set.
Newline/linebreak options
When PCRE is compiled, a newline default is selected. Which newline/linebreak is in effect affects where PCRE detects ^ line beginnings and $ ends (in multiline mode), as well as what matches dot (regardless of multiline mode, unless the dotall option (?s) is set). It also affects PCRE matching procedure (since version 7.0): when an unanchored pattern fails to match at the start of a newline sequence, PCRE advances past the entire newline sequence before retrying the match. If the newline option alternative in effect includes CRLF as one of the valid linebreaks, it does not skip the \n in a CRLF if the pattern contains specific \r or \n references (since version 7.3). Since version 8.10, the metacharacter \N always matches any character other than linebreak characters. It has the same behavior as . when the dotall option aka (?s) is not in effect.
The newline option can be altered with external options when PCRE is compiled and when it is run. Some applications using PCRE provide users with the means to apply this setting through an external option. So the newline option can also be stated at the start of the pattern using one of the following:
* (*LF) Newline is a linefeed character. Corresponding linebreaks can be matched with \n.
* (*CR) Newline is a carriage return. Corresponding linebreaks can be matched with \r.
* (*CRLF) Newline/linebreak is a carriage return followed by a linefeed. Corresponding linebreaks can be matched with \r\n.
* (*ANYCRLF) Any of the above encountered in the data will trigger newline processing. Corresponding linebreaks can be matched with (?:\r\n?, \n) or with \R. See below for configuration and options concerning what matches backslash-R.
* (*ANY) Any of the above plus special Unicode linebreaks.
When not in UTF-8 mode, corresponding linebreaks can be matched with (?:\r\n?, \n, \x0B, \f, \x85)
Backslash-R options
When PCRE is compiled, a default is selected for what matches \R. The default can be either to match the linebreaks corresponding to ANYCRLF or those corresponding to ANY. The default can be overridden when necessary by including (*BSR_UNICODE) or (*BSR_ANYCRLF) at the start of the pattern. When providing a (*BSR..) option, you can also provide a (*''newline'') option, e.g., (*BSR_UNICODE)(*ANY)''rest-of-pattern''. The backslash-R options also can be changed with external options by the application calling PCRE2, when a pattern is compiled.
Beginning of pattern options
Linebreak options such as (*LF) documented above; backslash-R options such as (*BSR_ANYCRLF) documented above; Unicode Character Properties option (*UCP) documented above; (*UTF8) option documented as follows: if your PCRE2 library has been compiled with UTF support, you can specify the (*UTF) option at the beginning of a pattern instead of setting an external option to invoke UTF-8, UTF-16, or UTF-32 mode.
Backreferences
A pattern may refer back to the results of a previous match. For example, (a, b)c\1 would match either "aca" or "bcb" and would not match, for example, "acb".
Named subpatterns
A sub-pattern (surrounded by parentheses, like (...)) may be named by including a leading ?P after the opening parenthesis. Named subpatterns are a feature that PCRE adopted from Python
Python may refer to:
Snakes
* Pythonidae, a family of nonvenomous snakes found in Africa, Asia, and Australia
** ''Python'' (genus), a genus of Pythonidae found in Africa and Asia
* Python (mythology), a mythical serpent
Computing
* Python (pro ...
regular expressions.
This feature was subsequently adopted by Perl, so now named groups can also be defined using (?...) or (?'name'...), as well as (?P...). Named groups can be backreferenced with, for example: (?P=name) (Python syntax) or \k'name' (Perl syntax).
Subroutines
While a backreference provides a mechanism to refer to that part of the subject that has previously matched a subpattern, a subroutine provides a mechanism to reuse an underlying previously defined subpattern. The subpattern's options, such as case independence, are fixed when the subpattern is defined. (a.c)(?1) would match "aacabc" or "abcadc", whereas using a backreference (a.c)\1 would not, though both would match "aacaac" or "abcabc". PCRE also supports a non-Perl Oniguruma
by K. Kosako is a BSD licensed regular expression library that supports a variety of character encodings. The Ruby programming language, in version 1.9, as well as PHP's multi-byte string module (since PHP5), use Oniguruma as their regular exp ...
construct for subroutines. They are specified using \g or \g.
Atomic grouping
Atomic grouping is a way of preventing backtracking
Backtracking is a class of algorithms for finding solutions to some computational problems, notably constraint satisfaction problems, that incrementally builds candidates to the solutions, and abandons a candidate ("backtracks") as soon as it d ...
in a pattern. For example, a++bc will match as many "a"s as possible and never back up to try one less.
Look-ahead and look-behind assertions
Patterns may assert that previous text or subsequent text contains a pattern without consuming matched text (zero-width assertion). For example, /\w+(?=\t)/ matches a word followed by a tab, without including the tab itself.
Look-behind assertions cannot be of uncertain length though (unlike Perl) each branch can be a different fixed length.
\K can be used in a pattern to reset the start of the current whole match. This provides a flexible alternative approach to look-behind assertions because the discarded part of the match (the part that precedes \K) need not be fixed in length.
Escape sequences for zero-width assertions
E.g. \b for matching zero-width "word boundaries", similar to (?<=\W)(?=\w), (?<=\w)(?=\W), ^, $.
Comments
A comment begins with (?# and ends at the next closing parenthesis.
Recursive patterns
A pattern can refer back to itself recursively or to any subpattern. For example, the pattern \((a*, (?R))*\) will match any combination of balanced parentheses and "a"s.
Generic callouts
PCRE expressions can embed (?C''n''), where ''n'' is some number. This will call out to an external user-defined function through the PCRE API and can be used to embed arbitrary code in a pattern.
Differences from Perl
Differences between PCRE2 and Perl (as of Perl 5.9.4) include but are not limited to:
Until release 10.30 recursive matches were atomic in PCRE and non atomic in Perl
This meant that "<!>!>><>>!>!>!>" =~ /^(<(?: <>, (?3), (?1))*>)()(!>!>!>)$/ would match in Perl but not in PCRE2 until release 10.30.
The value of a capture buffer deriving from the ? quantifier (match 1 or 0 times) when nested in another quantified capture buffer is different
In Perl "aba" =~ /^(a(b)?)+$/; will result in $1 containing "a" and $2 containing undef, but in PCRE will result in $2 containing "b".
PCRE allows named capture buffers to be given numeric names; Perl requires the name to follow the rule of barewords
This means that \g is unambiguous in Perl, but potentially ambiguous in PCRE.
This is no longer a difference since PCRE 8.34 (released on 2013-12-15), which no longer allows group names to start with a digit.
PCRE allows alternatives within lookbehind to be different lengths
Within lookbehind assertions, both PCRE and Perl require fixed-length patterns.
That is, both PCRE and Perl disallow variable-length patterns using quantifiers within lookbehind assertions.
However, Perl requires all alternative branches of a lookbehind assertion to be the same length as each other, whereas PCRE allows those alternative branches to have different lengths from each other as long as each branch still has a fixed length.
PCRE does not support certain "experimental" Perl constructs
Such as (??) (a callback whose return is evaluated as being part of the pattern) nor the (?) construct, although the latter can be emulated using (?Cn).
Recursion control verbs added in the Perl 5.9.x series are also not supported.
Support for experimental backtracking control verbs (added in Perl 5.10) is available in PCRE since version 7.3.
They are (*FAIL), (*F), (*PRUNE), (*SKIP), (*THEN), (*COMMIT), and (*ACCEPT).
Perl's corresponding use of arguments with backtracking control verbs is not generally supported.
Note however that since version 8.10, PCRE supports the following verbs with a specified argument: (*MARK:markName), (*SKIP:markName), (*PRUNE:markName), and (*THEN:markName).
Since version 10.32 PCRE2 has supported (*ACCEPT:markName), (*FAIL:markName), and (*COMMIT:markName).
PCRE and Perl are slightly different in their tolerance of erroneous constructs
Perl allows quantifiers on the (?!...) construct, which is meaningless but harmless (albeit inefficient); PCRE produces an error in versions before 8.13.
PCRE has a hard limit on recursion depth, Perl does not
With default build options "bbbbXcXaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" =~ /.X(.+)+X/ will fail to match due to the limit, but Perl will match this correctly.
Perl uses the heap for recursion and has no hard limit for recursion depth, whereas PCRE2 has a compile-time default limit that can be adjusted up or down by the calling application.
Verification
With the exception of the above points, PCRE is capable of passing the tests in the Perl "t/op/re_tests" file, one of the main syntax-level regression tests for Perl's regular expression engine.
Notes and references
Notes
[The core PCRE2 library provides both matching and match and replace functionality.]
[Sure the \x85 part is not \xC2\x85? (i.e. (?:\r\n?, \n, \x0B, \f, \xC2\x85), as U+0085 != 0x85)]
Caveat: If the pattern \xC2\x85 failed to work: experiment with the RegEx implementation's Unicode settings, or try substituting with the following:
* \x
* \u0085
References
[
Final release of PCRE1: https://lists.exim.org/lurker/message/20210615.162400.c16ff8a3.en.html]
[
Releases: https://github.com/PCRE2Project/pcre2/releases]
[
Exim and PCRE: How free software hijacked my life (1999-12), by ]Philip Hazel
Philip Hazel is a computer programmer best known for writing the Exim mail transport agent in 1995 and the PCRE regular expression library in 1997. He was employed by the University of Cambridge Computing Service until he retired at the end o ...
, p. 7: https://www.ukuug.org/events/winter99/proc/PH.ps
[
* Regular Expression - POSIX Standard (Google Search): https://www.google.com/search?num=100&q=%22Regular+Expression%22%7C%22Regular+Expressions%22%7C%22RegEx%22%7C%22RegExp%22+site%3Apubs.opengroup.org+inurl%3Aonlinepubs%2F9699919799+-intitle%3A%22Index+of+%2Fonlinepubs%22+-inurl%3Aidx+-inurl%3Acontents.html+-inurl%3Atoc.html+-inurl%3A9699919799.orig+-inurl%3A2008edition+-inurl%3A2013edition+-inurl%3A2016edition+-inurl%3A2018edition
* Utilities § Pattern Matching Notation: https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/utilities/V3_chap02.html#tag_18_13
* Base Definitions § Basic Regular Expressions: https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/basedefs/V1_chap09.html#tag_09_03
* Rationale § Regular Expressions: https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/xrat/V4_xbd_chap09.html#tag_21_09]
[
PCRE2 - Perl-compatible regular expressions (revised API) (2020), by ]University of Cambridge
, mottoeng = Literal: From here, light and sacred draughts.
Non literal: From this place, we gain enlightenment and precious knowledge.
, established =
, other_name = The Chancellor, Masters and Schola ...
: https://pcre.org/pcre2.txt
[
Differences Between PCRE2 and Perl (2019-07-13), by ]Philip Hazel
Philip Hazel is a computer programmer best known for writing the Exim mail transport agent in 1995 and the PCRE regular expression library in 1997. He was employed by the University of Cambridge Computing Service until he retired at the end o ...
: https://www.pcre.org/current/doc/html/pcre2compat.html
[
Quote PCRE changelog (https://www.pcre.org/original/changelog.txt): "Perl no longer allows group names to start with digits, so I have made this change also in PCRE."]
[
ChangeLog for PCRE2: https://www.pcre.org/changelog.txt]
See also
* Comparison of regular expression engines
This is a comparison of regular expression engines.
Libraries
Languages
Language features
NOTE: An application using a library for regular expression support does not necessarily offer the full set of features of the library, e.g. GNU grep ...
External links
* {{Official website
* PCRE - Development mailing list: https://groups.google.com/g/pcre2-dev
* PCRE - Bug Tracker: https://github.com/PCRE2Project/pcre2/issues
* Pattern Matching Using Regular Expressions (2010-03-02), by Nick Maclaren, Philip Hazel: https://www-uxsup.csx.cam.ac.uk/courses/moved.REs/paper.pdf
* pcre 8.43 (2019-04) - Windows Cygwin x86-64: https://www-uxsup.csx.cam.ac.uk/pub/windows/cygwin/x86_64/release/pcre/
Pattern matching
C (programming language) libraries
Perl
Regular expressions
Software using the BSD license