# [−][src]Crate aho_corasick

An implementation of the Aho-Corasick string search algorithm.

The Aho-Corasick algorithm is principally useful when you need to search many large texts for a fixed (possibly large) set of keywords. In particular, the Aho-Corasick algorithm preprocesses the set of keywords by constructing a finite state machine. The search phase is then a quick linear scan through the text. Each character in the search text causes a state transition in the automaton. Matches are reported when the automaton enters a match state.

# Examples

The main type exposed by this crate is `AcAutomaton`

, which can be constructed
from an iterator of pattern strings:

use aho_corasick::{Automaton, AcAutomaton}; let aut = AcAutomaton::new(vec!["apple", "maple"]); // AcAutomaton also implements `FromIterator`: let aut: AcAutomaton<&str> = ["apple", "maple"].iter().cloned().collect();

Finding matches can be done with `find`

:

use aho_corasick::{Automaton, AcAutomaton, Match}; let aut = AcAutomaton::new(vec!["apple", "maple"]); let mut it = aut.find("I like maple apples."); assert_eq!(it.next(), Some(Match { pati: 1, start: 7, end: 12, })); assert_eq!(it.next(), Some(Match { pati: 0, start: 13, end: 18, })); assert_eq!(it.next(), None);

Use `find_overlapping`

if you want to report all matches, even if they
overlap with each other.

use aho_corasick::{Automaton, AcAutomaton, Match}; let aut = AcAutomaton::new(vec!["abc", "a"]); let matches: Vec<_> = aut.find_overlapping("abc").collect(); assert_eq!(matches, vec![ Match { pati: 1, start: 0, end: 1}, Match { pati: 0, start: 0, end: 3 }, ]); // Regular `find` will report only one match: let matches: Vec<_> = aut.find("abc").collect(); assert_eq!(matches, vec![Match { pati: 1, start: 0, end: 1}]);

Finally, there are also methods for finding matches on *streams*. Namely, the
search text does not have to live in memory. It's useful to run this on files
that can't fit into memory:

use std::fs::File; use aho_corasick::{Automaton, AcAutomaton}; let aut = AcAutomaton::new(vec!["foo", "bar", "baz"]); let rdr = File::open("search.txt").unwrap(); for m in aut.stream_find(rdr) { let m = m.unwrap(); // could be an IO error println!("Pattern '{}' matched at: ({}, {})", aut.pattern(m.pati), m.start, m.end); }

There is also `stream_find_overlapping`

, which is just like `find_overlapping`

,
but it operates on streams.

Please see `dict-search.rs`

in this crate's `examples`

directory for a more
complete example. It creates a large automaton from a dictionary and can do a
streaming match over arbitrarily large data.

# Memory usage

A key aspect of an Aho-Corasick implementation is how the state transitions are represented. The easiest way to make the automaton fast is to store a sparse 256-slot map in each state. It maps an input byte to a state index. This makes the matching loop extremely fast, since it translates to a simple pointer read.

The problem is that as the automaton accumulates more states, you end up paying
a `256 * 4`

(`4`

is for the `u32`

state index) byte penalty for every state
regardless of how many transitions it has.

To solve this, only states near the root of the automaton have this sparse map representation. States near the leaves of the automaton use a dense mapping that requires a linear scan.

(The specific limit currently set is `3`

, so that states with a depth less than
or equal to `3`

are less memory efficient. The result is that the memory usage
of the automaton stops growing rapidly past ~60MB, even for automatons with
thousands of patterns.)

If you'd like to opt for the less-memory-efficient-but-faster version, then
you can construct an `AcAutomaton`

with a `Sparse`

transition strategy:

use aho_corasick::{Automaton, AcAutomaton, Match, Sparse}; let aut = AcAutomaton::<&str, Sparse>::with_transitions(vec!["abc", "a"]); let matches: Vec<_> = aut.find("abc").collect(); assert_eq!(matches, vec![Match { pati: 1, start: 0, end: 1}]);

## Structs

AcAutomaton |
An Aho-Corasick finite automaton. |

Dense |
State transitions that can be stored either sparsely or densely. |

FullAcAutomaton |
A complete Aho-Corasick automaton. |

Match |
Records a match in the search text. |

Matches |
An iterator of non-overlapping matches for in-memory text. |

MatchesOverlapping |
An iterator of overlapping matches for in-memory text. |

Sparse |
State transitions that are always sparse. |

StreamMatches |
An iterator of non-overlapping matches for streaming text. |

StreamMatchesOverlapping |
An iterator of overlapping matches for streaming text. |

## Traits

Automaton |
An abstraction over automatons and their corresponding iterators.
The type parameter |

Transitions |
An abstraction over state transition strategies. |

## Type Definitions

StateIdx |
The integer type used for the state index. |