Python Tutorial

Aho-Corasick is a kind of a dictionary-matching algorithm. This algorithm is used to search for words present in the keywords set. This algorithm is fast and efficient for finding words and their location. The Aho-Corasick algorithm builds an existing system and employs the TRIE notion.

A tree data structure is used to execute the technique. When we create the tree, it converts or attempts to convert it into an automaton, allowing us to complete or perform the search in linear time.

Time Complexity of the Aho-Corasick Algorithm

The Aho-Corasik algorithm searches words in O(X+ Y+ Z) time, where X is the text length, Y is the keyword length, and Z is the number of times a keyword occurs in the text.

Problem Statement of the Ahio-Corasick Algorithm

Let's assume we have the input text and an array of m words, a[ ]. We need to search the count of the words present in the input text.

Let x be the text length, and y be the number of characters in all the words. It means y = len(a [0]) + len(a [1]) + len(a [2]) + …. + len(a [z - 1]). Here, z is the number of input words.

We will understand this algorithm using an example in which we will take an input string and the set of words to be searched in the text string.

Example:

Input:

Text string: txt= "hellotheirshere"
The sample set of words: a[ ] = {"he", "hello", "she", "here", "their", "the"}

Output:

The Word "he" is found at index 0 to 1.
The Word "he" is found at index 6 to 7
The Word "he" is found at index 11 to 12
The Word "hello" is found at index 0 to 4.
The Word "the" is found at index 5 to 7
The Word "their" is found at index 7 to 11.
The Word "she" is found at index 10 to 12.
The Word "here" is found at index 11 to 14

Preprocessing of the algorithm

The Aho-Corasick Algorithm is divided into three different stages:

Go To
Output
Failure

Go To: This stage makes the tree using the keywords fed into the algorithm in a pattern. It goes to the main function, then set points, and then goes to the main root. It traces the margins of all the words in the array a[ ]. A 2-D array, gt[ ][ ], denotes the go function, in which we can store characters and states for further next states.

Output: This stage searches the words ending at a particular state. It is the result when the condition and the availability match. This function stores indexes of the words ending at current states. A 1D array, op [ ], represents the output function, in which we can store the words as a bitmap for the current state.

Failure: It searches backward transformation to find suitable keywords from the set. If the keyword does not match, then it will not count. When the current character lacks an edge in Trie, this function records every edge followed. The 1D array fl [ ] represents the failure in which we record the next state for the current state.

Implementation of the Aho-Corasick algorithm for Pattern searching in Python

Here is the implementation of the Aho-Corasick Algorithm in Python:

Code:

from collections import defaultdict
class Aho_Corasick:
    def __init__(self, words):
        self.maxStates = sum([len(word) for word in words])
        self.maxChar = 22
        self.out = [0]*(self.maxStates + 1)
        self.fail = [-1]*(self.maxStates + 1)
        self.goto = [[-1]*self.maxChar for _ in range(self.maxStates + 1)]

        for i in range(len(words)):
            words[i] = words[i].lower()
        self.words = words
        self.states_count = self.__build_matching_machine()
        
    def __build_matching_machine(self):
        m = len(self.words)
        state = 1

        for i in range(m):
            word = self.words[i]
            currentState = 0

            for char in word:
                c = ord(char)

                if self.goto[currentState][c] == -1:
                    self.goto[currentState][c] = state
                    state += 1
                currentState = self.goto[currentState][c]
        self.out[currentState] |= (1 << i)

        for c in range(self.maxChar):
            if self.goto[0][c] == -1:
                self.goto[0][c] = 0
            queue = []

        for ch in range(self.maxChar):
            if self.goto[0][c] != 0:
                self.fail[self.goto[0][c]] = 0
                queue.append(self.goto[0][c])

        while queue:
            states = queue.pop(0)

            for c in range(self.maxChar):
                if self.goto[states][c] != -1:
                    failure = self.fail[states]

            while self.goto[failure][c] == -1:
                failure = self.fail[failure]
            failure = self.goto[failure][c]
            self.fail[self.goto[states][c]] = failure
            self.out[self.goto[states][c]] |= self.out[failure]
            queue.append(self.goto[states][ch])
        return state

        def find_Next_State(self, currentState, next_inp):
            ans = currentSate
            c = ord(next_inp) - 97 
            while self.goto[ans][c] == -1:
                answer = self.fail[ans]
 
            return self.goto[ans][c]

        def search(self, txt):
            txt = txt.lower()
            currentSate = 0
            res = defaultdict(list)

            for i in range(len(txt)):
                currentState = self.findNextState(currentState, txt[i])

                if self.out[currentState] == 0: continue

                for j in range(len(self.words)):
                    if (self.out[currentState] & (1 << j)) > 0:
                        word = self.words[j]
                        res[word].append(i-len(word)+1)
            return result

if __name__ == "__main__":
    words = ["he", "hello", "she", "here", "their", "the"]
    txt = "hellotheirshere"
    aho_chorasick = Aho_Corasick(words)
    res = aho_chorasick.search(txt)

    for word in result:
        for i in result[word]:
            print("The Word", word, "is found at index", i, "to", i + len(word) - 1) 

Output:

The Word "he" is found at index 0 to 1.
The Word "he" is found at index 6 to 7
The Word "he" is found at index 11 to 12
The Word "hello" is found at index 0 to 4.
The Word "the" is found at index 5 to 7
The Word "their" is found at index 7 to 11.
The Word "she" is found at index 10 to 12.
The Word "their" is found at index 11 to 14

Explanation:

In this, we have used the default dictionary to store the output. We have set a maximum limit of characters and states. Using the three stages of the algorithm, we have preprocessed our data. Then, using loops and queues, we built the machine/ automata and then searched the set of words from the text string.

Next TopicAmazon Product Price Tracker using Python

← prev next →