Skip to content

Instantly share code, notes, and snippets.

@hackintoshrao
Last active July 31, 2020 19:32
Show Gist options
  • Save hackintoshrao/0e8d715d8739b12c67a804c7249146a3 to your computer and use it in GitHub Desktop.
Save hackintoshrao/0e8d715d8739b12c67a804c7249146a3 to your computer and use it in GitHub Desktop.
Steps to generate Dgraphs fulltext index tokens
  1. Install Go

  2. Download Dgraph repo using go get

    go get -u github.com/dgraph-io/dgraph
    
  3. Go the tok package directory inside Dgraph's source.

    cd $GOPATH/src/github.com/dgraph-io/dgraph/tok
    
  4. Turn the Go modules on

    export GO111MODULE=on
    
  5. Copy the following test into the tok_test.go file.

    package tok
    
    import (
            "testing"
            "github.com/stretchr/testify/require"
    )
    
    
    func TestGetTokensForFullText(t *testing.T) {
            val := "Let's Go and catch @francesc at @Gopherpalooza today, as he scans into Go source code by building its Graph in Dgraph!\nBe there, as he Goes through analyzing Go source code, using a Go program, that stores data in the GraphDB built in Go!\n#golang #GraphDB #Databases #Dgraph"
             // Call the fulltext tokenizer.
            tokens, err := (&FullTextTokenizer{lang: "en"}).Tokens(val)
            require.NoError(t, err)
            // Print the generated tokens.
            t.Logf("%v",tokens)
    } 
  6. Now, run the test, this should print the fulltext tokens for the matched tweet. Change the value of the val variable in the above test to find the fulltext tokens for a diffrent sentence or a text paragraph.

    go test -run=TestGetTokensForFullText -v
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment