Skip to content

Instantly share code, notes, and snippets.

@missinglink

missinglink/installation.md

Last active December 20, 2015 03:19
Show Gist options
  • Save missinglink/6063118 to your computer and use it in GitHub Desktop.
Save missinglink/6063118 to your computer and use it in GitHub Desktop.
Download ZIP
Working example of how to set up elasticsearch so that the _percolate index is being populated via a mongodb river. Tested with mongodb 2.4.4+, elasticsearch 0.9.0+, elasticsearch-river-mongodb 1.6.11
Raw
installation.md

Configure mongo for replication

  • Make sure you have sufficient disk space to hold the oplog ( can be several GB )
  • Edit /etc/mongodb.conf to add replSet = rs0
  • Restart mongodb
  • In the mongo REPL execute rs.initiate() ( this can take some time; log shows activity )
  • Try executing a query, your REPL prompt should now be rs0:PRIMARY>

MongoDB documentation: http://docs.mongodb.org/manual/tutorial/convert-standalone-to-replica-set/

Configuring elasticsearch mongodb river

  • Locate your elasticsearch plugin script /usr/share/elasticsearch/bin/plugin
  • Install elasticsearch-mongodb-river with the command below
  • Restart elasticsearch

Note: Always install the latest stable version from https://github.com/richardwilly98/elasticsearch-river-mongodb (at time of writing this is 1.6.11)

sudo /usr/share/elasticsearch/bin/plugin -url https://oss.sonatype.org/content/repositories/releases/com/github/richardwilly98/elasticsearch/elasticsearch-river-mongodb/1.6.11/elasticsearch-river-mongodb-1.6.11.zip -install river-mongodb
Raw
output.txt
peter@edgy:/var/www/cv/percolate$ ./example/test-percolate-river.sh
=== Drop example river ===
{"ok":true}
=== Clear percolator ===
{"ok":true,"acknowledged":true}
=== Drop example index ===
{"ok":true,"acknowledged":true}
=== Create example index ===
{"ok":true,"acknowledged":true}
-- Create product river for mongo_products --
{"ok":true,"_index":"_river","_type":"example","_id":"_meta","_version":1}
=== empty mongo ===
MongoDB shell version: 2.4.4
connecting to: percolate
=== insert mongo record ===
MongoDB shell version: 2.4.4
connecting to: percolate
MongoDB shell version: 2.4.4
connecting to: percolate
=== add manual entry to percolator ===
{"ok":true,"_index":"_percolator","_type":"example","_id":"insertcoffee","_version":1}
=== sleep 3 ===
=== ES index count ===
"count" : 3,
"_score" : 1.0, "_source" : { "query" : { "term" : { "name" : "foo" } }, "src": "manual" }
"_score" : 1.0, "_source" : {"_id":"51ee9b832e08e61e324be922","query":{"term":{"name":"foo"}},"src":"mongo"}
"_score" : 1.0, "_source" : {"_id":"51ee9b83e409dee0247a3566","query":{"term":{"name":"bar"}},"src":"mongo"}
=== check percolate results ===
percolate { "name": "foo" }
{
"ok" : true,
"matches" : [ "insertcoffee", "51ee9b832e08e61e324be922" ]
}
=== check percolate results - query filter ===
percolate { "name": "foo" } { "term": { "src": "mongo" } }
{
"ok" : true,
"matches" : [ "51ee9b832e08e61e324be922" ]
}
Raw
test-percolate-river.sh
#!/bin/bash
ES='http://localhost:9200'
echo
echo '=== Drop example river ==='
curl -s -XDELETE "$ES/_river/example"
echo
echo
echo '=== Clear percolator ==='
curl -s -XDELETE "$ES/_percolator"
echo
echo
echo '=== Drop example index ==='
curl -s -XDELETE "$ES/example"
echo
echo
echo '=== Create example index ==='
curl -s -XPUT "$ES/example"
echo
echo
echo "-- Create product river for mongo_products --"
curl -s -XPUT "http://localhost:9200/_river/example/_meta" -d '
{
"type": "mongodb",
"mongodb": {
"servers": [
{ "host": "127.0.0.1", "port": 27017 }
],
"options": {},
"db": "percolate",
"collection": "example"
},
"index": {
"name": "_percolator",
"type": "example"
}
}'
echo
echo
echo '=== empty mongo ==='
mongo percolate --eval "db.example.remove();"
echo
echo
echo '=== insert mongo record ==='
mongo percolate --eval 'db.example.insert( { "query" : { "term" : { "name" : "foo" } }, "src": "mongo" } );'
mongo percolate --eval 'db.example.insert( { "query" : { "term" : { "name" : "bar" } }, "src": "mongo" } );'
echo
echo
echo '=== add manual entry to percolator ==='
curl -s -XPUT "$ES/_percolator/example/insertcoffee" -d '{ "query" : { "term" : { "name" : "foo" } }, "src": "manual" }'
echo
echo
echo '=== sleep 3 ==='
sleep 3
echo
echo '=== ES index count ==='
curl -s -XGET "$ES/_percolator/example/_count?pretty" | grep '"count"'
curl -s -XGET "$ES/_percolator/example/_search?pretty" | grep '_source'
echo
echo
echo '=== check percolate results ==='
DOC='{ "name": "foo" }';
echo "percolate $DOC"
curl -s -XGET "$ES/example/something/_percolate?pretty" -d "{ \"doc\": $DOC }"
echo
echo
echo '=== check percolate results - query filter ==='
DOC='{ "name": "foo" }';
QUERY='{ "term": { "src": "mongo" } }';
echo "percolate $DOC $QUERY"
curl -s -XGET "$ES/example/something/_percolate?pretty" -d "{ \"doc\": $DOC, \"query\": $QUERY }"
echo
echo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment