MongoDB to Neo4j Using neo4j_doc_manager.

We had a requirement where we wanted to have all the data which is in mongodb to be replicated on neo4j to show few graphs. Here is quick way to demonstrate sync data between mongodb and neo4j.

We will be doing this in 4 phases.

Install MongoDB on Windows.
Install neo4j on Windows.
Install and Setup mongo-connector and neo4j_doc_manager.
Write a basic python script using pymongo to insert data to MongoDB.

Installing MongoDB.

Download.

First we download mongodb from MongoDB Download

Setup.

Install Above executable (or use below command in command prompt with Admin Mode).
```
 msiexec.exe mongodb-win32-x86_64-2008plus-ssl-3.6.3-signed.msi
```
Create directory.
```
 md \data\db
```

Start MongoDB with replica set.

 "C:\Program Files\MongoDB\Server\3.6\bin\mongod.exe" --replSet myDevReplSet

Open mongo shell and execute.
```
 rs.initiate()
```

Our MongoDB is ready to use. We can run some commands to check in mongo shell or use Studio 3T.

Testing.

Create Database.

use testdatabase

Test Query to insert data in to collection testCollections

db.testCollections.insert({ "session": { "title": "12 Years of Spring: An Open Source Journey", "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015." }, "topics":  ["keynote", "spring"], "room": "Auditorium", "timeslot": "Wed 29th, 09:30-10:30", "speaker": { "name": "Juergen Hoeller", "bio": "Juergen Hoeller is co-founder of the Spring Framework open source project.", "twitter": "https://twitter.com/springjuergen", "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg" } } );

Output.

WriteResult({ "nInserted" : 1 })

Search for data in the database testdatabase.

use testdatabase
db.testCollections.find()

Output.

{ "_id" : ObjectId("5ac4b4cc0774ac612ca06936"), "session" : { "title" : "12 Years of Spring: An Open Source Journey", "abstract" : "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015." }, "topics" : [ "keynote", "spring" ], "room" : "Auditorium", "timeslot" : "Wed 29th, 09:30-10:30", "speaker" : { "name" : "Juergen Hoeller", "bio" : "Juergen Hoeller is co-founder of the Spring Framework open source project.", "twitter" : "https://twitter.com/springjuergen", "picture" : "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg" }}

Installing neo4j.

Download.

Download neo4j from neo4j download.

Setup.

Install the above executable.
Create local DB

Create Local DB

Give Name

Manage DB to change Settings

Start the graph DB service.

Testing.

Creating a node in database.

create (Zubair:Person {name:"Zubair AHMED"})

Output.

Added 1 label, created 1 node, set 1 property, completed after 157 ms.

Search for node.

MATCH (n:Person) RETURN n LIMIT 25

Output.

{
  "name": "Zubair AHMED"
}

Setup `neo4j_doc_manager`.

Setup.

Installing depending python packages.

pip install neo4j-doc-manager

Export usernamd and password (by default this is neo4j:neo4j)

export NEO4J_AUTH=user:password

On windows we need to do this.

SETX NEO4J_AUTH "user:password" -m

On windows make sure you restart the terminal.

Execute Connector command.

mongo-connector -m localhost:27017 -t http://localhost:7474/db/data -d neo4j_doc_manager

On windows.

PS C:\Python27\Scripts> .\mongo-connector -m localhost:27017 -t http://localhost:7474/db/data -d neo4j_doc_manager
Logging to C:\Python27\Scripts\mongo-connector.log.

Checking Logs.

2018-04-04 14:43:01,062 [ALWAYS] mongo_connector.connector:51 - Starting mongo-connector version: 2.5.1
2018-04-04 14:43:01,063 [ALWAYS] mongo_connector.connector:51 - Python version: 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 20:25:58) [MSC v.1500 64 bit (AMD64)]
2018-04-04 14:43:01,065 [ALWAYS] mongo_connector.connector:51 - Platform: Windows-10-10.0.14393
2018-04-04 14:43:01,065 [ALWAYS] mongo_connector.connector:51 - pymongo version: 3.6.1
2018-04-04 14:43:01,078 [ALWAYS] mongo_connector.connector:51 - Source MongoDB version: 3.6.3
2018-04-04 14:43:01,079 [ALWAYS] mongo_connector.connector:51 - Target DocManager: mongo_connector.doc_managers.neo4j_doc_manager version: unknown

TESTING.

Go to the mongo shell and execute the below two commands.

use test
db.talks.insert( { "session": { "title": "12 Years of Spring: An Open Source Journey", "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015." }, "topics":  ["keynote", "spring"], "room": "Auditorium", "timeslot": "Wed 29th, 09:30-10:30", "speaker": { "name": "Juergen Hoeller", "bio": "Juergen Hoeller is co-founder of the Spring Framework open source project.", "twitter": "https://twitter.com/springjuergen", "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg" } } );

Now we can go to the neo4j terminal on http://localhost:7474/

Execute below command.

match (n) return n limit 25

Output.

{
  "name": "Zubair AHMED"
}
{
  "_id": "5ac4c9c5dd231d89523d41c4",
  "topics": [
    "keynote",
    "spring"
  ],
  "room": "Auditorium",
  "timeslot": "Wed 29th, 09:30-10:30",
  "_ts": 6540574406794543106
}

Output

So we can see all the nodes which were created on mongoDB.

Basic Python script to Insert into MongoDB.

Setup.

Installing pymongo using pip.

pip install pymongo.

Script.

Simple script to dump data info mongoDB.

from pymongo import MongoClient

# Get the handle on the MongoClient by default it will connect to localhost:27010
client = MongoClient()

# Get the test database which we have already created.
db = client.test

# Get Collection which we already have.
coll = db.talks

# Insert one row to the test collection.
results = db.talks.insert_one(
{
  "session": {
    "title": "ZUBAIR AHMED - Automate Everything",
    "abstract": "Automate Everything"
  },
  "topics":  ["keynote", "devOps"],
  "room": "Auditorium",
  "timeslot": "Wed 29th, 10:30-11:30",
  "speaker": {
    "name": "ZUBAIR AHMED",
    "bio": "Zubair AHMED is co-founder nothing new.",
    "twitter": "https://twitter.com/ahmedzbyr",
    "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg"
  }
}
)

On MongoDB.

Execute below command.

use test
db.talks.find()

Output.

{ "_id" : ObjectId("5ac4c9c5dd231d89523d41c4"), "session" : { "title" : "12 Years of Spring: An Open Source Journey", "abstract" : "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015." }, "topics" : [ "keynote", "spring" ], "room" : "Auditorium", "timeslot" : "Wed 29th, 09:30-10:30", "speaker" : { "name" : "Juergen Hoeller", "bio" : "Juergen Hoeller is co-founder of the Spring Framework open source project.", "twitter" : "https://twitter.com/springjuergen", "picture" : "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg" } }
{ "_id" : ObjectId("5ac4cbd7db0e4457c867e558"), "room" : "Auditorium", "topics" : [ "keynote", "devOps" ], "session" : { "abstract" : "Automate Everything", "title" : "ZUBAIR AHMED - Automate Everything" }, "speaker" : { "bio" : "Zubair AHMED is co-founder nothing new.", "twitter" : "https://twitter.com/ahmedzbyr", "name" : "ZUBAIR AHMED", "picture" : "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg" }, "timeslot" : "Wed 29th, 10:30-11:30" }

On Noe4j.

Execute below command.

match (n) return n limit 25

Output.

{
  "name": "Zubair AHMED"
}
{
  "_id": "5ac4c9c5dd231d89523d41c4",
  "topics": [
    "keynote",
    "spring"
  ],
  "room": "Auditorium",
  "timeslot": "Wed 29th, 09:30-10:30",
  "_ts": 6540574406794543106
}
{
  "_id": "5ac4c9c5dd231d89523d41c4",
  "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015.",
  "title": "12 Years of Spring: An Open Source Journey",
  "_ts": 6540574406794543106
}
{
  "name": "Juergen Hoeller",
  "bio": "Juergen Hoeller is co-founder of the Spring Framework open source project.",
  "twitter": "https://twitter.com/springjuergen",
  "_id": "5ac4c9c5dd231d89523d41c4",
  "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg",
  "_ts": 6540574406794543106
}
{
  "_id": "5ac4cbd7db0e4457c867e558",
  "topics": [
    "keynote",
    "devOps"
  ],
  "room": "Auditorium",
  "timeslot": "Wed 29th, 10:30-11:30",
  "_ts": 6540576683127209985
}
{
  "abstract": "Automate Everything",
  "_id": "5ac4cbd7db0e4457c867e558",
  "title": "ZUBAIR AHMED - Automate Everything",
  "_ts": 6540576683127209985
}
{
  "name": "ZUBAIR AHMED",
  "bio": "Zubair AHMED is co-founder nothing new.",
  "twitter": "https://twitter.com/ahmedzbyr",
  "_id": "5ac4cbd7db0e4457c867e558",
  "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg",
  "_ts": 6540576683127209985
}

Output

We are done.

Useful Links.

Share on

Twitter Facebook Google+ LinkedIn

MongoDB to Neo4j Using `neo4j_doc_manager`.

AHMED ZBYR

Installing MongoDB.

Download.

Setup.

Testing.

Installing neo4j.

Download.

Setup.

Testing.

Setup `neo4j_doc_manager`.

Setup.

Execute Connector command.

TESTING.

Basic Python script to Insert into MongoDB.

Setup.

Script.

On MongoDB.

On Noe4j.

Useful Links.

Share on

You May Also Enjoy

Datastream Workflow - Cloud SQL (MySQL) to BigQuery via Cloud Auth Proxy

Establishing a Datastream from Cloud SQL (MySQL) to BigQuery

Setting Up HashiCorp Vault for Secret Management

Data Export from Datastore and Firestore

MongoDB to Neo4j Using neo4j_doc_manager.

AHMED ZBYR

Installing MongoDB.

Download.

Setup.

Testing.

Installing neo4j.

Download.

Setup.

Testing.

Setup neo4j_doc_manager.

Setup.

Execute Connector command.

TESTING.

Basic Python script to Insert into MongoDB.

Setup.

Script.

On MongoDB.

On Noe4j.

Useful Links.

Share on

You May Also Enjoy

Datastream Workflow - Cloud SQL (MySQL) to BigQuery via Cloud Auth Proxy

Establishing a Datastream from Cloud SQL (MySQL) to BigQuery

Setting Up HashiCorp Vault for Secret Management

Data Export from Datastore and Firestore

MongoDB to Neo4j Using `neo4j_doc_manager`.

Setup `neo4j_doc_manager`.