Indexes in MongoDB – Sibeesh Passion

Continue With Learning Indexes in MongoDB

SibeeshVenu — Wed, 07 Mar 2018 16:02:02 +0000

[toc]

Introduction

This is the third article of the series “Learn MongoDB with me”, if you haven’t read my previous post on this topic, I strongly recommend you to find it here. This is the continuation of exploring the Indexes on MongoDB, we will be discussing about various MongoDB indexes which we can perform on our data. I hope you will find this post useful. Thanks for reading.

Learn MongoDB with me

You can see all the articles on this series below.

Background

Like I said, it is going to be the third part of the series. I believe that you have enough knowledge about Mongo DB now, If not, please consider reading my previous posts again.

Indexes in MongoDB

Let’s import a new collection, products first.

[
   {
      "id":2,
      "name":"An ice sculpture",
      "price":12.50,
      "tags":[
         "cold",
         "ice"
      ],
      "dimensions":{
         "length":7.0,
         "width":12.0,
         "height":9.5
      },
      "warehouseLocation":{
         "latitude":-78.75,
         "longitude":20.4
      }
   },
   {
      "id":3,
      "name":"A blue mouse",
      "price":25.50,
      "dimensions":{
         "length":3.1,
         "width":1.0,
         "height":1.0
      },
      "warehouseLocation":{
         "latitude":54.4,
         "longitude":-32.7
      }
   },
   {
      "id":4,
      "name":"Keyboard",
      "price":15.50,
      "dimensions":{
         "length":1.1,
         "width":1.0,
         "height":1.0
      },
      "warehouseLocation":{
         "latitude":24.4,
         "longitude":-42.7
      }
   },
   {
      "id":5,
      "name":"Doll",
      "price":10.50,
      "dimensions":{
         "length":5.1,
         "width":1.0,
         "height":7.0
      },
      "warehouseLocation":{
         "latitude":64.4,
         "longitude":-82.7
      }
   },
   {
      "id":6,
      "name":"Wallet",
      "price":5.50,
      "dimensions":{
         "length":1.1,
         "width":1.0,
         "height":1.0
      },
      "warehouseLocation":{
         "latitude":24.4,
         "longitude":-12.7
      }
   }
]

Please be noted that these are just dummy data, and it may sound illogical to you.

C:\Program Files\MongoDB\Server\3.4\bin>mongoimport --db mylearning --collection products --jsonArray --file products.json
2018-03-06T16:48:34.440+0530    connected to: localhost
2018-03-06T16:48:34.607+0530    imported 5 documents

C:\Program Files\MongoDB\Server\3.4\bin>

If you don’t know how the import command works, please read my previous posts where we have seen simple indexes. Now we have the data, let’s go perform Indexes.

Single Key Indexes

In one of my previous post in this series of article, I had mentioned about simple indexes. Here in this article, we are not going to talk about it, instead we will explore on other indexing option what MongoDB has. Sounds good? If yes, let’s continue. let’s go and see Multi key indexes

Multi Key Indexes or Compound Indexes

As the name implies, we are actually going to set indexes with more than one key element. On our products collection, we have some product documents right, what we a user needs to filter the same with the price and warehouse location. Yeah, we need to build a query.

MongoDB Enterprise > db.products.find({
... "price: {$lte: 16},
2018-03-06T17:10:15.005+0530 E QUERY    [thread1] SyntaxError: unterminated string literal @(shell):2:0
MongoDB Enterprise > db.products.find({
... "price": {$lte: 16},
... "warehouseLocation.latitude": {$gte: 60}
... })
{ "_id" : ObjectId("5a9e790a1ae1f955c1a70c4a"), "id" : 5, "name" : "Doll", "price" : 10.5, "dimensions" : { "length" : 5.1, "width" : 1, "height" : 7 }, "warehouseLocation" : { "latitude" : 64.4, "longitude" : -82.7 } }
MongoDB Enterprise >

We have got one entry according to our search, “price”: {$lte: 16} and “warehouseLocation.latitude”: {$gte: 60} that’s cool. Now let’s try to find out the execution status for the same.

Please be noted that we have used $lte and $gte which stands for “less than or equal to” and “greater than or equal to”, remember what I have told you before, “Mongo shell is cool and we can do anything with it”. Let’s find out the examined elements count for our preceding find query now.

db.products.find({ "price": {$lte: 16}, "warehouseLocation.latitude": {$gte: 60} }).explain("executionStats")

And if your query if correct, you will be getting a result as preceding.

"queryPlanner" : {
                "plannerVersion" : 1,
                "namespace" : "mylearning.products",
                "indexFilterSet" : false,
                "parsedQuery" : {
                        "$and" : [
                                {
                                        "price" : {
                                                "$lte" : 16
                                        }
                                },
                                {
                                        "warehouseLocation.latitude" : {
                                                "$gte" : 60
                                        }
                                }
                        ]
                },
                "winningPlan" : {
                        "stage" : "COLLSCAN",
                        "filter" : {
                                "$and" : [
                                        {
                                                "price" : {
                                                        "$lte" : 16
                                                }
                                        },
                                        {
                                                "warehouseLocation.latitude" : {
                                                        "$gte" : 60
                                                }
                                        }
                                ]
                        },
                        "direction" : "forward"
                },
                "rejectedPlans" : [ ]
        },
        "executionStats" : {
                "executionSuccess" : true,
                "nReturned" : 1,
                "executionTimeMillis" : 107,
                "totalKeysExamined" : 0,
                "totalDocsExamined" : 5,
                "executionStages" : {
                        "stage" : "COLLSCAN",
                        "filter" : {
                                "$and" : [
                                        {
                                                "price" : {
                                                        "$lte" : 16
                                                }
                                        },
                                        {
                                                "warehouseLocation.latitude" : {
                                                        "$gte" : 60
                                                }
                                        }
                                ]
                        },
                        "nReturned" : 1,
                        "executionTimeMillisEstimate" : 0,
                        "works" : 7,
                        "advanced" : 1,
                        "needTime" : 5,
                        "needYield" : 0,
                        "saveState" : 0,
                        "restoreState" : 0,
                        "isEOF" : 1,
                        "invalidates" : 0,
                        "direction" : "forward",
                        "docsExamined" : 5
                }
        },
        "serverInfo" : {
                "host" : "PC292716",
                "port" : 27017,
                "version" : "3.4.9",
                "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e"
        },
        "ok" : 1
}

You might have already noticed the value we have for totalDocsExamined , if you haven’t please check now. In my case it is 5, which means the query just examined all the records we have. Ah, that sounds bad right? What if we have millions of records on our collection, how long it is gonna take to fetch the results?

MongoDB Enterprise > db.products.createIndex({price:1, "warehouseLocation.latitude":1})
{
        "createdCollectionAutomatically" : false,
        "numIndexesBefore" : 1,
        "numIndexesAfter" : 2,
        "ok" : 1
}

Run your previous query now, and find out what is the value of docs examined.

MongoDB Enterprise > db.products.find({ "price": {$lte: 16}, "warehouseLocation.latitude": {$gte: 60} }).explain("executionStats")
{
        "queryPlanner" : {
                "plannerVersion" : 1,
                "namespace" : "mylearning.products",
                "indexFilterSet" : false,
                "parsedQuery" : {
                        "$and" : [
                                {
                                        "price" : {
                                                "$lte" : 16
                                        }
                                },
                                {
                                        "warehouseLocation.latitude" : {
                                                "$gte" : 60
                                        }
                                }
                        ]
                },
                "winningPlan" : {
                        "stage" : "FETCH",
                        "inputStage" : {
                                "stage" : "IXSCAN",
                                "keyPattern" : {
                                        "price" : 1,
                                        "warehouseLocation.latitude" : 1
                                },
                                "indexName" : "price_1_warehouseLocation.latitude_1",
                                "isMultiKey" : false,
                                "multiKeyPaths" : {
                                        "price" : [ ],
                                        "warehouseLocation.latitude" : [ ]
                                },
                                "isUnique" : false,
                                "isSparse" : false,
                                "isPartial" : false,
                                "indexVersion" : 2,
                                "direction" : "forward",
                                "indexBounds" : {
                                        "price" : [
                                                "[-inf.0, 16.0]"
                                        ],
                                        "warehouseLocation.latitude" : [
                                                "[60.0, inf.0]"
                                        ]
                                }
                        }
                },
                "rejectedPlans" : [ ]
        },
        "executionStats" : {
                "executionSuccess" : true,
                "nReturned" : 1,
                "executionTimeMillis" : 1089,
                "totalKeysExamined" : 5,
                "totalDocsExamined" : 1,
                "executionStages" : {
                        "stage" : "FETCH",
                        "nReturned" : 1,
                        "executionTimeMillisEstimate" : 310,
                        "works" : 5,
                        "advanced" : 1,
                        "needTime" : 3,
                        "needYield" : 0,
                        "saveState" : 2,
                        "restoreState" : 2,
                        "isEOF" : 1,
                        "invalidates" : 0,
                        "docsExamined" : 1,
                        "alreadyHasObj" : 0,
                        "inputStage" : {
                                "stage" : "IXSCAN",
                                "nReturned" : 1,
                                "executionTimeMillisEstimate" : 270,
                                "works" : 5,
                                "advanced" : 1,
                                "needTime" : 3,
                                "needYield" : 0,
                                "saveState" : 2,
                                "restoreState" : 2,
                                "isEOF" : 1,
                                "invalidates" : 0,
                                "keyPattern" : {
                                        "price" : 1,
                                        "warehouseLocation.latitude" : 1
                                },
                                "indexName" : "price_1_warehouseLocation.latitude_1",
                                "isMultiKey" : false,
                                "multiKeyPaths" : {
                                        "price" : [ ],
                                        "warehouseLocation.latitude" : [ ]
                                },
                                "isUnique" : false,
                                "isSparse" : false,
                                "isPartial" : false,
                                "indexVersion" : 2,
                                "direction" : "forward",
                                "indexBounds" : {
                                        "price" : [
                                                "[-inf.0, 16.0]"
                                        ],
                                        "warehouseLocation.latitude" : [
                                                "[60.0, inf.0]"
                                        ]
                                },
                                "keysExamined" : 5,
                                "seeks" : 4,
                                "dupsTested" : 0,
                                "dupsDropped" : 0,
                                "seenInvalidated" : 0
                        }
                }
        },
        "serverInfo" : {
                "host" : "PC292716",
                "port" : 27017,
                "version" : "3.4.9",
                "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e"
        },
        "ok" : 1
}
MongoDB Enterprise > db.products.find({ "price": {$lte: 16}, "warehouseLocation.latitude": {$gte: 60} })
{ "_id" : ObjectId("5a9e790a1ae1f955c1a70c4a"), "id" : 5, "name" : "Doll", "price" : 10.5, "dimensions" : { "length" : 5.1, "width" : 1, "height" : 7 }, "warehouseLocation" : { "latitude" : 64.4, "longitude" : -82.7 } }
MongoDB Enterprise >

Yeah, we got “docsExamined” : 1 , that’s the way to go. Go create some indexes on your top most queries, you can definitely see some magics over there. You can create up to 64 indexes on a collection in MongoDB, but you may need to create only few, only on your top result queries. What you can do is, whenever you are facing any performance issues on any queries, consider that it needs some tuning and definitely a Index. There are so many other complex Indexes, but widely used Indexes are single key index and compound index.

With that, we are done with this post. I will be posting the continuation part of this series very soon. Till then, bye.

Conclusion

Thanks a lot for reading. Did I miss anything that you may think which is needed? Could you find this post as useful? I hope you liked this article. Please share me your valuable suggestions and feedback.

Your turn. What do you think?

A blog isn’t a blog without comments, but do try to stay on topic. If you have a question unrelated to this post, you’re better off posting it on C# Corner, Code Project, Stack Overflow, Asp.Net Forum instead of commenting here. Tweet or email me a link to your question there and I’ll definitely try to help if I can.

Kindest Regards
Sibeesh Venu

Learn MongoDB With Me

SibeeshVenu — Thu, 22 Feb 2018 05:03:00 +0000

[toc]

Introduction

This is going to be a series of article on MongoDB. Here we are going to do some exercises with MongoDB, we will be talking about Mongo Shell, how can we configure MongoDB?, What are Indexes in MongoDB etc. We all know what an Indexes is, you might have already done that with any relational databases like SQL and MySQL . Have you every done indexing for your MongoDB? If your answer is “no”, no worries, here we are going to see indexes in MongoDB , if it is a “yes” please read this post and correct me if I am wrong anywhere. Let’s begin now.

Prerequisites

I hope you might have got a basic information about MongoDB, if not, I strongly recommend you to read these posts. Now that, you have a basic idea, I am assuming that you have already set up the environment for MongoDB development. Let’s recall what you might have done so far.

Install MongoDB
Set the environment variable for MongoDB
Start the MongoDB services

To set the environment variable for MongoDB, you may have to add a new path to the system variable path with the value as “C:\Program Files\MongoDB\Server\3.4\bin”, please note that the version number will be varied according to your MongoDB version. Once you are done the above steps, you should be able to start both Mongo server and Mongo shell from the command line interface.

Setting up MongoDB using CLI

Now let’s just open our command line interface, and create the data directory for Mongo. We will have to create a directory for the same. Please go along with the below commands.

md \data
md \data\db
mongod

Now let’s open a new CLI and run the command “mongo”, please do not worry about the warnings you are getting, as we are not working in production data, we may not need to secure and optimize it.

Exploring MongoDB

Once you are connected to MongoDB, by default you are connected to test DB. You can check that by running the command MongoDB Enterprise > db

Playing with Mongo Shell

Let’s just use a new database now.

MongoDB Enterprise > use MongoIndex
switched to db MongoIndex
MongoDB Enterprise >

Please be noted that the database MongoIndex doesn’t exist as of now, as we haven’t created it. Still, Mongo just switched our context to the new database. You can see this if you run the command show dbs

The database will be created once we insert any document associated with it. Now we are going to create a new collection called “User”, so once we made the entry to this collection, the database will also be created automatically. Let’s do that.

MongoDB Enterprise > db.users.insert({"name":"Sibees Venu"})
WriteResult({ "nInserted" : 1 })
MongoDB Enterprise >

Now if you run the “show dbs” command again, the database MongoIndex will show up. If you ever need to see the collections you have in the DB, you just need to run the command “show collections”.

MongoDB Enterprise > show collections
users
MongoDB Enterprise >

The MongoDB is very friendly when it comes to data, it doesn’t require any schema to get it started. The learning is so easy, am I right?

The other benefit of MongoDB is its JavaScript interpreted shell, where we can actually type JavaScript code and run. To test it out, let’s create a variable and use it.

MongoDB Enterprise > var name = "Sibeesh Venu"
MongoDB Enterprise > name
Sibeesh Venu
MongoDB Enterprise >

This way, we can interact with the database with a JavaScript program. Now let’s go ahead and creating a collection called “Numbers” and insert 26,000 rows in it. So how are we going to do that? Yes, you are right, we are going to write a for loop, the mongo shell gives that kind of flexibility. Let’s see that in action.

MongoDB Enterprise > for(i=0;i<=26000;i++){
... db.Numbers.insert({
... "number":i
... })
... }
WriteResult({ "nInserted" : 1 })
MongoDB Enterprise >

So we have done that. Note that, we are able to break the commands into multiple lines, this allows to break the complex codes to much readable format in the shell itself. Sounds good?

Even though we have inserted 26,000 rows, it always shows,"nInserted" : 1 this is because it is counting a number of operations, not the individual documents. Let’s see this by checking the count now.

MongoDB Enterprise > db.numbers.count()
0
MongoDB Enterprise > db.Numbers.count()
26001
MongoDB Enterprise >

Please note that it is case sensitive.

Indexes in MongoDB

Now if you need to see any particular record, you can always write the query in the shell as follows.

MongoDB Enterprise > db.Numbers.find(
... {"number":24000}
... )
{ "_id" : ObjectId("5a8d3be2020a0071d115cf62"), "number" : 24000 }
MongoDB Enterprise >

So in the query, we are using the function “find” with the filter “number: 24000”, so that the Mongo can return the record which has the number value as 24000. Now that we have got the output we needed, would you like to see what just happened in the background? To do so, we can use the function “explain()”.

MongoDB Enterprise > db.Numbers.find( {"number":24000} ).explain()
{
        "queryPlanner" : {
                "plannerVersion" : 1,
                "namespace" : "MongoIndex.Numbers",
                "indexFilterSet" : false,
                "parsedQuery" : {
                        "number" : {
                                "$eq" : 24000
                        }
                },
                "winningPlan" : {
                        "stage" : "COLLSCAN",
                        "filter" : {
                                "number" : {
                                        "$eq" : 24000
                                }
                        },
                        "direction" : "forward"
                },
                "rejectedPlans" : [ ]
        },
        "serverInfo" : {
                "host" : "PC292716",
                "port" : 27017,
                "version" : "3.4.9",
                "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e"
        },
        "ok" : 1
}
MongoDB Enterprise >

And, if you need to get more information about the execution, you can pass the parameter “executionStats” to the “explain” function.

The parameter is always case sensitive, you will get an errors as below, if you give it wrong. So please make sure you are passing executionStats not executionstats.

“MongoDB Enterprise > db.Numbers.find( {“number”:24000} ).explain(“executionstats”)
2018-02-21T15:12:34.197+0530 E QUERY [thread1] Error: explain verbosity must be one of {‘queryPlanner’,’executionStats’,’allPlansExecution’} :
parseVerbosity@src/mongo/shell/explainable.js:22:1
constructor@src/mongo/shell/explain_query.js:83:27
DBQuery.prototype.explain@src/mongo/shell/query.js:520:24
@(shell):1:1″

MongoDB Enterprise > db.Numbers.find( {"number":24000} ).explain("executionStats")
{
        "queryPlanner" : {
                "plannerVersion" : 1,
                "namespace" : "MongoIndex.Numbers",
                "indexFilterSet" : false,
                "parsedQuery" : {
                        "number" : {
                                "$eq" : 24000
                        }
                },
                "winningPlan" : {
                        "stage" : "COLLSCAN",
                        "filter" : {
                                "number" : {
                                        "$eq" : 24000
                                }
                        },
                        "direction" : "forward"
                },
                "rejectedPlans" : [ ]
        },
        "executionStats" : {
                "executionSuccess" : true,
                "nReturned" : 1,
                "executionTimeMillis" : 13,
                "totalKeysExamined" : 0,
                "totalDocsExamined" : 26001,
                "executionStages" : {
                        "stage" : "COLLSCAN",
                        "filter" : {
                                "number" : {
                                        "$eq" : 24000
                                }
                        },
                        "nReturned" : 1,
                        "executionTimeMillisEstimate" : 11,
                        "works" : 26003,
                        "advanced" : 1,
                        "needTime" : 26001,
                        "needYield" : 0,
                        "saveState" : 203,
                        "restoreState" : 203,
                        "isEOF" : 1,
                        "invalidates" : 0,
                        "direction" : "forward",
                        "docsExamined" : 26001
                }
        },
        "serverInfo" : {
                "host" : "PC292716",
                "port" : 27017,
                "version" : "3.4.9",
                "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e"
        },
        "ok" : 1
}
MongoDB Enterprise >

Now you can see more information on the execution of how much time it took for the execution and how many docs it is examined etc. If you have noticed, it has examined all the 26001 records and took 13 milliseconds. That’s just a case, that we had only less number of records in the table, what if, we have millions of records in it? And examining all the records would be a bad idea, am I right? So what do we do at that time? What would be a permanent solution for this? This is where the importance of Indexes comes into action.

Let’s create an Index for the number that we are going to search.

MongoDB Enterprise > db.Numbers.createIndex({number:1})
{
        "createdCollectionAutomatically" : false,
        "numIndexesBefore" : 1,
        "numIndexesAfter" : 2,
        "ok" : 1
}
MongoDB Enterprise >

Here the number is a special variable, not a string. As you can see, we had created the index. You can see that the property value of createdCollectionAutomatically is false, as the collection had already created and it didn’t have to create it again.

Let’s run our find query again.

MongoDB Enterprise > db.Numbers.find( {"number":24000} ).explain("executionStats")
{
        "queryPlanner" : {
                "plannerVersion" : 1,
                "namespace" : "MongoIndex.Numbers",
                "indexFilterSet" : false,
                "parsedQuery" : {
                        "number" : {
                                "$eq" : 24000
                        }
                },
                "winningPlan" : {
                        "stage" : "FETCH",
                        "inputStage" : {
                                "stage" : "IXSCAN",
                                "keyPattern" : {
                                        "number" : 1
                                },
                                "indexName" : "number_1",
                                "isMultiKey" : false,
                                "multiKeyPaths" : {
                                        "number" : [ ]
                                },
                                "isUnique" : false,
                                "isSparse" : false,
                                "isPartial" : false,
                                "indexVersion" : 2,
                                "direction" : "forward",
                                "indexBounds" : {
                                        "number" : [
                                                "[24000.0, 24000.0]"
                                        ]
                                }
                        }
                },
                "rejectedPlans" : [
                        {
                                "stage" : "FETCH",
                                "inputStage" : {
                                        "stage" : "IXSCAN",
                                        "keyPattern" : {
                                                "number" : 24000
                                        },
                                        "indexName" : "number_24000",
                                        "isMultiKey" : false,
                                        "multiKeyPaths" : {
                                                "number" : [ ]
                                        },
                                        "isUnique" : false,
                                        "isSparse" : false,
                                        "isPartial" : false,
                                        "indexVersion" : 2,
                                        "direction" : "forward",
                                        "indexBounds" : {
                                                "number" : [
                                                        "[24000.0, 24000.0]"
                                                ]
                                        }
                                }
                        }
                ]
        },
        "executionStats" : {
                "executionSuccess" : true,
                "nReturned" : 1,
                "executionTimeMillis" : 36,
                "totalKeysExamined" : 1,
                "totalDocsExamined" : 1,
                "executionStages" : {
                        "stage" : "FETCH",
                        "nReturned" : 1,
                        "executionTimeMillisEstimate" : 0,
                        "works" : 3,
                        "advanced" : 1,
                        "needTime" : 0,
                        "needYield" : 0,
                        "saveState" : 1,
                        "restoreState" : 1,
                        "isEOF" : 1,
                        "invalidates" : 0,
                        "docsExamined" : 1,
                        "alreadyHasObj" : 0,
                        "inputStage" : {
                                "stage" : "IXSCAN",
                                "nReturned" : 1,
                                "executionTimeMillisEstimate" : 0,
                                "works" : 2,
                                "advanced" : 1,
                                "needTime" : 0,
                                "needYield" : 0,
                                "saveState" : 1,
                                "restoreState" : 1,
                                "isEOF" : 1,
                                "invalidates" : 0,
                                "keyPattern" : {
                                        "number" : 1
                                },
                                "indexName" : "number_1",
                                "isMultiKey" : false,
                                "multiKeyPaths" : {
                                        "number" : [ ]
                                },
                                "isUnique" : false,
                                "isSparse" : false,
                                "isPartial" : false,
                                "indexVersion" : 2,
                                "direction" : "forward",
                                "indexBounds" : {
                                        "number" : [
                                                "[24000.0, 24000.0]"
                                        ]
                                },
                                "keysExamined" : 1,
                                "seeks" : 1,
                                "dupsTested" : 0,
                                "dupsDropped" : 0,
                                "seenInvalidated" : 0
                        }
                }
        },
        "serverInfo" : {
                "host" : "PC292716",
                "port" : 27017,
                "version" : "3.4.9",
                "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e"
        },
        "ok" : 1
}
MongoDB Enterprise >

As we had given the index on what exactly we are going to search, it just examined only that document when we run the query, that’s why the value of the property totalDocsExamined is 1. Indexing will not have many impacts on the database which has few records in it, but it has a massive effect on very large data sets which has millions of records in it. Using this simple Indexes can reduce the execution time to almost nothing.

With that, we are done with this post. I will be posting the continuation part of this series very soon. Till then, bye.

Conclusion

Your turn. What do you think?

Kindest Regards
Sibeesh Venu