Database – Sibeesh Passion

Foolproof Tips for Reading and Shrinking Transaction logs in SQL Server

Andrew Jackson — Thu, 07 Jun 2018 13:35:37 +0000

[toc]

Query-1: SQL Server log file growing unexpectedly?

“We are wondering why transaction logs keep on increasing?? It become so gigantic, the transaction log size get increase from 135GB to 350GB and that’s so annoying. My drive containing data store is losing space. I am using Full Recovery Model and the logs are backed up hourly. Please help me out why this is happening and what should be done to stop growing it too big?”

SOLVED: SQL Server Log File Getting Too Big

SQL Server Transaction logs records all activities of transaction logs. Each database in SQL Server instance consists of logs that records all the modification done in the databases. It is crucial component of the databases. Any modification, deletion can be tracked down in transaction logs.

There are various reasons of transaction log file getting expanded. The possible reasons are listed below:

Large transaction log file
Transaction taking too long to complete
Transaction operation fail & may start to rollback
Issue in performance
Blocking issue
Database participating in AlwaysOn availability group

What to do for stopping growing log file too big

Change the type of Recovery Model to Simple Recovery Model only if you are fine with not able to restore logs hence, failed point in time recovery. It is because if you are truncating your logs, you are breaking Transaction log LSN & if any disaster occur, you will not able to restore T-logs. Hence if you are fine with this situation, then change your Recovery model to Simple. This will not allow any extra growth of log file.
Take T-log backup every hour in Full recovery Model, your transaction log will keep on growing until you take backup and truncate of it. While taking hourly T-log backup, the logs will grow until one hour, but after this it would truncate all commited transaction.

Sometimes it might happen that you are in Simple Recovery Model, but your logs are still growing!!! Reason being is long running transaction. If your transaction taking long time when deleting thousand records in one delete statement, then logs will not be able to truncate until delete operation is done.

You can save yourself by maintaining proper size of log file or monitoring the usage of transaction.

By doing this, you can cut down the size of Transaction log.

Query-2 How To Shrink Transaction logs File in SQL Server?

“My log file size is growing continuously. Why is this happening?? I am lacking my disk space. I want to shrink my Transaction log file. Can anyone tell me how to shrink transaction logs in SQL Server. What will be the after effects if I shrink the log file.

SOLVED: Reduce SQL Server Database Transaction Log File

Relax!! There are several reasons for transaction log file growth. Long Running transaction, lack of taking log backup are some of the reasons which result in transaction log growth. Databases running for business purposes or production requires appropriate recovery model to be chosen for log management.

Log truncation frees up the log file space so that logs can reuse it. Truncation occur automatically if the database is backed up in Simple Recovery Model or after log backup when database is in Full Recovery Model.

Here I will resolve your shrink issue by two methods: By using SQL Server Management Studio and by using T-SQL.

Shrink By SQL Server Management Studio:

Right Click on your Database–>Tasks–>Shrink—>Files.
You will get the Shrink File Window, Change the File Type to Log.
There are three different actions under Shrink Action option. Either release the unused space, or shrink file to reorganize pages before releasing unused space, or empty file by migrating the data to other files in the same filegroup. Choose it according to your need.

Shrink By T-SQL:

You can simply shrink transaction log file by running DBCC SHRINKFILE after taking log back. If you are using Simple Recovery Model, run below code:

DBCC SHRINKFILE (TestDB_log, 1)

If your database is in Full Recovery Model, first of all, set it to Simple Recovery Model, Then run DBCC SHRINKFILE and then set it to full.

Caution: Setting database again in Full Recovery Model leads to loss of data in the Log. So, if you dont care about losing your log data, then definitely go for it!

Tips when you plan to shrink database or file:

Shrink operation found to be most effective after delete or drop table operation as it create lot of unused space.
If you are repeatedly shrinking your database and you notice that the size of database grows again. Alert!! Shrinking database is a wasted operation, as most database require free space for regular day-to-day operation. This indicates that the space that was shrunk was a part of regular operation.
You should not shrink your database or data file after rebuilding indexes. As shrinking increases fragmentation to extreme level.
Do not ON Auto_Shrink option. (Unless you have very urgent requirement)

Is Shrinking database Bad?

Well, Yes.

After shrinking operation,you will definitely be able to reduce the size of the database. But Wait!! Check your Fragmentation level first. It is found to be way too much!!!

Shrink operation increases the value of the fragmentation way too much. And higher fragmentation costs you poor performance of the database.

While if you are thinking that you can reduce fragmentation by using rebuild index, let me alert you, this will definitely reduce your fragmentation level. But Let me check the size of the database The database size increases way higher than the original.

Query-3 How to check Deleted Records in SQL Server?

“I am working in an organization and my manager assigned me work to check deleted database records of my database. How can I check my deleted records in SQL Server? Please Help me out from this situation.”

SOLVED: Ways to Check Deleted Records in SQL Server

You can track down your deleted database records by reading Transaction logs. And you can read it by using fn_dblog function. You can track down any DML as well as DDL activity like insert, delete , update, create operation by checking your transaction logs.

Fn_dblog function enables users to read all the database activity happening in SQL database. The function require begin and end LSN number for a transaction. You can easily trackdown the information of a person who did changes to your database. As LSN number are not in human readable form, SQL Server provide fn_dblog function for easy reading of transaction logs.

You can simply check your database activity by implementing below code:

USE ReadingDBLog; GO select [Current LSN], [Operation], [Transaction Name], [Transaction ID], [Transaction SID], [SPID], [Begin Time] FROM fn_dblog(null,null)

How to Read Transaction Log Quickly

One quick way to read SQL Server database activity is to use SysTools SQL Log Viewer. It read & analyse operation like insert, update, delete. The software previews all .ldf activities like transaction name, table name, query etc. You have option to choose the database from different mode ie Online & Offline mode. You will able to fetch it from live databases. If your database is in Simple Recovery Mode, the tool is capable to recover deleted database records. You will get three different option for exporting. You can export it as in SQL Server Database , as CSV or as Compatible Script.

Conclusion

The blog covers various users queries about transaction logs along with its possible solution. I have discussed why transaction logs keep on increasing, how to track down your transaction details of your database and how to shrink transaction logs. I have also discussed some quick tips when you shrink your database.

How to Copy Table Schema and Data from One Database to another Database in SQL Server

Andrew Jackson — Thu, 26 Apr 2018 12:41:04 +0000

[toc]

Problem:

A few days ago, while migrating data from one database to another, I encountered an interesting problem which user faced a lot.

“I have two databases, I have to copy tables data from one database to another. My query is how can I copy table schema from one database to another with its schema and data”

In this blog, we will learn the easiest solution on how you can solve that query.

Solution

There are plenty of methods on how you can copy table schema and data from one database to another like Generate script wizard, Import/Export Wizard etc. But these methods are quite lengthy and requires users time.

The easiest solution to copy schema’s table from one database to another is to use SysTools SQL Server Database Migration Tool. You will able to migrate your tables schema by performing few steps.

How to Copy Table Schema and Data from One Database to Another?

Launch SysTools SQL Server Database Migration Tool.

Click Open and load your database MDF file.

Choose the Scan Modes accordingly and select the SQL Server version of your .mdf file.

Select the database table from the left side, which you want to migrate to another database.

Click on Export for migration process.

You will get two options for exporting. One is SQL Server Database and other is SQL Server Compatible Script. Choose it according to your need.

After filling the Server credentials, Select the destination database as ‘Export to Existing database’

Software will fetch the databases and let you select the database on which you want to move the selected table.

You will get two options to export. Select ‘With Schema & Data’. This option will copy schema & data of your selected table from one database and export it to another and click Export.

You will get popup of Export complete.

Conclusion

The blog covers the easiest workaround of mostly faced users query on how to copy schema & data from one database to another. You can also use Import/Export Wizard, Generate script wizard etc for object migration using SQL Server. But these methods has some limitations. In Import/Export wizard, the indexes and the key will not get transferred, you need to generate scripts for that. The export wizard fails if foreign keys connecting the tables together is not in a correct order. ‘Generate script’ method is used to generate a single script for tables schema, data, indexes and keys. But again, the limitation is this method doesn’t generate tables creation script in correct order.

So one can opt for SysTools SQL Database Migration Tool for fast migration of tables schema from one database to another.

Continue With Learning Indexes in MongoDB

SibeeshVenu — Wed, 07 Mar 2018 16:02:02 +0000

[toc]

Introduction

This is the third article of the series “Learn MongoDB with me”, if you haven’t read my previous post on this topic, I strongly recommend you to find it here. This is the continuation of exploring the Indexes on MongoDB, we will be discussing about various MongoDB indexes which we can perform on our data. I hope you will find this post useful. Thanks for reading.

Learn MongoDB with me

You can see all the articles on this series below.

Background

Like I said, it is going to be the third part of the series. I believe that you have enough knowledge about Mongo DB now, If not, please consider reading my previous posts again.

Indexes in MongoDB

Let’s import a new collection, products first.

[
   {
      "id":2,
      "name":"An ice sculpture",
      "price":12.50,
      "tags":[
         "cold",
         "ice"
      ],
      "dimensions":{
         "length":7.0,
         "width":12.0,
         "height":9.5
      },
      "warehouseLocation":{
         "latitude":-78.75,
         "longitude":20.4
      }
   },
   {
      "id":3,
      "name":"A blue mouse",
      "price":25.50,
      "dimensions":{
         "length":3.1,
         "width":1.0,
         "height":1.0
      },
      "warehouseLocation":{
         "latitude":54.4,
         "longitude":-32.7
      }
   },
   {
      "id":4,
      "name":"Keyboard",
      "price":15.50,
      "dimensions":{
         "length":1.1,
         "width":1.0,
         "height":1.0
      },
      "warehouseLocation":{
         "latitude":24.4,
         "longitude":-42.7
      }
   },
   {
      "id":5,
      "name":"Doll",
      "price":10.50,
      "dimensions":{
         "length":5.1,
         "width":1.0,
         "height":7.0
      },
      "warehouseLocation":{
         "latitude":64.4,
         "longitude":-82.7
      }
   },
   {
      "id":6,
      "name":"Wallet",
      "price":5.50,
      "dimensions":{
         "length":1.1,
         "width":1.0,
         "height":1.0
      },
      "warehouseLocation":{
         "latitude":24.4,
         "longitude":-12.7
      }
   }
]

Please be noted that these are just dummy data, and it may sound illogical to you.

C:\Program Files\MongoDB\Server\3.4\bin>mongoimport --db mylearning --collection products --jsonArray --file products.json
2018-03-06T16:48:34.440+0530    connected to: localhost
2018-03-06T16:48:34.607+0530    imported 5 documents

C:\Program Files\MongoDB\Server\3.4\bin>

If you don’t know how the import command works, please read my previous posts where we have seen simple indexes. Now we have the data, let’s go perform Indexes.

Single Key Indexes

In one of my previous post in this series of article, I had mentioned about simple indexes. Here in this article, we are not going to talk about it, instead we will explore on other indexing option what MongoDB has. Sounds good? If yes, let’s continue. let’s go and see Multi key indexes

Multi Key Indexes or Compound Indexes

As the name implies, we are actually going to set indexes with more than one key element. On our products collection, we have some product documents right, what we a user needs to filter the same with the price and warehouse location. Yeah, we need to build a query.

MongoDB Enterprise > db.products.find({
... "price: {$lte: 16},
2018-03-06T17:10:15.005+0530 E QUERY    [thread1] SyntaxError: unterminated string literal @(shell):2:0
MongoDB Enterprise > db.products.find({
... "price": {$lte: 16},
... "warehouseLocation.latitude": {$gte: 60}
... })
{ "_id" : ObjectId("5a9e790a1ae1f955c1a70c4a"), "id" : 5, "name" : "Doll", "price" : 10.5, "dimensions" : { "length" : 5.1, "width" : 1, "height" : 7 }, "warehouseLocation" : { "latitude" : 64.4, "longitude" : -82.7 } }
MongoDB Enterprise >

We have got one entry according to our search, “price”: {$lte: 16} and “warehouseLocation.latitude”: {$gte: 60} that’s cool. Now let’s try to find out the execution status for the same.

Please be noted that we have used $lte and $gte which stands for “less than or equal to” and “greater than or equal to”, remember what I have told you before, “Mongo shell is cool and we can do anything with it”. Let’s find out the examined elements count for our preceding find query now.

db.products.find({ "price": {$lte: 16}, "warehouseLocation.latitude": {$gte: 60} }).explain("executionStats")

And if your query if correct, you will be getting a result as preceding.

"queryPlanner" : {
                "plannerVersion" : 1,
                "namespace" : "mylearning.products",
                "indexFilterSet" : false,
                "parsedQuery" : {
                        "$and" : [
                                {
                                        "price" : {
                                                "$lte" : 16
                                        }
                                },
                                {
                                        "warehouseLocation.latitude" : {
                                                "$gte" : 60
                                        }
                                }
                        ]
                },
                "winningPlan" : {
                        "stage" : "COLLSCAN",
                        "filter" : {
                                "$and" : [
                                        {
                                                "price" : {
                                                        "$lte" : 16
                                                }
                                        },
                                        {
                                                "warehouseLocation.latitude" : {
                                                        "$gte" : 60
                                                }
                                        }
                                ]
                        },
                        "direction" : "forward"
                },
                "rejectedPlans" : [ ]
        },
        "executionStats" : {
                "executionSuccess" : true,
                "nReturned" : 1,
                "executionTimeMillis" : 107,
                "totalKeysExamined" : 0,
                "totalDocsExamined" : 5,
                "executionStages" : {
                        "stage" : "COLLSCAN",
                        "filter" : {
                                "$and" : [
                                        {
                                                "price" : {
                                                        "$lte" : 16
                                                }
                                        },
                                        {
                                                "warehouseLocation.latitude" : {
                                                        "$gte" : 60
                                                }
                                        }
                                ]
                        },
                        "nReturned" : 1,
                        "executionTimeMillisEstimate" : 0,
                        "works" : 7,
                        "advanced" : 1,
                        "needTime" : 5,
                        "needYield" : 0,
                        "saveState" : 0,
                        "restoreState" : 0,
                        "isEOF" : 1,
                        "invalidates" : 0,
                        "direction" : "forward",
                        "docsExamined" : 5
                }
        },
        "serverInfo" : {
                "host" : "PC292716",
                "port" : 27017,
                "version" : "3.4.9",
                "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e"
        },
        "ok" : 1
}

You might have already noticed the value we have for totalDocsExamined , if you haven’t please check now. In my case it is 5, which means the query just examined all the records we have. Ah, that sounds bad right? What if we have millions of records on our collection, how long it is gonna take to fetch the results?

MongoDB Enterprise > db.products.createIndex({price:1, "warehouseLocation.latitude":1})
{
        "createdCollectionAutomatically" : false,
        "numIndexesBefore" : 1,
        "numIndexesAfter" : 2,
        "ok" : 1
}

Run your previous query now, and find out what is the value of docs examined.

MongoDB Enterprise > db.products.find({ "price": {$lte: 16}, "warehouseLocation.latitude": {$gte: 60} }).explain("executionStats")
{
        "queryPlanner" : {
                "plannerVersion" : 1,
                "namespace" : "mylearning.products",
                "indexFilterSet" : false,
                "parsedQuery" : {
                        "$and" : [
                                {
                                        "price" : {
                                                "$lte" : 16
                                        }
                                },
                                {
                                        "warehouseLocation.latitude" : {
                                                "$gte" : 60
                                        }
                                }
                        ]
                },
                "winningPlan" : {
                        "stage" : "FETCH",
                        "inputStage" : {
                                "stage" : "IXSCAN",
                                "keyPattern" : {
                                        "price" : 1,
                                        "warehouseLocation.latitude" : 1
                                },
                                "indexName" : "price_1_warehouseLocation.latitude_1",
                                "isMultiKey" : false,
                                "multiKeyPaths" : {
                                        "price" : [ ],
                                        "warehouseLocation.latitude" : [ ]
                                },
                                "isUnique" : false,
                                "isSparse" : false,
                                "isPartial" : false,
                                "indexVersion" : 2,
                                "direction" : "forward",
                                "indexBounds" : {
                                        "price" : [
                                                "[-inf.0, 16.0]"
                                        ],
                                        "warehouseLocation.latitude" : [
                                                "[60.0, inf.0]"
                                        ]
                                }
                        }
                },
                "rejectedPlans" : [ ]
        },
        "executionStats" : {
                "executionSuccess" : true,
                "nReturned" : 1,
                "executionTimeMillis" : 1089,
                "totalKeysExamined" : 5,
                "totalDocsExamined" : 1,
                "executionStages" : {
                        "stage" : "FETCH",
                        "nReturned" : 1,
                        "executionTimeMillisEstimate" : 310,
                        "works" : 5,
                        "advanced" : 1,
                        "needTime" : 3,
                        "needYield" : 0,
                        "saveState" : 2,
                        "restoreState" : 2,
                        "isEOF" : 1,
                        "invalidates" : 0,
                        "docsExamined" : 1,
                        "alreadyHasObj" : 0,
                        "inputStage" : {
                                "stage" : "IXSCAN",
                                "nReturned" : 1,
                                "executionTimeMillisEstimate" : 270,
                                "works" : 5,
                                "advanced" : 1,
                                "needTime" : 3,
                                "needYield" : 0,
                                "saveState" : 2,
                                "restoreState" : 2,
                                "isEOF" : 1,
                                "invalidates" : 0,
                                "keyPattern" : {
                                        "price" : 1,
                                        "warehouseLocation.latitude" : 1
                                },
                                "indexName" : "price_1_warehouseLocation.latitude_1",
                                "isMultiKey" : false,
                                "multiKeyPaths" : {
                                        "price" : [ ],
                                        "warehouseLocation.latitude" : [ ]
                                },
                                "isUnique" : false,
                                "isSparse" : false,
                                "isPartial" : false,
                                "indexVersion" : 2,
                                "direction" : "forward",
                                "indexBounds" : {
                                        "price" : [
                                                "[-inf.0, 16.0]"
                                        ],
                                        "warehouseLocation.latitude" : [
                                                "[60.0, inf.0]"
                                        ]
                                },
                                "keysExamined" : 5,
                                "seeks" : 4,
                                "dupsTested" : 0,
                                "dupsDropped" : 0,
                                "seenInvalidated" : 0
                        }
                }
        },
        "serverInfo" : {
                "host" : "PC292716",
                "port" : 27017,
                "version" : "3.4.9",
                "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e"
        },
        "ok" : 1
}
MongoDB Enterprise > db.products.find({ "price": {$lte: 16}, "warehouseLocation.latitude": {$gte: 60} })
{ "_id" : ObjectId("5a9e790a1ae1f955c1a70c4a"), "id" : 5, "name" : "Doll", "price" : 10.5, "dimensions" : { "length" : 5.1, "width" : 1, "height" : 7 }, "warehouseLocation" : { "latitude" : 64.4, "longitude" : -82.7 } }
MongoDB Enterprise >

Yeah, we got “docsExamined” : 1 , that’s the way to go. Go create some indexes on your top most queries, you can definitely see some magics over there. You can create up to 64 indexes on a collection in MongoDB, but you may need to create only few, only on your top result queries. What you can do is, whenever you are facing any performance issues on any queries, consider that it needs some tuning and definitely a Index. There are so many other complex Indexes, but widely used Indexes are single key index and compound index.

With that, we are done with this post. I will be posting the continuation part of this series very soon. Till then, bye.

Conclusion

Thanks a lot for reading. Did I miss anything that you may think which is needed? Could you find this post as useful? I hope you liked this article. Please share me your valuable suggestions and feedback.

Your turn. What do you think?

A blog isn’t a blog without comments, but do try to stay on topic. If you have a question unrelated to this post, you’re better off posting it on C# Corner, Code Project, Stack Overflow, Asp.Net Forum instead of commenting here. Tweet or email me a link to your question there and I’ll definitely try to help if I can.

Kindest Regards
Sibeesh Venu

Learn MongoDB With Me – Part 2

SibeeshVenu — Fri, 02 Mar 2018 08:18:23 +0000

[toc]

Introduction

This is the second article of the series “Learn MongoDB with me”, if you haven’t read my previous post on this topic, I strongly recommend you to find it here. This is the continuation of exploring the Mongo shells, we will be performing some commands on the Mongo shells. For easy reference I will try to add screenshots for each steps I am following. I hope it will help you come along with me. Thanks for reading.

Background

Like I said, it is going to be the second part of the series. I believe that you have enough knowledge about Mongo DB and how to set up it? How a Mongo shells can be used? If you are not able to answer these question yourself, please consider reading my previous posts again.

Mongo shells, the perfect CLI

We can do anything in the Mongo shell, to make the statement clear. I am going to perform some CRUD operations (Create, Read, Update, Delete) within Mongo shells. To do so, we can use Mongo import commands. The Mongo import commands can do the work for you, even if the data is in .tsv, .csv, .json etc. Let’s see those in action.

As a first step, let us see some documentation.

C:\Program Files\MongoDB\Server\3.4\bin>mongoimport --help

The above command will give you all the options available to get started.

Usage:
  mongoimport  

Import CSV, TSV or JSON data into MongoDB. If no file is provided, mongoimport reads from stdin.

See http://docs.mongodb.org/manual/reference/program/mongoimport/ for more information.

general options:
      /help                                       print usage
      /version                                    print the tool version and
                                                  exit

verbosity options:
  /v, /verbose:                            more detailed log output
                                                  (include multiple times for
                                                  more verbosity, e.g. -vvvvv,
                                                  or specify a numeric value,
                                                  e.g. --verbose=N)
      /quiet                                      hide all log output

connection options:
  /h, /host:                            mongodb host to connect to
                                                  (setname/host1,host2 for
                                                  replica sets)
      /port:                                server port (can also use
                                                  --host hostname:port)

kerberos options:
      /gssapiServiceName:           service name to use when
                                                  authenticating using
                                                  GSSAPI/Kerberos ('mongodb' by
                                                  default)
      /gssapiHostName:                 hostname to use when
                                                  authenticating using
                                                  GSSAPI/Kerberos (remote
                                                  server's address by default)

ssl options:
      /ssl                                        connect to a mongod or mongos
                                                  that has ssl enabled
      /sslCAFile:                       the .pem file containing the
                                                  root certificate chain from
                                                  the certificate authority
      /sslPEMKeyFile:                   the .pem file containing the
                                                  certificate and key
      /sslPEMKeyPassword:               the password to decrypt the
                                                  sslPEMKeyFile, if necessary
      /sslCRLFile:                      the .pem file containing the
                                                  certificate revocation list
      /sslAllowInvalidCertificates                bypass the validation for
                                                  server certificates
      /sslAllowInvalidHostnames                   bypass the validation for
                                                  server name
      /sslFIPSMode                                use FIPS mode of the
                                                  installed openssl library

authentication options:
  /u, /username:                        username for authentication
  /p, /password:                        password for authentication
      /authenticationDatabase:     database that holds the
                                                  user's credentials
      /authenticationMechanism:        authentication mechanism to
                                                  use

namespace options:
  /d, /db:                         database to use
  /c, /collection:               collection to use

uri options:
      /uri:mongodb-uri                            mongodb uri connection string

input options:
  /f, /fields:[,]*                  comma separated list of
                                                  fields, e.g. -f name,age
      /fieldFile:                       file with field names - 1 per
                                                  line
      /file:                            file to import from; if not
                                                  specified, stdin is used
      /headerline                                 use first line in input
                                                  source as the field list (CSV
                                                  and TSV only)
      /jsonArray                                  treat input source as a JSON
                                                  array
      /parseGrace:                         controls behavior when type
                                                  coercion fails - one of:
                                                  autoCast, skipField, skipRow,
                                                  stop (defaults to 'stop')
                                                  (default: stop)
      /type:                                input format to import: json,
                                                  csv, or tsv (defaults to
                                                  'json') (default: json)
      /columnsHaveTypes                           indicated that the field list
                                                  (from --fields, --fieldsFile,
                                                  or --headerline) specifies
                                                  types; They must be in the
                                                  form of
                                                  '.()'.
                                                  The type can be one of: auto,
                                                  binary, bool, date, date_go,
                                                  date_ms, date_oracle, double,
                                                  int32, int64, string. For
                                                  each of the date types, the
                                                  argument is a datetime layout
                                                  string. For the binary type,
                                                  the argument can be one of:
                                                  base32, base64, hex. All
                                                  other types take an empty
                                                  argument. Only valid for CSV
                                                  and TSV imports. e.g.
                                                  zipcode.string(),
                                                  thumbnail.binary(base64)

ingest options:
      /drop                                       drop collection before
                                                  inserting documents
      /ignoreBlanks                               ignore fields with empty
                                                  values in CSV and TSV
      /maintainInsertionOrder                     insert documents in the order
                                                  of their appearance in the
                                                  input source
  /j, /numInsertionWorkers:               number of insert operations
                                                  to run concurrently (defaults
                                                  to 1) (default: 1)
      /stopOnError                                stop importing at first
                                                  insert/upsert error
      /mode:[insert|upsert|merge]                 insert: insert only. upsert:
                                                  insert or replace existing
                                                  documents. merge: insert or
                                                  modify existing documents.
                                                  defaults to insert
      /upsertFields:[,]*            comma-separated fields for
                                                  the query part when --mode is
                                                  set to upsert or merge
      /writeConcern:     write concern options e.g.
                                                  --writeConcern majority,
                                                  --writeConcern '{w: 3,
                                                  wtimeout: 500, fsync: true,
                                                  j: true}'
      /bypassDocumentValidation                   bypass document validation

Insert data to MondoDB using Mongo shell

Now let’s say I have a following JSON data, and we are going to insert the same to our db collection.

[ 
{
 "color": "black",
 "category": "hue",
 "type": "primary",
 "code": {
 "rgba": [255,255,255,1],
 "hex": "#000"
 }
 },
 {
 "color": "white",
 "category": "value",
 "code": {
 "rgba": [0,0,0,1],
 "hex": "#FFF"
 }
 },
 {
 "color": "red",
 "category": "hue",
 "type": "primary",
 "code": {
 "rgba": [255,0,0,1],
 "hex": "#FF0"
 }
 },
 {
 "color": "blue",
 "category": "hue",
 "type": "primary",
 "code": {
 "rgba": [0,0,255,1],
 "hex": "#00F"
 }
 },
 {
 "color": "yellow",
 "category": "hue",
 "type": "primary",
 "code": {
 "rgba": [255,255,0,1],
 "hex": "#FF0"
 }
 },
 {
 "color": "green",
 "category": "hue",
 "type": "secondary",
 "code": {
 "rgba": [0,255,0,1],
 "hex": "#0F0"
 }
 }
]

To do so, we need to use the following command.

C:\Program Files\MongoDB\Server\3.4\bin>mongoimport --db mylearning --collection colors --jsonArray --file colors.json

Here, as you can see, we are providing the db name, collection, name, what is the data type of the file and finally the file name.

If you ever get the error as “Failed: open colors.json: The system cannot find the file specified.”, please make sure that the document is in the server folder, in my case it is “C:\Program Files\MongoDB\Server\3.4\bin”. If everything is fine, you will be able to see an output as preceding.

C:\Program Files\MongoDB\Server\3.4\bin>mongoimport --db mylearning --collection colors --jsonArray --file colors.json
2018-03-01T16:35:35.470+0530    connected to: localhost
2018-03-01T16:35:36.012+0530    imported 6 documents

C:\Program Files\MongoDB\Server\3.4\bin>

Reading the data from a collection in MongoDB

Now that, we have colors collection, let’s go and check whether the database has the data we are expecting.

C:\Program Files\MongoDB\Server\3.4\bin>mongo
MongoDB shell version v3.4.9
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.9
Server has startup warnings:
2018-03-01T16:24:43.793+0530 I CONTROL  [initandlisten]
2018-03-01T16:24:43.793+0530 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-03-01T16:24:43.793+0530 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-03-01T16:24:43.793+0530 I CONTROL  [initandlisten]
MongoDB Enterprise > use mylearning
switched to db mylearning
MongoDB Enterprise > show collections
chats
colors
messages
MongoDB Enterprise > db.colors.count
function (query, options) {
    query = this.find(query);

    // Apply options and return the result of the find
    return QueryHelpers._applyCountOptions(query, options).count(true);
}
MongoDB Enterprise > db.colors.count()
6
MongoDB Enterprise >

When you use count, make sure you are treating it as a function as count(). We have the count as 6, and that’s what we are expecting. Am I right? Don’t you think, that we should go fetch some data from that collection, yeah with some filter?

MongoDB Enterprise > db.colors.find({"type": "primary"})
{ "_id" : ObjectId("5a97de7f2fcdf731d255a19d"), "color" : "black", "category" : "hue", "type" : "primary", "code" : { "rgba" : [ 255, 255, 255, 1 ], "hex" : "#000" } }
{ "_id" : ObjectId("5a97de7f2fcdf731d255a19e"), "color" : "yellow", "category" : "hue", "type" : "primary", "code" : { "rgba" : [ 255, 255, 0, 1 ], "hex" : "#FF0" } }
{ "_id" : ObjectId("5a97de7f2fcdf731d255a1a1"), "color" : "red", "category" : "hue", "type" : "primary", "code" : { "rgba" : [ 255, 0, 0, 1 ], "hex" : "#FF0" } }
{ "_id" : ObjectId("5a97de7f2fcdf731d255a1a2"), "color" : "blue", "category" : "hue", "type" : "primary", "code" : { "rgba" : [ 0, 0, 255, 1 ], "hex" : "#00F" } }
MongoDB Enterprise >

We have successfully imported that data, and we have fetched the colors with type as primary. I think, there should be a custom color, which has the type as primary. Can we do that now?

MongoDB Enterprise > db.colors.insert({
... "color": "custome",
...  "category": "hue",
...  "type": "primary",
...  "code": {
...  "rgba": [255,1,255,1],
...  "hex": "#FF1"
...  }
... }
... )
WriteResult({ "nInserted" : 1 })
MongoDB Enterprise >

Now if you run our previous query, you can see an output as preceding.

MongoDB Enterprise > db.colors.find({"type": "primary"})
{ "_id" : ObjectId("5a97de7f2fcdf731d255a19d"), "color" : "black", "category" : "hue", "type" : "primary", "code" : { "rgba" : [ 255, 255, 255, 1 ], "hex" : "#000" } }
{ "_id" : ObjectId("5a97de7f2fcdf731d255a19e"), "color" : "yellow", "category" : "hue", "type" : "primary", "code" : { "rgba" : [ 255, 255, 0, 1 ], "hex" : "#FF0" } }
{ "_id" : ObjectId("5a97de7f2fcdf731d255a1a1"), "color" : "red", "category" : "hue", "type" : "primary", "code" : { "rgba" : [ 255, 0, 0, 1 ], "hex" : "#FF0" } }
{ "_id" : ObjectId("5a97de7f2fcdf731d255a1a2"), "color" : "blue", "category" : "hue", "type" : "primary", "code" : { "rgba" : [ 0, 0, 255, 1 ], "hex" : "#00F" } }
{ "_id" : ObjectId("5a97e3202c19f0e958477e06"), "color" : "custome", "category" : "hue", "type" : "primary", "code" : { "rgba" : [ 255, 1, 255, 1 ], "hex" : "#FF1" } }
MongoDB Enterprise >

Updating a document in MongoDB

Wow, we have the data now. And we performed, Create and Read operations on our DB. It is time for updating the document. Let’s go ahead and add a new property “manuallyCreated” to the color we have created. It is going to help us in finding this kind of entries easily.

MongoDB Enterprise > db.colors.update({
... "color":"custome"
... }
... ,{
... $set:{"manuallyCreated":"True"}}
... )
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
MongoDB Enterprise >

Let’s find all the manually created colors now, and yeah we know there is going to be one record.

MongoDB Enterprise > db.colors.find({"manuallyCreated":"True"})
{ "_id" : ObjectId("5a97e3202c19f0e958477e06"), "color" : "custome", "category" : "hue", "type" : "primary", "code" : { "rgba" : [ 255, 1, 255, 1 ], "hex" : "#FF1" }, "manuallyCreated" : "True" }
MongoDB Enterprise >

Let’s update the color name to “custom” instead of “custome”, sorry for the typo.

MongoDB Enterprise > db.colors.update({"_id" : ObjectId("5a97e3202c19f0e958477e06")},
... {$set:{"color":"custom"}})
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
MongoDB Enterprise >

Ah, I made a mistake. I ran the following query by mistake.

MongoDB Enterprise > db.colors.update({ "color":"custom" } ,{ $set:{"code.rgba" : [ 255, 1, 255]}} )
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })

Do you know, what that query just did? It just updated the rgba of our custom color to three point array “[ 255, 1, 255]”. That’s not what I wanted. Now what we can do? We need to add one value to that set. Let’s do that now.

MongoDB Enterprise > db.colors.update({ "color":"custom" } ,{ $addToSet:{"code.rgba" : "1" }} )
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })

MongoDB Enterprise > db.colors.find({"manuallyCreated":"True"})
{ "_id" : ObjectId("5a97e3202c19f0e958477e06"), "color" : "custom", "category" : "hue", "type" : "primary", "code" : { "rgba" : [ 255, 1, 255, "1" ], "hex" : "#FF1" }, "manuallyCreated" : "True" }
MongoDB Enterprise >

Please be noted that, there is one more update, which will just update the entire document. In this case, if you are not providing the value for each attributes, it will be overwriting the same. Let’s see an example.

MongoDB Enterprise > db.colors.update({"color":"red"},{"color":"test"})
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
MongoDB Enterprise > db.colors.find({})

Here we just give a command to update the color red to color test, as we have not provided other attributes as part of our query, after running the query, there will be only one attribute that is color.

MongoDB Enterprise > db.colors.find({})
{ "_id" : ObjectId("5a97f73f2fcdf731d255a1f1"), "color" : "black", "category" : "hue", "type" : "primary", "code" : { "rgba" : [ 255, 255, 255, 1 ], "hex" : "#000" } }
{ "_id" : ObjectId("5a97f73f2fcdf731d255a1f2"), "color" : "test" }
{ "_id" : ObjectId("5a97f73f2fcdf731d255a1f3"), "color" : "white", "category" : "value", "code" : { "rgba" : [ 0, 0, 0, 1 ], "hex" : "#FFF" } }
{ "_id" : ObjectId("5a97f73f2fcdf731d255a1f4"), "color" : "blue", "category" : "hue", "type" : "primary", "code" : { "rgba" : [ 0, 0, 255, 1 ], "hex" : "#00F" } }
{ "_id" : ObjectId("5a97f73f2fcdf731d255a1f5"), "color" : "yellow", "category" : "hue", "type" : "primary", "code" : { "rgba" : [ 255, 255, 0, 1 ], "hex" : "#FF0" } }
{ "_id" : ObjectId("5a97f73f2fcdf731d255a1f6"), "color" : "green", "category" : "hue", "type" : "secondary", "code" : { "rgba" : [ 0, 255, 0, 1 ], "hex" : "#0F0" } }
MongoDB Enterprise > db.colors.find({"color":"test"})
{ "_id" : ObjectId("5a97f73f2fcdf731d255a1f2"), "color" : "test" }
MongoDB Enterprise >

Deleting a document in MongoDB shell

So, we added a custom color, and later we found that it is no more needed in our document as we found an exact match with a different color. Now we need to remove the one we have added.

MongoDB Enterprise > db.colors.remove({"color":"custom"})
WriteResult({ "nRemoved" : 1 })
MongoDB Enterprise >

Let’s go and check again whether it is actually removed or not.

MongoDB Enterprise > db.colors.find({"color":"custom"})
MongoDB Enterprise >

The find query returns no records.

Deleting an entire collection

We have performed CRUD operations on a collection, what if we need to remove the entire collection?

MongoDB Enterprise > db.colors.drop()
true
MongoDB Enterprise >

With that, we are done with this post. I will be posting the continuation part of this series very soon. Till then, bye.

Conclusion

Your turn. What do you think?

Kindest Regards
Sibeesh Venu

Learn MongoDB With Me

SibeeshVenu — Thu, 22 Feb 2018 05:03:00 +0000

[toc]

Introduction

This is going to be a series of article on MongoDB. Here we are going to do some exercises with MongoDB, we will be talking about Mongo Shell, how can we configure MongoDB?, What are Indexes in MongoDB etc. We all know what an Indexes is, you might have already done that with any relational databases like SQL and MySQL . Have you every done indexing for your MongoDB? If your answer is “no”, no worries, here we are going to see indexes in MongoDB , if it is a “yes” please read this post and correct me if I am wrong anywhere. Let’s begin now.

Prerequisites

I hope you might have got a basic information about MongoDB, if not, I strongly recommend you to read these posts. Now that, you have a basic idea, I am assuming that you have already set up the environment for MongoDB development. Let’s recall what you might have done so far.

Install MongoDB
Set the environment variable for MongoDB
Start the MongoDB services

To set the environment variable for MongoDB, you may have to add a new path to the system variable path with the value as “C:\Program Files\MongoDB\Server\3.4\bin”, please note that the version number will be varied according to your MongoDB version. Once you are done the above steps, you should be able to start both Mongo server and Mongo shell from the command line interface.

Setting up MongoDB using CLI

Now let’s just open our command line interface, and create the data directory for Mongo. We will have to create a directory for the same. Please go along with the below commands.

md \data
md \data\db
mongod

Now let’s open a new CLI and run the command “mongo”, please do not worry about the warnings you are getting, as we are not working in production data, we may not need to secure and optimize it.

Exploring MongoDB

Once you are connected to MongoDB, by default you are connected to test DB. You can check that by running the command MongoDB Enterprise > db

Playing with Mongo Shell

Let’s just use a new database now.

MongoDB Enterprise > use MongoIndex
switched to db MongoIndex
MongoDB Enterprise >

Please be noted that the database MongoIndex doesn’t exist as of now, as we haven’t created it. Still, Mongo just switched our context to the new database. You can see this if you run the command show dbs

The database will be created once we insert any document associated with it. Now we are going to create a new collection called “User”, so once we made the entry to this collection, the database will also be created automatically. Let’s do that.

MongoDB Enterprise > db.users.insert({"name":"Sibees Venu"})
WriteResult({ "nInserted" : 1 })
MongoDB Enterprise >

Now if you run the “show dbs” command again, the database MongoIndex will show up. If you ever need to see the collections you have in the DB, you just need to run the command “show collections”.

MongoDB Enterprise > show collections
users
MongoDB Enterprise >

The MongoDB is very friendly when it comes to data, it doesn’t require any schema to get it started. The learning is so easy, am I right?

The other benefit of MongoDB is its JavaScript interpreted shell, where we can actually type JavaScript code and run. To test it out, let’s create a variable and use it.

MongoDB Enterprise > var name = "Sibeesh Venu"
MongoDB Enterprise > name
Sibeesh Venu
MongoDB Enterprise >

This way, we can interact with the database with a JavaScript program. Now let’s go ahead and creating a collection called “Numbers” and insert 26,000 rows in it. So how are we going to do that? Yes, you are right, we are going to write a for loop, the mongo shell gives that kind of flexibility. Let’s see that in action.

MongoDB Enterprise > for(i=0;i<=26000;i++){
... db.Numbers.insert({
... "number":i
... })
... }
WriteResult({ "nInserted" : 1 })
MongoDB Enterprise >

So we have done that. Note that, we are able to break the commands into multiple lines, this allows to break the complex codes to much readable format in the shell itself. Sounds good?

Even though we have inserted 26,000 rows, it always shows,"nInserted" : 1 this is because it is counting a number of operations, not the individual documents. Let’s see this by checking the count now.

MongoDB Enterprise > db.numbers.count()
0
MongoDB Enterprise > db.Numbers.count()
26001
MongoDB Enterprise >

Please note that it is case sensitive.

Indexes in MongoDB

Now if you need to see any particular record, you can always write the query in the shell as follows.

MongoDB Enterprise > db.Numbers.find(
... {"number":24000}
... )
{ "_id" : ObjectId("5a8d3be2020a0071d115cf62"), "number" : 24000 }
MongoDB Enterprise >

So in the query, we are using the function “find” with the filter “number: 24000”, so that the Mongo can return the record which has the number value as 24000. Now that we have got the output we needed, would you like to see what just happened in the background? To do so, we can use the function “explain()”.

MongoDB Enterprise > db.Numbers.find( {"number":24000} ).explain()
{
        "queryPlanner" : {
                "plannerVersion" : 1,
                "namespace" : "MongoIndex.Numbers",
                "indexFilterSet" : false,
                "parsedQuery" : {
                        "number" : {
                                "$eq" : 24000
                        }
                },
                "winningPlan" : {
                        "stage" : "COLLSCAN",
                        "filter" : {
                                "number" : {
                                        "$eq" : 24000
                                }
                        },
                        "direction" : "forward"
                },
                "rejectedPlans" : [ ]
        },
        "serverInfo" : {
                "host" : "PC292716",
                "port" : 27017,
                "version" : "3.4.9",
                "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e"
        },
        "ok" : 1
}
MongoDB Enterprise >

And, if you need to get more information about the execution, you can pass the parameter “executionStats” to the “explain” function.

The parameter is always case sensitive, you will get an errors as below, if you give it wrong. So please make sure you are passing executionStats not executionstats.

“MongoDB Enterprise > db.Numbers.find( {“number”:24000} ).explain(“executionstats”)
2018-02-21T15:12:34.197+0530 E QUERY [thread1] Error: explain verbosity must be one of {‘queryPlanner’,’executionStats’,’allPlansExecution’} :
parseVerbosity@src/mongo/shell/explainable.js:22:1
constructor@src/mongo/shell/explain_query.js:83:27
DBQuery.prototype.explain@src/mongo/shell/query.js:520:24
@(shell):1:1″

MongoDB Enterprise > db.Numbers.find( {"number":24000} ).explain("executionStats")
{
        "queryPlanner" : {
                "plannerVersion" : 1,
                "namespace" : "MongoIndex.Numbers",
                "indexFilterSet" : false,
                "parsedQuery" : {
                        "number" : {
                                "$eq" : 24000
                        }
                },
                "winningPlan" : {
                        "stage" : "COLLSCAN",
                        "filter" : {
                                "number" : {
                                        "$eq" : 24000
                                }
                        },
                        "direction" : "forward"
                },
                "rejectedPlans" : [ ]
        },
        "executionStats" : {
                "executionSuccess" : true,
                "nReturned" : 1,
                "executionTimeMillis" : 13,
                "totalKeysExamined" : 0,
                "totalDocsExamined" : 26001,
                "executionStages" : {
                        "stage" : "COLLSCAN",
                        "filter" : {
                                "number" : {
                                        "$eq" : 24000
                                }
                        },
                        "nReturned" : 1,
                        "executionTimeMillisEstimate" : 11,
                        "works" : 26003,
                        "advanced" : 1,
                        "needTime" : 26001,
                        "needYield" : 0,
                        "saveState" : 203,
                        "restoreState" : 203,
                        "isEOF" : 1,
                        "invalidates" : 0,
                        "direction" : "forward",
                        "docsExamined" : 26001
                }
        },
        "serverInfo" : {
                "host" : "PC292716",
                "port" : 27017,
                "version" : "3.4.9",
                "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e"
        },
        "ok" : 1
}
MongoDB Enterprise >

Now you can see more information on the execution of how much time it took for the execution and how many docs it is examined etc. If you have noticed, it has examined all the 26001 records and took 13 milliseconds. That’s just a case, that we had only less number of records in the table, what if, we have millions of records in it? And examining all the records would be a bad idea, am I right? So what do we do at that time? What would be a permanent solution for this? This is where the importance of Indexes comes into action.

Let’s create an Index for the number that we are going to search.

MongoDB Enterprise > db.Numbers.createIndex({number:1})
{
        "createdCollectionAutomatically" : false,
        "numIndexesBefore" : 1,
        "numIndexesAfter" : 2,
        "ok" : 1
}
MongoDB Enterprise >

Here the number is a special variable, not a string. As you can see, we had created the index. You can see that the property value of createdCollectionAutomatically is false, as the collection had already created and it didn’t have to create it again.

Let’s run our find query again.

MongoDB Enterprise > db.Numbers.find( {"number":24000} ).explain("executionStats")
{
        "queryPlanner" : {
                "plannerVersion" : 1,
                "namespace" : "MongoIndex.Numbers",
                "indexFilterSet" : false,
                "parsedQuery" : {
                        "number" : {
                                "$eq" : 24000
                        }
                },
                "winningPlan" : {
                        "stage" : "FETCH",
                        "inputStage" : {
                                "stage" : "IXSCAN",
                                "keyPattern" : {
                                        "number" : 1
                                },
                                "indexName" : "number_1",
                                "isMultiKey" : false,
                                "multiKeyPaths" : {
                                        "number" : [ ]
                                },
                                "isUnique" : false,
                                "isSparse" : false,
                                "isPartial" : false,
                                "indexVersion" : 2,
                                "direction" : "forward",
                                "indexBounds" : {
                                        "number" : [
                                                "[24000.0, 24000.0]"
                                        ]
                                }
                        }
                },
                "rejectedPlans" : [
                        {
                                "stage" : "FETCH",
                                "inputStage" : {
                                        "stage" : "IXSCAN",
                                        "keyPattern" : {
                                                "number" : 24000
                                        },
                                        "indexName" : "number_24000",
                                        "isMultiKey" : false,
                                        "multiKeyPaths" : {
                                                "number" : [ ]
                                        },
                                        "isUnique" : false,
                                        "isSparse" : false,
                                        "isPartial" : false,
                                        "indexVersion" : 2,
                                        "direction" : "forward",
                                        "indexBounds" : {
                                                "number" : [
                                                        "[24000.0, 24000.0]"
                                                ]
                                        }
                                }
                        }
                ]
        },
        "executionStats" : {
                "executionSuccess" : true,
                "nReturned" : 1,
                "executionTimeMillis" : 36,
                "totalKeysExamined" : 1,
                "totalDocsExamined" : 1,
                "executionStages" : {
                        "stage" : "FETCH",
                        "nReturned" : 1,
                        "executionTimeMillisEstimate" : 0,
                        "works" : 3,
                        "advanced" : 1,
                        "needTime" : 0,
                        "needYield" : 0,
                        "saveState" : 1,
                        "restoreState" : 1,
                        "isEOF" : 1,
                        "invalidates" : 0,
                        "docsExamined" : 1,
                        "alreadyHasObj" : 0,
                        "inputStage" : {
                                "stage" : "IXSCAN",
                                "nReturned" : 1,
                                "executionTimeMillisEstimate" : 0,
                                "works" : 2,
                                "advanced" : 1,
                                "needTime" : 0,
                                "needYield" : 0,
                                "saveState" : 1,
                                "restoreState" : 1,
                                "isEOF" : 1,
                                "invalidates" : 0,
                                "keyPattern" : {
                                        "number" : 1
                                },
                                "indexName" : "number_1",
                                "isMultiKey" : false,
                                "multiKeyPaths" : {
                                        "number" : [ ]
                                },
                                "isUnique" : false,
                                "isSparse" : false,
                                "isPartial" : false,
                                "indexVersion" : 2,
                                "direction" : "forward",
                                "indexBounds" : {
                                        "number" : [
                                                "[24000.0, 24000.0]"
                                        ]
                                },
                                "keysExamined" : 1,
                                "seeks" : 1,
                                "dupsTested" : 0,
                                "dupsDropped" : 0,
                                "seenInvalidated" : 0
                        }
                }
        },
        "serverInfo" : {
                "host" : "PC292716",
                "port" : 27017,
                "version" : "3.4.9",
                "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e"
        },
        "ok" : 1
}
MongoDB Enterprise >

As we had given the index on what exactly we are going to search, it just examined only that document when we run the query, that’s why the value of the property totalDocsExamined is 1. Indexing will not have many impacts on the database which has few records in it, but it has a massive effect on very large data sets which has millions of records in it. Using this simple Indexes can reduce the execution time to almost nothing.

With that, we are done with this post. I will be posting the continuation part of this series very soon. Till then, bye.

Conclusion

Your turn. What do you think?

Kindest Regards
Sibeesh Venu

SQL Log File Corruption and Possible Recovery Methods

Andrew Jackson — Mon, 12 Feb 2018 06:05:23 +0000

[toc]

Introduction

Structured Query Language (commonly known as SQL) is a programming language used for managing data held in relational database management system (RDMS) consisting of data definition language, data manipulation language and a data control language. The SQL database comprises of following three files:

Primary Database Files: Primary database file is the main database file (MDF) which points to another file in database and hence every database has one primary data file. The file is in the extension of .mdf
Secondary Database Files: When the data of a database exceeds than a secondary database file is created which stores the data that exceeds MDF limit. Hence, multiple secondary data files can be created for a single database. The file is in the extension of .ndf
Log Files: These files maintain a log of all the transactions done in SQL Server Database so that that information can be later used to recover the database. There must exist one log file for each database and it is possible that more than one log file can be created for a single database. The file is in .ldf file extension.

Log files (also known as transaction log) consist of actions executed on the database for database management to guarantee ACID properties over crashes or hardware failure. It is a file listing changes to the database, stored in a stable storage format

Problem

The log files of the SQL Server gets corrupted i.e. while the time we are performing some action upon SQL database, there exist some errors in between the process and due to which there is an interrupt in process.

Causes of Log File Corruption in SQL Server

Viruses or other Malicious Software- In computer system, many viruses can infect and damage the log files and makes them inaccessible.
Terminating System Abnormally- If system/application is quit abnormally, then files are prone to be corrupted or damaged.
Input Output Configuration- The I/O subsystem is a vital component of the database used to store system and user databases. Hence if the configuration is disturbed on enhanced that it may lead to corruption in log files.
Storage Size Issue- The biggest reason behind the corruption of log files is the storage size. In case the data exceeds the limit of LDF, corruption is likely to occur.

Errors which Occurs Due to Log File Corruption

Error Message 1: StartLogFiles: The error exists when the log files are unable to start because the system could not find the file specified. So try to diagnose and correct the operating system error, and retry the operation.

Error Message 2: File activation failure. The error occurs due to error in file on location ‘C:\ProgramFiles\MSSQLServer\MSSQL10_50.SQLEXPRESS\MSSQL\Log\ERRORLOG’.

Error Message 3: The error message displays that the transaction log has been deleted or lost due to a hardware failure of a system or any other reason. The log files cannot be redeveloped because users perform open transactions on file.

Error Message 4: Corrupted server of SQL database leads to corruption in backup of log files.

Error Message 5: When database log is corrupted and a user is attempting to attach the log file to the new server then an error message displays with a message that ‘Could not open new database (Name of Database). CREATE DATABASE is aborted.

Error Message 6: When the log database is attempted to attach but it gives an error while performing the attachment. The error displays one of the two number i.e. 9004 or 9001 notifying that you have to create a backup or it is necessary to rebuild the log.

How to Recover Corrupted Log File in SQL Server?

In order to repair a corrupt LDF file, use the WITH TABLOCK option for DBCC CHECKDB. It will recover the data from a corrupted LDF file that has been corrupted or damaged due to some reason such as logical corruption.

dbcc-checkdb

Conclusion

Through the above solution, we can recover SQL log file but not sure that the above procedure will recover complete log file data. However, there exists SQL Server transaction log recovery tool through which you can restore database transaction log.

Creating a Chat Application in Node JS with Express, MongoDB, Mongoose and Socket.io

SibeeshVenu — Mon, 04 Dec 2017 12:09:50 +0000

[toc]

Introduction

In this article, we are going to a chat application in Node JS with the back end MongoDB. We will also be using Mongoose for creating the MongoDB models and Socket.io for making multi directional chats on multiple client window. If you are really new to the Node JS, I strongly recommend you to read some articles on the same here. You can also see how you can create a sample Node application with MongoDB and Mongoose here. At the end of this article, I guarantee that you will be having some basic knowledge on the mentioned technologies. I hope you will like this article.

Source Code

You can always clone or download the source code here

Background

Creating a chat application is always an interesting this to do. And it is a good way to learn a lot, because you are creating some interactions on your application. And with the release of few technologies we can create such application without any hassle. It is much more easier than ever. Here we are also going to create a chat application. A basic knowledge of Node JS, MongoDB, JavaScript, JQuery is more than enough to create this project. So, please be with me. Let’s just skip the talking and start developing.

Setting up Node application

This step requires some basic knowledge, please see some of that here. So as mentioned in that article, we have successfully created our Package.json file and installed the required packages. Let’s review our package.json file.

{
 "name": "chatapp",
 "version": "1.0.0",
 "description": "A chat application in Node JS, which uses MongoDB, Mongoose and Socket.io",
 "main": "index.js",
 "scripts": {
 "test": "echo \"Error: no test specified\" && exit 1"
 },
 "keywords": [
 "Node",
 "JS",
 "MongoDB",
 "Mongoose",
 "Socket.io"
 ],
 "author": "Sibeesh Venu",
 "license": "ISC",
 "dependencies": {
 }
}

Let’s install the packages now by running the below commands.

npm install --save express
npm install --save mongoose
npm install --save body-parser
npm install --save socket.io

Now that we have all the dependencies added to the package.json file.

"dependencies": {
    "body-parser": "^1.18.2",
    "express": "^4.16.2",
    "mongoose": "^4.13.6",
    "socket.io": "^2.0.4"
  }

Creating an App using Express

Let’s create a file server.js and build an app using Express.

var express = require("express")
var app = express()
app.listen(3020,()=>{
 console.log("Well done, now I am listening...")
})

I hope you are getting the desired output, if not, please don’t worry, just double check the codes you had written.

Running our application on browser

Let’s run our application on port 3020 and see what we are getting http://localhost:3020/..

Yes you will get an error as Cannot GET / , no worries. To fix that you need to add the following code block to your server.js file

app.use(express.static(__dirname))

Creating Index page

Here I am going to create a HTML 5 page with JQuery and Bootstrap referenced in it.


Creating a Chat Application in Node JS with Express, MongoDB, Mongoose and Socket.io






    

    
        Start Chatting

As you can see, the page has two text boxes and one button. We will be creating some scripts very soon which uses these controls.

Start developing the chat app

Till now, it was all basic, and we have done it well. Now it is time to go and write some advanced codes. So, are you ready?

Create model from page data

Here we are going to create the model from the page data, that is, from the controls we have in our page. We will be using JQuery to do so.

Now that we have got the model from the user, let’s save it to the DB.

Model values on browser console

Setting up database

We are going to set up our database in mLab as mentioned in this article Using MongoDB on Node JS Application Using Mongoose. So let’s just require Mongoose and do the needed changes.

var express = require("express")
var mongoose = require("mongoose")

var app = express()

var conString = "mongodb://admin:admin@ds038319.mlab.com:38319/mylearning"
app.use(express.static(__dirname))

var Chats = mongoose.model("Chats", {
    name: String,
    chat: String
})

mongoose.connect(conString, { useMongoClient: true }, (err) => {
    console.log("Database connection", err)
})

app.listen(3020, () => {
    console.log("Well done, now I am listening...")
})

Creating a Post request

Now let’s create a post requests in our index.html file which will the post API in our server.js file.

index.html

function postChat(chat) {
        $.post("http://localhost:3020/chats", chat)
}

server.js

app.post("/chats", async (req, res) => {
    try {
        var chat = new Chats(req.body)
        await chat.save()
        res.sendStatus(200)
    } catch (error) {
        res.sendStatus(500)
        console.error(error)
    }
})

Let’s just run our application and test it out.

You may be getting an error as “(node:10824) DeprecationWarning: Mongoose: mpromise (mongoose’s default promise library) is deprecated, plug in your own promise library instead: http://mongoosejs.com/docs/promises.html””, here to fix this, we need to use the default promise instead of the Mongoose promise. Let’s change that. Add this code mongoose.Promise = Promise to our server.js file. Give it a try after setting it.

Still it is not working right, you are getting undefined at the place, var chat = new Chats(req.body) in our Post API . At this stage, we will have to use our body-parser packages, do you remember the package we have installed? Let’s just require that package var bodyParser = require("body-parser") add the preceding codes to our server.js file.

app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: false }))

Now this will convert the request to a JSON object by default. Give it a try again, I am sure you will get the actual value instead of undefined.

Creating a Get request for all the chat data

We have written codes to write our data into our database, we will have to show this data on our page right? Here we are going to write the get requests in out index.html page which will call the get API.

index.html

function getChats() {
     $.get("/chats", (chats) => {
         chats.forEach(addChat)
     })
}

function addChat(chatObj){
    $("#messages").append(`${chatObj.name} 
${chatObj.chat}`);
}

And call the function getChats() in document ready event.

server.js

app.get("/chats", (req, res) => {
    Chats.find({}, (error, chats) => {
        res.send(chats)
    })
})

Here we are passing {} to our find function, this means, we are going to get all the chats without any filter. Just run your application now, and check whether you are getting the chat messages you had sent.

Retrieving data from a MongoDB

Implementing Socket.io

Now the chat we are sending is getting saved to our database and we are able to load the same on page load. Is it the behavior of a perfect chat application? Absolutely no, a perfect chat application will be able to,

Show the chat message to the UI right after the data gets inserted to database
Show the chats in multiple clients, here in our application, if you are opening the URL in multiple browser instance, and if you need to show the chats to both instances, you will have to refresh the pages right? This is not a recommended way

That’s why we are going to implement Socket.io in our application.

Require the package

Unlike the other packages, adding socket.io to the application is a different process, we will have to require the http server first, then, set our app. You can see more information on socket.io here.

var http = require("http").Server(app)
var io= require("socket.io")(http)

Enabling the connection

To enable the connection, we need to use the function io.on.

io.on("connection", (socket) => {
    console.log("Socket is connected...")
})

Listen using new http server

Now that we have a new http server, and we should change our listen code to our new http.

var server = http.listen(3020, () => {
    console.log("Well done, now I am listening on ", server.address().port)
})

Changes in script

Let’s do listen for the event “chat” now in our html page.

var socket = io()
socket.on("chat", addChat)

Please do not forget to include the socket.io.js reference in our index page, other wise you may get an error as “Uncaught ReferenceError: io is not defined”

Emits the event

Once the above step is completed we need to make sure we are emitting the new event on our Post API.

app.post("/chats", async (req, res) => {
 try {
 var chat = new Chats(req.body)
 await chat.save()
 res.sendStatus(200)

 //Emit the event
 io.emit("chat", req.body)

 } catch (error) {
 res.sendStatus(500)
 console.error(error)
 }
})

Let’s just run our application in two browser instances and check for the output.

Socket.io Output

Please make sure that you are getting the chats in the second instance when you send it from the first instance of the browser and vice versa.

Conclusion

Your turn. What do you think?

Kindest Regards
Sibeesh Venu

Using MongoDB on Node JS Application Using Mongoose

SibeeshVenu — Sun, 03 Dec 2017 17:15:12 +0000

[toc]

Introduction

In this post, we are going to see how we can use a MongoDB on our Node JS application with the help of the package Mongoose. We will also be covering some facts about MongoDB so that as a reader, you will understand why we had chosen MongoDB as our backend. We will be going through some steps to install the required packages using a node package manager terminal, so please be with me. At the end of this article, I guarantee that you will be proficient on how to connect a MongoDB in our Node JS application. I hope you will like this article.

Why MongoDB?

Here, I am going to list down the reasons why I had chosen MongoDB.

MongoDB is a NoSQL database when I say NoSQL, it means that it doesn’t have any structure to be followed.
There are no relationships, we can perform the relation using separate packages like Mongoose
Everything is JSON, even the data and collections are stored as JSON
Since the data is stored as JSON, no more conversions are needed. We can directly use it in our application.

Source Code

You can always clone or download the source code here

Background

Node JS has become a trend now, thus some most used databases. One of the recommended database for a Node application is MongoDB. As we discussed a MongoDB doesn’t have any structure inbuilt, here we are going to use a package called Mongoose, with that we can create some structure for our Database. Let’s just skip the talking and start developing.

Setting up Node application

To get started with the Node JS, create a folder and name as per your wish, this is going to be our project directory. Here I am going to use one of my favorite editor, Visual Studio Code. Now please point to your project container, and run the command below.

npm init

NPM Init

This will create a file named “Package.json” in your directory. We will be adding all of our dependencies in this file very soon. The command may ask you some default questions which you need to answer. If you need to create the package.json file with default values in it, you may consider running the below command.

npm init --yes

Let’s review our package.json file.

{
  "name": "node-mongodb-mongoose",
  "version": "1.0.0",
  "description": "A Node application with MongoDB and Mongoose",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [
    "Node",
    "Mongoose",
    "MongoDB"
  ],
  "author": "Sibeesh Venu",
  "license": "ISC"
}

Getting the required packages

Now that we have the application ready, let’s get the required packages ready.

npm install --save mongoose

This will create a new folder npm_modules on your project directory where you can see the package Mongoose in it.

npm install --save express

The above command will add the package express to our application.

Creating the Node App

Now we can create our server.js file where we are going to write most of our application code.

var express = require("express")
var app = express()
app.listen("3010",()=>{
    console.log("I just started listening!.")
})

Now run the command node server.jsand make sure that you are getting a console as ” I just started listening!.”.

If you are getting an error as preceding, please make sure that you are running the application on an unused port.

PS F:\Visual Studio\Node.JS\Node-MongoDB-Mongoose> node server.js
events.js:136
throw er; // Unhandled ‘error’ event
^

Error: listen EADDRINUSE :::3000
at Object._errnoException (util.js:1031:13)
at _exceptionWithHostPort (util.js:1052:20)
at Server.setupListenHandle [as _listen2] (net.js:1367:14)
at listenInCluster (net.js:1408:12)
at Server.listen (net.js:1496:7)
at Function.listen (F:\Visual Studio\Node.JS\Node-MongoDB-Mongoose\node_modules\express\lib\application.js:618:24)
at Object. (F:\Visual Studio\Node.JS\Node-MongoDB-Mongoose\server.js:3:5)
at Module._compile (module.js:641:30)
at Object.Module._extensions..js (module.js:652:10)
at Module.load (module.js:560:32)”

Setting up database

Now that we have our application ready, let’s go and create our database. I am going to use mLab for creating the database. If you have not installed MongoDB on your machine, I strongly recommend creating a database in mLab. I had created a database there and have got my connection string as preceding.

mongodb://:@ds038319.mlab.com:38319/mylearning

We will be updating the connection string with actual DB user and password later. Please make sure that you are creating a user for your DB and remember the password.

Creating a Mongoose model

Let’s create a Mongoose model and set up our connection string now, which is going to be our collection. No worries, it is just a JavaScript model.

var conString = "mongodb://admin:admin@ds038319.mlab.com:38319/mylearning"

/**
 * Models 
 */
var User = mongoose.model("User", {
    firstName: String,
    lastName: String
})

Please be noted that, when you move the codes to production, make sure that your password and user fields on the connection string are separated from this file and encrypted

Now let’s connect our database.

mongoose.connect(conString, { useMongoClient: true }, () => {
    console.log("DB is connected")
})

Setup the data

We have our model ready, what else is needed now? Yes, you are right, we need data.

var dummyUser = {
    firstName: "Sibeesh",
    lastName: "Venu"
}

Inserting the data into MongoDB

We have got everything ready, we can insert the model now. Let’s do the saving logic now.

mongoose.connect(conString, { useMongoClient: true }, () => {
    console.log("DB is connected")
    saveData()
})

function saveData() {
    var user = new User(dummyUser);
    user.save();
}

Verify the data

Once you run your application, go to your database and check for the entries there. I am sure there will be a new collection “User” and our inserted data. As I am using mLab database, here is how my data saved.

MongoDB Data Insertion

If you have noticed, you can see there are an another property names “_id” in our database collection. This is a unique id generated by MongoDB for us, so no need to think about it.

Todo

As the purpose of this article is to show how we can insert a data to MongoDB, I called the save method right after the database is connected. In our real life, this is not a recommended way. You must create an Ajax post request and handle the same. I am leaving that task to you. Please try to do that and let me know. I would love to hear back from you.

Conclusion

Your turn. What do you think?

Kindest Regards
Sibeesh Venu

Learn to Repair Corrupt MDF File with Perfection

Andrew Jackson — Fri, 22 Sep 2017 17:07:34 +0000

[toc]

To get a damaged MDF file is not rare. However, the main cause behind it is that the file is prone to corruption. When this occurs, all data saved in the server database becomes inaccessible and therefore, it leads to data loss. In such circumstances, it is important to repair corrupt MDF file of server database storage.

This segment will discuss how to execute the repair process. In addition, it will also give you in-depth information about MDF file, reasons, and way to repair damaged MDF file to store it in healthy form. Read further to know more.

Quick Glance on MDF File

SQL Server MDF file is a relational management system of the database. Moreover, it is a primary data file of the database, which saves all the server data. Thus, you can even state it as the master database file of MS SQL Server. Every database of SQL Server would enclose at least one .mdf file.

It saves components like XML Indexes, Views, Indexes, Stored Procedures, Tables, Triggers, Rules, Keys, User Defined Functions, sparse columns, data types, file stream, the column set property data types. The .mdf file can be said as a primary element for managing SQL Server database.

Reasons of MDF File Corruption

There are various causes, which are answerable for damaging the primary file type of server.

Unexpected power failure.
Various bugs in server.
Defective Operating System.
Sudden shutdown of the machine.
Issues with hard drive
Virus outbreaks.

Thus, issues for your .mdf file revolving corrupt can be everything from malfunctioning of hardware to software and so it is necessary to fix corrupted MDF file.

Technique to Repair Corrupt MDF File

Method 1: Inbuilt Tool

There are some tools that make easy for users to fix corrupted MDF file of the server. Thus, make the saved data available. In fact, all these tools are series of commands in T-SQL programming language, which is called DBCC (Database Console Commands). The purpose of these statements in DBCC is to test physical and logical uniformity of MS SQL Server database files and fix troubling problems that continue.

DBCC CHECKDB is a command via which you can simply check the logical and physical integrity of all objects in precise MS SQL Server database. You can do it by implementing the mentioned operations successively:

Run DBCC CHECKALLOC command in the database.
Run DBCC CHECKCATALOG command in the database.
Run DBCC CHECKTABLE on every view & table in the database.
Verifying content of every indexed view present in the database.
Validating link-level constancy among table Metadata, file system directories files while saving varbinary (max) data utilizing FILESTREAM in the file system.
Confirming Service Broker data in database.

Once the above steps to repair damaged MDF file is completed then, if the application finds any issues of corruption or any errors, it recommends users have the usage of various recover options for fixing troublesome problems. The recover or repair options are:

REPAIR_FAST

It preserves syntax for compatibility of backward only; no repair actions are executed in definite. The syntax for this, Repair option is: DBCC CHECKDB (‘DB Name’, REPAIR_FAST).

REPAIR_REBUILD

This option of repair implements repair process, which scarcely has potentials of data loss. This can do quick repairs like repair of missing rows in non-clustered indexes, even time-consuming repairs like the rebuilding of indexes. The syntax is DBCC CHECKDB (‘DB Name’, REPAIR_REBUILD).

REPAIR_ALLOW_DATA_LOSS

This command creates an effort to fix all the issues, which are reported. However, it can root cause the loss of data as specified in repair command itself. The syntax is: DBCC CHECKDB (‘DB Name’, REPAIR_ALLOW_DATA_LOSS).

Limitations:

The specific database should be in single-user mode to be able to execute either of three commands of repair.
DBCC repair commands are not authenticated for memory-optimized tables.

Method 2: Trouble-Free Solution

There can be a possibility that when your SQL Server database MDF file is corrupt and when you try to connect to SQL Server you will find that it is marked as SUSPECT. During such scenarios or the above discussed, you will not be able to connect to the database. To repair corrupt SQL Server MDF database file best option is to restore from a recent full database backup available.

If no recent backup available in such cases best possible approach is to repair corrupted MDF file of Microsoft SQL Server 2000, 2005, 2008, 2012 and 2016 with SQL Database Repair Tool. One of the most highly used and recommended software to repair corrupt SQL MDF file with the assurance of 99% guaranteed recovery.

Observational Verdict

We have gone through the possible solution which you need to follow to recover a database in case MDF file get corrupted or database marked as SUSPECT. We serve you with the best of the best ways to repair corrupt MDF file. So, Hurry! What are you waiting for? You are raising the bar and we are taking it forward with the all in one and probably the best of solution repair damaged MDF database file.

Understand the Internal Structure of SQL Database File

Andrew Jackson — Thu, 23 Mar 2017 15:55:59 +0000

You must have thought how to store data in SQL Server? SQL Server is the platform, which provides a proper management of database. There is no doubt that it follows a strategy for the storage of all the files. All the users of SQL are aware with the fact that database objects are stored in MDF file. MDF files are responsible for the management of data in SQL Server. Therefore, in this article, we will learn the storage system of data in SQL Server data files.

Database Files and its Storage Structure in SQL Server

Database in SQL Server has two operating system files:

Data Files
Log File

Data Files

A data file consists of objects and data. Depending upon the needs, data files can be divided into two types:

Primary Database File

Each database has one primary data file. All the tables, database objects, indexes and views are stored in that file. The file extension of primary database file is .mdf.

Secondary Database File

This type is used at the time of exceeding the maximum allocated size for an individual file on Windows. It helps the database to increase in size. File extension for this file is .ndf.

Log File

Whenever users want to recover whole of the database in SQL Server, they require certain information, which are inside a log file. For one database there will be one log file and transactions of database are written on it even before to the data file. The file extension used by it is .ldf. LDF stands for Transactional Log file.

Inner Structure of Data File

Each database has its space to store the data, which is divided into different pages that are numbered from 0 to N. During the expansion of database files from its default size, those pages which are created newly are numbered from the last highest page number plus 1. In the similar way, at the time of shrinking of files it eliminates pages in decreasing order, which in the same manner starts from the highest page number. The basic unit of I/O for SQL Server operations is a page and each page is of 8 KB. A data file is a large array of pages. SQL Server stores different data with the help of multiple pages, Some of them are GAM, Index, IAM, Data, SGAM, and TextMix.

Page Addressing:

Now, if we analyze the pages addressing then we can see that every page in SQL Server is of equal size that is 8192 bytes. Each page has its own address, which is unique. It is a part of single database file. The ID of the file makes the first part of the unique address. Pages are numbered starting from 0. This is the mentioned format in which page address is written:

file number : page number

Data page is very essential among the page types. This is used for the storage of records to a database.

Structure of Data Page:

A data page contains three sections.

Page Header
Page Body
Slot Array or Row offset

Page Header (0-95 bytes)

The first 96 bytes of a particular page is known as page header, which consists of the information of the page.

That is, Page ID, No of records in that page and IDs of previous/next pages.

Page Body

It is that part where actual data is stored. It is followed by the free space and slot array.

Slot Array

Slot array, also known as off-set array, is an array of two-byte values, which is read by the SQL Server in reverse i.e. from the very end of the page. Slot array keep slot points to a byte in the page wherever a record begins. First slot of the record slot array is saved in the last two bytes of the page that points to the page of the first record saved at the page.

Conclusion

One of the essential things that should be kept in mind is that, In SQL Server, the database file (.mdf) store its data as per 8 kb pages and it is important for SQL Server to sustain the sequence of data pages as per their header information. In case of mismatching of information or the data is not stored in sequential order into data pages, there is possibility in the corruption of database or you may face SQL database error like 5172, which is a header corruption error.