The community will be in read-only from Tuesday 11:59pm (PST) to Wednesday 7:30am (PST)
The community will be in read-only from Tuesday 11:59pm (PST) to Wednesday 7:30am (PST)
Service Desk Practitioners Forum
cancel
Showing results for 
Search instead for 
Did you mean: 

Importing many CIs to my CMDB

Highlighted
Susana.gi_1
Super Collector

Importing many CIs to my CMDB

Hi all,
we are importing data from external sources to our CMDB to create new CIs. So, we are importing XML files of about 250.000 records.

We've seen that, as the CMDB is growing, the import proccess is going slower.

Is this the usual behaviour?

At the beginning, our first XML piece of 1Mb is imported in 3 minutes....De last piece of the XML file (1Mb) takes 45 minutes... it seems too much for me.

We are doing a parallel import, and all the slaves say the same.

Is this performance issue the most usual thing, because the CMDB is so big?

Thanks and best regards,
Susana
7 REPLIES
Oguz Kutlu Asi
Honored Contributor

Re: Importing many CIs to my CMDB

Hi,

Of course it will take more time if your CMDB is too big, but may be your database server (MS SQL?) and operating system need some tweaking. SQL server might be installed with default settings, operating system would be using virtual memory on disk because lack of physical memory after a while which makes all operation slow etc.

http://www.sql-server-performance.com/
http://msdn.microsoft.com/SQL/2000/learn/perf/default.aspx
What's right is right, whether or not God exists
Dan Ioan
Frequent Visitor

Re: Importing many CIs to my CMDB

Hi Susana,
Indeed, Oguz is right. Your CMDB is a huge database and is very important to check performance of your SQL server. Other recommendations regarding new CI data import:
1. Split your import files per category of CIs.
2. Don't use mapping fields with the same value per CI category! You can use later "update all" facility to set those fields to desired values.
3. If is possible, choose an Oracle data base solution on UNIX or Solaris platform to increase OVSD performance.
Best regards,
Dan
Susana.gi_1
Super Collector

Re: Importing many CIs to my CMDB

Sorry, I forgot to mention:
- HP-UX 11.11
- Oracle 9.0.2

Thanks,
Susana
Dan Ioan
Frequent Visitor

Re: Importing many CIs to my CMDB

Hi Susana,
In this case there is, almost, nothing to do!
Remember that this happens just once!
It is something normal. We have OVSD 4.5 with Oracle 10 and Solaris platform and a basic import for 1000 CIs with a mapping import with 3 fields (Search Code, Inventory Number and Barcode for ex.) takes 5 minutes!
Seems to be reasonable.
Try to import new CIs data when users aren't logged if importing XML file is huge.
Best regards,
Dan
Mike Bush
Honored Contributor

Re: Importing many CIs to my CMDB

To throw in another pennyworth - it will slow down because the behaviour is to first check for existance of the record (via its unique key) and only when confirmed absent does it create a record. Thus it is VERY important to ensure that your unique key is an INDEXED field (ie not a custom field or various others!).
Robert S. Falko
Honored Contributor

Re: Importing many CIs to my CMDB

Susana,

I think that Oracle can be tuned, too.

One of the standard tricks in loading large amounts of data into a database is to drop the indexes, and then recreate them afterwards. This generally is much faster. However, such an approach might impact the support you get from H-P, which frowns on tinkering with the DB.

When you created to database, did you size the objects to meet the expected, or are they creating a lot of extents along the way. That will slow things down a lot. How big are the extents being created?

-Josh
Jason C. Basket
Super Collector

Re: Importing many CIs to my CMDB

My suggestion, if you are concerned with affecting Service Support during business hours, is to test a command line import with about 10 records and then setup task scheduler to import the rest after hours.

sd_exchange import < input file>< sername>< password>< mapping >< import log>< temp dir> < reconciliation change list>

See page 152 of Data Exchange Administrator's Guide for more on this..


Jason
//Add this to "OnDomLoad" event