Many have undertaken a DIY project of installing a home security system and have been successful.
First, make an honest appraisal of your abilities. If your picture is posted with a "Do Not Sell To" at the local home improvement stores or electronic suppliers, you may be getting in over your head.
If you are going to install professional grade home security system equipment, the installation and programming instructions will presuppose certain levels of knowledge and experience. The equipment is not plug-and-play and there will not be step-by-step instructions or a tutorial on how to design and install a security system. You will, no doubt, have to gain the some of the required knowledge from research and reading. Generally, some understanding of basic electricity (no more than you learned in high school) would be beneficial.
Also, consider whether you have the necessary time available to install a home security system and the attendant learning curve. If you do not have ample time or if you are pressed for time and in a rush, you can avoid costly mistakes and a lot of frustration by having the system installed by a local alarm company.
Second, you'll need to audition the equipment and decide which way of three you are going: hardwired, wireless, or hybrid.
A hardwired home security system will be the most difficult to install unless you are pre-wiring a new or gutted house. It is generally the cheapest for hardware and equipment, but will take the longest to do. One benefit is you can mix and match detection devices to get the best install for your application. You're going to be limited usually to the manufacturer's control panel and keypads for the application. Generally, a hardwired system has less upkeep and servicing over time compared to its counterparts. It'll also never truly become obsolete, unless wiring issues develop over time.
A wireless home security system is the easiest to install but the most costly. The tradeoff in time is device size, limitations of being able to mix and match devices, as well as eventual obsolescence. The main manufacturer's wireless devices are extremely reliable and effective, provided you follow the manufacturer's installation instructions. When "auditioning" equipment, be sure to compare the amount and types of devices offered by a manufacturer. Some offer only a handful of devices, while others offer a complete spectrum that will allow you to customize your install. The maintenance has improved, but sooner or later, you'll have to replace batteries and contend with obsolescence, where you'll have to replace all the devices if you upgrade or update and the new panel doesn't support the units.
Hybrid is a combination of the two, and is quite common now. Generally, most will hardwire all the easy devices, or devices where selection flexibility is needed, and install wireless devices for the difficult or "impossible" locations, or where a single unit would negate the need for 2 hardwired units, such as double hung windows.
Almost every "better" system on the market will have a separate control unit, keypad, siren, and peripherals. Try to avoid installing an "all in one" unit if possible, since their design is less secure than a hardwired separate component system. Of course steps can be done to make an all in one more secure, but it's generally better to go the more secure route to begin with. Unless you're limited by a lease or rental agreement, you're going to be drilling holes, installing screws and running a couple of wires, even for the "all in ones" and tabletop units.
Now that the differences have been generically explained, you'll have to factor in the installation in your premises. You're going to need basic hand tools, a drill, as well as some specialized tools. Long drill bits, such as 12" all the way up to 72" may be needed, as well as a "fish tape" and a wiring staple gun. You can't use a traditional flat staple gun to secure the wires in this case, since that method will damage the integrity of the wire. Romex staples, wire tacks, and other fasteners specifically designed for wire can be used. Generally, the purchase of a good "crown" staple gun, such as an Arrow T-25 with appropriate staples will make your job go easier and neater.
On top of the additional specialized tools, you will need to be comfortable drilling and making holes in your house or property. The potential for "collateral" damage is high, even for an experienced installer, since most times, you can't see what's inside the wall/ceiling/floor where you are drilling. Plumbing, electrical, and ductwork can all show up in places where you don't want it. The main thing in this case is to always look prior to drilling wherever possible. A small knowledge of how a house is framed and put together is also somewhat beneficial, but not 100% necessary.
After all of this, not that we're trying to sway your decision, you'll have to factor in the knowledge of the equipment and potential for false alarms from improperly installed or located equipment. Also, if the equipment needs to be moved, you'll also have some repairs to contend with.
Part of the cost of installing the system is, most definitely, the labor and insurances involved in doing such work, as well as the cost of the equipment, tools and expendables. The other part of the cost is paying for the knowledge of a good installer for proper device selection and location, as well as programming the system for proper and appropriate operation. It's a do-able process, but part of the discount of DIY involves the discount you receive on the labor. Nothing is worse than an improperly installed system that constantly falses and is no longer used because of those reasons.