I did all of the mods that you are talking about to a N6.0, oops, not the addition of the Boom. I worked with the boomlet and never had a problem flattening out the main or getting the sail shape i wanted, or close enough for my skill level.
That being said, I just threw a boom on my N5.0 to get a little more control of the foot of the sail and for easier mast rotation control.
As far as the Sq Top Main and self tacking Jib, Awesome! No helm problems but much better control of the power in the sail. A very small amount of weather helm upwind.
The problem that you will run into is when you add the Kite. The boat will have way too much power up front. I was always pushing on the stick. On a lake with fairly smooth water, not too bad. But in the ocean, with big waves that you are constantly climbing and surfing, and the apparent wind swinging for and aft, it was a lot of work! Also, the boat wanted to dive the bows unless the mast was raked waaay back.
I now have an I20(ie, more modern hull shape) and the things that I notice other than the buoyancy in the bows are, Balance under Kite, and it seems that the wide hulls allow the boat to pick up speed much faster than the 6.0. The 6.0 had a much smoother ride, and went through waves like a hot knife through butter(intermittently more whetted surface). The I20 (more modern hull shape) sits on top,(less whetted surface). The I20 will turn on a dime, losing very little energy, whereas the 6.0 had to push her rounded hulls sideways through the water, losing a lot of energy. Not a huge deal until the top of your boat tries to accelerate much faster than the bottom of your boat wants to go!
A good friend had a 5.8 with the same mods, we were always very close. In fact, he got GPS readings of over 28 mph for about 10 seconds on his 5.8 under Hooter, double trapped, (he was pushing 270lb, his crew 160lb), and then an incredibly quick and spectacular pitch pole. I think winds at the time were around 25-30mph. I know, GPS reliability. But that's what it said.
On both boats, Upwind, the mast would keep its shape pretty good. The 6.0 had double diamonds. (after I got caught in winds clocked at the ranger station 1/4 mile away at 56mph. We were triple handed and double trapped. Snapped a Single diamond wire and of course, the mast. Insurance bought me a NA mast with double diamond wire.NICE!). The double diamond handled the Kite, no problem. I could have hung the kite off the mast head with no problems.
The 5.8 mast( and also my earlier N5.7 with the same setup) on the other hand, would take a hard left turn at the Hound. Neither ever snapped, but it was uuuugly! I would estimate that I had seen the 5.8 mast bend off +5-6 feet under kite and double trapped. When they finally looked up and saw what the mast was doing, they backed off.

I guess what im getting at is, congratulations on the new boat! You will have a blast! Do all the mods you want and do them well and well thought out. Have a blast on your new 5.8. Its an awesome boat. You will come out of it with a greater understanding of what makes these boats tick, and what doesn't.
But do it with the understanding that you could more easily sell the boat and for the money you would have put into it, grab a newer F18 with three sets of sails and a faster design and a clean tramp.