You're pretty much right when you say it can be trail and error to develop a niche. Some just happen to stumble across them by mistake and it takes others many, many hours trying, and there is of course absolutely no guarantee of success at all, no matter how much work you put in.
My own personal opinion, and there will be plenty of those that will disagree, but with the popularity of research tools such as Terapeak, and the massive influx of people wanting to get into the market, it makes it almost impossible to maintain a niche market for very long at all.
If you are selling a product that others have the ability to find, they'll be looking for it as soon as their research identifies the under supplied market that all your hard work has discovered, and there goes the niche.
I call it Tow Trucking or Plowing the Field, and there's nothing wrong with that. We're all in business to make money and we should all be using any tool available to us to achieve it!
I look at areas such as hand made products, something very unique as niche type products now, something that you have an inside line on, but not something the other 200,000 can get easy access to.
I've always been a fan of second hand, especially on eBay. You can find all sorts of things are garage sales (yard sales), and pay very little for them, even less if you can talk a good deal, and there you have a chance of having products for sale that others can't, or aren't even interesting in competing with as it requires them to actually leave the PC for a few hours and use their legs.
So many seem to get into this mind set that just because they are selling online, they have to source online, and then they all wonder why it's so hard to find a product to sell with any sort of margin left. If you're a gold fish in a bowl with 10,000 others (it's a very large bowl), and there are only 3000 fish flakes available, then your chances of getting one of those flakes are great!
The good news is we're not gold fish, we even have legs, and sometimes they can be of advantage in this game lol