This boils down to simple mathematics. Most of those DJs worked their asses off to build a name for themselves and have a decent fan base that supports them. When a promoter thinks about booking a DJ, they think about how many people they can bring to the show with their popularity, and how hard they promote the event. Notice I didn't say skill.
I'll use myself as an example: A promoter knows that even at my lowest draw, I can guarantee that at least 30 or 40 people will show up to either see me spin, or they know that the type of show I'd get booked at is something they'd most likely enjoy. Plus I hype the show up and post about it online a bunch, so even though some people may not recognize the rest of the lineup, they now know what type of show it is going to be. You'd be surprised at how many people only know their local DJs, but not the major headliners.
30 people paying $20 for a party makes $600 for the promoter, at the absolute least. Since my booking fee is $200, they are guaranteed to at least triple their profits on that particular investment, if not more. Usually it is way more, but I'm doing a low estimate. Since my name is recognizable, even if I personally don't do shit to promote their part, or if their party falls on an off day like Monday, or if there is direct competition across the street, they know I'll play a good set, and boost attendance. I'm not trying to sound conceited, it's just the reality of the numbers.
If you can't offer anything besides your DJ skills, I'm sorry to say that you are not a commodity in the eyes of a promoter. There are thousands of DJs in the Bay Area alone, with major talent, but it takes more for this shaky music business to thrive. You need to build a following. Promoters these days, (unfortunately) see this as a business before anything else. Some of them are so burned out, and out of touch, that they wouldn't even know good underground music if it hit them in the face. Hell, it hits them in the face all the time. You know how many people submit their music to them daily? Promoters look at numbers and popularity. They are so consumed by it that all they really know is what's popular. They'll check your twitter account before they listen to your music. I fucking guarantee it! I have a saying: "Never trust a promoter who doesn't DJ."
As a fan, if you want to do something about it, then stop supporting promoters that book the same people all the damn time, and let them know why. You should be spending $25 on a well rounded lineup. Not just one huge headliner and a bunch of people you've seen 30 times before. Once promoters see they are losing business because they are not diversifying, they will change their business model. Keep supporting the grassroots events like Ritual or Wormhole, who continue to support the community by giving newcomers, and unknowns a chance to shine.
I wrote another article similar to this, but in that one, I wrote about why oversaturation and repetitive bookings can be a negative thing. You can read it here: LOCAL DJ OVERSATURATION