It isn't so much the "big advance". I don't hold my breath for such things. I do want readership. Getting published is important, I just think I really did something right with this book. I believe in it (and hate that I'm starting to doubt it).
Sometimes it's not whether your book is good or not, but whether the agent or publisher is a good fit. The more commercial you go (agent/publisher) the more they want market, and lit fic tends to have a smaller market to begin with. Many agents, to be honest, even if their guidelines say they are looking for lit fic, they're really looking for the next Rowling or Grisham. Agents are looking to make a buck, and they're looking for books that will give them that. It's not to say there really is no market for lit fic or contemporary. Jamie Ford sold his. But it's going to take a bit more effort -- so don't let 50, or even 100, rejections get you down. As long as you haven't exhausted the list yet, you keep going. I know it's a cliche, but it really does only need ONE yes. I know as writers we all want to get instant recognition, or 50 rejections do feel like 50 personal denial ("you suck") sometimes, but again, don't let that get you down.
Also, there's nothing wrong with niche market publishers, indies, small presses. You may not have the kind of readership you'd like, but they're legit ways of getting started. And it doesn't mean your book is not good because you're published by a small press. Gosh, I hope people don't think I'm a hack because I'm published by Behler. You can always resell the rights once you get established. Grisham sold his first novel to a small press (which is now out of business). It wasn't until his second book (and a subsequent movie starring Tom Cruise) that he became a household name. And to this date, he still thinks his first book (A Time To Kill) was his best book. So going with a small press doesn't mean your book is not good -- you just need to find the right venue (for the time) for it. I always tell Grisham's story not to comfort myself, but to show that there are many paths to success.
That said, I do know some people don't take me as seriously because I'm not published by Random House or St. Martin's. I try not to dwell on it. To me, it's like saying unless you're employed by IBM or Microsoft, you're not a good IT person. That's a bunch a crock. But I don't really care at this point. Writing is a career for me; at least I can say to them: I'm published, are you?
I just keep hearing "start at the top and work your way down the list of agents that represent your genre" and I am not sure I have done enough of that yet. Which leads me to wonder if it is my letter, my premise or my agent matching skills. or maybe it is just the wrong market for this book right now.
Absolutely start from the top. Work your way down. And as Uncle Jim said, "do it until Hell doesn't want it."
Where you stop is up to you though. When I had about 60 rejections I told myself I'd go with smaller presses. It was a conscientious decision I made. I don't really regret that decision.
But I think you have the right approach, and it's not a bad idea to revisit the query/etc. to make sure you do refine your process. I had 15 rejections (and one request for full that didn't go anywhere) before I realized my query sucked. I reworked it, 12 drafts total. And I kept refining it. By the end of my process my "request" rate was about 1 in 5 -- not that bad, considering my book is niche market stuff (lit/contemporary/romantic/personal journey). I did get impatient and went directly to the small publishers -- and I got bites immediately; it seems like they're more hungry for fiction like mine than the big agents/big houses.
But it's a good thing to reassess your query or approach once in a while. My point is, don't second guess yourself though. I hate to see you ask questions like: "Maybe this story is just not that good..." You should never think that. It's okay if you think your query needs work or rework -- so workshop it some more. Critique it. Totally rewrite it if you must, if you think it's the problem (since I don't know what your rejections say, I have no idea -- but when I reworked mine, I based that on the feedback I got... I even cut out 15,000 words because I had enough rejections that told me the story started too slow...) It's even good idea to workshop your ms. some more if you think it can be improved upon. Did you have betas? Who were your betas? Did you have both writers and readers for your critiques? Do you have published authors read through your ms.?
An interesting anecdote: when I first workshopped The Pacific Between, I got some mixed feedback. Some readers just loved the whole thing, but there was one reader who told me, "in the beginning, the plot didn't go anywhere and your protagonist wasn't really relatable." I got mad (the golden word syndrome, perhaps, or just being a naive jerk). I thought she was wrong because how could she be right if my other readers loved it? Well, long story short, it turned out that she was right, and the agents agreed with her: the book started at the absolutely wrong place. I ended up cutting 15,000 words (7 chapters) and reworked the beginning, and it sold. And secretly I appreciated that first reader for being candid and critical with me, even though I was not receptive at the time.