I helped start an anime convention (AWA) in Atlanta in 1995, before Japanese cartoons were really any sort of pop culture thing, and it's still going, so I guess we did something right. I also helped organize a UFO/conspiracy culture/hacker show in 1991-1992 that kind of fizzled out, so I have a little experience in what NOT to do, as well. I now live in Toronto, and in both cities I've staffed fan conventions and seen shows rise and fall for various reasons.
You want to start with a small, manageable show. When I say "manageable" I mean a show you can manage physically, in that you can personally make sure all the advertising is distributed, all the guests show up, all the events happen on time, the dealers have what they need, the attendees get their badges, and registration gets their money. I also mean a show you can manage financially, in that you can pay for it without a second mortgage or having to hope an extra 500 people show up on Sunday.
When we started AWA, we budgeted assuming we'd get 200 people. We figured there were 200 people in the fandom community of our city that could be counted on to show up to just about any fan convention. We spent as little money as possible on our first year. Our program books were black and white copies assembled by hand fanzine-style, and our guests were industry folks, comic artists we knew, and local fans, all whom would show up without asking for speaking fees. We booked a very cheap facility and only booked the exhibit space we needed and we didn't have T-shirts or a con suite or any of the catering events that hotels love to throw at conventions. When the convention started on Friday afternoon, before the first walk-in bought the first walk-in badge, we'd already paid for the entire show via advance registration and vendor table fees.
It's vital to honestly estimate how many people will come to your show. You can base this on how well other first year shows do in your market, and other metrics like population, other fan events in the area, nearby colleges, etc. You'll want to lowball that estimate and spend money based on your lowest figure. It's much better to have booked a slightly too-small function space and be crowded and busy; the event seems more fun, more exciting, and it makes your show look attractive and successful, instead of a few fans wandering around inside a cavernous, empty convention center that got booked because the organizers fooled themselves.
Everything is negotiable; hotel rates, convention center rates, guest speaking fees, everything. Hotels will give you better rates on meeting space if you can promise (and sell!) a certain number of room nights. Convention centers aren't as flexible and will generally cost more. Convention centers may require you to engage further contractors to handle table setup, AV equipment, etc. Get quotes and do research while working out your costs.
I should mention the ridiculous turnover rate in the hospitality industry. It's nuts. People cycle through hotel positions with astounding rapidity. Speaking to the same person more than twice in a row is rare. If you do establish a good relationship with somebody at a convention center or hotel, hang onto it.
And of course get everything in writing. Make sure you have a printed copy of the contract in hand from the very beginning of setup to the very end of take-down, and that includes maps and diagrams of how every room is to be set up, table and chair layouts, power drops, everything. Count on it - there WILL be problems with the facility where everybody isn't on the same page, and having that page in your hand is a tremendous asset.
Plenty of people - potential guests, vendors, fan groups, local celebrities, you name it - will approach your show with what they think are great ideas for events or appearances. Generally they will want you to do most of the actual work while they reap the benefits, so be wary. Filling panel slots with fan-run events is a great way to round out your schedule and involve the community; just make sure to be very clear on what your organization is providing and what they will need to provide themselves. Most conventions have some sort of panel event form that panelists submit, and this is a great idea to keep everyone on the same page.
Again, you'll have countless people coming up to you with great ideas that you should implement. Well, "you" are busy with other things. Keep focused on YOUR goals. There's always next year for these people and their extraneous events. Of course sometimes these volunteers wind up producing terrific events and becoming valuable members of your team. Gambles sometimes pay off. Have a back-up plan for when they don't.
Like the saying goes - never buy what you can rent, never rent what you can borrow, never borrow what you can steal. You don't need a state of the art HD surround sound video room, you don't need to buy 10 TVs and 30 video game systems, you don't need to cater meals for 300 people all weekend long (these are all things people will ask you to do for them. "No" works great as an alternative). If you buy AV equipment, consoles, speakers, or anything else, guess what? They're yours now, and they have to be somewhere the 51 other weekends a year you aren't running a convention. Unless you own a warehouse, that costs $$$. There are many fine event equipment rental outfits in every large city - some of them probably provide services to your favorite conventions. Do your research.
In my opinion, one of the most successful shows I've seen is TCAF in Toronto; it's a free-to-the-public event held over 2 days in the main branch of the library downtown. TCAF works with the city and receives support from the library and other municipal organizations. The vendors and exhibitors are curated heavily and they do a ton of business, and the general public can wander through and enjoy themselves without having to stand in line for badges. Not every city can handle a TCAF but their model is worth emulating, I think. If you can work with the city and put together a package that appeals to the community at large, that's a win for everybody. Don't be afraid to think outside the box and move beyond the typical comic-con cliche.
What about guests? Here's the big secret about guests: they don't really matter that much. Sure, they look great on your advertising, but with few exceptions that's their only benefit to you. It's a great deal for THEM - they get a free airline trip, a hotel room and meals, events where they can pretend to be big stars, and a table they can sell autographs and merchandise from. For you, it might be nothing but headaches and babysitting. You need to do strict cost-benefit analysis on every single one of your guests and be blunt about how much they're going to cost you and how much your show will benefit from their appearance. Do you want to shell out travel and lodging for a guest whose big Q&A panel only attracts flies? Certainly, there are guests who are worth the time and money, and part of running a convention is knowing who those people are and how much trouble they're going to be to get. Pay attention when visiting other conventions and see who clicks with the crowd and who doesn't. If you honestly have no idea of how big a draw a particular guest is going to be, perhaps running a fan convention isn't for you.
The truth is, fans will attend a fun event no matter who the guests are. You aren't selling a guest or a book or a physical object, you're selling an intangible - an experience, the experience of shopping in a dealer's room, of seeing people in costumes, of participating in discussions with fellow fans about favorite movies or TV shows or comics, of playing games with friends and strangers. Guests are only a small part of that package.
What about competition? In terms of another organizer putting on a show in your territory, your best response is to make YOUR show the BEST show you can possibly make it. Let the other guy go his own way. If you must mention other shows, make every public statement polite and professional. If you're going to promote your convention at his event, then be friendly and polite and smile and wish him all the best, and in return, make sure his experience at your show is a positive one. Maybe you can turn a potential competitor into an ally. Fan conventions need each other for networking and advertising, for staff, for vendors; conventions don't exist in a vacuum.
Let's face it; running a convention is a pain in the ass. Your phone rings all the time with idiots asking stupid questions, every teeny tiny detail will need your attention, every decision you make will be second-guessed and debated by pedantic fanboys and fangirls and fanwomen and fanmen, and if you happen to make any money at all out of the convention, it will need to be plowed right back into next year's show. The week leading up to the show you'll be a nervous wreck, when the actual show happens you'll be in a state of low-grade, out-of-body-experience panic, and when it's over you'll be totally destroyed, and then the grind starts all over again. If your competition wants to take that on himself, then he's welcome to it. It's a big world and there is room for conventions all over the calendar and all over the map and you can't waste any time worrying about anybody else's convention, you have your own thing to worry about.
So, to sum up:
1. start small and manageable
2. work with the city & community
3. don't be afraid to say no
4. get it in writing
5. cost-benefit analysis on everything
6. worry about your own show, not anybody else's
7. good luck.