I haven't been awfully near the border, but I imagine there's a very large and very conspicuous Border Patrol presence and consequently very little of whatever wild-West shenanigans you might be worried about.
The rest of San Diego and environs are as safe as the equivalent area in any other city. I wouldn't be parking an Escalade on the street at night in a badly-lit industrial area, and the area between La Jolla and Mission Bay is the barrio (with all that entails), but other than that I can't think of any areas I'd explicitly recommend staying away from.
Do be on high alert when driving: coming from the Los Angeles area, I have to say that SD traffic is the worst I've ever dealt with. Here's the thing: LA drivers are a-holes. They drive aggressively, but fairly predictably. SD drivers, by contrast, just seem to be distracted all the time, and as a result they're
very erratic on the road. Lots of no-signal merges, lane drift, etc. If you make it north of Mira Mesa, you'll get to contend with oblivious golfers driving luxury SUVs badly and their elderly mothers driving luxury SUVs even more badly.
Fun stuff to do?
It's neat to watch Ospreys fly around NAS Miramar - just make sure you're near an overpass so you can take shelter in case they start smoking...
The USS Midway is anchored in San Diego Bay and functions as a museum. Haven't been yet, but chances of it not being completely awesome approach 0%.
If you or the kids are into bright colors and goofy motifs (and as long as displays of Mexican culture don't make you froth at the mouth), I highly,
highly recommend you make a visit to Lucha Libre Taco Shop (just call it Lucha Taco for short) in Mission Hills. There's cheap meter parking across the street which goes free after (IIRC) 6PM, and you can reserve the
Champion's Booth online at their website (
www.tacosmackdown.com ) Fair warning: tiny venue and high demand, so grab seating as it becomes available and expect to be in line for ten minutes or so to order. Worth it.
If you have any specific questions, feel free to PM me and I'll try to help you out.