The Nature of Theory

So what is "theory", or more broadly: what does a theorist do?

"What's the good of Mercator's North Poles and Equators,
Tropics, Zones, and Meridian Lines?"
So the Bellman would cry: and the crew would reply
"They are merely conventional signs!

"Other maps are such shapes, with their islands and capes!
But we've got our brave Captain to thank:
(So the crew would protest) "that he's bought us the best--
A perfect and absolute blank!"

And I don't mean the endlessly inane "but it is `just a theory'" that one hears from the media and certain idiots about broad fields of intellectual activity with decades or centuries of established foundations; I don't have the energy for that during finals week... I mean, what is done when a "theorist" does a "theory" on some small(ish) problem in a sub-field.

This is a slightly thorny issue for a couple of reasons: within physics, particle field theory and in particular the infamous quest for "the Theory of Everything" has somewhat appropriated the phrase - "theoretical physics" has a dangerous tendency to be synonymous with "particle theory". In astrophysics there is a similar tendency for "theory" to be synonymous with "cosmological theory" or at least theory of large scale structure formation. And despite my strange appearance as a founding member of the Grand Challenge Cosmology Consortium, the one thing I am not is a theorist on large scale structure (well, I sometimes backslide, but only when lead astray by fun collaborators).
The other reason is that, in my ever so humble opinion, there are several different ways of doing theory - whereas a distinct number of my colleagues, at least at the knee-jerk response level, might contend there is only one way to do theory (their way of course; naturally, my way is way better...).

So, what are we doing? Well, there are two root approaches:
explain an observation - "postdiction" - sadly that is what we do most of the time, mostly due to a horrendous lack of imagination and bravura by theorists. This is sorta easier, and sorta more boring, but in reality what most of us do most of the time. Even if only inadverdently. And so it should be; science is data drive, we "observe reality", in so far as we care to ponder anything beyond naive positivism (philosophy is for beer drinking, working requires pragmatism or at least enough of a disconnect to separate the subjectivist or theosophic quandries from the practicalities of actually getting on with it). And, theories, quite properly ought to be tested against reality where practical (some interesting theories can't be, yet, but that doesn't mean they're not interesting; ok, they may be more metaphysical, but who are we to deny the reality of metaphysics).
or, sometimes, we "predict". This is fun, exciting, and sometimes testable. It is also almost always incorrect. That is why theorists who make correct predictions tend to get famous and rich, or at least famous, or maybe ignored and rudely treated, but at least they have the internal satisfaction of knowing they were right all along. Or possibly just delusional.
Prediction, famously, is hard.

Ok, so "how" do we do this?
Well, this is where it gets tricky:

there is the "algebraist" to borrow a useful phrase from Banks. These are people who know all the special functions in Abramowitz and Stegun; they recognise the most obscure ODEs and their closed form solutions, and they are wizzes at plowing through many pages of eqns (or Mathematica output) and keeping track of all the factors of 2 and signs.
These are very useful people; this has historically been a very good way of solving problems. Simplify it to a known PDE with initial or boundary conditions, look for closed form special solutions, or guess (hopefully unique) solutions.
If it works, you either have a solved problem, or at least an approximate solution that captures the qualitative nature of the true solution.
The catch? To get to a solvable problem, you often have to make so many approximations or simplifying assumptions, that you're solving an irrelevant toy problem. Bummer.
My attitude is that you want at least one person like this in a department or research group, and to judiciously collaborate with them if the chemistry is good. Some places and people feel everyone should be like this, and that is what they produce.

there is the computer modeler. So we don't want to overapproximate, but the problem is intractable. No problem, we brute force it (or better still invent improved solution schemes) numerically. There is an increasing tendency to do this as computers become more powerful and easier to use with black box codes. I have a deplorable tendency to do this myself.
When it works, you not only discover something about the problem, maybe even a solution, you also get beautiful graphics that can outshine reality (proudest moment: carrying a b/w printout of a simulated system, and have an experienced observer look over my shoulder and say "which system is this?".
The problem: well, oversimplification of assumptions - this is more insidious in some ways then in the "find an ODE" approach. Numerical subtleties (errors) especially when using "black box" codes other people supplied. And a tendency to "parametric studies" (which can degenerate into "salami science"). Since you've gone to the trouble of getting the code going, debugging, generating initial conditions etc., once you're good to go, you can just keep running while tweaking the initial conditions.
Case study: a cruel but correct referee's report I got, which noted that I had "predicted" all possible behaviours of a particular system, and there was not testable prediction left... Ouch. Quickly corrected of course, since the idea was to see which part of the explored parameter space was consistent with eventual observed outcome (ie future data would test which model of all possible models was correct), but the attitude that had crept in was bad. Bad theorist. Oh, and there is a philosophical trap for people to fall into, when they start agonising over "emergent behaviour" and "complexity". Beware of buzzwords. As with subjectivists, there is a good point there, maybe even a coffee point rather than beer point. But lets keep it under control shall we.

perturbed individuals: these are people who did too many quantum mechanics homework problems. You look for an exact theory, and then look for a perturbation solution to the approximate problem. Or some series expansion in a funky space where you're guaranteed to have some solution spanning your problem. Guilty. It is too easy, for well behaved problems you're guaranteed to make progress (so don't try this with quantum gravity...). But this approach has limits, and may give misleading results. It also leads to the Hunting of the Lagrangian - a fun but sometimes futile endeavour, which at least leads to papers being published. It is not true that the universe can be described entirely if we just find the perfect Lagrangian. The subtleties of mid 19thC mechanics are elegant, but it is a formalism with some limits.

phew, almost done: so, what remains (except whatever bits I forgot)? Well, in astrophysics, the "order of magnitude" approach can work wonders, and is on a given day, my favourite. It sometimes works like magic. It sometimes fails like magic. It is also a "school of theory" approach, you can see the lineage in many papers (I like to think, immodestly, as an offspring, not yet much of a progenitor). In the purest approach, it is a matter of writing down ab initio the equations governing a system, and abstracting the qualitative solutions, without necessarily solving the full dynamical or asymptotic state. In its crudest, it is an estimate of which scales matter, and converting derivatives to ratios (this sometimes works in astrophysics, drives some other physicists nuts, it is fun), and getting out a number, any number...
If you do it right, you may suddenly find a solution accurate to 15 significant figures. If you do it wrong, you may be off by a factor of million or more - (we'll ignore the cosmological constant here). In messy physics there is the lesson of the Reynolds number. Scaling isn't everything.

Enough for now, I'll rethink and regurgitate on this some other time.