To cut a long story short, I was upset with the insecurity and family incompatibility of an astronomy career. Therefore, I concluded to quit astronomy research and look for a job in industry. Being an expert on statistics and machine learning, I figured I would encounter little or no problems to find a job outside astronomy. I was grossly mistaken.
Failure 1: The perfect candidate
In my numerous applications and job interviews, a clear pattern emerged: Industry almost always searches for the perfect candidate - the milk-giving, egg-laying, wool-growing pig. They would always request subject-specific experience. I had years of experience in astrophysical data analysis - but what about business data analysis? What about financial data analysis? No? Sorry, we found someone more experienced (less in statistics but more in the origin of the data).
Failure 2: Unrealistic expectations
Closely related to Failure 1, industry often has ridiculous expectations. I remember one job-ad which went quite to the extreme: A consultant company was looking for a physicist/mathematician with a PhD and several years of scientific research experience to be hired as Junior (!) Business Intelligence Analyst.
Failure 3: You start from scratch
Closely related to Failures 1 and 2, industry almost always refuses to acknowledge any previous working experience as a PhD student or post-doc in some 90% of all cases. It does not matter if you spent 5 years on machine learning or data mining - it was in astrophysics but not in the field that particular company is working on. When changing from astronomy to industry, you are essentially expected to fall back on your master degree level, everything you did since being dismissed. That may be fine directly after your PhD but after a few post-docs it is rather annoying.
Failure 4: Unprofessional
This failure hit me out of the blue. I always believed that, because a lot of money is involved, industry would be much more professional than academic science with all its inefficiencies. But this view misses another important aspect. While industry is indeed much more efficient than academic science, I was shocked by the lack of professional expertise that is evident in many companies I have visited - no matter how small or large the company and no matter what its business. Let me illustrate this point with the most extreme example: In one of the (large web-based) companies I applied to, the head of the business intelligence department was an economist without any qualification in data analysis. You can imagine that the job interview had a few hilarious moments when that person attempted to question my expertise.
In general, having years of experience in up-to-date machine learning methods, I was almost fascinated - almost - what infantile and out-dated methods are used by the vast majority of companies. Industry is even worse than astronomy in this respect and I often had the impression as if the last decades of development in statistical methods had actually never taken place! Another example was another head of some development department who said the client wants an answer so he gets an answer - if the client wants a better answer he must come a second time and pay again. This is not really in accord with my own quality standards. I figure that 95% of statistics jobs in industry would either bore me to death or let me go crazy trying to tell people what nonsense methods they are using.
Failure 5: Beggars cannot be choosers
Some companies (say 20-30%) will regard you as a petitioner - and you should feel honoured that they sacrifice some of their precious time talking to you in an interview. You would be surprised what lack of good manners I had to witness during interviews. For instance, one interviewer's mobile phone ringing during our conversation, him taking the call and leaving the room without any word of excuse - even if only for the sake of appearance. It falls in line with Failure 3 that industry tends to disrespect your experience and expertise. Interviewers from the human-resource department can be particularly annoying, because they try to provoke you and play other psychological games with you. In my experience, the mere presence of a human-resource interviewer is already a good indication that this company lacks professionality, or at least does not realize which issues about you are relevant and which are not. In my opinion, the only useful role for human resources in a job interview is to fix the details of your new working contract after the important persons agreed to hire you.
Failure 6: Salary
"Money is not everything" say those who usually have a lot of it. Seriously, money always is an issue, though not always the most important one. An underpayed job can be an option, if colleagues are nice and if the job is interesting. I naively expected that industry jobs would pay much more than research ones. However, reality is different. If industry does not acknowledge your experience, why should they pay you properly? In fact, I actually had two job offers which did accept my expertise but which still did not offer a net gain of income. Earnestly, it is non-trivial to find an industry job which pays better than a full-time post-doc position. You can earn more money, say, in the insurance or banking sector, but any real-world Industry will usually pay you less.
Industry is shockingly unprofessional in many respects and a lot of jobs will never even remotely exploit your skills and expertises. Personally, I am not willing to dismiss my years of experience in data analysis and start as a fresher, casting pearls before swine. Unfortunately, if you insist on the acknowledgement of your skills, this will leave you only with a handful of potential jobs. Those are primarily jobs as Industry Scientist. Those jobs do exist and they are broadly comparable to academic science. However, those jobs are very rare. If you want to find such a job, then watch out for companies that explicitely search for physicists or which have hired physicists before. They will usually know what your skills are worth. You can also filter job-ads for professionality. For instances, a job-ad which asks for "in-depth programming skills" and simultaneously requests "MS Office experience" but no other programming languages is probably not worth applying to.
All in all, I decided to stay in academia after being thoroughly disenchanted with Industry. I came to the conclusion that academic career - despite its insecurity and family incompatibility - is the lesser evil. Especially because many industry jobs are not half as secure and family compatible as you may think. Although, of course, trying the academic career and failing after 5 post-docs makes the fall-back to master level an even harder landing than it is already today. We will see ...