Statistics, Data Science, and Data Mining Discussion Thread (Business Intelligence, Analytics, etc)

[amathew]

Great thread guys. I just came across it. The type of topics discussed on SF never stop surprising me.

In response to the above, how do you differentiate between EFA and PCA? I hear people reference PCA as something different from Factor Analysis quite a bit; my understanding is that it's incorrect to do so, but I'm curious to see what you think.

My understanding is that Factor Analysis (whether Exploratory or Confirmatory) is the general technique. Component Analysis (also PCA) and Common Factor Analysis are two methods of extracting factors for Factor Analysis, not separate techniques.

Let's compare that to Multiple Regression analysis. MR is the analytical technique. Stepwise Estimation and Forward Addition/Backwards Elimination are model estimation methods (similar to PCA's role in Factor Analysis). No one refers to Stepwise Estimation as its own technique; it's one option that you can use to create a MR model, but people (erroneously) refer to PCA as a separate technique from Factor Analysis, rather than one possible extraction method that can be used for Factor Analysis.

Exploratory and Confirmatory Factor Analysis are uses of Factor Analysis for certain ends. PCA is an extraction method for Factor Analysis, not a separate technique "geared towards a similar 'type' of problem."

Thoughts?

When I specified "type" of question, I was merely referencing that FA and PCA were options for a given type of question. For example, to find the indicators of job satisfaction from survey responses. I always considered FA to be an extension of PCA, but that the two were separate techniques. Both FA and PCA use different estimation procedures to identify the structure underlying a dataset. Given that I never use FA or PCA and have a limited working knowledge of both, I'm going to take the easy route and suggest that you have a look at the following link.
http://stats.stackexchange.com/ques...ctor-analysis-and-principal-component-analysi

In regards to your comparison to MR...MR is the analytical technique and least squares, partial least squares, maximum likelihood estimation, etc are the estimation techniques. I don't think of stepwise regression as an estimation technique but a variable reduction technique. Stepwise can be implemented using backwards or forwards selection, but it's not directly related to the MR estimation process...once again, stepwise is something that I don't use, so don't hold me to it.

 

[amathew]

How do you handle communicating complex statistical ideas to people? And how does one get better at it?

I was recently criticized for this after a presentation where I talked about testing for causality and counterfactual analysis without properly describing what that was in very plain terms. Basically, I was told I should try to explain this in such a way that a 5 yr old would understand.

 

[clee1982]



How do you handle communicating complex statistical ideas to people? And how does one get better at it?

I was recently criticized for this after a presentation where I talked about testing for causality and counterfactual analysis without properly describing what that was in very plain terms. Basically, I was told I should try to explain this in such a way that a 5 yr old would understand.

I remember my old adviser use to tell me a joke

In academia if you present something and people can't understand then people assume they're stupid
In industry if you present something and people fail to understand they assume you're stupid

I think everyone who is technical have the same problem (myself included).

 

[kindalonely]

How do you handle communicating complex statistical ideas to people? And how does one get better at it?

I was recently criticized for this after a presentation where I talked about testing for causality and counterfactual analysis without properly describing what that was in very plain terms. Basically, I was told I should try to explain this in such a way that a 5 yr old would understand.

This happens a lot, and I am reluctant to change how I present my findings. Some mathematical maturity is certainly required to correctly interpret my work, and when my audience doesn't have that, I wonder why I am presenting in front of them at all. Obviously in a business setting they don't need to know how the key theorems I use can be proven, but I expect them to be able to understand the statements of the theorems. Theorem A says B, and because of B we have C. It's OK that they've not heard of A (I am sure a lot of practitioners don't know ergodic theory or Wold decomposition), but they should be able to understand B (after I have stated B!) for the meeting to be worthwhile.

When the audience lacks the prerequisites to understand the topic or and methods (e.g. they don't know what t-stats are!), then perhaps all they are looking for is just a plot in Tableau.

But then I just do what my boss asks me to, and I end up in a meeting full of non-sequiturs and inaccurate/wrong claims. It is not that I cannot communicate complex ideas in simple terms, but to replace that one mathematical term I may have to introduce paragraphs and paragraphs of simple ideas to make the statement correct.

When I do coding work the problem is less irritating. I have used this and that for my applications to be able to communicate with each other, and you think applications are somehow born with the ability to 'talk'. People may wonder why it is taking you so long to accomplish a goal that can be stated in one sentence. At least the results are the same, even though your bonus is not.

 

[clee1982]

Don't want to sound too harsh but I think that's "your" problem. When I was younger I used to be somewhat cynical, i.e. I think the only way to correctly present mathematical idea is to give the full pictures, everything else is a "compromise" or half baked truth, but in reality is at the end of the day people wants conclusion and base on evidence they can understand.

 

[kindalonely]

Oh you misunderstood me. My point was that the techniques you use should be accessible. If it's a non-technical audience, just use visualization (e.g. plotting it in Tableau). If it's an OK audience (e.g. math/physics majors), go ahead and present your findings. If it's a bunch of technical PhD's, write it up and send the paper to them. You should only need to dumb things down when you re-present your work to a different audience.

 

[amathew]

So I quit my job about 2.5 months ago and have since had eight job offers for everything from data scientist to business analyst roles. Job market
is insane right now. Of course, I turned down all eight of them and am holding out for something at a tech start-up or ad-tech company in the midwest. Being
picky, but after my last employer, I almost have to be.

 

[brokencycle]



So I quit my job about 2.5 months ago and have since had eight job offers for everything from data scientist to business analyst roles. Job market
is insane right now. Of course, I turned down all eight of them and am holding out for something at a tech start-up or ad-tech company in the midwest. Being
picky, but after my last employer, I almost have to be.

I see you're in Minneapolis too. The unemployment rate around here is near zero. Every retailer I go into is hiring, and I even saw a billboard truck driving around advertising $11/hr to work at Home Depot.

 

[Reggs]

EDIT: nevermind

 

[horndog]

My company is hiring a Data Scientist, someone with a strong stats background, if anyone is interested. DC area, Federal Healthcare IT (Centers for Medicare Medicaid services) contractor mainly.

 

[amathew]

Reviving this, though there doesn't seem to be much interest around stats or similar positions here on SF.

I've since left my old job to be a statistician (data scientist) at a tech company in the Bay Area. Have moved back
to a lot of stuff that interests me like time series analysis, market response models, splines, and causal inference. Fun
times...

 

[otc]

Hadooping that big data through the cloud with your synthetic control methods?

 

[amathew]

Ha...I'm in the medium sized data world thankfully, Makes life a lot easier

 

[clee1982]

I work on small data in the world, just millions... Though the dimension we end up chopping them into make it hard...

 

[Sir Humphrey Appleby]



I see you're in Minneapolis too. The unemployment rate around here is near zero. Every retailer I go into is hiring, and I even saw a billboard truck driving around advertising $11/hr to work at Home Depot.

Is the job market there still that hot?