[Stats] Ideas, Requests etc.

[Zethar]

We were discussing stats in #rainwave, and it turns out that people are curious beings, and it can't be the case that I am the only one curious. As a good professor knows, for each question that gets asked in class there are a dozen students with the same question but are too scared to ask in class, so I have decided to make a thread regarding ideas and questions about Rainwave [stats].

I for one, am curious about the following: Is there a correlation between song rating and the number of requests that it gets? If so, how large is it?

While mulling over this problem, I realize there are plenty of issues which need to get tackled first. For example, it's pretty apparent that the longer a song has been in the database, the longer it has had time to accrue requests, so we need to account for time. To account for time, the most reasonable thing to do is to model it as a Poisson process and determine the rate and determine if there is correlation between that and song rating. However, we should then check that a Poisson process is a reasonable model (that is, its other assumptions are not violated, like the fact that the rate doesn't change over time significantly) etc.

I'm not a very good programmer (my code is a dense black box full of lambdas), but I will be receiving my bachelors in statistics later in the year and hope that I have the cooperation of the other members of the community to get these burning questions answered.

[Rob]

If it can be made into an Excel formula, jonfunk can and/or will do it.

[Zethar]

I'm not sure if Excel does Poisson regression, but I would be more than willing to, when I have the spare time, do the statistical analysis needed if necessary.

[Rob]

It sure does: http://support.microsoft.com/kb/828130

[jonfunk]

Good to have a genuine statistician on board! I'm a Civil Engineer that likes to dabble in stats, but it's all very superficial stuff that is mostly finding, visualizing, and plotting Rainwave data. As soon as you start talking about 'check that a Poisson process is a reasonable model', you move past where I'm at and into new and exciting territory. ^_^

I've created a post with an index of previous statsy type things I've done here. Feel free to browse and give feedback!

(And yes, anything I've done in there in terms of analysis or visualization almost exclusively has been with Excel)

[Zethar]

New and exciting stuff indeed. I'm a little busy this week and it'll take some time to actually read everything to give feedback, but hey, I'm sure we can tackle interesting questions.

[jonfunk]

I for one, am curious about the following: Is there a correlation between song rating and the number of requests that it gets? If so, how large is it?

Questions for Rob / William: Do we track the number of requests a song receives? Is it a static number or is there some request table we can use to build the relationship of requests vs. time?

[Rob]

We have both a static number for quick stats displaying, as well as a full table of request history.

[fredy31]

There's probably no column for that but I would like to see the songs that do the most denies.

Basically it's the song that gets voted in the most when its against a request (so denying the request)

[Zethar]

I do apologize for reviving an old thread. I had been busy in the meantime and this mostly had been sitting on backburner, and this is still the most relevant thread for this.

After a recent comment by someone elsewhere after seeing me tear through data, I would like to make a slightly unorthodox suggestion: provide the statistician with all the data, and see what models and data could be gotten from it. This would give me a better understanding of what sort of questions are within our ability to answer given that information.

There's a lot of information and questions which are to be answered about people's music tastes, the popularity of songs, etc., but without knowing what we have there is virtually no way to start tackling those questions, and it would be a shame if it turns out that the answers are within grasp, obscured.

[Rob]

Zethar: We're currently debating the ethics of sharing our entire database in the moderator forum. With API calls, it'd be trivial to de-anonymize users. It's arguable that it's only rating/favourite data on an online radio site, but trust is important regardless of the perceived importance of keeping the data private. Since we don't have a privacy policy at all, let alone one that states we can share your (anonymized or not) data, I am personally not inclined to open the entire data set to the public. To only you is a different question that we haven't decided on yet.

(for the record, William and I are the only ones with full access to all data, and jonfunk is the only 3rd party to have received a full data dump)

[Zethar]

I understand the concern. Thank you for taking the time and effort to carefully consider the ramifications of such courses of action. Should you decide on allowing me access to any of the data, please have it on my word, oath, and legal signature that I shall not share the data with anyone else.