Skip to content

October 9, 2011

3

Acupuncture verses Coin Toss

by juju2112

I ran across a lady in my area on Google Plus recommending a chiropractor/acupuncturist:  Dr. Max Norris of Ozark Herb & Spice. She said anyone suffering should try him out. Browsing his website, he lists quite a lot of conditions that he can cure through acupuncture, including:

  • Migraine headaches
  • Stress
  • Diabetes
  • High blood pressure
  • ADD
  • ADHD
  • PMS
  • Menopause
  • Fibromyalgia
  • Anxiety
  • Insomnia
  • Depression
  • Vertigo
  • Tinnitus
  • Epilepsy
  • Arthritis

That’s quite an exhaustive list. But how well do these claims hold up to scientific scrutiny? Being limited in time, I decided to focus on just one claim, depression. I asked Dr. Norris what sort of studies had been done for this. He provided me list of 12 studies backing up the claim. I don’t have time to read up on 12 studies, so I asked which one had the best evidence. Since I never got a response, I decided to pick the very first one.

The Efficacy of Acupuncture in the Treatment of Major Depression in Women. Psychological Science, September, 1998; Vol. 9, No. 5: pp. 397-401.

In this study, the experimenters broke people up into three groups.

  • Specific treatment – these people would receive acupuncture specifically designed to treat depression.
  • Non-specific treatment – these patients would receive real acupuncture, but the treatment would targeted towards something besides depression.
  • Wait list – These people would remain on a wait-list for 8 weeks, and after that receive 8 weeks of treatment.

So, it seems like they made a reasonable effort to blind the study and have control groups to measure against. That’s good.

Results from the abstract:

Following treatments specifically designed to address depression, 64% of the women (n = 33) experienced full remission. A comparison of the acute effect of the three 8-week treatment conditions (n = 34) showed that patients receiving specific acupuncture treatments improved significantly more than those receiving the placebo-like nonspecific acupuncture treatments, and marginally more than those in the wait-list condition.

According to the abstract, they received a 64% success rate out of 33 people.  However, If you read further into the study, it says:

“Patients were randomly assigned to one of three conditions (specific treatment: n = 12, nonspecific treatment: n = 11, or wait list: n = 11) and to one of the acupuncturists).”

So, actually, there were 38 total in the entire experiment. 5 dropped out, leaving 33 total. Only 12 were in the group that got the real treatment.

That sample size seems pretty low. Don’t scientific studies need large sample sizes to have accurate results? Are the results in this study really better than flipping a coin? I’m no scientist, and I don’t understand statistics. Being a programmer, I decided to write a program to test this. This is coded in python. It flips a coin however many times you tell it to and outputs the results.

from __future__ import division
import random

num_heads = 0
num_tails = 0
times_flipped = 0
num_flips = 12

for i in range (num_flips):
    result = random.randrange(2)
    if result == 0:
        num_heads += 1
    else:
        num_tails += 1

percentage_heads = num_heads / num_flips * 100
percentage_tails = num_tails / num_flips * 100

print str(num_flips) + ” iterations”
print str(num_heads) + ” heads,  ” + str(num_tails) + ” tails”
print “%.2f%% heads”  % percentage_heads + “, %.2f%% tails”  % percentage_tails

I ran through 10 tests. Here are the results:

12 iterations
6 heads,  6 tails
50.00% heads, 50.00% tails

12 iterations
7 heads,  5 tails
58.33% heads, 41.67% tails

12 iterations
5 heads,  7 tails
41.67% heads, 58.33% tails

12 iterations
4 heads,  8 tails
33.33% heads, 66.67% tails

12 iterations
7 heads,  5 tails
58.33% heads, 41.67% tails

12 iterations
4 heads,  8 tails
33.33% heads, 66.67% tails

12 iterations
9 heads,  3 tails
75.00% heads, 25.00% tails

12 iterations
7 heads,  5 tails
58.33% heads, 41.67% tails

12 iterations
8 heads,  4 tails
66.67% heads, 33.33% tails

12 iterations
4 heads,  8 tails
33.33% heads, 66.67% tails

Wow! Over half the time, we get results of 64% or more on one side or the other, purely through simulating a random coin toss. I’m not trained in statistics, but this would seem to me to show that, although the 64% success rate seems good, it’s really no better than random chance.

What happens if we increase our sample size of coin tosses to 10,000? See below:

10000 iterations
4929 heads,  5071 tails
49.29% heads, 50.71% tails

10000 iterations
4999 heads,  5001 tails
49.99% heads, 50.01% tails

10000 iterations
4991 heads,  5009 tails
49.91% heads, 50.09% tails

10000 iterations
4955 heads,  5045 tails
49.55% heads, 50.45% tails

10000 iterations
5041 heads,  4959 tails
50.41% heads, 49.59% tails

10000 iterations
4970 heads,  5030 tails
49.70% heads, 50.30% tails

10000 iterations
5037 heads,  4963 tails
50.37% heads, 49.63% tails

10000 iterations
4963 heads,  5037 tails
49.63% heads, 50.37% tails

10000 iterations
4961 heads,  5039 tails
49.61% heads, 50.39% tails

10000 iterations
5074 heads,  4926 tails
50.74% heads, 49.26% tails

As you can see, when the number of coin tosses is 10,000, not once do we get a result higher than 64%. In fact, we don’t even get a result of 51%. Maybe there is something to this large sample size thing.

I’m in no way a scientist, of course. I’m not even good at math. So, I remain humble and ready to admit mistakes if anyone finds any. But it seems to me, with my uneducated opinion, that the results of this study are no better than random chance. If I find the time and am not shown to be completely incompetent in the results above, then I’ll explore the other studies listed.

Link:

http://128.196.99.80/JJBAReprints/Allen%20Schnyer%20Hitt%20Psych%20Science%201998.pdf

Advertisements
Read more from Local Stuff
3 Comments Post a comment
  1. Oct 10 2011

    It’s very difficult to do research on something that is unquantifiable. Although I think that the protocol of this experiment was legit, assessing symptoms of depression is in itself very subjective. If 10 people asked one depressed female how she was feeling in one specific moment, I’d say she’s give at least 5 different answers. You understand what I am saying. If you would like more information you can contact me. Although I have evidence-based information, one must keep in mind that the typical RCT does not suit well the advent of acupuncture research. Insufficient evidence in this sphere does in no way render the practice inefficacious.

    Reply
    • juju2112
      Oct 10 2011

      Yeah, but it doesn’t prove it, either. People are using the study to show people it works when the study doesnt say that.

      I was concerned with the part you mentioned, as well. Not to mention the total lack of randomization of patients between the groups.

      I have an idea of recreating this study exactly, except instead of acupuncture, have something totally ineffective. I wonder of it would give similar results.

      I wouldn’t mind hearing more if what you mentioned. But of course I have 11 other studies in the queue to be examined.

    • juju2112
      Oct 10 2011

      Why arent Randomized controlled trials well-suited to acupuncture?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Note: HTML is allowed. Your email address will never be published.

Subscribe to comments

%d bloggers like this: