Andrew Wilkinson

Random Ramblings on Programming

Accessing FitBit Intraday Data

with 35 comments

JoggingFor Christmas my wife and I brought each other a new FitBit One device (Amazon affiliate link included). These are small fitness tracking devices that monitor the number of steps you take, how high you climb and how well you sleep. They’re great for providing motivation to walk that extra bit further, or to take the stairs rather than the lift.

I’ve only had the device for less than a week, but already I’m feeling the benefit of the gamification on FitBit.com. As well as monitoring your fitness it also provides you with goals, achievements and competitions against your friends. The big advantage of the FitBit One over the previous models is that it syncs to recent iPhones, iPads, as well as some Android phones. This means that your computer doesn’t need to be on, and often it will sync without you having to do anything. In the worst case you just have to open the FitBit app to update your stats on the website. Battery life seems good, at about a week.

The FitBit apps sync your data directly to FitBit.com, which is great for seeing your progress quickly. They also provide an API for developers to provide interesting ways to process the data captured by the FitBit device. One glaring omission from the API is any way to get access to the minute by minute data. For a fee of $50 per year you can become a Premium member which allows you do to a CSV export of the raw data. Holding the data, collected by a user hostage is deeply suspect and FitBit should be ashamed of themselves for making this a paid for feature. I have no problem with the rest of the features in the Premium subscription being paid for, but your own raw data should be freely available.

The FitBit API does have the ability to give you the intraday data, but this is not part of the open API and instead is part of the ‘Partner API’. This does not require payment, but you do need to explain to FitBit why you need access to this API call and what you intend to do with it. I do not believe that they would give you access if your goal was to provide a free alternative to the Premium export function.

So, has the free software community provided a solution? A quick search revealed that the GitHub user Wadey had created a library that uses the urls used by the graphs on the FitBit website to extract the intraday data. Unfortunately the library hadn’t been updated in the last three years and a change to the FitBit website had broken it.

Fortunately the changes required to make it work are relatively straightforward, so a fixed version of the library is now available as andrewjw/python-fitbit. The old version of the library relied on you logging into to FitBit.com and extracting some values from the cookies. Instead I take your email address and password and fake a request to the log in page. This captures all of the cookies that are set, and will only break if the log in form elements change.

Another change I made was to extend the example dump.py script. The previous version just dumped the previous day’s values, which is not useful if you want to extract your entire history. In my new version it exports data for every day that you’ve been using your FitBit. It also incrementally updates your data dump if you run it irregularly.

If you’re using Windows you’ll need both Python and Git installed. Once you’ve done that check out my repository at github.com/andrewjw/python-fitbit. Lastly, in the newly checked out directory run python examples/dump.py <email> <password> <dump directory>.


Photo of Jogging by Glenn Euloth.

About these ads

Written by Andrew Wilkinson

December 30, 2012 at 1:22 pm

35 Responses

Subscribe to comments with RSS.

  1. Thanks so much for writing this (and for updating it…) Very helpful to look at my raw data.

  2. I am hoping to use Andrew’s library. Would anyone please help me skip steps in learning how to use Git and Python by telling me why I get a syntax error when I run this code in Python Client?:
    python /examples/dump.py

    rickpack

    April 21, 2013 at 3:18 pm

    • I am giving up now, but at least I learned a fair amount about Python and its programming community. Anyone who wishes to use Andrew’s module and comes from an inexperienced position like me might experiment with the latest version of Python 2 instead of Python 3. If you decide to go with Python 3 like I did (latest final version is the greatest?), this site can help with installing modules that may be necessary to use Andrew’s module: . Also, errors encountered on commas appear to be fixed in Python 3 by replacing the comma with an open parenthesis and then closing it appropriately. I hope this saves someone the few hours I spent bumbling around.

      rickpack

      April 21, 2013 at 8:37 pm

      • Hi,

        Sorry to hear you were having trouble. I haven’t tried it on python 3. I’ll try to find some time this week to make it work on both 2 and 3.

        Andrew

        Andrew Wilkinson

        April 21, 2013 at 9:31 pm

      • I had the Problem that running it as described above yielded the following error:

        Traceback (most recent call last):
        File “examples/dump.py”, line 14, in
        import fitbit
        ImportError: No module named fitbit

        I moved the dump.py from the example folder to the parent folder, so it lies beside the fitbit folder.
        I now run: python dump.py

        works. Python is 2.7 on OS X (from macports)

        Heine

        April 22, 2013 at 8:15 pm

      • Hi,

        I’ve updated the script so it works with both Python 2 and 3. If you do a git pull it should start working for you. Let me know if you have any trouble!

        Thanks,
        Andrew

        Andrew Wilkinson

        April 25, 2013 at 9:18 pm

  3. Thanks for your work.
    I had a problem due to the time format. As I’m European I use 24hour Time Format.
    That leads to a crash: ValueError: time data ’23:16′ does not match format ‘%I:%M%p’

    I had to change line 104 in client.py to the following:
    timestamps = [datetime.datetime.strptime(e.attrib['description'].split(‘ ‘)[-1], “%H:%M”) for e in elements]

    now it works like a charme.
    Thanks again!

    Heine

    April 22, 2013 at 8:19 pm

    • Hi Heine,

      I’ve made a couple of fixes so you should be able to run the dump script as I wrote in my blog post, and it will support the european time format.

      Thanks for letting me know about the issues!

      Andrew

      Andrew Wilkinson

      April 25, 2013 at 9:13 pm

  4. Thanks Andrew, your script is a great relief for a python n00b like me. I got it to export my intraday fitbit values for this year back until feb 7, 2013. I also have older data (feb 6 and before that). How can I get the older data as well? The script ended without error message. Thanks!

    Holger

    May 9, 2013 at 10:43 am

  5. Love the github code, Andrew! Thanks for sharing this! I was wondering if you are plotting the data or using it in any way? I’m thinking of using scipy/matplotlib to work up some of the data. Wondering if you have any tips?

    Big cheers!
    -Allen

    AllenH

    June 19, 2013 at 6:33 am

    • Hi,

      No, I’m just saving the csv files for backup purposes. The fitbit website had enough graphs for my needs!

      Andrew

      Andrew Wilkinson

      June 19, 2013 at 7:02 am

  6. Hi,

    I have been trying this out today and it looks like Fitbit may have changed things again. I get data for sleep, calories etc. but not for steps. This is using the dump example.

    Is it still working for others?

    Thanks

    Al

    Al

    August 27, 2013 at 8:53 am

    • Hi Al,

      I checked my data last night and the steps data is downloading correct. Can you confirm that the other csv files have data in them, but steps.csv is just full of zeroes?

      Thanks,
      Andrew

      Andrew Wilkinson

      August 29, 2013 at 7:59 am

      • Hey Andrew, I’m having the same issue where the steps data is an empty file and then the other files are downloaded but only contain 0 values any suggestions?

        Paul

        October 16, 2013 at 4:42 am

  7. Hi there, just to confirm that the steps csv is full of zeroes also (it’s the only one, all others have data in them).

    Thanks for your time putting the script together though.

    R

    Richard

    November 10, 2013 at 3:23 pm

    • HI again,

      Just to say that I fiddled a little with the code, and commented out (see below) the condition in the dump_day method that checked for zero steps to continue on, this then populated the steps.csv file for me correctly (when compared with the same fitbit data on the dashboard).

      # Assume that if no steps were recorded then there is no data
      #if sum([s[1] for s in steps]) == 0:
      # return False

      Thanks again,

      R

      Richard

      November 12, 2013 at 11:15 pm

  8. Not sure if anyone is still reading this, but there’s no issue tracker open on Github….

    Was initially getting “NameError: global name ‘HTTPError’ is not defined” — which may be caused by differences between Python versions of versions of urllib; adding “from urllib2 import HTTPError” to client.py helps.

    Now getting “NameError: global name ‘HTTPError’ is not defined” — which seems to be because Fitbit is now returning the HTML ellipsis character in its results, which is not defined in XML and thus not understood by xml.etree.ElementTree…. Not sure what the fix is there.

    Whatever the cause, it’s successfully grabbing data for the first day (ie, today), then dying. Looks promising otherwise; thanks for keeping this alive!

    TD

    January 10, 2014 at 4:25 pm

    • Hi TD,

      Thanks for pointing this out, I have just pushed a fix for this to GitHub.

      Thanks,
      Andrew

      Andrew Wilkinson

      January 14, 2014 at 9:17 pm

  9. Hi Andrew,

    thanks for your great script to get the fitbit data.

    Additionaly i want to use the dump2sqlite.py script. But i have problems in indentifying the proper credentials from the cockies (Firefox 27.0 both on Win and OSX 10.9.1). It would be very helpful to get assistance for gaining the information.

    Thanks.

    Uwe

    February 15, 2014 at 1:06 pm

    • Hi Uwe,

      I’m afraid I don’t use the dump2sqlite script so I’ve not updated it. If you look through the commits I’ve made to the other script it should be relatively straight forward to apply them to it as well.

      I’m happy accept pull requests!

      Andrew

      Andrew Wilkinson

      March 3, 2014 at 9:26 pm

  10. Andrew, thank you for this great post. I was successfully capturing intraday data, however, when I went to run the script today, I received an invalid username or password error. I haven’t changed my credentials or the process by which I run the script. I was wondering if you had any thoughts on this. Any guidance would be greatly appreciated.

    Thanks,
    Jonathan

    Jonathan

    March 3, 2014 at 3:27 pm

    • Hi Jonathan,

      Is this still failing for you? I’ve just checked my download process and it’s working fine. Perhaps there was a temporary issue at fitbit? How often are you running the script? If you run it too often them you might get blocked.

      Andrew

      Andrew Wilkinson

      March 3, 2014 at 9:21 pm

      • Andrew,

        Thank you for the guidance. I hadn’t ran it in a few days; however, I tried an alternative account and the script ran fine. Is there any simple way to get my account “unfrozen”? Or, will it eventually unfreeze?

        Thanks again,
        Jonathan

        Jonathan

        March 4, 2014 at 12:29 am

      • I’m now seeing the same issue. I suspect FitBit have changed something on the site. I’ll try to fix it this week.

        Andrew Wilkinson

        March 4, 2014 at 4:03 pm

  11. Thanks Andrew. Much appreciated.

    Sincerely,
    Jonathan

    Jonathan

    March 5, 2014 at 5:58 pm

    • Hi,

      I’ve just committed a fix so if you git pull you should be good to go. Fitbit have started redirecting users to an https site, which broke the script.

      Andrew

      Andrew Wilkinson

      March 8, 2014 at 2:14 pm

      • Andrew,

        Again, thank you so much. The fix worked perfectly.

        Jonathan

        Jonathan

        March 10, 2014 at 10:00 pm

  12. Hi there. This is amazing. It is just what I have been looking for. Everything else has been a huge nightmare but I got this one working within minutes. Thank you!

    katystreet

    March 25, 2014 at 11:49 pm

  13. Thanks for writing the library, I used the dump.py example to get intraday step data for visualizing in Processing. You can see a screenshot here: https://www.flickr.com/photos/watz/13938381021/

    If you want my Processing code I’ll be happy to send it to you, I’ve linked to the GitHub repo wherever I’ve posted it.

    mariuswatz

    April 21, 2014 at 4:58 am

  14. […] struggled for a while to figure out how to download my data. Finally I found this post which outlines how Andrew Wilkinson does it in python and includes a link to his code. His code […]

  15. Hi Andrew, I was using your (fixed) script to obtain intraday data from fitbit. Unfortunately the script stopped working. It looks like opener = build_opener(HTTPCookieProcessor(cj)) in the login method returns a http 500 error.

    Chris

    June 20, 2014 at 11:39 pm

    • Hi Chris, yes, FitBit seem to have changed something that has broken the script. I’ll try to fix it in the next few days.

      Andrew Wilkinson

      June 21, 2014 at 5:52 pm

    • This problem has been solved. Just git pull and you should be good to go.

      Andrew Wilkinson

      June 23, 2014 at 8:38 pm


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 102 other followers

%d bloggers like this: