Accessing FitBit Intraday Data

JoggingFor Christmas my wife and I brought each other a new FitBit One device (Amazon affiliate link included). These are small fitness tracking devices that monitor the number of steps you take, how high you climb and how well you sleep. They’re great for providing motivation to walk that extra bit further, or to take the stairs rather than the lift.

I’ve only had the device for less than a week, but already I’m feeling the benefit of the gamification on FitBit.com. As well as monitoring your fitness it also provides you with goals, achievements and competitions against your friends. The big advantage of the FitBit One over the previous models is that it syncs to recent iPhones, iPads, as well as some Android phones. This means that your computer doesn’t need to be on, and often it will sync without you having to do anything. In the worst case you just have to open the FitBit app to update your stats on the website. Battery life seems good, at about a week.

The FitBit apps sync your data directly to FitBit.com, which is great for seeing your progress quickly. They also provide an API for developers to provide interesting ways to process the data captured by the FitBit device. One glaring omission from the API is any way to get access to the minute by minute data. For a fee of $50 per year you can become a Premium member which allows you do to a CSV export of the raw data. Holding the data, collected by a user hostage is deeply suspect and FitBit should be ashamed of themselves for making this a paid for feature. I have no problem with the rest of the features in the Premium subscription being paid for, but your own raw data should be freely available.

The FitBit API does have the ability to give you the intraday data, but this is not part of the open API and instead is part of the ‘Partner API’. This does not require payment, but you do need to explain to FitBit why you need access to this API call and what you intend to do with it. I do not believe that they would give you access if your goal was to provide a free alternative to the Premium export function.

So, has the free software community provided a solution? A quick search revealed that the GitHub user Wadey had created a library that uses the urls used by the graphs on the FitBit website to extract the intraday data. Unfortunately the library hadn’t been updated in the last three years and a change to the FitBit website had broken it.

Fortunately the changes required to make it work are relatively straightforward, so a fixed version of the library is now available as andrewjw/python-fitbit. The old version of the library relied on you logging into to FitBit.com and extracting some values from the cookies. Instead I take your email address and password and fake a request to the log in page. This captures all of the cookies that are set, and will only break if the log in form elements change.

Another change I made was to extend the example dump.py script. The previous version just dumped the previous day’s values, which is not useful if you want to extract your entire history. In my new version it exports data for every day that you’ve been using your FitBit. It also incrementally updates your data dump if you run it irregularly.

If you’re using Windows you’ll need both Python and Git installed. Once you’ve done that check out my repository at github.com/andrewjw/python-fitbit. Lastly, in the newly checked out directory run python examples/dump.py <email> <password> <dump directory>.


Photo of Jogging by Glenn Euloth.

Advertisements

Author: Andrew Wilkinson

I'm a computer programmer and team leader working at the UK grocer and tech company, Ocado Technology. I mostly write multithreaded real time systems in Java, but in the past I've worked with C#, C++ and Python.

40 thoughts on “Accessing FitBit Intraday Data”

  1. I am hoping to use Andrew’s library. Would anyone please help me skip steps in learning how to use Git and Python by telling me why I get a syntax error when I run this code in Python Client?:
    python /examples/dump.py

    1. I am giving up now, but at least I learned a fair amount about Python and its programming community. Anyone who wishes to use Andrew’s module and comes from an inexperienced position like me might experiment with the latest version of Python 2 instead of Python 3. If you decide to go with Python 3 like I did (latest final version is the greatest?), this site can help with installing modules that may be necessary to use Andrew’s module: . Also, errors encountered on commas appear to be fixed in Python 3 by replacing the comma with an open parenthesis and then closing it appropriately. I hope this saves someone the few hours I spent bumbling around.

      1. I had the Problem that running it as described above yielded the following error:

        Traceback (most recent call last):
        File “examples/dump.py”, line 14, in
        import fitbit
        ImportError: No module named fitbit

        I moved the dump.py from the example folder to the parent folder, so it lies beside the fitbit folder.
        I now run: python dump.py

        works. Python is 2.7 on OS X (from macports)

      2. Hi,

        I’ve updated the script so it works with both Python 2 and 3. If you do a git pull it should start working for you. Let me know if you have any trouble!

        Thanks,
        Andrew

  2. Thanks for your work.
    I had a problem due to the time format. As I’m European I use 24hour Time Format.
    That leads to a crash: ValueError: time data ’23:16′ does not match format ‘%I:%M%p’

    I had to change line 104 in client.py to the following:
    timestamps = [datetime.datetime.strptime(e.attrib[‘description’].split(‘ ‘)[-1], “%H:%M”) for e in elements]

    now it works like a charme.
    Thanks again!

    1. Hi Heine,

      I’ve made a couple of fixes so you should be able to run the dump script as I wrote in my blog post, and it will support the european time format.

      Thanks for letting me know about the issues!

      Andrew

  3. Thanks Andrew, your script is a great relief for a python n00b like me. I got it to export my intraday fitbit values for this year back until feb 7, 2013. I also have older data (feb 6 and before that). How can I get the older data as well? The script ended without error message. Thanks!

  4. Love the github code, Andrew! Thanks for sharing this! I was wondering if you are plotting the data or using it in any way? I’m thinking of using scipy/matplotlib to work up some of the data. Wondering if you have any tips?

    Big cheers!
    -Allen

  5. Hi,

    I have been trying this out today and it looks like Fitbit may have changed things again. I get data for sleep, calories etc. but not for steps. This is using the dump example.

    Is it still working for others?

    Thanks

    Al

    1. Hi Al,

      I checked my data last night and the steps data is downloading correct. Can you confirm that the other csv files have data in them, but steps.csv is just full of zeroes?

      Thanks,
      Andrew

      1. Hey Andrew, I’m having the same issue where the steps data is an empty file and then the other files are downloaded but only contain 0 values any suggestions?

  6. Hi there, just to confirm that the steps csv is full of zeroes also (it’s the only one, all others have data in them).

    Thanks for your time putting the script together though.

    R

    1. HI again,

      Just to say that I fiddled a little with the code, and commented out (see below) the condition in the dump_day method that checked for zero steps to continue on, this then populated the steps.csv file for me correctly (when compared with the same fitbit data on the dashboard).

      # Assume that if no steps were recorded then there is no data
      #if sum([s[1] for s in steps]) == 0:
      # return False

      Thanks again,

      R

  7. Not sure if anyone is still reading this, but there’s no issue tracker open on Github….

    Was initially getting “NameError: global name ‘HTTPError’ is not defined” — which may be caused by differences between Python versions of versions of urllib; adding “from urllib2 import HTTPError” to client.py helps.

    Now getting “NameError: global name ‘HTTPError’ is not defined” — which seems to be because Fitbit is now returning the HTML ellipsis character in its results, which is not defined in XML and thus not understood by xml.etree.ElementTree…. Not sure what the fix is there.

    Whatever the cause, it’s successfully grabbing data for the first day (ie, today), then dying. Looks promising otherwise; thanks for keeping this alive!

  8. Hi Andrew,

    thanks for your great script to get the fitbit data.

    Additionaly i want to use the dump2sqlite.py script. But i have problems in indentifying the proper credentials from the cockies (Firefox 27.0 both on Win and OSX 10.9.1). It would be very helpful to get assistance for gaining the information.

    Thanks.

    1. Hi Uwe,

      I’m afraid I don’t use the dump2sqlite script so I’ve not updated it. If you look through the commits I’ve made to the other script it should be relatively straight forward to apply them to it as well.

      I’m happy accept pull requests!

      Andrew

  9. Andrew, thank you for this great post. I was successfully capturing intraday data, however, when I went to run the script today, I received an invalid username or password error. I haven’t changed my credentials or the process by which I run the script. I was wondering if you had any thoughts on this. Any guidance would be greatly appreciated.

    Thanks,
    Jonathan

    1. Hi Jonathan,

      Is this still failing for you? I’ve just checked my download process and it’s working fine. Perhaps there was a temporary issue at fitbit? How often are you running the script? If you run it too often them you might get blocked.

      Andrew

      1. Andrew,

        Thank you for the guidance. I hadn’t ran it in a few days; however, I tried an alternative account and the script ran fine. Is there any simple way to get my account “unfrozen”? Or, will it eventually unfreeze?

        Thanks again,
        Jonathan

  10. Hi there. This is amazing. It is just what I have been looking for. Everything else has been a huge nightmare but I got this one working within minutes. Thank you!

  11. Hi Andrew, I was using your (fixed) script to obtain intraday data from fitbit. Unfortunately the script stopped working. It looks like opener = build_opener(HTTPCookieProcessor(cj)) in the login method returns a http 500 error.

  12. Hello Andrew,

    In the dump_day function, steps = c.intraday_steps(date) keeps returning zero steps, even though in the fitbit.com dashboard I can clearly see activity at the respective date. This results in no data being dumped, of course.

    Could you please look into it? Am I doing something wrong?

      1. Right, I logged some more data for today and it works now. Might be because yesterday I had just done my first sync and the API wasn’t yet seeing it.

        Thanks for checking!
        Grig

  13. Andrew, I found your script the other day and was finally able to download my data today. Thanks for providing the dump.py script. I am mostly interested in seeing my sleep data, but I’m sure I’ll make use of the rest of it too.

    You indicated that you have a Fitbit One (or had). Did you figure out how to download the information for floors? I’m curious what the values for that are, or how I would find that intraday information.

    Thank you (and wadey),

    Inez

  14. Great piece of code – does just what I want. I’m definitely not a Python developer but know enough about coding in general to get this to work. I’m running Python 3.5 on Windows and found a couple of minor issues in the client.py which I got round by making the following changes (which may or may not be the best solution):
    Line 9: HTTPError = urllib.error.HTTPError
    Line 91: return ET.fromstring(data.strip().replace(b'”…”‘, b'”…”‘))

    Also, whenever the day changes in the Sleep data there was a carriage return missing – which I couldn’t resolve in the code but fixed easily enough by editing the files being produced

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s