Mark Koester

Personal Blog. A Data-Driven Life. Minding the Interstitial Spaces since 2007.

How to Export, Parse and Explore Your Apple Health Data With Python

Most of us walk around carrying a small, sensor-infused computer. We call these devices “smartphones,” and they have more computing power and memory than the Apollo Space Capsules did when they went to the moon. Our phones contain sensors that detect movements, determine magnetic north, and even pinpoint us in relation to rotating satellites.

Our smartphones are incredible mini-trackers that can be used for both good and bad. On the good side, they can be used to help us know more about our health and behaviors. On the bad side, a lot of talk centers on privacy concerns, especially in relation to social media and internet usage but also go back to revelations about government surveillance and our smart phone data too. People seem worried about privacy and personal data, even though few know what data they actually have.

We should promote greater data protection and privacy, but we shouldn’t ignore the incredible opportunities we can gain from personal data too. So, while the bulk of the discussion these days is about personal data is on the negative’s, like data leaks and data privacy, I believe it’s a good time to try to understand the actual data we do have and how personal data and self-tracking might be used for self-improvement and even self-transformation.

For example, one of the most robust repositories about human health is on our smartphones, wearables and activity trackers. Leveraging a few sensors, our phones and wearables are able to interpret our movement patterns and tell us how many steps we took, how many stairs we climbed, how often we stood up, and many other activities. If you use a wearable with a Heart Rate Sensor, you can also capture your resting, active and sleeping heart rate and even know how long you slept too.

There are various ways and reasons why people track their lives, but when it comes to recording their daily movements, the most common method is with a wearable, activity tracker or smart watch. According to a Statista infographic, the most used wearables today are Fitbit, Apple Watch, Garmin, Mi-Band from XiaoMi, and Fossil. Interestingly, there are dozens of other devices with a much smaller marketshare but which offer an additional array of sensors to track other data points like blood pressure and HRV.

I recently created an open source project called Quantified Self Ledger. These are a collection of Python scripts that help to collect, process and aggregate data from various services like Fitbit, Apple Health, RescueTime and more. The initial goal is to collect and aggregate various self-tracking data. The end goal is to build a personal data dashboard and hopefully one day leverage it for more sophisticated data science and machine learning. In this post, I want to look at Apple Health. For example, how to export, parse and do some data analysis on your Apple Health data using Python. In later posts, we will look at a few other data points and tracking services.

If you are an Apple user, then your iPhone has been tracking your steps and a host of other health metrics. Some are directly recorded by the phone. Others are logged via other health apps that store their data into the Apple Health repository. If you also regularly wear an Apple Watch during the day, during workouts and at night, then you have even more data, like Heart Rate, VO2 Max, and possibility even Sleep.

In this post, we will be exploring Apple Health Data. First, we will look at some methods for exporting your Apple Health data, either using Apple’s raw export or an aggregated version using QS Access app. Second, we will then use some code to parse and process our raw Apple Health logs into more usable formats. Third, we will do some data exploration and data processing, so we can understand patterns and trends. Finally, we use this data to create some data visualizations in Python.

Whether you are merely curious or are trying to use tracking to support lifestyle changes and better habits, hopefully by the end of this post, you’ll understand what data you are collect and hope to start engaging with that data.

Why People Self-Track: Research on the Motivations Behind the Quantified Self and Self-Trackers

According research in 2016, sociologist Deborah Lupton estimates that there are well-over 160,000 tracking apps available in the app stores, including both for Android and Apple phones. This includes both explicitly tracking apps like Nomie and and various health and wellness apps like Strava and RunKeeper.

While we have yet to see a ubiquitous world of activity trackers, there are also dozens of wearables devices today like the Fitbit, Garmin, Jawbone UP, Nike+ Fuel, MiBand, and Apple Watch as well as dozens of other targeted devices and tools for quantifying your health and fitness.

Tracking and personal observation date back centuries. You can find strands of self-improvement through self-examination in both Ancient Greek and Ancient Chinese philosophers. Proceeded by the confessional writings of Saint Augustine of Hippo and Jean-Jacques Rousseau, the Victorian era was notable for the proliferation of personal diaries and journals, which allowed for a narrative format of self-reflection. Today’s digital age has not really changed the human quest, to borrow a phrase, to know thy self. We simply have more more tools and manners, both passive and active, to track our body, mind, time, environment or whatever. In short, it’s easier than ever to track a life.

Several centuries after Socrates declared the “unexamined life not worth living” is its digital equivalent, the “Quantified Self,” a neologism, a meetup, a movement and a life philosophy, whose tagline is “self-knowledge through numbers.” Considered one of the founders of QS, Gary Wolf is also one of the most active writers on the topic. His piece, The Data-Driven Life in the New York Times in 2010, captures the core of what self-trackers are pursuing as well as how diverse and divergent the QS movement is.

For example, one aspect is a technologically infused attempt at understanding human behavior. As he writes, “Ubiquitous self-tracking is a dream of engineers. For all their expertise at figuring out how things work, technical people are often painfully aware how much of human behavior is a mystery.”

As a journalist at Wired, Wolf has been chronicling the QS movement and its characters for nearly a decade. He subscribes to the idea that what today’s self-trackers are doing is not that different than what humans have been doing for centuries: personal observations. A few things have changed though. As he puts it:

Four things changed. First, electronic sensors got smaller and better. Second, people started carrying powerful computing devices, typically disguised as mobile phones. Third, social media made it seem normal to share everything. And fourth, we began to get an inkling of the rise of a global superintelligence known as the cloud.

Quantified self enthusiasts, self-trackers and just curious technologists can now leverage technology to deepen and widen their ability to observe and quantify themselves. But that still begs the question: Why do people track? Why Self-Tracking? Why pursue a quantified self?

In this post, I want to explore what motivates people to track their lives. Whether it’s a quantified self adherent or simply someone tracking their weight, health or fitness, a lot of people are tracking their lives today, and there hundreds of ways to do it. To help understand the space more, we will look the general categories tracking falls into. We will then look at a couple of research papers that attempt to survey and define the QS and self-tracking community. The goal of these papers is to understand what motivates someone to pursue self-tracking and create their self-tracking projects and experiments.

How to Track Your YouTube Watching (and Understand It)

How much time do you think you spend watching online videos like Youtube each day?

Be honest and take a guess.

While you probably have a pretty good sense of the type of content you are watching on YouTube, do you know how much time you spend watching?

You are likely underestimating the amount of time you spend watching online videos. According various studies, the average adult watches videos and TV for an hour longer than they estimate.

In short, we think we watch less TV and online videos than we actually do.

Personally, there have been nights where I’ll guiltily realize at 2am or later that I’ve just spent several hours streaming dozens of entertaining videos but wish I hadn’t. I’ll tell myself again and again “just one more video” as the dawn nears. I have oscillating feelings about YouTube, since it’s a free medium to watch amazing content, entertainment and educational.

If you have ever wondered what and how many videos you are watching online, you are in luck because you can transform Google’s YouTube Watch History into tracking statistics that tell you what you are watching and how much. Unlike other forms of tracking, it doesn’t require you to set anything up to access this tracking data.

The reality is that people spend a lot more time on YouTube: about an hour or more per day, according to Google. In comparison to my own tracking data, which I’ll go into detail on how you can get these stats too, I have watched well over 45 days worth of videos on YouTube in my life, and in the past two months, I’ve spent over a day and half each month watching YouTube videos or about 73 minutes per day.

While a lot of blogs and people online talk about protecting your privacy and about deleting your history on this or that service, I highly encourage you to collect your data before you delete it. Google and YouTube data is particularly interesting, since it can tell you a lot about your online behavior. You can see trends in what you watched during different times of your life. Collecting your Youtube watch history is also a great way to save as well as extend and augment your digital memory.

Our relation with technologies like smartphones, social media, apps, etc. is not always as simple as accept or reject. Instead, there can be a nuanced understanding and a conscious engagement. By tracking and analyzing our usage of certain technologies, we can better understand our engagement and make conscious choices about how and how much we we want use a technology like YouTube.

In this post, I want to share some ways to track and quantify your Youtube watching as well as a few steps you can try change and improve your Youtube usage. We will look at four different techniques for tracking your Youtube video watching. The first way is provided by YouTube app and requires zero setup, but it does require you to note your watching once a week. The next two ways are quite simple and require minimal setup but provide only a limited view of your Youtube History. The fourth way will require some Python Code and Data Analysis in order to gain the most complete data on your Youtube History.

Hopefully this techniques can help you not only understand how much and what you watch YouTube, but change it too.

PS – Check out my 30-day No YouTube Challenge for what happened when a took a break from online video watching.

How I Write (2018): My Favorite Tools and Apps for Writing

So, you want to write? And you’re looking for different tools to make your writing easier, better organized or adopted for a new publishing format?

I’ve published nearly 300 blogs and articles over the last several years, and, while the tools aren’t as important as the time, attention and process your put into writing, I’ve come to like and get pleasure out of certain writing software. These are the tools that put me into writing zen.

Whether you are working on the next great American novel, creating a work report, penning a poem, or just striving to get a shareable blog post published, here are my favorite tools for writing for writers.

This post has been updated and revised. Please checkout Version 2.0 that now includes The Archive and Typora.

A Matter of Fecal Matter: A 31-Day Scatological Self-Tracking Experiment

I tracked my poop for a month. Here is how I did it, what I did to track and process the and, in the end, what I learned from a matter of fecal matter.

First off, I did not literally touch or photograph my poop during this experiment. What I did do is log each and every time I passed a stool.

There are some few alternative ways to track your poop. Interestingly, there is a lot of talk in health and self-tracker space around the “gut microbiome” testing. This is where you analyze the bacterial makeup of your fecal matter, and there are several commercial companies that offer this service. The New York Times seems to love writing about it (here is a good article to get you started here). One of my favorite podcasts, The Quantified Body, has several in-depth episodes on the microbiome too. . It’s a topic that merits a separate experiment and discussion.

Similarly, in case you didn’t know it, there are already quite a few apps dedicated to tracking and logging your excrement. Apps like Poo Keeper, Poop Tracker and others, let you log and rate your poop using the Bristol Stool Scale (BSS). Developed over 20 years ago, this 7-type poop categorization system has become the gold standard for the clinical evaluation of your poo. This short medium post offers a great intro into the Bristol Stool Scale and poop tracking.

For this self-tracking experiment, I decided to keep things simple. I used a generic tracking tool called Hindsight to keep a log of my body waste over a 31-day period. Basically, each time I pooped, I took a few extra seconds to log the activity. Unfortunately I have yet to find a passive way to track my poop (yet).

While there are probably better ways to spend my time and plenty of other more “appropriate” experiments you can do to quantify your life or track your health, poop tracking provided an interesting and amusing opportunity to test out this new lifelogging tool, to practice my skills in data analysis and data visualization, and to learn a few more things about myself and my poop.

In this post, we will be looking at lifelogging and data visualization of fecal excrement over a month-long period, using Hindsight app.

Know Thy Blood: Common Questions and Answers About Blood and Blood Testing

Think of blood tests like a scoreboard.

On the one extreme, your lab results can help tell you and your doctor if you have a disease and they can give you early warning signs of future health problems. Blood testing get used throughout treatment to check on your responses and measure any side effects.

At the other extreme, biomarkers and blood tracking in general can indicate areas that are ok but not optimal. These tests can provide feedback on how to optimize towards improved wellness and longevity. If you are like me, then you can use your blood biomarkers to guide your health towards not just normal but optimal health.

There are a ton of reasons for regular blood testing. And, not surprisingly, blood testing has becoming a popular tool for self-trackers, biohackers, athletes and anyone striving for improved health in general. In my opinion, blood testing should be something everyone does regularly.

In this on-going series of posts on blood tracking and biomarkers, we are looking at how blood tests and other biomarker data can be use to to help self-trackers and people in general understand their health.

Unfortunately, blood testing, tracking and biomarkers can be quite confusing. Frankly most stuff in the medical space is rather intimidating. Almost intentionally so. There are hundreds of terms and a never ending range of opinions when it comes to our health. Fortunately, I think a basic and useful understanding of blood testing can be grasped relatively quickly, especially with the help of technology.

In this post, I want to answer many of the common questions about blood, blood testing and tracking, The first part focuses on questions like what is blood, what is a blood test, and how often to get your blood tested. In the conclusion, we will look at an initial answer to the most important question: Which blood tests you should get? Hopefully by the end of this post you should have a basic understanding of your blood and on how to get started with your blood tracking.


Dude, I’m not a doctor. If you aren’t sure, ask a real one. This post is not meant to be taken as professional medical advice. This is strictly my observations and opinions on what matters and doesn’t. While a lot of research and experimentation has gone into this, please seek professional medical advice along with your own personal research before doing anything stupid. Now on with the show.

How to Track Your Money and Finances: Where Your Money Goes Is Where It Grows

If I had to reduce my principles on finances and money to a single expression, I’d put it this way: Where Your Money Goes is Where It Grows. Or to spin it more negatively, where your money goes might just be where it dies too.

Most American don’t take much interest in their money and finances, at least in terms of their savings. “Thirty-four percent of workers have no savings whatsoever; another 35 percent have less than $1,000; of the remaining 31 percent, less than half have more than $10,000. Among older workers between 50 and 55, the median savings is $8,000,” according to an article in Inc on retirement. The stark reality is that most Americans have little or no savings.

People should expect more out of their money. The idea is that if you don’t know where your money is and what you are doing with it, then it’s really hard to expect much of your money. If you spend all your money and never save or invest, then your money doesn’t really work for you. But if you spend less than you earn and invest as much as you can, then your money becomes a partner. Your money begets more money.

Obviously different people have different life situations, but I believe everyone should strive for some form of financial security. That translates to savings. If you are following some basic common sense principles of personal finance, which we covered in a previous post, then you know that you should spend less than you earn, avoid (or pay down) debt, save, and invest in a diversified, low cost portfolio.

The gap I see in these simple principles is developing an awareness of your money, by which I mean tracking your money and having a budget.

For me, the answer to most things I’m trying to figure out is tracking it. I’m admittedly a pretty obsessive tracker, including time, productivity and health. Finances is no different in the role of tracking. Fortunately, most of finances and banking have gone digital, and there is a plethora of really great tools and apps to help you track, budget, save and invest.

Doubly awesome, if you mostly are using digital payments and banks online, there is a good chance most of your tranaction history is already being tracked. You just need to take ownership over the data, process it, and then start to leverage it.

In this multipart series on personal finance tracking, we have been looking at the principles of good financial behavior, how to track your finances (current post), and how to set trackable goals for saving and investing.

In this post, I want to look at how to track your money. I strongly believe that tracking of your money can help you gain an awareness of where you money goes. Then with a bit of budgeting you can set targets and build towards your savings and investments.

In the first part, we will review several great tools to track your money. Some are completely passive, though you’ll need to occasionally recategorize certain transactions. Other tools require you to actively log each and every expense, which while taking more effort, carry the benefit of greater awareness. Good money tracking can even be and sometimes should be done in a spreadsheet too. So we will look at some basic ways to visualize your financial data using a spreadsheet or existing tools.

Ultimately, the goal of tracking your money, creating a budget, and establishing good habits is to help you reach your goals. With tracking you can position yourself for the next stage of your financial life: saving and investing (future topic).

Heart Rate Variability: Science and Advanced Concepts for Understanding a Key Health Indicator

Ever wondered how stressed you are? Heart Rate Variability (HRV) is one practical way to objectively quantify your stress and health. While some amount of stress can be good, being in a constant, long-term state of stress can be very bad for our bodies and minds. HRV helps you objectively understand the state of your body and what factors trigger a stress response in you.

As a long-time biohacker and self-tracker, HRV has become one of my favorite biomarkers to track. It allows me to understand my overall state of stress on a day-to-day scale as well as contextualize what’s affecting my stress. It’s even something I can improve.

Quite simply: By measuring your HRV and capturing contextual factors like sleep, exercise, lifestyle stress, drinking, etc. you can understand your physiological stress.

What is Heart Rate Variability? Unlike Heart Rate (HR), which is the average number of beats of your heart in a minute, HRV is measuring the variance of intervals between heart beats. You can then use your calculated HRV to know if your heart rate is showing higher or lower variability in the moment and in comparison to your baseline average (typically last 7 days). Somewhat counterintuitively, lower variability is a sign of an increase in stress and activation of your sympathetic nervous system, and higher variability is a sign of being a state of rest and activation of your parasympathetic nervous system.

Your body is a complex system, and it deploys a number of systems to keep you alive and responds to its environment and internal states. Its goal is homeostasis or a kind of living balance. In order to maintain homeostasis and stay alive, your body responds to different situations through its different cells, organs, and organ systems. One of the most important is your nervous system, which transmits to different parts of the body to control actions.

In terms of the science that backs up HRV as a valuable biomarker it all comes down to how HRV acts as a proxy to a subset of your nervous system called Automatic Nervous System (ANS) and its two main branches, the Sympathetic Nervous System (SNS) and Parasympathetic Nervous System (PSNS). In order to better understand HRV and how stress works, we need to understand how HRV relates to your ANS.

In this post, which is a continuation of a previous post on HRV as a biomarker and how to track your HRV, we will be looking at Heart Rate Variability from a more scientific angle. Specifically, in order to understand our HRV scores and what factors affect our HRV, we need to understand how our nervous system acts through these two different subsystems. One of the main reasons HRV is such a key health indicator or biomarker is its ability non-invasively to tell us about the state of our automatic nervous system. For example, are we in a state of so-called “rest and digest” or in a mode of “fight or flight”?

Let’s dig in.

Common Sense Money Management: What Are Your First Principles of Personal Finance?

The basics of money and money management seem undeniable: spend less than you earn, avoid debt, save, and invest in a diversified, low cost portfolio.

How we behave with our money can be the difference between a successful, happy and stress-free life and an unsuccessful, unhappy and stressed one. Whether we like it or not, money matters.

Unfortunately personal finance can be a real struggle for many people and too many people let so-called “experts” manage their money, instead of spending a bit of time to first learn and make their own informed money decisions; and second to develop reliable instincts and habits for long-term financial success.

In my opinion, everyone should learn about money and understand how to budget, save and invest. The basics are not that complicated really. It’s consistent actions that are harder, and without consistent actions, you miss out on what Einstein allegedly called the most powerful force in the universe: compound interest.

In this three-part series on personal finance tracking, we will be looking at the basics of good financial behavior, how to track your finances, and how to come up with goals for saving and investing.

I’m no financial expert nor genius on money, but I have managed to create a life I love; a life where I save much more than I spend and invest it in ways that create returns. Good financial “ hygiene” and some tried-and-true investment strategies have enabled me to spend my time and money on the activities and things I truly enjoy without stressing out about money all the time too.

In this first part, I’ll share a few of the things I’ve learned about personal finance and money management. I call these “first principles,” and I think having simple go-to rules is a key part of being money smart. These are mine, and some of yours might be the same or the they might vary.

An Experiment in Mood Tracking: How I Tracked My Mood and What I Learned

Can we and should we track our moods? And if we could track our moods what would we learn?

For the last couple months, I’ve been running an experiment in mood tracking. For two separate two-week periods, I recorded my mood score five or six times per day by selecting a variable smiling, frowning or neutral face. I logged my mood a total of 134 times.

What were the results of my mood tracking data? Am I moody? Not really. On average, my mood has been either “ok” or “good.” There were a few instances of a neutral mood (I’m not a morning person), and a couple outliers where I was very happy. As life happens, on one occasion, I was “not ok.”

In this post, I’d like to share how I tracked my mood and some of the lessons I learned through this experiment.