Apple Watch VP — It Has to Work in 2 Seconds!

(upbeat music) – I'm Kevin Lynch. I work on software across
Apple watch and health. – And I am Deidre Caldbeck, and I am on the Product Marketing team for Apple watch and health. – And it's cool that Deidre
had been off the paddleboard and came into the, yeah– – No, it's underneath my feet. (Kevin laughs) I'm just very stable. (Kevin laughs) – I know we didn't get to see
you dance again this year.

(Kevin laughs) Kevin, we didn't get to see
you and Jules do a duet. And I did file a radar,
expect to behavior dancing. – Off camera, that was
all off camera this year. (Kevin laughs) – I'm actually really curious. If we go back to the beginning earliest days of the watch, when you were first read in or disclosed and you heard that Apple
was making a watch, what went through your head? What was your immediate reaction? – (chuckles) Well, for me,
that was actually day one when I started at Apple,
the morning I started, the story was we wanna build a watch, and we've got to get going. In fact, we wanna get going right now. We're feeling like we got to go fast. And actually I was advised to
not even go to orientation. Like you don't even have
time to go to orientation just get going, you know
Apple well enough already and get started. And so it was a very rapid start. The thing that went through my mind, there's always, of course
occasionally rumors about what Apple's may be working on.

But when you look at Apple watch, I felt like that was inevitable. Like to have Apple make a watch, I felt like that was the
first thing that went through my head was of course,
oh we're gonna do that. 'Cause if you look, I love
the history of computing. If you look at the history of computing, it's a series of evolution of smaller and smaller computing devices that are more and more powerful. And it goes from mainframes
to mini computers, to desktop computers, laptops,
portables, mobile phones. And so that evolution of power and administration leads
you to, well, what's next, what's next after the phone? And that really leads you to
something as small as a watch. And the watch is a natural
location on your body. If you start thinking about
something that's even smaller than a mobile phone, which
is already pretty small, it's like, well, where would you put it? How would you carry it? How would you look at it? And so attaching it to
your arm was a really really natural place to do that.

So that was really interesting. And so of course, going
through my head then was this is gonna be an
incredibly challenging project. How do you even get a
compelling interaction on a really tiny display? And battery life, of course
is gonna be an issue, which is a major challenge. And so designing around that was from an engineering perspective, really interesting
challenge, which I love. And then also this great opportunity, and really a sense of responsibility for working on a device that
is actually on your body, like you wear with you, like that responsibility
is really significant. How do we make something that people will feel really good
about that supportive, and empowering for them, not bugging them. So that kind of philosophy around how we design all of our interactions is really sensitive to that. – Unlike Kevin, I had no
idea what was going on. Apple is very good at making sure that we keep our surprises, surprises for both employees and our users. And so I was working on iOS, and actually what I worked on initially was the health app
because I worked on iOS, and Kevin's team was
working on the health app, but I was on the iOS
product marketing side.

And so we were working out
how to help tell the story of this whole new health app in iOS eight. And what I thought was amazing was we were gonna be able to talk about how you could bring all of
your health data together from third-party apps and devices, and little did I know (laughs) that one of those devices
was soon to be Apple watch. So it was actually a really
exciting thing for me, as you said, to be read in
or sworn in on the secret of Apple watch much closer to
when we actually announced it.

So I really wasn't told about it until I think just a few
weeks before the announcement. And I was very fortunate
to be one of the people to show it off to our
audience at the event, in the hands-on area. So I learned about it,
honestly, just probably the same way you did (chuckles) Renee, and that was a really neat thing for me to work at Apple and hear
about it like everyone else. So it was cool. – My dad worked at IBM and
he got an Apple two plus we wouldn't have to drive downtown to use the mainframes. And I had an iPhone. I wouldn't even have to go back to my Mac to do certain really important things. And then I got an Apple watch and it can't do as much as an iPhone, but it can do these brief frequent, but really critically important things without me even having
to reach into my pocket or reach onto the desk for my iPhone. And it occurred to me that
it took until like iOS five, and iCloud before the iPhone
was really that independent from the Mac or the PC running iTunes, but systematically over the years with on-device logic
and on-device app store, and bringing LTE over and just continuing now you have the family sharing where you can set up a watch for somebody, you are systematically just
increasing the functionality.

And I know that Apple is not shy about cannibalizing their own devices. You'd much rather do it than
have somebody else do it, but it is that like a…
Not a predatory process where you're looking at
the iPhone and going, "What can we do next?" But is that like more of an organic thing where as time and technology allows, you'll just keep building
up the functionality and the things that we're
capable of doing from our wrist? – The work we're doing
there is really how can we empower people with Apple watch, and in different situations whether you're near your phone
or away from your phone.

And you've seen us doing that more and more over time, for sure. But really the main focus is how all of our devices work really well together. And it kind of, when you
interact with each one of them what are you trying to accomplish? And each of the devices has its own kind of personality and way
of going about things. So like you were saying with Apple watch, the little brief
interactions is really core to how Apple watch works. And we started with that and
that's how it's different than like you're using your Mac, you'll tend to use it for longer
periods of time and phone. Like you were saying less than your Mac but more than your watch. So there are these kind
of interaction times that are very different
across the devices. And when we started
working on Apple watch, we thought that time
of interaction might be maybe something like 10 seconds. We were kind of estimating what those typical interactions might be like. And it was really interesting when we started actually living
with the watch internally, we started learning that really, you wanna do those interactions even in shorter times than that.

More like two seconds
rather than 10 seconds. And that's a tall order to do some of these tasks in two seconds. We weren't sure how we could do that. So we paused what we
were doing for a moment, we had everybody go and brainstorm for a bunch of different
areas of the system. How could we possibly make
those happen in two seconds? Like no rules. Like you can do whatever you want. You can change the
interaction with the system, you can make shortcuts,
you can take stuff out, whatever it is, make it two seconds. And we made a list of those things, and that was super productive. One of the examples of
that is the messages app, when you receive a message, it used to be initially when you replied you had to kind of go into reply, and then go to a compose
view and then press send. And that took more than two seconds. Now in messages, which has
been this way for a long time now in the watch, when
you receive a message, you can just scroll up
with your digital crown, there's a bunch of suggested replies that we've generated for you, and there's some you can edit if you want, and you just tap them.

So scroll and tap, and
soon you receive a message that scroll and tap you can totally do in less than two seconds. And so we went across
and did a bunch of that same for like wallet, doing Apple pay, like just a whole bunch
of areas of the system, we went and tuned in. And that type of investment in time and engineering makes a huge difference. And it's sometimes like what are you removing more
than what are you adding? – That's one of the things I
love most about Apple watch, because like the, the downside of focus is sometimes tunnel vision, but it really seems like
over the last few years even though health and fitness
have become so popular, have gotten so much attention, the original sort of
promises of the Apple watch also included payments and authentication, and remote control and communication
all these other things.

And you've been systematically
improving those as well whether it's walkie-talkie
or now the Intercom system, the new home app, way more robust. And one of the things
that interest me the most is keys and IDs, because
this just takes… It both simplifies the process
of authentication, identity and control, but also greatly empowers it. Can you talk just a little bit, about how those new technologies work? – This is an area we've been
working on for many years and it's really exciting.

The infrastructure is
really well understood now, and we're at the beginning
of it kind of rolling out this generation of access to your home, your office, your car, hotel rooms. So each of these
categories has its own way of implementing this and
then we've designed it on Apple watch so you can
just use tap to access, just like you do with Apple pay using NFC. You can make them so they
have express mode turned on, we call it, so you can
just tap and access, whatever it is you're using.

And with the proximity
of NFC, there's an intent you're waving your device
in front of the lock to cause it to unlock. So there's not this guesswork about whether it's unlocking or not. You still have a gesture that's
involved to do the unlock. So I think we've got really
a great approach to this now and super excited for
it to start rolling out. – Previously, we talked a lot
about how messages and photos, and things like that were
really what kept you connected. And now we have this
evolution of the watch where it's your identity. It allows you to control
the things around you and access the places you love.

And I mean, if you think
about Apple watches, your identity we started with Apple pay, when it's on your wrist,
that's your identity. And we've recently used
it as your identity for when you wanna unlock your iPhone and you're wearing a mask. And now with these watch USAID updates, with things like state ID
and the home app redesign and these other wallet features, it's really starting to be
more of this identity access, and control where it's
the safety and security of a device that's always on you. So it's really perfect for
all of these capabilities. – I'm always continuously impressed and inspired by the
accessibility technologies that Apple brings to
all of their products. But I think that demo of assistive
touch on the Apple watch, it just next level blew people like straight up science fiction away. And I was wondering like, how do you sort of realize
all of this potential and sometimes maybe even take it further than anybody imagined.

– Yeah, these are deep collaborations across multifunctional teams. So it's the design,
accessibility, engineering storytelling, marketing, it's all the different teams
work on projects like this. And it's the combination that
really makes them awesome. And on that one, there
were lots of explorations like more gross movements of
your arm, shaking your arm rotating your arm, lots
of things like that. Those require energy as well. So we thought about like
how much energy does it take to do these different gestures? And if you're using assistive touch, you wanna be really thoughtful
about how much energy every time and every interaction. And so we settled on a
couple that work really well from a low energy human
energy perspective, but are really hard to sense. And the two are there's four
signals, clench, double clench and then we did pinch and double pinch. And the sensing of that is
happening on your watch. And it's using the gyro,
the accelerometers looking at micro movements of your arm
when you do those gestures, it actually uses the
heart rate sensor as well. It gets the imaging
feedback from from your arm included with the gyro
and the accelerometer.

So those are all blended
together with machine learning and we've created models
to capture these gestures. This wouldn't have been
possible a few years ago, and it's just, as we're
starting to understand how we can use these approaches
in terms of building software. And also the capability of the performance of Apple watch has gotten so incredible, we can run these things live now. And we've been working on accessibility in Apple watch since day one, we've first shipped Apple watch. We were thinking about how
to enable people with this and it's been such a pleasure working across all the teams on
this and the team members, and the assistive technology
area have just really been brilliant and have
been working as part of the integrated Apple watch
team since the beginning. – I love it, I love your dedication, like the plural your
dedication to accessibility and also health, just the way
you've been ramping up health.

And please correct me if I'm wrong, but Apple watch has been
a big focus of health, but now you have a mobility
and a sort of tying in the iPhone in a way that's
really Apple watch style like using the sensor fusion
and the machine learning on device and taking all of that, and giving us really useful
preventative information. And in my head, I'm thinking, well it's the Apple pencil
one day gonna be able to tell me I have RSI or carpal tunnel, like is this the beginning of some great Apple
ecosystem health thing, but I'm curious like when
you see this escalating across multiple Apple devices now, Deidre, well, first, can you talk about bringing this kind of technology to the iPhone, and maybe how people will use it or get the most out of using it? – Yeah, I mean, I think
you said it perfectly when you said that we do kind of think about our entire ecosystem of devices, and the capabilities within
each of those devices.

And really our goal is
always to empower people to better manage their health. And whether there are
sensors on the watch, or the phone, or Apple
pencil that can offer that empowerment for our
users, that's actually where we'll focus the
investment and the energy. And, you know, with walking steadiness, it's something that, as you
said, we really wanted to think about how can we prevent these falls. We introduced fall detection
with Apple watch series four. And of course it's been
incredibly rewarding to hear the stories from our users who have benefited from that feature. But if you think about it, gosh wouldn't it be better if that fall had never occurred in the first place.

And we know that Apple watch
has amazing health sensors, the motion sensors, the heart rate sensor, even we're using the microphone for health features with the noise app, but the iPhone also has as
you know, motion sensors. And in this particular case, the motion sensors on the phone and where you actually carry your iPhone, typically in your back
pocket or your front pocket or maybe a shoulder bag
or a cross body bag, those sensors actually can
pick up those subtle signals from your hip movement
that can actually give us the fidelity that we
we need to provide you with those mobility metrics, that can then give you the signal for when you might be at an
increased risk of falling.

So of course the watch similarly
has powerful motion sensors and some of those mobility metrics are from the watch things
like stair ascent speed and stair descent speed. We can only get that from the watch, because of the way the
watch is positioned, and you often may not
have your phone with you when you're going up and down stairs. And so we're trying to
think about the best place to get the best data, to
provide the best experience. And in this case, the iPhone
was really best positioned for this and that certainly
gives us the opportunity to offer it to all of the iPhone users, and not just Apple watch users, but we know that combination of phone and watches is really powerful for a lot of these health features.

– Always felt to me like the
phone was like the star ship and the watch was like the shuttle craft. – Ooh. – And you can just get so much done by going back and forth– – I love it, I love it. – Between those things. – Walking steadiness
actually came initially from fall detection. So we're doing fall
detection in Apple watch, we're all working on
it and it's incredible how that's been helping people, and we still get letters about that, but we were thinking
as we're working on it, well how can we prevent
people from falling? Like, rather than just
detecting that they're falling can we actually stop them from falling? And that just led through this
really interesting journey of discovery and brainstorming about while we may not be able to stop you
in the moment from falling– – (indistinct) very carefully,
walked around disclosing any potential anti-gravity
fall prevention technology, (Kevin laughs) that you might have in
play after you teased us so mercilessly with the teleportation, a couple of years ago, and then Craig escalated
to portal technology, through his aperture
science app just this week.

So I appreciate that your
whatever vibranium casing you're using is still in the future, but you're trying to give
us tools we can use today. – In the meantime exactly
doing what we can, exactly. – Yes. – We actually did this really deep work that has taken years of sensing these different attributes
that Deidre was mentioning. And then doing studies where we partly use the Apple heart movement study for this was about a hundred
thousand people in that. And so we were able to use the
insights that we've gathered from people's movements in that study, to actually see what would
be predictive of falls. And so that's what this walking
study this metric came from, was the modeling of those metrics. And then seeing which ones
indicated most clearly that you had potential to fall. And that was supplemented
by studies specifically on this topic too, where
we had cohorts of people, where we were doing traditional
walking studies with them, like observations and
questionnaire filling out and things like that, kind of traditional
characterization of potential falls. And then we tracked those users with, you know, they were part of the study, and as people did fall over
the next kind of year or two, we were able to go back, and look at those authentic falls, and see what the indicators were earlier and what led to those.

And you can't fake that you
can't just do prep fake falls, you have to have actual falls. So it takes a long time to
really get that pre-work done, see it actually happening
and model it correctly. – It feels like we're
getting into like phase two of all of this technology,
where in the beginning you were just building
up more and more sensors and more and more data. And maybe labs is like the current ultimate expression of that.

But now that you have
all of this rich data you're providing things like trends, but not only are you
looking at the trends, you have this Edward Tufty inability to take all of this medical information, and make it just so human digestible, like, just in terms of the interface and the graphing and all of that, how do you sort of deal with these ancient sometimes very inhuman data sets, and make them so people who
are legitimately stressed like they have health anxiety
can get so much benefit from just seeing it easily displayed maybe for the first time? – Yeah, so trends is like,
how do we draw insights across all kinds of different
data types in a simple way.

And again, we use a lot of modeling from the studies that we've been doing to look at patterns and the significance of those patterns for people. And so we basically have built systems that do statistical
relevance of these insights. So not just showing you
numbers that maybe bigger or smaller, but which ones might matter. So there's a lot of
thoughtfulness by data type about the variances in which types of variances might matter
for which data types. And then we look at
comparing the past month, through the past six months, and then we look for the changes that might be notable in that time. And there was also a lot of work on like, how many of these trends
might we show people? We don't wanna show people too many trends 'cause it's kind of overwhelming, but also you don't wanna
like not show any trends.

And so we did a lot of
experimentation on tuning that in to get it to a point where it
was a useful relevant trends without being too many. – One of the things I'm curious about I get asked this question all the time and I'm sure you do as
well now, especially that you have iPhone
now involved in health. And I was joking about
the Apple pencil before, but for example, the iPad, when you look across the Apple ecosystem and you start thinking
about a future where, you know, maybe it
doesn't make as much sense to have these particular
features on an iPad because you don't carry it the same way.

It's not with you the same amount of time, it doesn't have the same sensors or connectivity all the time, but maybe it's really great for reporting or for a big screen to
see all these infographics that you're doing such a good job on, is health something that
you really do look at in terms of the entire Apple ecosystem? – Yeah, absolutely. I mean, if you think
about the health features we have today, there are
obviously several on Apple watch and iPhone, there's also
some health features with Air Pods and some of
our audio products, right? I mean, so there's
absolutely an opportunity for us to leverage the
ubiquity of our devices, to discover new ways that
we can empower people to better manage their health.

And we'll continue to
investigate those areas, we'll continue to invest in those areas. I think with the new features
that we introduced this year, we are excited to hear
from more and more users who are going to take advantage
of these great features, for example, with health sharing, we know that we hear
from so many of our users that they're feeling
this this burden really, or this overwhelming
sense to care for both their children and their aging parents. And to be able to do that in a way that, just have it in one
place in their health app where they can see their own health data, but then also the health
data of their loved ones.

And in a way that's private and secure, where not only do their
family members feel safe when they're sharing their data, but also the people that
they're sharing with know that it's a secure connection
between the two of them. – I love that so much. And I love your approach to
privacy and security so much, because I think it's easy to be extremist like Info Sec extremist
about these things. Yes, you wanna keep your
health data private, but if you're an absolutist about it, you can't share it with your doctor or with your kids or your parents data, or maybe you're not physically
capable of using that, but your caregiver is the one you want to, and I love that you're being so, like you're doing informed consent which to me is everything.

But when people are willing to do that, you're empowering them to use that data, beyond just locking it up and making it actually
functional for them. – Exactly, and I think to your
point about informed consent that's why the way we
built the experience, it was with all of our features
privacy is at the core, you are in complete control
over the data you share and who you share it with. You can of course stop
sharing at any time. You can choose very granularly which pieces of data you wanna share. If you've had a chance to
go through the experience, you can even see a
preview of what the person you're sharing with will be able to see, so that you can be sure
that's what you wanna share. And it's not just control, right? It's also transparency. So you'll never share data
that you can't see yourself, and all of those principles are at the core of these features.

And we'll continue to ensure that that's the case
in the future as well. – One of the things we've
done which is wonderful is we've added kind of this discipline to the mix of making products,
which is the clinical side. So clinicians working with
great health backgrounds, working on and that's
really helped us make some great strides in the work that we're doing here around health, and have it be something
that's super relevant medically to people, especially when we
do the doctor sharing stuff, like how do we make a dashboard
that a physician can look at and understand very quickly, 'cause these interactions
tend not to be very long.

And so you've got to really understand someone's information really fast. And so we put a lot of energy into the design of those views, so that that would be
super relevant to a doctor. And on the privacy side of that, that was really interesting too, because the end point of that
is not an Apple device usually it's a computer running
electronic health record system, and they're viewing the patient's records, they're inside the health institution. And so how do we get this information from their Apple watch and their iPhone to show up in that other device? And yet, have it be completely
end-to-end encrypted and secure in a way that
Apple can't see the data. And we're actually
serving that to that EHR through a web view
inside of those systems. And so we generate a view of the data that comes from the person's
phone into this web view and download it onto the EHR so they can see it right in place. And it's in context, it
knows which person's data to bring up 'cause of the current person they're viewing the rest of
the information for in the EHR.

But the encryption of that data is handled with an encryption key
that's shared between the user's phone and
that health's institution that Apple doesn't have. And so the package of data is encrypted on the user's device, sent
via the cloud down to the EHR, and you get this encrypted
package of data with them. If you looked at it in
transit, you couldn't tell what it was, it's this encrypted blob, but it shows up and it's
decrypted at the last second inside the browser of the doctor's view.

So that's where it kind of unpacks itself and shows it in the view. And then when the doctor
stops looking at it it's not there anymore and it's encrypted. – I have this beautiful dream where one day when the world stops ending, I'm back in South Bay with all of you and maybe I'm at bitter and
sweet having a red velvet latte. And my Apple watch goes off and alerts me to a potential health concern, and I'm just a Canadian on US soil.

So I go to the local clinic and I can just tap my Apple watch and it immediately authorizes
like all my health records, all my allergies, my medication, everything that has to do with me, maybe it Apple pays my copayment or my insurance deductible. It does everything that
I would otherwise have to sit there with endless clipboards, endless phone calls, in a
panic having to go through. And it handles that as
easily as it may be handled at transit interaction today.

And so my question is not so
much about future technologies but about when you're
approaching all of these things, how much of it to you is
just like yearly updates versus the stars that you see
hanging above the horizon? – We're at the beginning of
this work in health I would say, even as we've done a
number of things already, there's so much potential here. And then some of the things
you just outlined are lots of great challenges that
still exist in the world around health and sharing information. And we're at the beginning. So the work we do is long-term. So we do work that, you
know, may show up kind of on an annual basis or so, but that work is really
backed by in some cases, years of work, that just
happened to align at that time. So there's a pattern of that.

Some of it's short-term,
some of it's long-term, but our thinking is
definitely long-term about what we can do here for
Apple watch and for health. And for both of those
things, it's early still, like we've done a bunch of great stuff, but the ideas are still ahead
for us to do a lot more. It's just really exciting. – Deidre I mean, Tim Cook
said that health is one of the things that Apple's
gonna be remembered for. So zero pressure, right? – Zero pressure. But if you think about it, there's the, you know, we talk about this a lot, but it really warrants continued emphasis. We are just so moved by these letters we hear from our users. And I think that's why
Tim makes these comments, because we are all just so fortunate to be able to read these
letters and you hear, oh gosh, I had no idea
I had this condition and I I bought an Apple watch, because I wanted to get my messages when I'm on a run or I
bought an Apple watch, because I wanted to be
motivated to work out more.

And I had no idea I had this, this issue, or I was out kite surfing, and I had an issue and I remembered I had a cellular Apple watch (chuckles) that's also water resistant. And so I could just make a phone call. And so I think the way
I think about it is we in some ways have already
made quite an impact on our users lives in a
way that I don't think we would have imagined, when we were first building Apple watch. But I also think to exactly Kevin's point, this is just the beginning, because those stories are really
what inspire us to do more and to invest in these areas where we think we can
make a real difference in people's lives. And so we're so excited
to continue to do this. – I've said a few times that I think, and I don't want Cayenne
or Craig to gift it to me, but I've said several times
that I think that Apple watch is just the most important
device that Apple has ever made, because it like, yes, a
PC, a phone can save lives, but the Apple watch has so many features that are designed just for
the purpose of improving, preventing, helping, contacting for life.

And I think just thank
you, you personally, and the teams for all the
work and all the effort and all the thoughtfulness and compassion, and empathy that you've
put into each year, every year, making it just a better and better device, not just for me, but like many people, I've
given it to my parents, I've given it to my siblings, and it's made a tremendous,
tremendous difference. So sincerely thank you. – Thank you, Renee, it's like an honor. The whole team feels this
excitement and dedication to working on this project for very much the reasons
you're talking about, and it feels like responsibility as well. It comes back to the
Apple watch being worn. I mean, it's on your body,
it's there all the time.

And that's a unique thing. And a lot of goodness can come from that and a lot of support for people. So I've never worked on something as meaningful and
compelling as this project, I never have worked on something where we constantly get users writing to us about how this has
affected their lives, I've worked on some really
great things over time, but this one really is
special in that regard. And it just keeps coming, Deidre, we get these stories all the time, and we share them across the team. So people write, Tim, Tim forwards them around or to us, like we collect all these letters
up, it's the emails mostly and the whole team reads them, and that's super inspiring. And some of them are great letters, some of them are like, Hey,
you guys can do better on XYZ. We share those too,
which is really helpful if you do write those in, know that actually we read those, and it really affects our thinking about what we were
working on in the product.

So yeah, it's an amazing
thing to be working on this..

Add Comment