21 Mar 2019

Data Starvation

After the retirement of my dad, he started to apply for his pension benefits, I realised how analogue the Indian bureaucratic system is. Everything is still done on paper and files. Even if banks are very digital but a paper copy is being maintained considered more trustworthy.

I have studied and worked with numerous open data sets available freely for the past 3 years while in the US. Most of those data sets were about something or the other in the US. The availability of structured data about employment, GDP, waste disposal in rivers and such is taken for granted there. In fact, the city of Boston, where I was living, has a data portal dedicated to making structured data openly available. Even then, the people working with these data sets often wanted more data to infer information.

I, being unaware, presumed that, in this era of the fourth revolution, data is available everywhere in the world similarly. But then, I came back to India, my home country. I spent some time in a meditation retreat, a little more in exploring some parts of the country that I hadn’t visited before. Then it was time to start working again on a project that I wanted to work on. This involved collecting the political affiliation history of the MPs (members of parliament) of the country. I think it is valuable for the citizens to know that candidates of each political party were not always part of that party. It could be even more valuable to know a few outliers who change their affiliations often.

Collecting all that data is incredibly hard. In fact, I couldn’t even find a list of current MPs and their respective constituencies as a CSV file. I looked up many government websites, websites of so called “data agencies”, journalism websites and much more. At this point, I manually entered each row in the excel sheet from the wikipedia list. Then googling each name on the list for their history and adding it to the excel file. I was doing all of that and losing motivation to do it further. And finally, I gave up.

Along with the previous data collection process, I also started looking for other data sets that I thought might be interesting. For instance, unemployment is a major issue in the upcoming election, I tried to look for data sets for that. I did find OECD and world bank data for the topic, but there are multiple organisations that publish reports cited by news channels e.g. NASSCOM and CII. I wanted to include them as well.

All of the data released by the Indian government, NSSO, CII and similar agencies is hidden deep inside reports published online as PDFs. One can choose to read through 100 – 500 pages of each of them (on another note: they are also herculean-ly difficult to find on the websites mentioned). This made me realise how the offices operate in India and how the bureaucratic process like getting pension benefits takes so long.

India might have come a long way in terms of standards of living of people. In fact, the standard of living of a person with a white collar job in the cities is virtually identical to the one living in the so-called “first world”. But my observation of how the government works makes me realise how far behind we are. If the government, people, academics, journalists and researchers don’t have access to readily available data about pressing questions, how can we imagine ourselves to progress? It is true that the world of private companies is much better in any country; government offices will probably always have some level of formality associated to them, but if 2016 data is published in 2017 which takes another year to understand for the general public something is very wrong with our democracy.

It will not be fair to leave without mentioning the respect I have for the people who study government data in India.

08 Jul 2018

The Namesake (by Jhumpa Lahiri)

Being an Indian who moved to Boston for my post graduation, it was a lot to learn. Even after 3 years of living here, I am not done learning about lifestyle in America. I understand and relate to so many of the things in this book. The book follows a person named Gogol whose parents move from West Bengal (a state in eastern India) to Boston (in Massachusetts, north-east USA) from before his birth to his middle-age.


My mom passed away after 2.5 years of my move, although I was in India at that time… but still I relate to Ashima and Ashoke leaving their families behind… losing all members one by one while trying to fit into American culture.

I personally feel it is very courageous of people who leave their own home to pursue “something” even when they are not specifically running away from home (like a war, in some parts of the world). It makes me wonder when people, back in Punjab, think of moving to the parts of the world, where they don’t even understand the language, or understand very little of it.

Courage is probably not even the right word. It takes more than courage… it takes determination, sacrifice, curiosity and incredible endurance against loneliness. And then when all is done, you find a person to live with and make a family. You see your kids growing up and being part of the culture you can never completely be part of, trying to keep them as close to your own culture as you were. You aren’t lonely anymore, and yet you aren’t home.

Later like Gogol’s parents accept that their kids will be Americans no matter what, they celebrate American festivals for them, they accept their American-ness e.g. by cooking those foods for them etc. And yet, when twice or thrice a year those ‘American’ kids are asked to be part of a Bengali party, they despise it. The loneliness comes back, but in the end you deal with it… it’s too late now to change your own kids, and too soon for them to understand you. And in this way your whole life is gone.

It’s an incredible book for someone who wants to empathise with immigrants or is curious about their experience. The way America looks from India, ‘the land of dreams’ idea, is nothing like it actually is. It is a trap for one generation so that their next generation can live in ‘the land of dreams’.

08 Jun 2018

Enabling sleep analysis

I work for Health Behaviour and Informatics lab at Northeastern University. The lab conducts studies for development of wearable technologies. The goals of these studies varies from changing/developing habits of the participants to testing the feasibility of a particular commercially available wearable or method for health tracking. One of the projects deals with improving sleep habits and understand their relation to stress recovery of the body. The stress is known to affect sleep quality and circadian rhythm. Better sleep patterns may help with stress management as well.

Sleep quality can refer to many variables which include percentage of time in light/deep sleep with respect to total time in bed, time it takes to fall asleep, resting heart rate of the person. Circadian rhythm is a biological process that makes our bodies realise when to eat and when to sleep. When we travel between timezones, our circadian rhythm is disturbed and hence we experience jet-lag.

The sensor (and data collection)

Emfit Quiet sleep sensor was used for this particular study. It is a non-intrusive sensor that goes under the mattress on which a participant sleeps. It is connected to the internet and makes a reading of the heart-rate and respiration-rate every 4 seconds. This data can be accessed using the API, provided by Emfit, after the end of a sleep session. The NUCoach system developed by our lab allows for integration of this sensor and fetches data every few hours. I get this data through the lambda function triggered by NUCoach platform itself. After processing the data, it is stored back on NUCoach database for coaches and participants to access.

The project

Most projects entail having participants who are being coached by a coach based on the data being collected by different sensors. This coaching aims to create a healthy habit in the participant in order to live a healthier lifestyle. For instance, in this particular project, coaching participants to go to bed at the same time everyday can help regulate the circadian rhythm. Their food habits, exercising habits also come into the picture in a larger context but this study focusses, majorly, on sleep. Since participants are the source of the data, we must give them access to the data in order to help them see their progress. My contribution to the project can be classified in 2 parts.

  1. Coach side
  2. Participant side

Coach side

The coaches are interested in many aspects of the same data. My job was to visualise this data in a consumable way in order to help them to decide on feedback to the participants. The sleep classification data (awake/light/deep sleep) during the sleep session was visualised from beginning to and end.

The timeline is created in a way that each row is a 24 hour period (a day) but the day starts from noon to next noon. One of the advantages of this view is that a coach can visually see that the client did (or did not) got to bed at the same time. The day starts from noon in order to keep all parts of a sleep session together. It is very rare that someone sleeps beyond noon. While keeping the midnight to midnight (left to right) timeline, the session was broken into 2 rows.

The other addition to this view is are the grey histograms on the right. Initially they were planned to be on a different view, which came up once the coach clicked on a sleep session. But I decided to add them on the same view as the landscape view of the web enables that. The 3 columns on the right are histograms for resting heart-rate, sleep efficiency and time it took each night to go to sleep. All this information can display the trends of sleep quality to the coach and help them make decisions on how to intervene with the participant.

Participant side

The participant uses a phone app in which has many other features, but let’s focus on the display of the information for this particular project. The phone space is limited as compared to web (sure, a coach can try to access client’s information on the phone, but the NUCoach platform does not encourage that). Showing a participant their sleep data is done in a scrollable list form, where each session has some details. These details are chosen in a way that when a coach talks to a participant about something that they observed in their own interface, the participant is able to observe that as well. The participant may not able to see the time it took them to sleep, they definitely have to “read” it from the chart or from the text, but if and when a coach points to something they will be able to observe it from their interface as well.

06 Jun 2018

Can we ever?

She had just had a nightmare. But this time it was different. It wasn’t the usual running away from her co-workers, and work. But it was unusual, since she succeeded; not by running away but rather there were no co-workers or work. That didn’t happen in a very hypnotic way, it just was. So, it wasn’t a nightmare really, she thought, but in a way it was. What will she do? Work never really ends. Does it? There is always more to do. Even if you like your job a lot, you still don’t like it every day. And her, she disliked her job. She didn’t only not-like it, but disliked it. But since it wasn’t a nightmare, what woke her up?, she wondered.

Nevertheless she was awake now, at 5:07am. Usually she wakes up at 5:40am precisely, then goes for a workout. So she was still in her sleep time according to her schedule, which she had followed religiously for 5 years now. She likes to plan everything because, she thinks, if she doesn’t do it her hours will fall apart, which eventually will become days, weeks, months… years? (maybe). In any case, not able to use her time perfectly meant she is losing an incredibly important part of her life. That’s all life has for us, our time, isn’t it?

But today her sleep had betrayed the schedule that she worked hard to establish, teaching her sleep her schedule. She couldn’t fall asleep anymore but still she waited for the alarm to go off. Not that she needed alarm anymore, but she set it for a situation if her biological clock fails her. She got out of the bed and carried on with her perfect routine. It was perfect until she reached her office. No wonder everyone was so chaotic on her walk to the office today. Her office building had developed cracks, from this morning’s earthquake. Being an old building, everyone thought this will happen in the next earthquake except they imagined a crumbled building not a cracked one. Suddenly from the crowd her co-workers found her.

“Hey, looks like we won’t have to work today”, said one.
“Is your apartment building ok? Are you ok? Did you feel it?”, asked another, worried about his boss.
“Yeah! Everything is fine”, wondering if it was the earthquake that woke her up.

As crazy as it may sound, it felt. This building was the only one in the city that was affected by the earthquake, which was just of magnitude 3 on richter scale. The city had seen worse, but apparently this was the last straw for that building.

Most of her colleagues didn’t seem affected at all. They were though, they were happy. They had a paid vacation basically, until the regional managers of the company were busy acquiring a new office space in the city. Nobody had been hurt in the event, so the city didn’t have a phase of resilience, residents were glad about that. Everything went on as it would. Her colleagues were happy and made plans to have a vacation, visit their family or friends, or just catch up on sleep or books. But she was upset, not only not-happy but unhappy to the point where she couldn’t stop thinking.

Depending on the new office location, she had to find a new place, move her stuff. More than that, she was worried if she will get a place with bedroom window facing east, so the sun could wake her up; if she will get a place from where she can just walk to the office; if she will find a place with a coffee shop with coffee she likes and ambience that she liked to read in, on weekend afternoons; if she will find a place that has small backyard for her composting setup; or if she should move at all? Sure she could just use the city metro service to work. But that meant having to leave for the office earlier than she did, waiting for the train; and what to do while waiting? And would she have to wake up earlier as well to leave early? What about her sleep schedule? …and on and on the thoughts went.

And there she sat looking out the window, while her colleagues enjoyed the days. She wasn’t running away from her co-workers. There were no co-workers, no work either. No job to dislike. Just like the dream that morning. She spend the time worrying about these immediate future decisions she had to make. Spent? rather wasted time. The order she setup was upset now, disturbed as she never planned for the disturbance. But can we ever?

20 Apr 2018

In the afterlife

In the afterlife, you don’t really exist. Your soul, so to say, exists. But it is still looking for the purpose. The life-long question of ‘Who am I?’ is still not answered. However, a different question ‘Does God exist?’ or ‘What is God like?’ is answered. Surprisingly for the soul it is not answered by God.

When you die, the soul goes into a different world where it meets up with other souls. It doesn’t really ‘go’ to a different world, it just realises it’s existence in the other world. During the transition the soul feels like it’s falling on to a heap of other souls. But as the ground gets closer, it feels just like falling on a planet like Earth which is over-populated with souls. Getting even closer reveals it’s not really over-populated, just that souls are connected somehow.

Finally, your soul falls onto the planet. Your soul looks at the people who pick you up. Your soul tries to ask them usual questions that any surprised person would ask: ‘Who are you?’, ‘Where am I?’. But they don’t seem to understand your language. Eventually, the soul decides to learn their language to survive. As your soul progresses, one day you ask: ‘how did I come here?’ Their response: ‘You mean, how were you born?’ They answer by saying: ‘God gave you to us’.

This makes you wonder as by that time, you don’t really remember anything about how you ended up there. The cognitive effort of learning a completely new language of another world, all by yourself, was so much that the brain found it more efficient to just forget other things. You aren’t you anymore, your soul is you. The question is answered but you don’t understand the answer anymore. The belief that God brought you into this world becomes stronger as you grow and then you die again.

15 Mar 2018

Data visualisation is an abstraction

Beers and Burrows define data visualisation as visual representation of data and datasets which communicate precise information and values1. In that sense, anything that does not lead to precise information and values in the dataset is not a data visualisation. As the information design is evolving it is becoming harder to define data visualisations, for instance force layouts are not precise representations of the data. Force layouts are an abstraction to a level that it is almost impossible (if not impossible) to get to the actual data underlying the relationships between the nodes. The distance between the nodes in a balanced force layout is irrelevant, it’s just the best layout the indeterministic algorithm could find. The edge between 2 nodes symbolises relationship between them , but it is not apparent how ‘closely’ those nodes are connected. Some visualisations try to encode the ‘closeness’ in the visual stroke weight of the edge or colour of the edge. From visual perception studies, we know that for humans (target audience) position, distance, length is more relevant as compared to thickness or colour.

Is there a loss of data when we turn it into geometric shapes/mathematical representations?

One could easily argue that even with Excel sheets there is loss of data because of our inability to absorb all the numbers at once, that argument would be true. Visualisations are not perfect tools that can make humans process all the data at once in their head, in fact that inability led to invention of information design field. This loss of data, or let’s call it data abstraction, is more important these days than ever before because of how ordinary people (not data experts/literates) engage with visualisations. Data visualisation has become a medium to make arguments (climate change visualisation), represent news(US election 2016), marketing, advertisement and much more.

According to Periscopic, a data visualisation studio, data can be used to do good. The claim is incredibly abstract and hard to measure. Visualisation designers believe that they can do good with data by being true to datasets2. As observed by The work that visualisations do2, one of the designers said:

Sometimes it happens [that clients want to tell stories that are not present in the data], but then we just show them that they story isn’t there and that we cannot force the data to tell that story.

One of the interviewee in the same paper said: “The data can only be the shape it is”. The intentions of visualisation designers may be appreciable and as neutral as possible, but visualisations they produce still might be hiding the truth (partially or completely3). The unintentional lie of omission might always happen, as a designer is also human and cannot perceive the data all at once. Sometimes even a design decision can motivate omission of important data. After all data visualisations are just another abstraction of the data we have.

Measuring data abstraction

Like Edward Tufte’s idea of measuring data-ink ratio is quantifying the idea of information density in an data visualisation. Could there be a way to measure the abstraction?

10 Jan 2018

Visual parameter mapping in data visualisation

Since the first ever data visualisation, we as humans have been trying to map data to visual features of different kinds of shapes. The use of shapes have evolved over the centuries, and so have the methods to collect data. The data is/are becoming more complex with every passing hour. The raw data from innumerable sources — satellites, telescopes, marine sensors, GPS in our phones, security systems, economics, crime, refugee crises, trade, agriculture, energy, wearables, online advertisement, photos we click, blogs we write, texts we sent — is becoming more and more immense. And so are our visualisations. In fact, human visual perception has reached it’s limits to interpret all the data we collect at once; so have our brains gave up trying to understand the numbers.

What does it mean when Syrian Observatory of Human Rights say that about 475,000 people have been killed?

For comparison, the population of Boston, Massachusetts, USA is estimated to be 617,594 in 2016; and it’s the 10th most populated city in USA. That number is also the approximate population Costa Rica and Ireland individually.

Here is a visual explanation of that 475,000 people.

Power of parameter mapping

Of course, encoding data into visual parameters has the power to explain those numbers to us humans. That power has allowed not only experts to understand data in a better way, but also the people who cannot grasp the comparisons done with numbers. The famous Florence Nightingale’s chart helped the people at that time to understand the cause of mortality with much more clearity, and as we know today that it wasn’t the first report created by her and her team.

Rose diagram image

There were other ways that these numbers were communicated with the administration, but it was hard for anyone to pin point the cause. So visual parameter mapping made a huge difference to the soldiers in the field in those days.

It can be argued that the field of data visualisation exists just so we can find ways to map data values to the visual parameters in an effective way. It doesn’t always work the way we would like it to work, that’s why we need to design the visualisations.

Non-visual parameter mapping

Visual parameter mapping is not always the best way to represent the data. It’s one of the ways that visualisation creators represent data. Although most data visualisations are done by visual parameter mapping one way or the other, still there can be cases where parameter mapping is not present in the visual representation at all. To start the discussion, I would like to mention the sound of wikipedia, which uses visual parameter mapping but not for every part of the visualisation. In fact, the project can be experienced even without a visual interface (without a screen).

Link visual creation from data is called data visualisation, the audio creation from data is called data sonification?

Let’s consider IBM News Explorer, the complex network visualisation in the middle of the page is completely non-deterministic—every time we load the page the same data would look different. The parameter mapping is done behind the scenes that further calculates the indeterministic layout that we finally see on the screen. It is not obvious how to understand data from the visual encoding being used. The closer nodes are more closely related and the farther ones are less. But how closely related are they, how do we even know the measure of closeness? Nevertheless, it helps us navigate through the innumerable articles that can be there on the internet about any given topic.

10 Nov 2017

Preamble of today's needs

But even while Rome is burning,
There’s somehow time for shopping,
at IKEA…
See, when I moved out of the house earlier this week
toting my many personal belongings in large bins and boxes, and 50 gallon garbage boxes,
my first inclination was, of course, to purchase the things I still “needed” for my new place…
You know, just the basics, a shower curtain, a bed,
and, oh, I need a couch and a matching leather chair,
and a love seat, and a lamp, and a desk and a desk chair,
and another lamp for over there.
And, oh yeah, don’t forget about the sideboard,
that matches the desk and
a dresser for the bedroom.
And oh, I need a coffee table and a couple end tables,
and a TV stand for the TV I still need to buy.
And now that I think about it,
I’m going to want my apartment to be
my style, you know, my own motif.
So I need certain decorative to spruce up the decor.
But wait, what exactly is my style?
Do these stainless steel picture frames
embody that particular style?
Does this replica Matisse sketch
accurately capture my edgy but professional vibe?
Exactly how edgy am I?
What espresso maker defines me as a man?
Does the fact that I am asking these questions
preclude me from being a,
quote, “man’s man”?
How many cups, and plates,
and bowls should a man own.
I guess, I need a dining room table, too…
And a rug for the entryway, and bath mats.
What about that thing that’s sort of like a rug
but longer?
Yeah, a runner.
I’m gonna need one of those.
And I am also gonna need — 
Hell, what else do I need?

07 Oct 2017

Art of the march

screenshot of the webpage

This project is an archive of the posters that people carried and displayed at March for Women in Boston in 2016. It is a representation of political dissent of people. It represents a method of persuasion citizens try out in a civil/political discourse. This project was done in collaboration with Dietmar Offenhuber, Nathan Felde, Alessandra Renzi, and Siqi Zhu.

The posters were placed (in a way mounted) on the fence of the Boston Common at the end of the march. These were collected by Dietmar Offenhuber, Nathan Felde, and Alessandra Renzi. Later on all of them were digitised (photographed). The entirety of this archive was made online, including the data that was created by manually tagging each of approximately 6000 posters. The project is still in progress. Check it out at .

23 Sep 2017

Where are we taking Liberty?

With the end of World War II, the empires of colonial era started to break, giving rise to a lot of sovereign states liberating the residents of the planet. But those states were divided in different ways keeping the conflicts alive. These conflicts gave rise to modern day warfare, arguably even worse for the people living in conflict zones, making sure that liberty stays unequally/unfairly distributed.

Considering that even USA (self-proclaimed most liberal country) has compromises on distribution of liberty of people. Even though policies are in place for equality and fairness and justice, socially (in practice) it’s not like that. It’s still dependent on your social interactions. Most recently, this came to light when current president banned transgenders from serving in the army, which I consider to be an example of social inequality influencing policies just because biased people came to power.

Role of internet

In 90’s internet started getting popular giving voice to people who socially are discriminated against. It gave rise to a lot of movements asking for equality. Most famous examples being the Black Lives Matter, Arab Spring, Feminist and LGBTQ movements. Turns out we needed them all along. On internet, people felt more liberty to say it out what they feel would not be listened to (or considered a taboo). People felt empowered when they found communities online with opinions similar to their own. They feel part of the society as internet enables them to find people like them which was previously almost impossible because of social and physical barriers.

Even if someone is one in a million, there are 3600+ internet users like that person… internet users being more than 3.6 billion. So there is a very high likelihood of finding someone with similar opinions as you.

The decline

The decline of trust in this family liberal place, called internet, started when people learned about hacking and Julian Assange told the world how our own data is not safe on internet. How far we are from controlling it. Later this fact was reinforced by Edward Snowden, and the trust was all lost. Statements from Facebook and Google didn’t help at all. Even normal people started to be scared of having a digital footprint (even now they are). No matter how much security each of the digital services provide, it’s hard to gain back that trust.

It takes years to build trust. A moment to break it.

Services like WhatsApp, Signal and Telegram providing end-to-end encryption, making that their marketing strategy is the world we live in. Our data should be ours, security shouldn’t be a feature but a foundation. Trust of users like us goes away even further when CEO of Google comes to stage and lies by saying: “Farmer and Professor see the same results for same keywords”.

The hope

Last year when Apple stood against the court’s orders and decided not to break into an iPhone, the hope in us came back. Even though they could do it, but making that software once would mean that it exists and anyone’s phone could be broken into. Tim Cook, CEO of Apple, told the press boldly that they have decided to defy the orders. A lot of Apple users felt pride in what Apple stands for once again (not everyone though). This debate turned into a revolution both for and against Apple. But what Apple was trying to do was important. Internet is one of the places (if not only) where everyone literally has equal rights. Apple was asking not to spoil that. Many companies including Google, Facebook and Amazon supported Apple giving us hope of controlling our own data hence encouraging liberty on internet. Because no control means we can’t express freely. But that hope wasn’t there to last long.

The decline of hope

Recently Apple removed a lot of VPN apps from Chinese iPhone AppStore. This action was done to comply with China’s regional censorship laws. VPNs allow users within a firewall to pretend that they are outside the firewall, as a result avoid the firewall restrictions. In case of China, the Great Firewall has isolated Chinese people from the viewpoints of the rest of the world, stopping communication & hence taking away liberty of citizens to freely access internet as they please, restricting their access to internet (which won’t harm anyone and its against net neutrality). Controlling what information citizens are allowed to access is in my opinion taking away liberty (like in a prison, books and movies are scrutinized).

China is not only country doing it, Russia and Iran are other famous examples. Almost every country censors something, be it books, movies or internet. Censorship is the first step towards discouraging people’s right to information. The act of banning VPN services is a step further into discouraging that right. More importantly Apple is playing a role in it now. And no-one is talking about it in the more liberal world as it doesn’t directly affect us. Hence taking liberty distribution a step towards colonial era.

page 1 of 3
next >