Blog

Liberating annotations in Personal Documents from the Kindle

UPDATE: It seems that now the notes from personal documents also show up in the Your Highlights page. So there is no more need for all the tricks below. I’ll keep this post here just for historical reasons anyway.

UPDATE 2 (01/04/2014): I’ve been making more tests and it looks like only my Paperwhite started uploading the notes from Personal Documents, after the latest firmware update. However, that didn’t happen for any personal document, just to one book specifically. I’m still investigating what made that book special in Amazon’s view… But for now, it seems like the post below is, after all, still relevant.

Now that I am using my Kindle to read academically, I’ve realized that I will need to make a very good use of the Notes and Highlights feature. I am still working out a system on how to make my notes more useful in my reading (which will become a post on its own), but before I put such a huge effort in note taking, I want to make sure that the notes will be easily retrieved and safely stored outside of the Kindle.

For books that have been bought from Amazon, getting the notes out of the Kindle is not a problem at all. Amazon has a page that shows all your notes from all your Kindle books, taken in any of the devices that you have connected to your account (you have to be logged in to Amazon to be able to see it). You can just copy and paste from that page and that’s it. Or even better, you can clip it to Evernote.

The problem is when you are annotating documents that have not been bought from the Amazon store – what Amazon calls “Personal Documents”. For those documents, notes and highlights can be synchronized across devices, which is very handy, but unfortunately they are not uploaded to the magic page with all your highlights.

“My clippings.txt”, almost a solution, but not quite

The “My clippings.txt” file is great and there are amazing tools that extract the notes from this text file and allow you to export them to many formats (this one is my favorite).

For those who don’t know this, each time you take notes in your Kindle device (not in the desktop or mobile apps), the device writes the note to that text file. It is a one-way operation, though; more like a back-up. If you delete a note from the book, it will not be deleted from the “My clippings.txt” file. If you edit the file and put it back in the Kindle, it will not carry your edits back to the book annotations.

But there are two problems, at least for me, with relying on this text file for my note-liberation needs. One is that this file is device-dependent, so the notes that I take while reading a Personal Document in another device will not show up on my Paperwhite “My clippings.txt”. The second problem is that the Kindle reading apps (for desktop and mobile) do not produce such a file (otherwise it would just be a matter of merging different files and exporting them).

In practice, this means that any notes that I take while reading my Personal Document from my Android tablet will appear on my Paperwhite while I’m reading the book, but they will not be exported in text format to outside of the devices.

.mbp and .empb, the annotations file

As I said above, the “My clippings.txt” file is merely a backup of your notes. The notes themselves are stored in a “side” file, named exactly like the e-book file itself but with a different extension: mbp, which is a file that holds Mobipocket annotations (Mobipocket was bought by Amazon and now it has basically disappeared).

These files are not text files, and although I haven’t found any software that can edit them directly (except from the Kindle for PC itself, which opens them without troubles together with the original book, but doesn’t allow for bulk exporting of all the notes), there is a software that some nice soul posted online that is able to extract the mbp notes to text.

(One tip for those using the MBP note extracter above: edit the file mbp.parameters.txt and remove the “#” from the beginning of the line that shows BKMK_PAGE_FACTOR=150 – as this will write the location of the notes to the file.)

But there is one problem here. Up until late 2012, these files were just as Mobipocket had created them, and using tools such as the one posted above or software like Buzzworthy, you could simply extract the notes and live happily ever after. But since then, Amazon started encrypting the notes (file extension .mbp1 or .embp or .mbs) and those software no longer work. Not fun.

The solution to this issue is posted on Buzzworthy’s website: you have to downgrade the Kindle app in your mobile/tablet in order to have your notes in the non-encrypted format to be able to read the .mbp file.

Wrap up

So, to sum up all that is written above, this is how I am managing my notes in my Personal Documents in all my Kindle apps/devices (one Kindle Paperwhite, one Android Smartphone, one Android tablet, and two Kindle for PC installations):

1) I make sure my Personal Document is synchronized across my devices by using “Send to Kindle”. Although it is possible to add personal documents directly to my device(s), I use Send to Kindle, so that the document and the notes are synchronized across all the devices (Cloud and Desktop readers, unfortunately, do not sync personal documents). If you are sending the document via e-mail, make sure to add the word “CONVERT” to the subject of the message.

2) To extract the notes I take directly in the Paperwhite, I export the “My Clippings.txt” file to Clippings Converter and from there to Evernote. And because this is the simplest method, I am trying my best to do all my annotations there. But if I need to make notes on the phone or the tablet, it’s not a big problem. Read on.

3) I keep the Kindle App in my Smartphone with an old version of the software (namely 3.6.0.87). This way, I can get the unencrypted .mbp file out of the phone if I need, and as I sent the document via Send my Kindle, this file will have all my notes, even from the other devices with newer Kindle apps. I just have to make sure that the notes have been synced before transferring the file from the phone to the computer (via USB, Dropbox, Bluetooth, whatever works). The .mbp files will be inside the “kindle” folder, together with the books themselves.

4) If I need to extract those notes from the .mbp file, I use the handy software mobipocket notes file extractor to get them safely out of the Kindle ecosystem. Setting the “BKMK_PAGE_FACTOR=150” preference makes the software print out the locations of the notes as well.

5) If I need to see the notes in my Desktop, I copy the personal document and the unencrypted .mbp file manually from the phone to the Kindle for PC, by simply dropping those files in the “My Kindle Content” folder in the computer. Then I can open the book and all my markings in my Desktop are there (and this is very useful while cleaning up my notes on Evernote Desktop, using a proper keyboard, and with the book and notes open right in front of me).

I tried adding notes in the desktop and sending the .mbp file back to the phone to see if the new notes would sync and show up in the other devices, but unfortunately this didn’t seem to work. I could see the note in the phone, but it was not replicated to the other devices. It only became visible in the Paperwhite once I edited it on the phone directly. So I consider the installation of Kindle for PC as read-only – I never edit the notes there so that they won’t get lost.

—

So that’s it. It’s a lot of pain to go through just to get your Personal Documents’ notes out of the Kindle. It bothers me a lot to think that this is Amazon’s way of pushing us towards buying our e-books from them so that we can enjoy the much better synchronization possibilities that Amazon books have in that matter. I have always preferred to buy my books from Kobo and convert them to .mobi using Calibre to send to the Kindle, but I don’t know if I want to go through all this trouble all the time… I can’t wait for Open Annotations in epubs to become a reality and to have some impact in Amazon’s way of dealing with this. After all, those are my notes and I want to make sure they are easily readable!

The next post will be more about the quality of the notes and how I am planning to use them in aiding me in reading critically using an e-book reader.

Reading academic books in the Kindle

I love my Kindle as much as I am able to love a physical object, and I honestly prefer reading in a Kindle than reading a “real” book. Ever since the first time I ever heard the idea of e-readers, I already knew that I would love them, and it never even crossed my mind that I would have any kind of longing or nostalgia for the smell of a real book in my hands, the colors of the cover, the weight of the paper, or anything like that.

However, now that I am reading academic books in the Kindle (not just articles, but whole books), I am starting to feel a bit at a loss. Not that I think that Kindles are not up to the task, but I am searching for a way to make the task of reading critically more productive in my e-reader.

What is good about e-readers

In my opinion, there are three main and two bonus reasons that make me think that e-readers are better than “real” books:

1) It has a embedded dictionary. As a non-native English speaker, I can’t even begin to express how much the dictionary has helped me with my vocabulary. I think I have an acceptable English vocabulary, but by reading fiction I noticed that I still lack so much! I am learning so many new words now that I don’t need to do anything else than to hold my finger on the screen to see what those words mean. Before, in the “real” books time, I would just guess from the context and move on. And although I haven’t been reading books in my native tongue, I am pretty sure there are plenty of words for me to learn in Portuguese as well.

2) It is able to synchronize with my phone. I don’t even need to take my Kindle along with me to be able to read anywhere. I just sync the books on my phone and I can keep reading wherever, even during those five minutes waiting in the supermarket line. Very handy with those books that I can’t put down.

3) It holds all the books I need at any given time. I can have more than one book with me at once. No issues with having to leave books behind because they were too heavy or thick to fit in my purse or suitcase.

Bonus reason I: reading in the dark. I am the happy owner of a Kindle Paperwhite, and I just love to be able to read in bed in lazy weekend mornings, while my dear SO is still asleep. I get to enjoy my book without disturbing him at all. Other couples have mentioned this benefit when one likes to read in bed before going to sleep and the other one gets disturbed by bedside lamps.

Bonus reason II: “Time remaining” feature. In the beginning I was a bit annoyed with Kindle’s “Locations”, because I had no practical way to figure out how much I had left until the end of the book (I never really got used to the mental calculations needed to convert locations into pages), but now my Kindle shows the time left to finish the chapter and the book, if I want. I found that a very interesting and useful feature (especially now that I know how to reset the reading time, if I feel like the estimation it gives me is wrong). In fact, I actually miss that feature now that I’m reading The Glass Bead Game in the “real” book format!

What’s not so good

But now we come to the part I don’t like about reading in the Kindle. For fiction and those very linear non-fiction books, I think the Kindle is just perfect. You start and keep reading until the end, stopping eventually to do things like eating and going to work. When it comes down to academic reading – those books full of self references, tables, images, footnotes – going back and forth is very impractical. And yet, going back and forth is required if you want to understand the author’s arguments.

I am trying to figure a way around this shortcoming of the e-readers, so that I don’t need to print the 300 pages of the book I’m reading now in order to understand it.

Highlights and notes have been my friends, just like in university times. In addition, I found that the Kindle exports my notes and highlights in a text format, and it is easy enough to extract those notes to other tools such as Evernote (my favorite note-taking software), and then I can search them very easily. It would be even easier if Amazon let us see the notes online also from books that were not bought at their store but rather uploaded directly or sent to the Kindle.

Just the fact of highlighting and commenting on the text already increases my understanding rate, but by no means it is enough. I’m now going to experiment in making summaries from the notes from each chapter and printing them out (or having them in my computer or tablet beside me), to help understanding the argument in the following chapters. If anything good comes out of that, I will write about it here.

EC-TEL 2013: wrap up

From September 18th to 20th, I had the opportunity to join the Eight European Conference on Technology Enhanced Learning (EC-TEL 2013), in Paphos, Cyprus. While the main discussions were not directly related to my own research at the moment, it was certainly productive to hear what is going on in the field of TEL, to broaden the views on the topic and, of course, to spot some possibilities for collaboration.

Scaling up and sustainability – The main topic of the conference was “Scaling up learning for sustained impact”. The slogan reflects the attention that the TEL community has been giving to the fact that so many of the advances in the area end up not surviving for long. Implementing technology for education in the real world is a completely different beast than setting up a controlled experiment or a pilot project, and in rare occasions do the initiatives presented in an academic setting survive the end of funding and/or when managers stop monitoring the project.

But, as it happens, the issue of sustainability was given more importance in the keynote speeches, in the panel discussion and in the workshop EC-TEL meets ECSCW (which took place after the EC-TEL conference). Unfortunately, most of the presentations that I attended still followed the same pattern of limited experiments, with limited results. And it is understandable: implementing large scale studies in the real world is easier said than done. After all, as Peter Brusilovsky sharply noted in the panel discussion at the end of the conference: “we’re not scaling because we’re not paid to scale. We’re paid to publish papers”. Sad but true. Still, an interesting take-home message: designing something that does not survive in the real world is almost as bad as not designing anything at all. And one extra challenge: even for successful projects, the pressure from the government and the need to comply with standardized testing and assessment can be very difficult to face.

Serious Games – EC-TEL offered some interesting tracks on Serious Games. Apart from our tutorial on Seamless Assessment for SGs, there was a whole scientific session dedicated to SGs on the last day of the conference, titled “Games – Serious and Fun”. Games for learning were also somewhat present in other works, where researchers showcased the use of gamified applications for children in tablets, tabletops and smartphones. [The proceedings of the conference are already available here.]

In the “Games – Serious and Fun” session there were some interesting projects: the EMuRgency project with its HeartRun mobile game; the University of Cadiz’s cooperative game for learning German language “The Hidden Room”, presented by Anke Berns; and the Business Process Modelling game for health case institutions called ImPROVE, developed by INESC-ID. Albeit interesting, these experiences were somewhat limited to their specific fields, and still did not employ Learning Analytics or in-game assessment – they relied in pre and post-questionnaires or in the observation of the game being played by the target audience. That said, all the games had received positive initial evaluations and are now in phase of improving the implementations based on the feedback and results obtained so far.

The work presented by the Kassel University, in Germany, differed from the other four presentations in which it did not focus on one single game, but instead it presented an effort to define a standardized description of learning processes to achieve learning objectives. The modelling language proposed by them is quite similar to UML diagrams, and it focus on reusability of the components for specific learning outcomes. This work, in particular, is the research that I found most relevant for the work that I am developing currently.

Collaboration in learning – The topic of collaborative learning was also quite relevant in the conference. Many of the experiences described highlighted the role of collaboration in learning. One of the most interesting concepts for my own work in collaborative games was the Metafora Project, which, among other objectives, aims to define a visual language to help in planning and following up on the process of learning collaboratively. Its emphasis in the importance of planning and reflecting on the activities as the crucial point for effective collaboration was an eye-opener for me and one more point to include in the investigation of collaborative SGs.

—

All in all, the experience in Cyprus was definitely a good one. Apart from Paphos being a gorgeous place, I do think that this visit will result in good advances for my work here in Italy.

EC-TEL: Tutorial on Seamless Assessment in SGs

As I mentioned previously, I participated in the Eight European Conference on Technology Enhanced Learning (EC-TEL 2013), in Paphos, Cyprus. I helped on the organization of the Tutorial on Seamless Assessment in SGs, offered by partners of the GaLA project.

(For a more complete description of my experience in the conference, check out this post.)

Our tutorial took place in the morning of September 18th. It was a small group – out of the 16 people who indicated that they would participate, only 7 actually showed up. Our idea was to have a short theoretical introduction and then proceed with a hands on demonstration of Learning Analytics for Serious Games (SGs) in action.

In the theoretical part, we explained the importance of in-game assessment, as opposed to the traditional pre and post-tests. It is possible to do better with SGs than to use them just as an alternative way of presenting content; we are moving in the direction of using the game itself to assess student competences. Michael Kickmeier-Rust gave an overview of assessment based on the Competence-based Knowledge Space Theory (CbKST), in which a knowledge domain can be represented in a finite set of atomic competences and a prerequisite relation between those competences, and a person’s competence state is described by a subset of those competences in the domain. We also presented the GLEANER (Game Learning Analytics for education research) framework developed by UCM-e in Madrid.

The hands-on part was more interesting, as the participants had the chance of seeing what in-game assessment looks like. Kickmeier-Rust gave an overview of a few tools implemented to support in-game assessment and their application in games. Afterwards, the participants had the opportunity to play a game connected to the implemented GLEANER service and see, in real time, their answers being analyzed and reported by the service. It is a simple game and the reports the service offers are simple as well, but it was a good sneak-peek on what in-game Learning Analytics can look like.

The most interesting part, in my opinion, was to see which tools are already implemented and available for use, especially the ones developed under the NEXT-TELL project. The participants of the tutorial seemed also interested in the service-based approach, as it allows for the reuse of software parts in their games with small effort.

For more information:

“What’s your PhD about?”

Such a simple question, and one so difficult to answer. I believe that one must be ready to explain it in 30 seconds, no more. If you can’t explain it in one breath, then it’s probably not clear enough even to yourself.

This week marks the end of my first year of the PhD, and only now I can finally say what I want my PhD to be about in the elevator-pitch format. But – there’s always a but – I’ve been working in so many apparently unrelated topics that it gets a bit complicated to explain how do I intend to get from A to B within the 3 years of my program.

I’m having the opportunity to participate in an academic conference, and I’m getting asked this question all the time, over coffee and lunch break, so I thought it was more than the time to get my pitch straight.

So here’s my PhD topic:

To create a tool to automatically capture and represent over time the emotional states of players in a collaborative Serious Game, using as input their interactions in a chat-room like environment within the game. The tool will be implemented as a service, so that the information is exposed and available for other services, in particular for teaching/learning analytics reporting tools and for real-time in-game adaptation. This data will be modeled and represented in an open way, so that this information can be aggregated with data about players’ emotions coming from other sources (gaze, body language, intonation, etc.), contributing to the accuracy of the model.

I’m still tracing my path to get there, and so far I have been concentrating in smaller parts of this larger goal. Here is what I have been doing in these last 12 months of my studies:

  • Collaboration and cooperation in Serious Games: Although the term “collaboration” seems to be getting more and more in vogue, it seems like in the field of games and SGs there’s some confusion about what exactly does it mean. One of the current focus of my work has been to understand collaboration and cooperation in SGs and how do the game mechanics and learning objectives are interwoven in this specific setting. The next step will be then to define which emotions in Collaborative games would be relevant to try to capture and represent in the service.
  • SGs Analysis and Design: I will need to create a representation of the emotional states of players during the game, and to compare this information to the flow of the game. We have been investigating how to make this representation, representing the game flow and its mechanics.
  • Services for SGs: But for a service for SGs to be useful, it needs to exist in an environment that is fertile for services. There needs to be standards, and common languages, and repositories and meaningful ways of describing services, so that other people can take advantage of them, use them and help improve them. I have also been involved in trying to define this environment, initially represented as a catalog for services for SGs, but that could eventually evolve into some sort of framework(1) for the development of service-based SGs. UPDATE: The Services Catalog is now available online!

So this is what I have been doing. Step by step, hopefully I’ll be able to get to the end of the program with some nice tool to the SG development community to use.

—

(1) I don’t like the term “framework”, as it sounds too vague and seems to be used way too liberally. Or mostly because I am never sure what exactly is it supposed to mean. But, for the lack of a better term, I’ll stick to it here.

A mind map of open standards relevant to Serious Games

It’s been less than two months since I started my PhD. I’m a new student, and my broad research topic, Serious Games, is also new to me. Unsurprisingly, I’m still unable to see clearly to where I should point my future research, and even less so to identify the thread that connects what I had been doing before and what I’m going to do next.

It’s still time to make sense of all the information that there is around me and to identify what has already been done. This can be a daunting and frustrating task… also because sometimes it seems all the good ideas are already being pursued!  (Doesn’t it always seem like that when you start reading the literature?)

In any case, in order to start moving in some direction, I decided to try to map what open standards relevant to serious games, gaming platforms, learning analytics and learning management systems exist already. The goal is to stand on the shoulder of giants, not to reinvent the wheel, right?

My first step into that direction was to create a mind map to start collecting links to existing work that I can find. This map is a work in progress, but I’m sharing it here nevertheless. If anyone has anything to add, please let me know in the comments or on Twitter (I’m @carvalho_mb).


Create your own mind maps at MindMeister

And this is where it starts…

I am one of those people. Yes, those, who consider themselves writers.

Even though I have never published anything I have written – other than, of course, academic work – I know I am a writer. Because I see my world in prose. When I am especially happy, the prose turns into poetry. I describe things in my head with words; there are always labels for everything. I love to use these abstract little symbols to put order in the mess, to understand, to create.

This is the main reason why I decided to write a blog about my PhD. It will be a way of putting some order in the next three years of my life.

So, welcome to this space! I hope these words can be as useful to you as they are to me :)