Planet Knight-Mozilla OpenNews

I’m not big on offering advice when it comes to learning how to code.  Everyone learns differently and has different goals; my experience isn’t necessarily instructive.  But I seem to be getting asked the same question more and more often: someone wants to be able to make cool things for the web, and they don’t know how to get started.  Here are some thoughts on how to keep your head on straight while you’re trying to learn. Take them all with a big grain of salt.

Work on an idea you’re excited about.

As a learning tool, there’s nothing more powerful than having an idea that you’re genuinely excited about.  There are two big reasons for this:

1. Learning how to code is full of exhilarating lightbulb moments, but it’s also full of hours of banging your head against the wall, not understanding why something doesn’t work or what to do next.  You will get stuck.  You will get frustrated.  Being excited about what you’re building will help you power through those times.  Rather than lose interest when you hit the wall, you’ll go above and beyond to find a solution.
   
   
2. You’ll care about doing it well.  You’ll learn a lot more when you’re interested in the end product not just as proof that you did it, but as a project you wanted to build for a reason.  You’ll think about the details, tradeoffs, and design considerations.  You’ll question your assumptions.  You’ll refine it over time instead of checking it off the list as soon as it satisfies the bare minimum.

Take your time.

There’s a whiff of infomercialism in the air these days, this notion that if you take the right online course or buy the right book, you can just skip ahead to being a master coder.  The 8 Minute Abs version of learning to code is like the 8 Minute Abs of…well, abs.  It’s an enticing fiction, the notion that as long as you’re really clever about the process and you buy the right accessories, you can skip most of the actual doing.

It’s true that it’s easier than ever to do amazing things quickly, and that you’ll have lots of bursts of insight that make it all feel quick and easy.  You’ll add a few lines of code and make something great happen on the screen, and you’ll be ready to take on the world.  But this is a long, gradual, and humbling learning process.  There aren’t a lot of shortcuts for building up the context, the “why” behind different approaches and frameworks and the code underneath them, and that’s what will allow you to go off-script and improvise awesome stuff.

You’re not checking off a box when you learn to code.  You never stop learning.  But that’s part of the fun!

Don’t overload on tutorials.

Tutorials are easy to find.  In five minutes of Googling, you could grab 10 of them on every coding topic you care about.  But they won’t stick nearly as well as hands-on practice and experimentation.  And when it comes to code, it’s much easier to read about the “how” than the “why” (in part because it’s much easier to write about the “how” than the “why”).  You want healthy portions of both (one of the peculiarities of learning to code for the web: you’re constantly learning in both directions on the ladder of abstraction, learning new tricks you don’t fully understand and learning more about how your old tricks actually work).

Tutorials are great sometimes, especially when you have a well-defined task you’re trying to figure out, but use them sparingly.  Get your tutorials on a just-in-time delivery system.  Don’t just go on a shopping spree and expect to download all that information into your brain.

Don’t worry too much about the “right” way to do things.

As you learn to code, you’ll probably feel self-conscious about whether you’re doing things the way you’re “supposed” to.  You’ll come up with some weird approach that does what you want but you’ll be certain that it’s an absurd solution and that if you were a real coder you could do it the correct way.  Coders actually have a word for this situation: we call it “coding.”  

There are lots of reasons it’s easy to feel self-conscious, especially if you’re just starting out:

Find a community.

Don’t be a hero and try to power through the learning process on your own, surviving only on twigs and berries and O’Reilly books.  Your fellow learners are your best resource (and remember, all coders are learners).  Make friends with fellow beginners, but with more experienced coders too.  Go to meetups.  Ask lots of questions.  Get feedback.  Offer feedback.  Like somebody else’s work? Tell them so, and tell them why.  And don’t forget to share your own work and the lessons you’ve learned.  You’ll have much more to teach others than you realize.

There’s always more to learn.

Learning to make stuff for the web means going at your own pace and getting more comfortable with the fact that there will always be a lot left to learn.  For every thing you master, you’ll also find out about ten other things you didn’t even know you didn’t know.  And the web moves fast - even if you could learn it all, by the time you finished, so much more would be possible.  So don’t get overwhelmed.  Just worry about the next thing you need to learn, have some fun, and don’t be afraid to get in over your head.

Some fine print: besides being generally skeptical of my advice, you should keep my biases in mind.  I’m an accidental web developer who just sort of learned along the way because I had things I wanted to make.  That may not suit you.  Also, learning by doing without a grand plan works pretty well for the web, but don’t assume the same is true of programming generally, especially complex or high-stakes things.  If you’re serving up code to millions of users, managing a bunch of important databases, or writing software for banks or ballistic missiles, you should probably get some real computer science education and care about the “right” way to code things.

May 13, 2013 11:06 AM

Data is a buzzword nowadays. Whether it’s sifting Big Data to influence business, or the promise of Open Data to transform government, or Data Analytics winning elections, data is constantly in the news. But one thing that gets glossed over in all the buzz is that data is hard. Really, really hard. One of the hardest parts is cleaning, standardizing, and formatting data in a way that journalists and others can start to work with. These are real challenges faced by newsrooms and we’re hoping to make some of that a little easier with two new Code Sprints we’re happy to announce today.

First up: Dedupe

One of the biggest problems with data sets is figuring out if information in one set of data is the same as information in another. When you have a small set of data, the work is pretty straightforward. But as your rows increase, the work becomes daunting. Derek Eder and Forest Gregg at Chicago’s DataMade have been working on an automated process for deduplification of data, and we’re happy to help get it to a state where running it through huge datasets is as simple as a few calls from the command line.

A clear early use for the tool is in deduplifying campaign finance records, which can often be a slog. We’ve recruited the help of Derek Willis and others from the New York Times—a href=”who know something about

The DataMade team have done a great deal of heavy lifting already—“we’ve solved the most of major engineering challenges of scaling up on large datasets,” DataMade’s Eder says—but getting a lower barrier to entry on the tool is time and money well spent. If you can program Python, you can fork and start running Dedupe today. If you want to wait for the simplified version, we’re expecting development to wrap up early this summer.

Next up: FMS Parser

The US Treasury releases a statement of, essentially, the Federal Government’s checkbook every day at 4pm EST. Unhelpfully, they release it as a straight-up text file or a PDF. Newsroom developers and info-hackers Cezary Podkul, Burton DeWilde, Thomas Levine, Jake Bialer, Brian Abelson, and Michael Keller started work on scraping and parsing that daily statement at the Bicostal Datafest earlier this year.

The team got far enough along at the Datafest that they approached us about helping to turn it into an open API that any newsroom developer can access. With our Code Sprint grant, the team will take this once nearly-inaccessible dataset and transforming it into an easily accessible API that returns machine-readable JSON. In this time of cutbacks and budget wrangling, the FMS Parser should offer developers and journalists a new way to dive deeply into governmental spending.

The tool should see some immediate use too, as the team of developers working on it include newsroom developers at Reuters, the Daily Beast, and the Huffington Post (along with our Knight-Mozilla Fellow at the New York Times). While it’s still being developed, you can fork and follow at the FMS Parser Github repo.

Onward

A month ago I announced a reimagined Code Sprint application process, and we’re excited to help tools like this get the funding and attention they need through it. We’re always looking for developers and newsrooms with great ideas they want to build (along with newsrooms that want to betatest them), so please drop a line. Let’s do this!

May 07, 2013 02:41 PM

May 02, 2013 10:03 PM

April 10, 2013 09:00 PM

A little more than two weeks from now, Erika Owens, Greg Linch, and I will be holding an “unconference” at BarCamp News Innovation to talk about the theory, methods, and implications of media impact measurement. We’re hoping to have a pretty focused discussion, centered on implementation. In particular, I’ll be talking about tangible experiments I’ve been carrying out with the New York Times as my laboratory.

In preparation for the discussion, I’m hoping to get your thoughts and concerns about impact measurement (read up here) – what have we done right so far and what’s missing from the conversation? If you’re interested in these questions, make sure to join Greg and I’s google group and follow along. See you on April 27th!

April 10, 2013 04:00 AM

Last weekend, Stijn and I visited inkLink13, where I presented a few ideas about using data to cover the European Union. The meet-up was a first encounter between Budapest's tech and journalism communities. As both groups shared some of their ideas for the future of news in Hungary, the discussion was given a special weight by the actions of the Orban government, which has shown little regard for the freedom of the press and routinely rejects requests for information.

I picked the EU as a subject for two reasons: first, because of its undeniable influence on the everyday lives of all of its citizens, whether they live in Budapest, Berlin or Barcelona. Second, because even in 2013, covering the processes and discussions within the Union - especially inside the Brussels bubble - is not routine for many news organizations in the EU. Even Spiegel Online, with a journalistic staff of 150, does not yet have a permanent presence in Brussels.

While this is a real challenge to journalism in the member states, it is also an opportunity: the EU offers a blank slate in terms of government oversight. This creates a chance to re-design the technical and institutional architecture of how we cover government: What role can technology in general (and data in particular) play?

In this context, I think five aspects deserve some thought:

• Building the right data feeds ("publisher/subscriber accountability"). When I think about the role that data can take in monitoring government activity, static databases increasingly seem like a dead end. To be relevant for news reporting, data must be delivered in feeds, just as in wire services or social media. Information in such services must be current, well-prepared, filtered and easy to use both for human editors and for machines trained to handle those events which can be expressed through sets of rules.

  To some extent this comes back to the question of human-assisted reporting and the use of heuristics to answer key journalistic questions (the biggest, the fastest-growing, etc. items in a dataset). But I think that constructing automated feeds in journalism also makes a great deal of economic sense.

  For most data investigations, the effort required for automation largely overlaps with having a clean and reproducible "chain of evidence" in the first place. The ability to continuously update and perform an analysis is directly related to how well it was done in the first place. A monitoring workflow requires additional investment, but clever use of cloud computing and open source tools can help to turn these projects into cash cows, stories that keep on giving.

  There is already a growing number of such feeds, but they're rarely considered as news sources by journalists. stf's parltrack is one example, OpenTED and LobbyFacts and ItsYourParliament don't offer suitable, continuous streams of data.

  This rift is growing, presumably caused by diverging skill sets and language. Filling it will be beneficial to both sides, the feed providers as well as the journalists.

• Cooperate across countries and communities. This is a natural result of the complexity of the topics and the methods needed to cover them. FarmSubsidy.org - as well as the recent OffshoreLeaks investigation - are great examples for this, and the argument in their favor within the EU is simple. Most (data-driven) projects require some investment but can then be adapted to each of the 27 member states to provide stories for national media.

  At the same time, cooperations between coders and journalists are necessary, although by no means simple. I'm involved in a few such efforts, and differences in approaches, commitment, resources and interest have made each of them project management nightmares. Making sure that there are sufficient resources and a common approach to managing and planning work seems important in order to hacks/hackers cooperations more successful in the future.

• Use your rights. Those covering the EU need to make more intense use of the rights to information that have been established for citizens. Probably the most impressive example of this has been Access Info's investigation into the practice of extraordinary rendition flights within the EU. Their team leveraged differences in FoI regimes across member states to gain access to documents and communications even in cases where the country producing the documents had rejected the request.

• Report actions, not words. The goal of reporting on any government should be to break through political rhetoric. This applies particularly to data-driven reporting, where we need to move away from pre-framed datasets - such as national statistics - towards those data sources which can serve as evidence to actual behaviour: transactional spending, environmental regulations violations, lobbying expenditure.

  Getting the facts is made easier by the Commission's desire to measure it's member's activity in great detail: whether the handling of environmental affairs, consumer protection or state aid to companies and banks: the EU is the country of the ten thousand databases, in which regulation is often based on surprisingly thorough metrics (which, of course, don't always support the eventual decision).

• Tell a story. Facts can only have an impact when they are used to make a point about the real world. Understanding the EU also means understanding the technocratic language and process of Brussels. Going native in this environment is easy, especially when using a tool as complex and prone to "expertism" as data. So, more than ever, trends and patterns need to be translated into human scale and connected to the political reality of citizens.

  Again, FarmSubsidy.org is a good example: while they were able to make a strong case for the mis-allocation of the agriculture and fisheries funds, their story of big food companies receiving heavy subsidies was not strong enough to significantly influence the political gridlock.

  One of my favorite narrators from the Brussels bubble is Ronny Patz. Together with his colleagues at the BloggingPortal, he tells an updated kind of story in which Europe is considered as a real political place - more than just a trading ground for member state interests. Discussing Europe purely in terms of its members is, in many ways, now inadequate and distorting. In this regard, his blog and his approach are brilliant inspirations for journalists covering the EU.

For news organizations that don't properly cover the EU yet, it's high time to act. Still, this new environment should also be used for experimentation - with technologies, in collaborating with other organizations and in telling different types of stories.

The complexity of decision-making and regulation in the EU makes these approaches necessary, but in many cases they may proove to be useful at home, too - in national politics and even a city government.

April 08, 2013 07:00 AM

Back at the Hacks/Hackers Media Party in Buenos Aires, I announced the creation of Code Sprints—funding opportunities to build open-sourced tools for journalism. We used Code Sprints to fund a collaboration between WNYC in New York and KPCC in Southern California to build a parser for election night XML data that ended up used on well over 100 sites—it was a great collaboration to kick off the Code Sprint concept.

Originally, Code Sprints were designed to work like the XML parser project: Driven in concept and execution by newsrooms. While that proved great for working with WNYC, we heard from a lot of independent developers working on great tools that fit the intent of Code Sprints, but not the wording of the contract. And we heard from a lot of newsrooms that wanted to use code, but not drive development, so we rethought how Code Sprints work. Today we’re excited to announce refactored Code Sprints for 2013.

Now, instead of a single way to execute a Code Sprint, there are now three ways to help make Code Sprints happen:

• As an independent developer (or team) with a great idea that you think may be able to work well in the newsroom.
• As a newsroom with a great idea that wants help making it a reality.
• As a newsroom looking to betatest code that comes out of Code Sprints.

Each of these options means we can work with amazing code, news organizations, and developers and collaborate together to create lots of great open-source tools for journalism.

I always think real-world examples are better than theoreticals, so today I’m also excited to announce the first grant of our revamped Code Sprints will go to Jessica Lord to develop her great Sheetsee.js library for the newsroom. Sheetsee has been on the OpenNews radar for a while—we profiled the project in Source a number of months back, and we’re thrilled to help fund its continued development.

Sheetsee was originally designed for use in the Macon, Georgia government as part of Jessica’s Code for America fellowship, but the intent of the project—simple data visualizations using a spreadsheet for the backend—has always had implications far beyond the OpenGov space. We’re excited today to pair Jessica with Chicago Public Media (WBEZ) to collaborate on turning Sheetsee into a kick-ass and dead-simple data journalism tool.

For WBEZ’s Matt Green, Sheetsee fit the bill for a lightweight tool that could help get the reporters “around the often steep learning curve with data publishing tools.” Helping to guide Jessica’s development to meet those needs ensures that Sheetsee becomes a tool that works at WBEZ and at other news organizations as well.

We’re excited to fund Sheetsee, to work with a developer as talented as Jessica, to collaborate with a news organization like WBEZ, and to relaunch Code Sprints for 2013. Onward!

April 02, 2013 01:43 PM

EDIT THIS DOCUMENT

What’s in here?

This bibliography contains a smattering of pieces, long and short, that are relevant to the study of media impact. From blog posts and case studies to fully-developed frameworks for analysis, reading selections from this list should quickly get you up to speed with the field.

Where do I start?

There are so many good pieces in here, but these are three of my favorites:

• Tofel, Richard J., “Non-profit journalism – Issues around impact: A white paper from ProPublica,” February, 2013. http://s3.amazonaws.com/propublica/assets/about/LFA_ProPublica-white-paper_2.1.pdf

• Stray, Jonathan, “Metrics, metrics everywhere: How do we measure the impact of journalism?,” Nieman Journalism Lab, August 17, 2012. http://www.niemanlab.org/2012/08/metrics-metrics-everywhere-how-do-we-measure-the-impact-of-journalism/

• Linch, Greg, “Quantifying impact: A better metric for measuring journalism”, January 14, 2012. http://www.greglinch.com/blog/2012/01/14/quantifying-impact-a-better-metric-for-measuring-journalism/

Can I add something?

Sure! Just go here and edit the document using Draft – a great new tool for versioned writing. Once you add something, I’ll update this website to reflect those changes.

This format is ugly!

I know, I’m horrible at design. Feel free to take all these links and make something pretty and more useful!

Impact Reading List

---

Abelson, Brian, “HI Score: Towards a new metric of influence,” June 27, 2012. http://harmony-institute.org/therippleeffect/2012/06/27/hi-score-towards-a-new-metric-of-influence/

Ad Council, “Overview of Ad Council research & evaluation procedures,” Ad Council, Date Unknown. http://www.adcouncil.org/Impact/Research/Overview-of-Ad-Council-Research-Evaluation-Procedures

Anderson, C.W. et al, “Post-Industrial Journalism: Adapting to the Present,” Tow Center, Fall 2012. http://towcenter.org/wp-content/uploads/2012/11/TOWCenter-Post_Industrial_Journalism.pdf

Barrett, Diana and Leddy, Sheila, “Assessing Creative Media’s Social Impact”, The Fledgling Fund, December 2008. http://www.thefledglingfund.org/wp-content/uploads/2012/11/Impact-Paper.pdf

Blakely, Johanna (Norman Lear Center), “Research study finds that a film can have a measurable impact on audience behavior”, February 22, 2012 http://www.learcenter.org/pdf/FoodInc.pdf

Blakely, Johanna, “TedX Phoenix – Movies for a change”, February 12, 2012. http://youtu.be/Pb0FZPzzWuk

Bornstein, David, “Why we need solutions journalism,” Skoll World Forum, 2012. http://skollworldforum.org/debate-post/why-we-need-solutions-journalism/

Brock, Andrea, et al./ Center for Effective Philanthropy, “Room for Improvement: Foundations’ Support of Nonprofit Performance Assessment,” 2012 http://www.effectivephilanthropy.org/assets/pdfs/Room%20for%20Improvement.pdf

Caulkin, Simon, “The rule is simple: Be careful what you measure”, February 9, 2008. http://www.guardian.co.uk/business/2008/feb/10/businesscomment1

Chinn, Dana et al., “Measuring the online impact of your information project”, Knight Foundation / FSG Social Impact Advisors, May 31, 2011. http://www.knightfoundation.org/media/uploads/publication_pdfs/Measuring-the-Online-Impact-of-Information-Projects-092910-FINAL_1.pdf

Clark, Jessica, “Find needs and five tools for measuring media impact,” May 11, 2010. http://www.pbs.org/mediashift/2010/05/5-needs-and-5-tools-for-measuring-media-impact131.html

Clark, Jessica and Van Slyke, Tracy, “Investing in Impact”, The Center for Social Media, May 12, 2010. http://www.centerforsocialmedia.org/sites/default/files/documents/pages/Investing_in_Impact.pdf

Clark, Jessica and Abrash, Barbara, “Social justice documentary: Designing for impact,” Center for Social Media, September 2011. http://www.centerforsocialmedia.org/sites/default/files/documents/pages/designing_for_impact.pdf

Community Wealth Ventures, “How nonprofit news ventures seek sustainability”, Knight Foundation, October 2011. http://www.knightfoundation.org/media/uploads/publication_pdfs/13664_KF_NPNews_Overview_10-17-2.pdf

Channel 4 BritDoc Foundation, “Evaluation”, http://britdoc.org/real_good/evaluation

Duros, Sally, “Impact counts for hyperlocal news, but how to count it?”, August 6, 2012. http://www.blockbyblock.us/2012/08/06/impact-and-what-it-is-for-community-and-hyperlocal-news/

Fisch, Shaolm M. and Truglio, Rosemarie T., “G is for growing: Thirty years of research on children and Sesame Street,” Children’s Television Workshop, 2000. http://www.amazon.com/Growing-Thirty-Research-Children-Communications/dp/0805833951

Fox, Steve, “Why are we spending so much time ‘Measuring the Impact of Journalism?’” March 30, 2012. http://umassjournalismprofs.wordpress.com/2012/03/30/why-are-we-spending-so-much-time-measuring-the-impact-of-journalism/

FSG Social Impact Advisors/ John S. and James L. Knight Foundation, “Measuring the Online Impact of Your Information Project: A Primer for Practitioners and Funders,” 2010. http://www.fsg.org/tabid/191/ArticleId/428/Default.aspx?srpush=true

FSG Social Impact Advisors/ John S. and James L. Knight Foundation, “IMPACT: A practical guide to evaluating community information projects,” February 2011. http://www.knightfoundation.org/media/uploads/publication_pdfs/Impact-a-guide-to-Evaluating_Community_Info_Projects.pdf

Gates, Bill, “My Plan to Fix The World’s Biggest Problems”, Wall Street Journal, January 25, 2013. http://online.wsj.com/article/SB10001424127887323539804578261780648285770.html

Gates Foundation, “A Guide to Actionable Measurement,” 2010. http://www.gatesfoundation.org/learning/Documents/guide-to-actionable-measurement.pdf

Gigli, Susan, “What Is ‘Disruptive Metrics’“, March 20, 2013. http://www.intermedia.org/2013/03/20/what-is-disruptive-metrics/

Gill, Kathy E., “Carnival of Journalism: How To Measure What Matters?”, April 4, 2012. http://wiredpen.com/2012/04/04/carnival-of-journalism-how-to-measure-what-matters/

Gordon, John, “See, say, feel, do: Social media metrics that matter,” Fenton, Date Unknown. http://www.fenton.com/resources/see-say-feel-do/

Graves, Lucas, “Traffic Jam: We’ll never agree about online audience size,” September 7, 2010. http://www.cjr.org/reports/traffic_jam.php?page=all

Graves, Lucas et al., “Confusion online: Faulty metrics and the future of digital journalism,” Tow Center, September, 2010. http://brianabelson.com/assets/impact_papers/faulty_metrics.pdf

Green, Daniel, “Eyeballs and Impact: Are we measuring the right things if we care about social progress?,” Skoll World Forum, 2012. http://skollworldforum.org/debate-post/eyeballs-and-impact-are-we-measuring-the-right-things-if-we-care-about-social-progress/

Harmony Institute, “Waiting for Superman: Entertainment evaluation highlights”, Harmony Institute, May 2011. http://harmony-institute.org/wp-content/uploads/2011/07/WFS_Highlights_20110701.pdf

Heedy, Lucy and Keen, Sara, “SROI for funders,” New Philanthropy Capital, September 2010. http://www.thinknpc.org/?attachment_id=815&post-parent=4924

Hickman, Blair et al., “Best of MuckReads 2012”, ProPublica, December 31, 2012. http://www.propublica.org/article/best-of-muckreads-2012

International Center For Journalists, “An evaluation of the Knight International Journalism Fellowships”, ICFJ, Date Unknown. http://www.knightfoundation.org/media/uploads/publication_pdfs/Evaluation_of_Knight_ICFJ_Fellowships_final.pdf

International Center For Journalists, “Evaluation field manual and tools for the Knight International Journalism Fellowships”, ICFJ, January 2011. http://issuu.com/kijf/docs/icfj_knight_international_evaluation_manual

Johnson, John, “A New Approach to Making Films That Matter,” January 11, 2013. http://www.good.is/posts/a-new-approach-to-making-films-that-matter

Karlan, Dean et al., “More than good intentions: How a new economics is helping to solve global poverty”, Dutton, 2011. http://www.amazon.com/More-Than-Good-Intentions-Economics/dp/052595189X

KETC, “Facing the mortgage crisis: People, connections, resources”, Spring 2008. https://mediaengage.adobeconnect.com/_a938034862/p90449908/

Kohavi, Ron, et al., “Trustworthy online controlled experiments: Five puzzling outcomes explained”, Microsoft, 2012. http://www.exp-platform.com/Documents/puzzlingOutcomesInControlledExperiments.pdf

Kramer, Mark and Kania, John, “Collective Impact”, Stanford Social Innovation Review, Winter 2011. http://www.fsg.org/tabid/191/ArticleId/211/Default.aspx?srpush=true

Kristof, Nicoholas D., “Getting smart on aid,” May 18, 2011. http://www.nytimes.com/2011/05/19/opinion/19kristof.html?_r=1&partner=rssnyt&emc=rss

King, Gary et al., “Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference,” Political Analysis 15:199–236, 2007. http://gking.harvard.edu/files/gking/files/matchp.pdf

Linch, Greg, “Quantifying impact: A better metric for measuring journalism”, January 14, 2012. http://www.greglinch.com/blog/2012/01/14/quantifying-impact-a-better-metric-for-measuring-journalism/

LFA Group: Learning for Action / Bill & Melinda Gates Foundation / John S. and James L. Knight Foundation, “Deepening Engagement for Lasting Impact: Measuring Media Performance and Results,” Februrary, 2013. (Forthcoming)

Los Angeles Times Data Desk, “Complete guide to the LAFD data controversy”, Los Angeles Times, April 12, 2012 (Ongoing). http://timelines.latimes.com/lafd-data-controversy/

Mayer, Joy and Stern, Ruben, “A resource for newsrooms: Identifying and measuring audience engagement efforts” June 3, 2011, http://www.rjionline.org/sites/default/files/theengagementmetric-fullreport-spring2011.pdf

McKinsey & Company, “Learning for Social Impact”, http://lsi.mckinsey.com/

National Center for Media Engagement (NCME), “Measuring Public Media’s Impact: Challenges and Opportunities”, March 2013. http://mediaengage.org/CommunicateImpact/measure3.cfm

Ní Ógáin, Eibhlín et al., “Making an impact: Impact measurement among charities and social enterprises in the UK”, New Philanthropy Capital, October 2012. http://www.thinknpc.org/publications/making-an-impact/making-an-impact/

Peters, Jeremy W., “Some Newspapers, Tracking Readers Online, Shift Coverage”, The New York Times, September 5, 2010. http://www.nytimes.com/2010/09/06/business/media/06track.html

Peters, Jeremy W., “A Newspaper, and a Legacy, Reordered”, February 11, 2012 http://www.nytimes.com/2012/02/12/business/media/the-washington-post-recast-for-a-digital-future.html?pagewanted=all&_r=0

Pilhofer, Aron, “Find the right metric for news”, July 25, 2012. http://aronpilhofer.com/post/27993980039/the-right-metric-for-news

Priem, Jason et al., “The Altmetrics Collection”, Public Library of Science, 2012. http://www.ploscollections.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0048753

Ramsay, Clay et al., “Misinformation and the 2010 Election: A Study of the US Electorate,” December 10, 2010. http://www.worldpublicopinion.org/pipa/pdf/dec10/Misinformation_Dec10_rpt.pdf

Reisman, Jane et al., “A handbook of data collection tools: Companion to ‘A guide to measuring advocacy and policy’“, Organizational Research Services, 2007. http://www.organizationalresearch.com/publicationsandresources/a_handbook_of_data_collection_tools.pdf

Richardson, Breeze, “Measuring community engagement: A case study from Chicago Public Media,” Reynolds Journalism Institute, December 1, 2011. http://www.rjionline.org/blog/measuring-community-engagement-case-study-chicago-public-media

Rosenblum, Michael, “How to Quantify the Impact of Journalism,” March 30, 2012. http://www.nyvs.com/blog/user/michael/How-To-Quantify-The-Impact-of-Journalism

Salagnik, Matthew J. et al., “Experimental study of inequality and unpredictability in an artificial cultural market” Science 311, 854, 2006.http://brianabelson.com/assets/impact_papers/SalagnikWatts2006.pdf

Search, Jess, “Beyond the box office: New docuemntary valuations,” Channel 4 BritDoc Foundation, May 2011. http://www.documentary.org/images/news/2011/AnInconvenientTruth_BeyondTheBoxOffice.pdf

Sparkwise: http://sparkwi.se/

Spittle, Andrew, “Defining new metrics for journalism,” April 28, 2012. http://andrewspittle.net/2012/04/28/new-metrics/

Stray, Jonathan, “By the numbers, American journalism failed to inform voters,” December 29, 2010. http://jonathanstray.com/american-journalism-failed-to-inform-voters

Stray, Jonathan, “Designing journalism to be used,” September 26, 2010. http://jonathanstray.com/designing-journalism-to-be-used

Stray, Jonathan, “Does Journalism Work?,” December 14, 2010. http://jonathanstray.com/does-journalism-work

Stray, Jonathan, “Metrics, metrics everywhere: How do we measure the impact of journalism?,” Nieman Journalism Lab, August 17, 2012. http://www.niemanlab.org/2012/08/metrics-metrics-everywhere-how-do-we-measure-the-impact-of-journalism/

Stuart, E.A., “Matching methods for causal inference: A review and a look forward,” Statistical Science 25(1): 1-21, 2010. http://biostat.jhsph.edu/~estuart/Stuart10.StatSci.pdf

Tofel, Richard J., “Non-profit journalism – Issues around impact: A white paper from ProPublica,” February, 2013. http://s3.amazonaws.com/propublica/assets/about/LFA_ProPublica-white-paper_2.1.pdf

TRASI: Tools and Resources for Assessing Social Impact: http://trasicommunity.ning.com/

Ward, Nicholas/ Infomart, “Un-juking the stats: Measuring journalism’s impact on society,” October 17, 2012. http://www.infomart.com/2012/10/17/un-juking-the-stats-measuring-journalisms-impact-on-society/

Ward, Nicholas/ Infomart, “We become what we measure: Developing impact metrics for journalism,” October 3, 2012. http://www.infomart.com/2012/10/03/we-become-what-we-measure-developing-impact-metrics-for-journalism/

White, John Myles, “Bandit Algorithms for Website Optimization: Developing, Deploying, and Debugging”, O’Reilly Media, 2012. http://oreilly.com/shop/product/0636920027393.html?bB=g

WITNESS, “WITNESS performance dashboard,” WITNESS, December 2009. http://www.witness.org/sites/default/files/downloads/witness-dashboard-evaluation-2010.pdf

Zuckerman, Ethan, “Metrics for civic impacts of journalism,” June 30, 2011. http://www.ethanzuckerman.com/blog/2011/06/30/metrics-for-civic-impacts-of-journalism/

---

April 02, 2013 04:00 AM

More and more, newsrooms large and small are adopting digital and data-driven techniques to engage readers with their work. News applications, or “large web-based interactive databases that tell a journalistic story using software instead of words and pictures”, have emerged as the standard medium through which journalists present the outcomes of data-driven projects. From Dollars for Docs to Dogs of NYC, news apps present readers with interactive experiences that offer them opportunities to engage more deeply with a story than traditional article-based formats. Given the common structure of news apps (discussed below) and the existence of well-developed tools for tracking user-level events on a website, I propose a simple framework for measuring engagement with news applications.

What makes a news app “successful”?

In a recent InsideThunderdome live chat, Scott Klein, editor of News Applications at ProPublica, wrote, “what I like to see in a news application is a way to see the “far” view (in other words, the big national picture) and the “near” view (how the big national phenomenon relates to me personally).” Put another way, the far view “lets you know why you should care, what the story is, what the national phenomenon is and how places compare to each other.” The near view, on the other hand, walks you “through levels of abstraction down to the very specific – your town, your street, your school. So if we’re successful now you know the national picture, why you should care, and what it’s got to do with you.”

Echoing this philosophy, Michael Keller, Senior Data Reporter at Daily Beast says, “we think of interactives as tools to help people to see how a story might matter to them. Here, ‘success’ means drawing people into the story and, if we’re lucky, also building in ways to hear how it affected them.”

Other indications of what makes a “successful” news app come through job ads. For instance, PBS’s News Hour is looking for someone to “develop news applications that allow users to explore the stories behind data.”

From these examples, a pattern emerges: a “successful” news application is one that presents a reader with a story and offers them an opportunity to dig deeper. Given this common structure of news applications, it should be possible to create a simple metric that captures the degree to which readers use an app to move from the far view to the near view. Such a metric might help newsrooms begin to assess readers’ level of engagement; defined here as the intersection of what readers want and what newsrooms want readers to do.

How might we measure this?

Over the past couple of years, journalists, editors, and news room developers have begun to openly criticize the poverty of traditional metrics like page views and average time spent on page. Most of their frustration lies in the inability of these metrics to capture deeper ideas like attitudinal, behavioral, or legislative change - in a word: impact. But while automated metrics may never fully capture these important considerations, they could help us get at some idea of overall reader engagement with a news application.

Event tracking

One step past page-views and time spent on page are more granular web metrics that capture individual events. From Google Analytics, Omniture, to WebTrends, many free and paid-for analytics services offer access to raw event-level data, or individual actions taken on a given page. “Events” can be created with small tags in your site’s JavaScript (see how to do this with Google Analytics). By tagging elements of a project associated with clicking through a site, entering a search term, watching a video, or sharing on social media, event tracking can be used to track a users path from the ‘far’ to the ‘near’ of news apps.

Identifying users

The difficult (and perhaps troubling) aspect of event tracking is the necessity of identifying individual users of a site. The New York Times is lucky enough to have substantial traffic from registered users. In these cases, collecting user-level event data over time and across multiple devices is simplified through logins. However, IP addresses or IDs extracted from cookies, small bits of information that sites use to track their users, can serve as proxies. While these can be deactivated on most modern browsers, or through services like Tor, in most cases they can represent an “individual user.” In turn, reconciling this information allows us to study the scope of each user’s interaction with an app.

A case study: The Red Carpet Project

In late January, the New York Times interactive team released an app for exploring 15 years of outfits on the Red Carpet. The app represents a new take on the slide show, enabling filtering, searching, and sharing of certain slices of slides. Built into the app is an implicit logic of engagement: while the images are on showcase, readers are encouraged to explore the dresses and suits curated by editors and filter the images by time, style, and color. If all goes well, the readers share their own selection of outfits on social media.

Tagging news apps

In The Red Carpet Project, the far view are the images themselves - what the app would have been had it been merely a slide show. Readers move closer to the near view as they filter the outfits, explore the editors’ selections, and share the site on social media. By tagging each of these events with unique ids, we can isolate users whose behavior on the site was most indicative of “engagement.”

Exploring the data

Given the data for these events, we can now measure the most-used elements of the app. In the figure above, we group events into simpler categories. Note that “faves” connote instances where users selected their favorite outfits without necessarily sharing them on social media. We find that over 95% of the events captured were associated with viewing an image - the “far view.” However, without knowing the degree to which each reader used all possible features, we cannot approximate levels of engagement.

By counting the number of unique features each reader used, we get a better idea of the degree to which users explored the app’s near view. We immediately see that readers rarely engaged with more than one or two of the apps features. Taken together these two graphs beg the question: what was the difference between readers who only viewed slides and those who used additional features?


In this view, we see that more than half of the users viewed only slides and 5% clicked every slide without engaging other features. Readers that only clicked on slides also spent less time on the page than readers that explored more of the site. While such readers may have used the app extensively, they didn’t reach its “near view.” In creating a metric for news apps, we should account for these cases, assigning penalties for behaviors that suggest a lack of engagement.

Creating a metric of engagement

With these insights, we begin to get a sense of the things we can measure to approximate a user’s overall engagement with the app. While we’d like to give weight to the raw number of events – after all, rapidly clicking through 500 images without filtering is its own form of engagement – we also want to assign influence to features that capture whether a user reached the app’s near view. A metric of engagement for this app might looking something like this:

While I’ve simplified the math here – the variables are also transformed to decrease the influence of outliers and re-scaled to increase interpretability – the abstracted equation represents a simplified measurement which captures the logic of engagement embedded in the site. In this case, the corresponding density plot reveals a bifurcated distribution of user experiences, with one half barely engaging with the site (1-20), and the other half dispersed over a long-tail (20-100). However, we might imagine alternate situations where more/less users explored the deeper intricacies of the app. In these cases we would expect the distribution of engagement to look much different.

What might we do with this?

Metrics, while abstractions, are useful in that they provide a single number one can use to track progress. As Bill Gates wrote in a recent Wall Street Journal article (while referencing William Rosen’s The Most Powerful Idea in the World), “Without feedback from precise measurement … invention is ‘doomed to be rare and erratic.’ With it, invention becomes ‘commonplace.’” Without a metric for news apps, our only tools for gauging success will be conjecture and anecdote - a great irony given the meticulousness with which data is cleaned and analyzed to construct these projects. With a standard, interpretable metric, editors and journalists can begin to judge whether their intuitions match up with their readers behavior; analysts can compare apps over time and across news organizations; developers will come to value simplicity of presentation and ease of use over technical complexity; and, perhaps most importantly, news organizations will begin designing their digital offerings with users in mind. While the measurements proposed here are just a rough sketch, I hope they will lead to an open discussion on the development of meaningul metrics for journalism.

March 18, 2013 04:00 AM

At NICAR 2013, there were lots of great sessions how to work with data effectively, talking about things like where to find good data and how to do sound statistical analysis for journalism.  There’s also an important and time-consuming middle piece: you just got some very raw data, and before you can do interesting things with it, you need to field strip it and reassemble it into something usable.  As someone who has a lot of  experience of wrangling messy datasets of all shapes and sizes, I figured I’d share some advice on how to avoid doing all the stupid things I’ve done at some point.

Don’t change your original files/tables

If you have a big Excel file or a MySQL table that needs cleaning, don’t just go in there and start drilling holes in the wall.  Make a copy and work on that instead.  It will make it easier to start over if you screwed up, and, more importantly, you can compare what you’ve got at different stages of cleaning and mutation to make sure it matches up not just with the previous step in your cleaning, but with the starting data (spoiler alert: it won’t).

Spot check everything

As soon as your dataset is large enough that you can’t read it all line-by-line (which is to say, just about every dataset), you have to start spot checking.  Any assumptions you have about the nature and consistency of the data are just a guess, and they’re probably wrong.  SELECT * FROM `TABLE` ORDER BY RAND() LIMIT 5 is one my most frequent SQL statements (for non-SQL people: “show me 5 random rows from the table”).  Use it, love it.   Get a few random records at a time and read them closely.  See if anything seems off.  Do this early and often.  It will prevent dumb mistakes but there’s also a bonus: it will give you much better insight into your data.

Check for weird values

Look out for empty values, and especially for pseudo-empty values.  Excel files from government agencies are unendingly creative when it comes to weird placeholder characters, and you’ll get data that uses - or * or ~ to fill a blank space.  For bonus points, check for extreme value lengths too.  If most of the street addresses in your data are about 75 characters long and one of them is 450 characters long, strange things are afoot at the Circle K.

For every categorical column, check the list of values that exist in the data.  Make sure everything that should be there is, and anything that shouldn’t be isn’t.  GROUP BY (or its twin SELECT DISTINCT) should become your new best friend.  For every numerical column, at least sanity check mins and maxes.  Check the ranges of every special data type (e.g. ZIP codes).  Don’t assume that all dates will be formatted the same, or even be dates.

When you’re GROUPing things, use COUNT() to see how many items there are in each category.  See if all the numbers pass your basic smell test.  If they don’t, you probably screwed something up, but if you’re lucky it means there’s a story there.  Either way, you want to know.

Text data sucks.  Be careful about whitespace, capitalization, and character collation in text columns.  There could be random whitespace in your cells that breaks your ability to compare or group them.  You know that “Paris” and “Paris ” are the same, but your database doesn’t.  Use TRIM() where it’s reasonable to do so.  And remember Veltman’s Law of Character Sets:

Veltman’s Law of Character Sets: there will be exactly one é somewhere in your data and you won’t find it until it ruins everything.

Worry about your column types

Whether you’re using Excel, a relational database, or something else, make sure that your columns/fields are the type you want, and ALL of the data going in matches those types.  If you don’t, your data is going to get squeezed through a special decorative mold when you import it and all your candy cane integers will become snowflake text and you won’t even know it’s happening.  This is especially important if your data is going to pulled out the other side later to power some sort of cool app, where it’s going to go through the variable type wringer again.

Pop quiz: which of the following things evaluate as false in whatever language your developers use?
0, “0”, 0.0, “0.0”, [], {}, [{}], null, undefined

Answer: don’t let this question come up.

Do dry runs before you change things

Are you changing all 50,000 rows in a table with a complicated UPDATE statement?  Try updating a few of them first and inspecting the results.  Make sure your clever operation with five nested REPLACE()s and an inscrutable WHERE clause does what you think it does.  Or maybe you’re auto-generating a thousand database queries with a nifty script you wrote.  Run a version that prints the queries without executing them first and read the output.  SELECT liberally, INSERT/UPDATE conservatively.

Clear is better than clever

There’s no prize for writing the world’s most beautiful JOIN or most deeply-nested Excel formula, so don’t try.  If you’re a journalist trying to whip some data into shape, it doesn’t matter if you take a few extra computing cycles.  You are not working with Big Data.  Processing power is cheap.  Storage is cheap.  Your time is not.  Trying to be clever will make things harder to understand for you and for others, and it increases the chances that you’ll do something really stupid.  When clever things fail, they tend to fail insidiously (I’m looking at you, crazy regexes).  Keep your operations simple.  Work stepwise.  Break complicated operations into multiple simple ones.  It creates a better trail so that when the results are all screwed up (they will be), you can figure out why.

Keep track of what you’re doing

Out of any piece of advice on this list, this is the one I’ve most consistently failed to heed.  Don’t be like me.  Keep a log of what you’re doing to the data.  It can be a paper notebook or a slick automatic query logger or file version control with commit messages or a friendly parrot.  It doesn’t matter.  Knowing what exactly you’ve done to your data is important for troubleshooting and undoing mistakes, but it’s essential for quality journalism.

This post isn’t about how to get data, how to analyze it as a journalist, or how to do mindbending statistic analysis.  For information on those, consult smart people like Jill Riepenhoff, Chase Davis or Jennifer LaFleur. It’s also not about “the right way” to do things (I tend to do them “the wrong way”) or what software you should be using (I’m using SQL as an example but the general rules apply anywhere).  For information on that, talk to the entire internet.  These are meant as general rules of engagement to prevent disaster when you’re reshaping and refining your data on a deadline.

March 05, 2013 06:04 PM

It was all the way back last summer when I first made mention of the OpenNews Learning project. The idea was to assemble some of the best minds in the journalism-code world to help create case studies around the journalistic problemsets that developers come across in the newsroom. The trick of assembling great people is that they’re in demand, and when you’re talking about journalism coders, trying to get a project going in a timeframe that included both the Olympics and the US Presidential Election, well… you’re going to want to find a new timeframe.

Which is why today, I’m absolutely ecstatic to announce that that timeframe has been found and it’s next week.

That’s right: starting next week, we’ll be launching OpenNews Learning as a new section on Source. It will be a regularly updated section of case studies that dig deep into the thinking, design, ethics and execution of code in journalism, written by the very people that know this world best.

Kio Stark, who’s heading up our learning team in part because she’s written the book on independent learning, writes more on her blog today:

OpenNews Learning works by example, through case studies written by a stellar set of journalist-developers, designers and hackers about projects they’ve worked on, describing the hairiest coding problems and hidden ethical issues they’ve come up against. You’ll find out how they solved them, and more importantly where they didn’t. You’ll see where there are opportunities to kick ass and take names to keep information free and make democracy more democratic.

Follow @source on Twitter for the final announcement of when we’re live, and get ready to learn some amazing things.

February 27, 2013 03:42 PM

I’ve been working with the Knight-Mozilla OpenNews project since October, and we are thrilled to announce next week’s launch of OpenNews Learning. We’ve created a place for awesome civic-minded developers to learn more about how journalists and journalism work, and to get acquainted with the hardest problems journalist-developers get to hack on. It’s the place to find out about the landmines they encounter behind the scenes in the process of making amazing maps, visualizations and data-enabled stories.

OpenNews Learning works by example, through case studies written by a stellar set of journalist-developers, designers and hackers about projects they’ve worked on, describing the hairiest coding problems and hidden ethical issues they’ve come up against. You’ll find out how they solved them, and more importantly where they didn’t. You’ll see where there are opportunities to kick ass and take names to keep information free and make democracy more democratic.

First up are cases by Jacob Harris of the New York Times, on how much you can and can’t learn from Federal data sets on food safety; developer Adrian Holovaty on data parsing problems in the journalistic context; Miranda Mulligan, executive director of Northwestern University Knight Lab, on the significance of color decisions in mapping and visualization; Jacqui Maher of the New York Times on making lightning fast sense of the deluge of data in the 2012 Olympics; and journalism professor Matt Waite of the University of Nebraska on the unexpected ethical snarls of making an app from police records.

Join the community, learn the ropes, hack the news.

February 27, 2013 10:15 AM

February 25, 2013 11:15 PM

Every hour, Spiegel Online serves more than half a million visitors. To make that work, all content has to be served via a CDN. For data-driven applications that means: no dynamic queries can be served easily, data needs to be static. This doesn't need to be a showstopper for great content, sites like the UNDP data explorer demonstrate that often, a set of JSON file is enough to power a great project.

DataFreeze facilitates the creation of such applications by freezing relational data from a SQL database into a set of easy-to-use JSON and CSV files. What data is included gets controlled by a Freezefile - a simple YAML or JSON file that specifies queries, output file names and formats. A sample Freezefile would look like this:

common:

  database: "postgresql://user:password@localhost/operational_database"
  prefix: project/data/
  format: json

exports:

  - query: "SELECT id, title, date FROM events"
    filename: "index.json"

  - query: "SELECT id, title, date, country FROM events"
    filename: "countries/{{country}}.csv"
    format: csv

  - query: "SELECT * FROM events"
    filename: "events/{{id}}.json"
    mode: item

Of course, all of this is not a very hard problem to solve - but DataFreeze does it in a clean and simple way that invites you to document your workflow so it becomes repeatable by others.

Check out the GitHub repository: spiegelonline/datafreeze.

February 25, 2013 08:00 AM

Last week I spent a very romantic Valentine’s Day at Mozilla’s London office participating in the BBC News Connected Studio event.  The program was started in 2011 as a way to find and develop innovations for BBC digital offerings, with separate events for different products like BBC News, BBC Sport, and Children’s BBC.  Each Connected Studio starts with a goal in mind (here’s the brief for News) and then goes through three stages:

1. Creative Studio is an all-day event open to all comers.  Participants get a sort of “State of the Product” address from key BBC staff for context,  and spend the rest of the day brainstorming and fleshing out possible product innovations.  Teams can get guidance from BBC staff experts and book audience feedback sessions, where they have a chance to try out their idea to a focus group of actual BBC users.  The day culminates with 3-minute pitches from each team to a panel of judges, and a handful of pitches are invited to move on to the Build Studio.
2. Build Studio is a two-day affair in which the invited teams roll up their sleeves and prototype their idea, giving a 10-minute demo and presentation to a panel of judges at the end of the event.  As with the Creative Studio, BBC staff are on hand to answer questions and help point you in the right direction.  In the case of News, this was three weeks after the Creative Studio.
3. Pilot is the final stage, where the BBC selects the best Build Studio proposals and ponies up real money (up to £50,000) and BBC staff time to bring them to life.  They’ll work with you to build out and integrate your idea, do a live pilot with a portion of the audience, and, if it passes muster, fold it into the core product, where it will go out to hundreds of millions of people around the world (in 27 languages!).

It was fascinating to see an approach that was trying to tap into the same potential of traditional “hack day” formats but with a more targeted, incubational spin.  Key differences include:

• Institutional Support - At Connected Studio, you don’t just get WiFi and a free lunch; you get access to a variety of BBC staff to guide you along and explain what separates a fun idea between friends from something that an organization like the BBC can use.  They also provide audience focus groups, servers, and IT support for whatever you build.
• Goal-setting - The BBC invests so much in these events because they’re looking for concrete outcomes that improve their products, and they structure the event to keep things pointed towards that goal.  Everyone is preparing for a real presentation where they’ll stand up in front of a room full of Very Serious People and pitch the value and feasibility of what they’ve been building all day.  The results were the most thoughtful and substantial show-and-tell of any hack event I’ve attended.
• Feedback - Rather than working largely in isolation and then coming back up for air at the end, you have access to a strong feedback loop throughout Connected Studios.  Teams are in frequent contact with BBC folks and each other, answering questions and providing updates.  You get thoughtful, written notes from the judges about what they liked in your pitch and what questions need to be addressed before the next stage.  You also go through trial-by-non-developer in the audience sessions.  The value of these feedback mechanisms was tremendous, and something that I’d love to see more attention to in hack days of all shapes and sizes.

I thought Connected Studio was a great event, but to compare it to a typical hack day would be misleading; the motivation behind it is totally different.  The real takeaway for me is that there’s a lot of room to play with the structure of an event where people get together to make stuff.  There’s a natural tendency for organizers to emulate whatever they’ve seen done before, especially if it seemed to work last time, but it’s useful to take a step back and think about WHY you’re putting together an event in the first place and then figure out the structure that best promotes that.

February 21, 2013 11:53 AM

That was quick. My first week as an OpenNews fellow at Spiegel Online is already coming to an end today. I've spent these first days exploring different aspects of the organization, inevitably failing to give a succinct explanation of what kind of a creature I am and what sort of trouble I plan.

What I've been trying to convey is that I am a technologist who is excited about news. When I applied to join the Knight-Mozilla programme last year, it was because of the many intriguing discussions I've had as an open data advocate with journalists from around the world. But my interest in this goes beyond data as a means for storytelling. News technology is information architecture for the public sphere, and the web means we'll have to completely re-think it.

Of course, there's been little space for such lofty talk as I've been on a marathon to meet as many of the different teams at Spiegel as I could this week. During this, I've tremendously enjoyed joining some of the editorial meetings and getting a first-hand impression of the sort of debate that goes into framing the news of the day. Spiegel's famous fact-checkers also keep amazing me, they are the folks I'm hoping to learn about data journalism from.

At the same time, I'm beginning to think about the different projects that I will work on over the coming months. That, of course, is a problem of abundance: there are tons of things I wanted to try before I arrived in Hamburg, and everyone I've met since has also been full of ideas they've wanted to share and cooperate on. The kanban board I've started so that I can keep track already starts looking a bit packed, but this also means that there are also some themes starting to emerge.

• Create stories. I've decided that the most direct way I can find out what tools are useful for digital journalism is to actually get my feet wet. Thankfully, Nicola, my primary contact at SpOn, is an editor in the science department. She has provided me with plenty of leads, so I'll aim to dig up some data and see what I can learn from it.

  Further down the road, Germany's September federal elections will be the topical focus of my fellowship: I want to experiment not just with methods to cover the election, but focus on ways in which the contested issues and the key differences in the party platforms can be explained interactively.

• Break articles. Journalism should focus on stories, and articles are just a paper world hack to that end. Still, it seems to me that the ways in which we're delivered developing stories (is there any other kind?) have hardly changed.

  Spiegel, with its vast archive of neatly contextualized information, is in an amazing place to tell stories that develop not just for a few hours, but across weeks, years and even decades.

  I've already started some work around entity extraction based on a scrape of SpOn, and I hope to catch up soon with Stijn, my fellow fellow at the Guardian, and Annabel, at Zeit Online, about other ways in which he intends to liberate stories from their dead tree shackles.

• Make robots. Technology should make life simpler and help us to stay on top of things. With databases becoming a source of news, that has to mean that we need tools that help us to find stories based on any heuristic we can imagine.

  The notion of human assisted reporting seems extremely powerful to me, and January's OpenNews onboarding allowed a team of fellows to develop some ideas on how we could make it work. Again, the secret motive here is that I may get to collaborate not just with the awesome team of fellows, but also with my friends at OKFN.

At the same time, I want to stay open and learn much, much more about how news is produced here, and what could be done to make that process more effective. Maybe I'll end up doing something completely different. This freedom to tinker with news technology over the next year is what I am most amazed by.

February 08, 2013 08:00 AM

Calculating "crosswordiness":

A post I wrote for the Guardian Datablog on how we can think about and compute the “crosswordiness” of different words using the New York Times crossword puzzle, a dictionary, and Google Books N-Grams.  (What is “crosswordiness”?  Read it to find out!)

February 07, 2013 06:08 PM

Last week I made Infinite Street View, a little hack for viewing a stream of random street view snapshots near a place.  If you’re curious about how it was made, read on!

Background

I often find myself using things like Flickr and Google Street View to get a quick visual sense of a place I haven’t been.  Sometimes it’s for a constructive purpose, like trying to narrow an apartment search or deciding on a neighborhood to stay in during a vacation.  Other times it’s just to satisfy my curiosity about a place that came up in a book or in conversation.  Unfortunately, photo sites and Street View both have their limitations for this sort of casual exploration.  Sites like Flickr are heavily biased towards coverage of the major landmarks, and if you’re talking about a tourist-friendly destination, you’re going to see hundreds of sunsets and retina-burning HDR shots.  Street View is better on that score, but there’s no good way to get lots of views at a glance.  You have to drag that little yellow homunculus onto the street and then plod along frame by frame.

Last week it finally occurred to me that building something to give you a clusterbomb of street view images in an area would be a pretty straightforward hack.  The relevant pieces are all available from Google through APIs, so it was just a matter of writing some JavaScript and a UI to present results.  I quickly settled on some form of infinite scroll as a good fit so that a user could just keep getting additional frames until they were satisfied.

Building it

The page itself couldn’t be simpler: it’s a text input and a dropdown with some preset radius values, and that’s it.  All the meaningful stuff happens in the JavaScript, which works something like this:

Wrinkles

• The text input uses the Google Places Autocomplete to geocode user input as you type, and if you get no autocomplete results and submit the form it does a direct Google Places lookup instead.  There is one aspect of the standard behavior that bugged me, which is that it doesn’t passively select the top result as you type, so if you type part of a place, see the result, and hit ‘Enter,’ it won’t select the result for you.  Because of the structure of the Places Autocomplete events and results, standard key listeners don’t get the job done, but eventually I found a workaround that grabs the top result and geocodes it when the enter key is pressed (trying to force input to an autocomplete isn’t always desirable, but in this case it’s preferable because there’s a tight relationship between a place being known to Google and having Street View imagery).
• Getting a random point near a given latitude and longitude requires a bit more math than just adding a random number between r and -r to both latitude and longitude, because you should adjust for the curvature of the earth (longitudes converge at the poles).  This probably wouldn’t have mattered on the scale of a few miles that this site deals with, but I’ll take any excuse to dredge up high school trig!  If you want to geek out about spherical point randomization, haversines, and the like, check out Wolfram MathWorld’s writeup.
• I spent a bit of time trying to put together an alternate mobile version that takes up the full device screen width instead of an arbitrary fixed width, but because so much of what happens occurs post-load in JavaScript, and because I suck at responsive design, I struggled getting this to work and put it on the back burner.
• I experimented with having the images float to the left to occupy all available space, but I found it was a little overwhelming; I like the neat linear flow of the current single-column approach.

Publishing it

I sent out a tweet when I finished the initial version, and it turned out I wasn’t only the one who enjoyed this flavor of armchair tourism.  It only took about 30 minutes for me to hit the free daily limit for the Google Street View Image API (25,000 calls), and 4 hours for me to hit the PAID daily limit (500,000 calls).  I had to board up the site for the remainder of the day until Papa Goog gave me my next allowance.

As with all things I’ve done, I got lots of valuable feedback right away.  I had skipped two forehead-slappingly necessary features in particular: making the images links to the appropriate spot on Google Maps, and adding a permalink hash so people could share URLs for places.  Now, when the page loads, it checks for a permalink hash and loads results based on that location and radius if they’re provided (here’s the view around my old neighborhood in San Francisco).  A user can also click on any image to see a Google Maps split screen of that exact location.

What I’d do differently next time:

• Write my own autocomplete wrapper that works off the Google Places autocomplete and genericizes it, so I have easier access and finer control over the autocomplete’s events and properties.
• Seduce and marry a lonely billionaire, so I can buy unlimited Google API credit.
• Make a mobile version that works.
• Make the permalink hash live-update instead of only getting referenced on the initial page load.  I tried to do this, had some trouble, and didn’t have time to power through it.

Links

1. Infinite Street View
2. Google Places Autocomplete API
3. Google Street View Image API
4. Wolfram MathWorld: Sphere Point Picking

February 07, 2013 05:34 PM

February 05, 2013 10:55 PM

For an upcoming project for NewsBeastLabs on the gun debate, I’ve been monitoring statements representatives have made on the topic. As President Obama prepared to unveil his proposal for gun control last Wednesday, Michael Keller and I were curious to see the reactions of representatives to the highly publicized announcement. Given the degree to which breaking news is now reported (and responded to) on social media, we thought it would be useful to build a bot to log officials’ comments on certain issues and present them in real-time. Such a tool could be used by news rooms to engage their readers on a continuous basis by aggregating and serving content from members of particular communities or who serve on different committees.

@RepsGunTweets was born.

We were inspired by the work of 2013 Mozilla-Knight OpenNews fellows who recently built a prototpe for an app called “if (this) then news,” a news-oriented take on IFTTT – a site for linking triggers from gmail, twitter, dropbox, and other services to actions on the web. Applying this logic to news coverage, the fellows created the shell for a tool that would monitor live data streams, detect important events, and issue notifications. As Vice President Biden took the mic, we started furiously coding up a bot that would follow the twitter accounts of US Representatives and retweet any comment that included “gun”, “assault weapon”, “firearm”, or other relvant keywords. After a couple hours of missteps and headaches, we eventually got @RepsGunTweets up and running. In the last ten days, the bot has logged 307 tweets; two-thirds of which came in the first three days. We’re still analyzing the conversation but one interesting observation is representatives who are not in favor of gun control tend to link to longer explanations of their position on their website instead of tweet a comment.

Under the hood

At its core a retweet bot is a pretty simple tool: Follow a feed, find what matters, and serve it back up under a single account. The harder part is figuring out how to accurately communicate with Twitter’s API. Using tweepy for python we were able to easily access twitter’s numerous methods. All we needed to provide it with were the the consumer key, consumer secret, access token, and access token secret for an application generated on http://dev.twitter.com/apps). The bot follows CSPAN’s member of congress list and applies a regular expression for the desired keywords and retweets any matches.For even more technical info, including an easy way to create your own retweet bot, check out this Github page.

January 29, 2013 05:00 AM

Team 2013 on the streets of Cambridge

I’m still reeling from the amazingness that was the 2013 Knight-Mozilla Fellowship Onramping we held at the MIT Media Lab two weeks ago. Our fellowships are different than many because our fellows spend most of their time apart—they’re embedded in their host news organizations, working alongside reporters and newsroom developers—so we wanted to make sure that before they got swept up in the hustle of the newsroom, that they first learned more about each other and start to etch pathways of collaboration that will deepen over the course of the year.

We decided on the Media Lab because it’s a place that’s filled with the exact spirit of experimentation that we wanted to kick the year off with, and thankfully our friends at the Center for Civic Media were able to give us a great spot off the central atrium to set up camp.

Getting heads-down at the MIT media lab

In planning the week, we knew we wanted to hit a good balance gaining shared experience and knowledge and giving everyone the freedom to hack together. We had stuff we needed them to know (how to file an expense report, for instance), stuff we wanted them to learn (how to feel comfortable really diving into need finding in the newsroom), and stuff we hoped would happen (they’d realize just how valuable each one of them is to each other). And we had four days to do it.

The first two days were heavy on talking. It was “Fellowship 101” on day one, where we also had Dan Schultz and Laurian Gridinoc from our 2012 class and representatives from some of the newsrooms that hosting fellows this year on hand to be able to answer questions about the fellowships from many angles. Day two we decamped to the Boston Globe, where friends from the design firm IDEO lead sessions on the fundamentals of human-centered design. In order to start our fellows’ year off with a good grounding in how to observe need inside the newsroom, they then took what they learned and performed need-finding interviews with Globe staffers. The insights they gained through those conversations continued to resonate throughout the rest of our time together.

Annabel Church shows what she’s been up to on Saturday night

Friday and Saturday we moved from talking to making. Since a major part of the Knight-Mozilla Fellowships is for the fellows to feel free to experiment, create, and try new things, we wanted to give ample space and time for exactly that. We only had one rule for the hack weekend: No solo endeavors—the fellows had to work together on stuff. And they did, in small pairs, in larger groups, and in whole-room ideation, it was amazing to watch a group of near-strangers coalesce into a community of peers, of friends, and of collaborators.

The week ended with piles of Indian food and lots of new friends at a meet-the-Fellows get-together on the Media Lab’s fifth floor. We invite folks from the Lab, as well as from around Boston’s robust media innovation community. The fellows got to show off things they’ve been working on, and we all got to play a few robust rounds of Werewolf before heading back to our hotel rooms to collapse.

We focus a lot on community here at OpenNews—the big, sprawling, amazing community that creates the code that’s transforming journalism every day. Those four days in January at MIT was an opportunity focus on a much smaller community: the community of fellows who, over the course of the next year, will not only help journalism on the web make exciting new leaps, but will also become forever a part of each other’s lives. It was a moment to focus on the things we can build together, the ways we can change the world, and the ways we’ll change each other as well. The next year together is going to be amazing.

January 28, 2013 06:03 PM

Last August, I was reading the ProPublica Nerd Blog and came across a post about the Knight-Mozilla OpenNews Fellowship, which sends developers to leading newsrooms around the world to help build the future of journalism. As a lifelong news nerd and web tinkerer, it seemed like a perfect fit. Digging into data, looking for stories, and building visualizations has been something I mess around with on rainy weekends and late at night when I have a project idea I can’t let go of. This would be a chance to spend all day doing what I already do for fun. Applying was a no-brainer.

Long story short, in October I got an email from the Godfather, Dan Sinker, naming me the 2013 OpenNews Fellow at the BBC in London. I’ll be starting later this month, and I couldn’t be more excited.

MozFest

In November I had the chance to attend MozFest and meet some of the best news developers in the business.  I was genuinely blown away by what I saw. Back in my native Silicon Valley, people love to talk a big talk about the tech “community.”  Sometimes they won’t shut up about it. But at MozFest I saw a real professional community in action, a group of people with a profound admiration and respect for each other’s work who love to share knowledge and push boundaries together. I just couldn’t get over the sheer brainpower and positive energy on display.

The atmosphere on the 9th floor at MozFest also reminded me of some of the reasons I love news development more generally:

• There’s no playbook. The limit of what’s possible is constantly being redefined by clever, creative people. Every single day I see something new and amazing.
• Everything is a team effort. So many people in the news developer community have taken the show your work mantra to heart. I love that when everyone hits a wall, the first person to make it over sticks out a hand and pulls the rest of us over and then it’s on to the next wall. Even the smartest people in the room are constantly learning from their peers.
• Things move fast. In the newsroom you’re expected to do great work on short notice and tight deadlines. You have to learn to sketch out an idea, build it, test it, and ship it in a matter of hours. Mark Surman talked about the “fuck it, ship it” mentality at MozFest, but personally I like the Lorne Michaels version best: “The show doesn’t go on because it’s ready, it goes on because it’s 11:30.”

Challenges

Along with all the excitement, the news developer universe has plenty of challenges to deal with, and a big part of my fellowship year will be trying to more deeply understand those challenges and how we can meet them, things like:

• The cultural divide. How do we ensure a fruitful marriage between the culture of the newsroom and the norms of software development? How do we get developers excited about journalism and journalists excited about code?
• Code in the attic. If a JavaScript library is open-sourced in the woods, and no one’s around to use it, does it make a sound? The chaotic nature of progress in this world means that there are lots of dead ends, duplicated efforts, and tools to solve non-existent problems. How do we build platforms that flourish instead of gathering dust?
• Knowledge-sharing. How do we effectively synthesize the lessons learned from the hundreds of experiments being run in parallel to figure out what works (Source is a good start)? How do we gather best practices and avoid this:
  
  

Goals

As I get ready to start on what I’m sure will be an amazing experience, I wanted to sketch out some goals I have in mind for the year ahead:

• Write something. I used to know how to write a sentence, but that muscle has atrophied in recent years. Now I can’t stare at a blank page without breaking into a cold sweat. But since I’ll be spending a lot of 2013 trying to make sense of the news development landscape, I might as well be thinking out loud. I hope to use this blog to talk through my questions, ideas, and lessons learned and to take a look under the hood of some of some news apps.
• Teach something. Before the year is done, I’d love to find a way to give back to the community by teaching aspiring news developers or proto-webmakers something useful, whether in person at events or through online tutorials.
• Understand the dynamics of a large news organization. Moving from solo late night hack sessions to being a tiny cog in a 23,000+ person organization will be a huge learning experience. A world leader like the BBC offers unique resources and opportunities, but also unique constraints. I hope to figure out what that means for the OpenNews mission.
• Build something together with the rest of the 2013 fellows.  My 7 co-fellows are some of the smartest and most interesting folks I know, and each one brings something totally different to the table. I hope that even though we’ll be fanning out across the globe, we can combine into a news development Megazord and build something awesome this year.
• Learn the state of the art. There’s a long list of specific software platforms, tools, and methods that I’ve been meaning to try. Hopefully this year will give me the time and opportunity to master some of them, though I also want to be careful not to place too much attention on technical specifics at the expense of the big picture.
• Give in to Twitter. I’ve been a conscientious Twitter objector for years, but it’s clearly where a lot of the conversation is happening in this world, so I’m determined to get on the bandwagon. I’ve only really been paying attention for a month or two but I’m already starting to understand why what I used to see as downsides actually make it a great medium for certain uses. That being said, if I ever tweet a photo of my dinner, please punch me in the face.
• Figure out how to explain OpenNews to my grandmother.

I can already tell that the biggest problem this year will be the number of hours in the day. There’s so much I want to sink my teeth into, and no way I’ll get to it all. Fortunately if everything goes well, 2013 is just the beginning.

Coming Soon

Some topics I’ll try to tackle here in the coming months:

• How do people actually learn to code?  And when we ask whether journalists should learn to code, or how to teach them to code, are we asking the wrong question?
• How do we stamp out code snobbery and make news development a big tent?
• What are the secrets to successful conferences and hackathons?
• Brief post-mortems on some of my past projects. What was I thinking (or not thinking)?
• Deep dives into some of my favorite news apps and infographics.
• What possibilities does code in the newsroom open up for photojournalism?

January 18, 2013 07:36 AM

Two worlds colliding, or that is how it felt for me when I first started to understand the potential of web technology and journalism combined. Colliding perhaps isn’t the best word, but neither is it a bad word to describe the coming together of these two worlds, it suggests friction, reaction even explosions and that it will take some time for the dust to settle.

Beginnings

I am sad to say I’m no longer an OpenNews fellow - my time has expired and with heavy heart I shall remove the title from my bio. But let’s back up a bit and look at how this all started.

media is instrumental in shaping people’s opinions

My attention was first drawn to the OpenNews (then MoJo) project in the Summer of 2011. Being a curious sort of chap, I wondered what sort of shape this initiative would take. I was curious because it combined two of my interests - media and technology. I believe that media is instrumental in shaping people’s opinions and — perhaps somewhat naively — I believe that technology holds the keys to the democratisation of information and media, especially the inter-connectedness that web-based technology provides.

Optimism aside, ‘techno-journalism’ struck me as a very interesting intersection in which to work and so I enrolled in the Knight Mozilla News Challenge - a process designed to embed five people with technical backgrounds into newsrooms.

Six months later I was an OpenNews fellow working with Al Jazeera English. I had opted to work  on a remote basis with the understanding that I work on-site as often as was required.

Ten months after starting, my fellowship has drawn to a close and this is what I have learned.

Travelling is Important but so is Doing the Work

I’ve often wondered how those people who are in a continual state of transit ever get any work done. I guess there’s a knack to working on the hoof that I’ve yet to acquire, but while I’m travelling I’m often too excited or exhausted to be a very effective developer, at least on the shorter trips. That’s not to say travelling isn’t important. There is no substitute for meeting people in real life and I especially found the month I spent at Al Jazeera HQ in Doha very worthwhile. Other highlights included the Civic Media Conference in Boston where I was made to feel at home at MIT Media Lab and The Guardian offices in London where all the fellows got to work as a team.

I did get some work done at the various hack days I attended, but really travelling for me was all about meeting people and building up relationships, most crucially with other fellows. All my trips were worthwhile and indeed served to inspire and motivate me to actually do the work.

Developers in News Organisations are Constantly Shipping

I have blogged about this more than once - because of the fast-paced nature of news, if you are a developer in a newsroom you will find yourself imbibed with the “Fuck It, Ship It” mentality.

your work is transient - of the moment.

The good thing is, most stuff you work on doesn’t have to be perfect, it just needs to be good enough to get the message across. To all intents and purposes your work is transient - of the moment. That is not to say that the time that you are iterating hard against a deadline isn’t stressful, it is. It’s also very exciting.

Developers in News Organisations Make Cool Shit

Some of the best developers I know work for news organisations. Jeremy Ashkensas, creator of CoffeeScript and Backbone.js works for the New York Times and so does Mike Bostock who created D3.js. The Miso Project was taken on by Irene Ros and Alex Graul working at The Guardian - all important open source tools that all types of developers can take advantage of. In fact there are so many new tools that OpenNews launched Source to document it. 

Developing More Than just Applications

When I first arrived in Doha, I was told that it was easy to make a difference at Al Jazeera as there was always so much that needed doing. Actually Al Jazeera English (where I was placed) struck me as a very agile organisation - if you had a good idea, it seemed that you could often run with it. Due to my background and the discussions I had with people at various events, I’d amassed a few ideas for Al Jazeera to try out. I was curious about most aspects of the aljazeera.com website - and was eager to discuss issues such as content management, styling and even content. What I noticed and liked about the people there, was that they were not afraid to — and indeed were very good at — harnessing and taking advantage of technologies, services and concepts that were not necessarily home grown.

Pic: Internet Archive’s TV News Cloud

The use of Creative Commons licensing for much of their content was particularly inspiring and so was the grasping of the role of social media in news, never more apparent than during the Arab Spring.

I felt that there was an opportunity to bring AJE and other organisations together. Through various shared interests I was already in touch with Internet Archive. Being broadcasters as well as online producers AJ create a lot of video based content and are actively looking to get their content viewed on the US mainstream channels. Through closed captioned content they can not only meet the accessibility requirements for US TV they can produce content that can be archived at internetarchive.org. This will also allow developers to expose this video content in new and exciting ways, but that’s another story.

Another couple of potential matches I made some headway in making was to bring Al Jazeera together with Shoutabout.org who facilitate social action and Globaleaks.org who provide a secure whistle-blowing framework. 

When Nikki Usher a journalist and assistant professor at GWU contacted me asking if I could introduce her to anyone at Al Jazeera, I was happy to. From that came a visit and an enlightened article for Nieman Lab on my colleagues in the interactive team 

In the last days of my fellowship, I was very happy to introduce Al Jazeera to Nicola Hughes. Another OpenNews fellow working with The Guardian, Nicola flew to Doha to pass on her knowledge in the form of a Data Journalism course. I helped out when needed but my main effort was to bring the two parties together.

So I’m happy to say that thanks to Al Jazeera being very open to working with others, I was able to foster a few relationships.

Mucking In

Matchmaking aside, I did spend a fair amount of my time actually making things. I used the knowledge I had built up from working on jPlayer to create media based interactive pieces ranging from YouTube powered slide shows to Interactive Transcripts of the US presidential debates.

I’m fascinated by the emotion that can be conveyed by audio

My first foray was to create a contextual take on a documentary piece. The idea was that the viewer could choose to view (and share) extra information about the program. Ingredients included jPlayer, Tabletop.js and judicial use of an iFrame to not only sandbox the content but also allow it to be embedded in other publications.

I’m fascinated by the emotion that can be conveyed by audio and its role in storytelling. I’d been following what a UK startup ThisIsMyJam had done to allow people to select music from YouTube and share their favourite ‘Jams’. This is very shrewd as YouTube is an excellent music repository. Sure, the quality may be lower than usual, but the quantity of musical material it houses — and crucially can be accessed without an account — is second to none.

Inspired by a chat I had with Showkat Shafi a talented and — judging by his stories — very courageous photographer, I had the idea to create a service that allowed anybody to choose an audio track from YouTube and play a slide-show along to the music. I wanted to include that slow zoom and fade and partly succeeded. AJE ended up using it slightly differently to how I had envisaged by uploading their own audio commentary to YouTube. Again, jPlayer and Popcorn were used.

The US elections provided a global opportunity to create new forms of visualisations and interactivity. I say global as it seemed that every news organisation in the world was trying to come up with something new. My part was to work on an interactive tool that allowed the reader to analyse, discover and share parts of the debate. It was an interesting challenge as there were four debates and we got to improve the format in response to feedback each time. I also took the opportunity to play about with simple data visualisation. So this time Popcorn, jPlayer and a bit of D3.

This is how my work is logged on aljazeera.com 

Unfinished Business

Right back at the start of my fellowship I began writing what was to be an op-ed on Firefox OS for aljazeera.com. I’m not the only developer to be encouraged to do write for AJE. Dick Olson - — a Drupal developer for AJE by day — recently got the opportunity to follow his passion for motor-sport and write up some interviews he had with some of his heroes. In my case as there was no real urgency to complete, this got put on the side and is still not quite finished. Thankfully the subject is still very much relevant and I look forward to finishing it off soon.

Playing about with D3 and generally getting closer to data journalism inspired me to start writing my own visualisation of the recent Israeli-Palestinian conflict. This remains work in progress but I’d love to create a generic comparison tool when I have time.

Reusability

As Dan Schultz — my colleague at The Boston Globe — touches on in his recent blog-post - one of the most important parts of an open developer’s remit is not only to release the code but to release it an a format that others can easily adapt. The issue is the next project is usually just around the corner and the luxury of time to work on genericising a piece of code all to often scarce. I admit despite my best intentions, I never fully got around to doing this.

OpenNews is Evolving

Help, support and encouragement are at hand. Anticipating the vital life-blood of development that is code, a source-code repository known simply as Source was launched and put in the capable hands of Erin Kissane which means you also get excellent news related articles into the bargain. Additional motivation is provided by Code Sprint grants, so if you’re a developer working in a newsroom and want to make that last bit of code generic you should definitely consider the grants and of course putting your code on Source.

What’s Next?

To be honest, I’m not exactly sure. It is very exciting to see the quality of interactive journalism coming out of news organisations like the New York Times and The Guardian right now. The Guardian have a whole section dedicated to Data Journalism and The New York Times are coming out with gems like 512 Paths to the White House and their recent excellent interactive piece - Snow Fall which received more than 3.5 million page views. It’s also exciting to see fellow developers who formerly worked in other areas get bitten by the journalism bug.

I’m looking forward to maintaining my relationship with The Knight Foundation, Mozilla, Al Jazeera and the rest of the OpenNews fellows and continuing to work in this very extraordinary space. I’d love to push hard on Hyperaud.io which has been a running theme for me throughout this fellowship. I’d also like to continue help improve our jPlayer JavaScript Media Library that happily is now being used by news organisations globally.

Here’s to all the Punks

It’s all been a fantastic opportunity and I’d like to thank the following people for making 2012 the epic year it really was for me. First and foremost Dan Sinker for support and encouragement and for being both Visionary Druid and Down-to-earth Punk. Thanks also to Erika Owens, fearless OpenNews community manager and organiser of epic journeys and accommodation Mohammed Haddad - my colleague and friend at Al Jazeera and all the other fine people working there, of which of course there are simply too many to mention. Thanks also to the Knight Foundation for funding and Mozilla Foundation for support.

Lastly but not leastly a big thanks to my family, (not my real family - I’ve thanked them privately), fellow fellows (my own Punk Rock band) Dan Schultz, Nicola Hughes, Laurian Gridinoc and Cole Gillespie. It’s been amazing working with you all. Thank you!

January 14, 2013 02:02 PM

In November, in advance of the announcement of our amazing slate of 2013 Knight-Mozilla Fellows, I wrote a pretty thorough look back at everything the OpenNews project has accomplished in 2012. Erika Owens, our community manager, recently published a great look back at a year of our hack days. And so now, on the last day of the year, writing another year-in-review post about the OpenNews project doesn’t seem necessary. It’s simple enough to sum up our 2012 in two words: fucking awesome. And a preview of 2013 really just requires a single additional word: incredibly fucking awesome.

Instead, I want to widen the focus a little bit, off our project and on to the community it’s a part of since, at the end of the day, that’s why we do what we do: to strengthen, support, and help build the journalism code community. Source, our hub for that community has been publishing year-in-reviews all week. These looks back highlighting some of the inspirational work that’s helped to push journalism on the web forward this year. I want to take a moment to add my own picks to that list:

• Snow Fall: The team of developers, designers, and interactive journalists assembeled at the New York Times is unparralleled. Outside of some of the biggest companies in tech, I’m not sure there’s a more concentrated collection of raw talent than the folks sitting in the Times building in New York right now. They’ve managed to produce a staggering number of incredible pieces this year, but none has gotten the (deserved) attention more than Snow Fall. An engrossing piece of long-form journalism made all the more immersive through smart use of CSS layering and time-based media triggers. There have been a number of “Is the future of journalism?” pieces written about Snow Fall already—to which it’s easy to answer with a resounding “no.” It is a new option for the present, a new way of thinking about presenting and engaging an audience. That they had to completely break their aging CMS to do it right is the real thing to consider closely.
• NPR Big Board: I’ve written before that major elections are the Super Bowl of news applications, and 2012 certainly proved that right. There were incredible pieces created by all the major news organizations, but the one I ended up relying on on the night of the US Presidential Election was the Big Board created by the newly-formed NPR News Apps Team. Instead of building a map like most election results, the NPR team decided to simply display numbers on the screen. As it turned out, that was exactly what a person who was obsessively checking the results (as I was) needed. It’s no surprise that the team assembled by NPR would come up with an approach that turned election result display on its head—they’re all veterans of other news apps teams. Want to see just how strong this community is getting? Read that sentence again: news organizations can now create teams from scratch comprised of veteran talent. More of this. Now.
• Tabletop.js: My single favorite codebase this year was Tabletop.js. A simple idea beautifully executed, Tabletop lets you easily use a Google Spreadsheet as the backend to run a data-driven web page. That’s all, no additional bells, no unnecessary whistles. Created in collaboration with WNYC, it’s been used by news apps developers around the world since its introduction in February. The killer feature is that you don’t have to train anyone on the backend: if they know how to use a spreadsheet, they know how to populate your app with data. Tabletop is also the secret sauce on my *second* favorite codebase of the year, Jessica Lord’s Sheetsee.js, which takes the same it’s-the-database-stupid approach to data visualizations.
• Opened Captions: For pure geeky wow-factor, I’m just as enamored by Opened Captions, built by 2012 Knight-Mozilla Fellow Dan Schultz, today as I was the day he first showed it to me. A smart hack that takes the closed-captioning data coming out of CSpan (copyright issues be damned) and turns it into a fully-functional real-time API, Opened Captions simply produces an unrelenting stream of text as fast as it’s spoken. The possibilities of something like this—from real-time data augmentation with supplementary material to real-time fact checking—is pretty amazing. That it was a hack produced in just a couple days is a great reminder that sometimes fast, cheap, and out-of-control is exactly the right way to do things.
• Data Journalism Handbook: Finally, 2012 saw the publication of the Data Journalism Handbook—an excellent guide to the basics of working with data that was started at the Mozilla Festival in 2011 and has contributions from some of the best in the business. Most notable about the Handbook to me is that it’s a creative-commons licensed project that’s actively looking for localizers and people to produce updates. That’s exactly the kind of collaboration that an engaged community can create. That it’s also quickly become a defacto standard for journalism schools and is being used to open up more newsrooms to the idea of news code is proof of just how vital collaboratively-produced documentation can be.

In putting together this list of five, I kept adding and then removing tons of other great things. 2012 brought so much amazing work that there’s too much to include. Here’s to the incredible work done in 2012, and—even more so—to the incredible work still to come in 2013. Let’s do this.

December 31, 2012 04:56 PM

The planets have finally aligned on one of my early assignments at The Boston Globe. The project is called Quizzler, and it is by no means going to change anything. It’s a quiz system—something the producers ultimately want because it will generate page views. It has been done.

This post is not about Quizzler, it is about my quest to answer the question “why are we building this from scratch?” It’s about observed realities regarding cross-newsroom collaboration, insights from upper management of The New York Times, and some major hurdles for open source in legacy media organizations. Prepare to explore the deep, dark, and relatively unspoken depths of technological openness in newsrooms.

We want something similar to…

I was introduced to Quizzler back in August. That first meeting was generally uneventful; we sat in a room. I listened to Miranda Mulligan skillfully duke it out with the project’s newsroom sponsor to explain that no, the first version won’t have custom “you are a 95% Vampire” sharable Facebook messages. I listened to the sponsor vocalize concern that there would never actually be a second version. I decided that both of them were probably right.

Eventually someone said something so shocking that I literally spat out my drink and fell out of my chair at the same time. It wasn’t intended to stand out—I don’t even know who said it. Ready? Brace yourself. Here it is: “Have you seen the Academy Awards tool by The New York Times? Eventually we will want something similar to that.” No wait that wasn’t it.

The New York Times is the parent company of The Boston Globe. They own the Globe in the same way humans own their children.

OK here’s the exchange.

Me: “Can we use some of their code?”

Someone: “We would have to pay them for that.”

Me:

Their response implied two things. First, that The New York Times would charge their kid for the digital equivalent of food. Second that the anticipated costs were high enough that it would be cheaper to rebuild this tool from scratch (again) than it would be to explore the possibility of reusing existing code.

Before you call child protection services, hold on. The situation is complex.

EDIT: To be clear, I quickly learned that the Times would not have charged us a dime.

Actually, this sounds completely reasonable

“Meh.” you say, “so The Boston Globe and The New York Times don’t share code, what’s the big deal?” A fair response, but trust me when I say the deal is big. If the deal was a rapper it would be notorious.

Here’s why: Starting from an existing code base instead of starting from nothing is often the difference between “having time to innovate” and “not.” If you are using technology as a core part of your business and you aren’t set up to experiment then you’re doing it wrong and you will become obsolete.

Borrowing code is kind of like being airdropped into the middle of a marathon; sure, you have to take a moment to figure out where you are and what direction to go, but now you have time to run in circles laughing like a crazy person before winning the race.

There’s more! If you borrow code then you are more likely to be familiar with what the rest of the world is doing. If you share code then you are going to build your systems with an emphasis on reuse and extensibility (i.e. correctly). If you regularly borrow AND share code then you are building a community around whatever it is you do.

What I’m trying to say is that if newspapers can buy into the mantra of openness—even just internal openness—they can kill about thirty birds with one stone.

But they usually don’t.

Why not? Are they idiots?

There are many reasons these organizations don’t trade bytes, none of which have to do with the original “we would have to pay for it” claim.

Reason 1: Wildly Different Technology Stacks

I lied to you earlier when I said the Globe was like a child to the Times–they’re more like middle-aged lovers. They didn’t grow up together or meet in college. They are two independent entities that recognized their love later in life, which means they have fundamentally different infrastructures.

One uses Java and PHP, the other uses Python, Ruby, and NodeJS. They have incompatible content management systems. They disagree on deployment policies, quality control processes, needs, and third party libraries. It’s like they come from two stubborn families that speak completely different languages and eat very different foods. They aren’t going to start casually sharing cook books.

Reason 2: Internal Politics

If a full team dedicates three months to creating a new public-facing interactive, will they want to just give it away? If you are a manager do you want to rely on favors from an external team to accomplish your goals? If you are a coder do you want to be judged for the quick last minute hacks you had to throw into the project?

The answer to these questions, and many more like them, is “hell no.”

Reason 3: Moving Costs and Learning Curves

Most technologies are dirty piles of duct tape with a shiny chrome finish. This makes them difficult to deploy and hard to understand. This is especially true among newspapers.

Packaging code in a way that strangers can use could take hours, days, or weeks depending on how much the developers cared about portability when they built it. I’m basically describing the difference between moving a campsite and a home. Newsroom developers don’t tend to have camping on the brain when rushing to meet looming deadlines.

Words from On High

Fine, so there are real reasons that code sharing between the Globe and the Times is a lost cause, but what does that mean for the industry? If financial allies with serious resources don’t share code, what are the chances that other newsrooms around the world will look outside their walls for help? Maybe this is why so many open source journalism tools are gathering dust.

I talked to Rajiv Pant (CTO) and Marc Frons (CIO) of The New York Times about code sharing and the role of open source in their company. For context: the Times is very progressive compared to other newsrooms when it comes to innovation and openness. They have a blog dedicated to their open source inititatives, there is a suite of APIs that provide civic data, and they do a good job of telling people about what they do.

Unfortunately they are also leading an industry that is forced into “deadline driven technology” and without a supportive institutional strategy, open source and reusable code are just nice-to-haves. Developers must ask themselves if they have time to meet the organization’s needs while also contributing to open source. Sometimes this means the same tools get built multiple times, but such is the nature of deadlines. Plus, as Marc was quick to point out, reinventing the wheel can be a good thing so long as the new one is slightly different.

But wouldn’t it be nice if all these new wheels could be used again and improved upon over time? Rajiv identified three factors that a project needs in order to be realistically used again by an organization like the Times.

Your code has to be…

1. Established – Is it safe to rely on your creation? How long will your project stay active, and how long after you move on will it stay useful?
2. Extensible – Your solution won’t meet all needs. How easy is it to improve? What kinds of features can be added?
3. Easy to Integrate – Will this play with existing systems and tools? Can it be skinned to look like it belongs?

In short, it doesn’t matter how powerful you think your code is: if it is difficult or risky to adopt, it will stay an orphan.

None of those points should come as a surprise, but they should probably be considered gospel to anyone developing anything—open or closed—in any newsroom. Just ask yourself “would the Times use this if they needed it?” If the answer is yes then you’ve made something that will last; otherwise you might as well get out the broom now.

Note: Since I’m sure you are worried, the Times doesn’t actually charge the Globe for code. And yes, we are writing Quizzler from scratch.

December 31, 2012 02:56 PM

All the public conversations right now around higher education are getting it wrong. Everyone’s talking about the free access to college courses that Massive Open Online Classes (MOOCs) are creating. They talk about the potentially detrimental effects platforms like Coursera, Udacity, and MITx, might have on the higher education industry. Whether or not these online classes count for anything in terms of credentials. The economic value of college itself. That’s all important. But no one is talking about learning.MOOCs work like this. Online, you can take a class taught by a professor from a leading university. Harvard, Stanford, MIT. Any field you choose. Comp Sci to Ancient History, take your pick. You get custom-recorded lectures, auto-graded quiz assignments, and in some cases, peer-evaluated written work. In some cases if you do well enough, you can get a “certificate of completion.” They courses are run in realtime, with assignments due on a weekly basis and lectures released in order, also on a weekly basis. Sound a lot like school? That’s because it’s trying to be, and that’s a huge mistake, a sadly missed opportunity.

The model here is designed from the perspective of putting teaching online. That’s not the future of education. No one is talking about learning because the people who are talking here and designing systems are education reformers. Reform is not nearly enough to change education now. We need a revolution. We have to start thinking of all this as putting learning online.

What we’re getting now in open classes detaches teaching from physical classrooms and tuition-based enrollment. What MOOCs should be working toward is more radical—detaching learning from the linear processes of school. That’s not the goal of the designers of MOOCs, but it absolutely should be.

What would this detached model of learning with access to the resources of school look like? It looks like the forms of independent learning I’ve been researching and writing about for Don’t Go Back to School. People getting the resources to learn what they want to learn, in contexts in which that knowledge or skill is necessary to the learner, or something they are passionate to learn. Some people learn very well in the constrained modes of school, and that’s fine. But the model of independent learning allows people to learn the way they need and want to. As one among many examples, let’s say you’re taking a class in modern poetry. You’re doing this because you like to read it and want to understand its context, or there’s one poem you love and you want to read more. Neither of these is necessarily well served by a linear class structure. What if you could start with the poem you love and find your way backward and forward and sideways from there, using syllabi or timelines and lectures to contextualize the poem. You could find sideways paths to other contexts of the poem, history, what’s going on in art, or politics, to other writers and poetry. You end up with a broad, insightful, deep, and pleasurable learning experience and outcome. You remember what you’ve learned. That’s what independent learning is like, and that’s what school disallows. That’s the assumption we need to model open education on.

This is important for learners’ experience and for feeding their motivation. It’s also important for a changing job market. Increasingly, at the leading edge of hiring in some businesses and at the center of hiring for other professions, the ability to learn on the job, independently and quickly, is the most crucial qualification.

MOOCs have other challenges besides ditching linear formats. The most important condition for independent learning reported in my research is learning in the context of a community. MOOC designers make only a token effort to incorporate the social aspect of learning, with giant discussion forums that produce crowds, not learning communities.

If MOOC designers start taking a lesson or two from the way learners learn outside of school, they could drive genuinely useful reform in higher education. Right now they’re just putting vinegary old wine in giant new bottles. That’s not enough.

December 30, 2012 09:39 PM

It’s incredible how much has changed in a year. It’s been about a year since I started to really think about news and technology seriously and about ten months since I took on the role of an Knight-Mozilla OpenNews fellow. While on this journey, I’ve felt a gradual and positive change in the world of journalism and its relationship with technology.

Application

Technology applied is a wonderful thing and much credit must go to OpenNews lead Dan Sinker for realising that the world of journalism was ripe for its application. I see Dan as modern-day Druid, this is thanks in no small part to his spectacular beard in which he keeps kitchen utensils and the smaller of his gardening tools, but it chiefly comes down to his leadership style; he’s more a co-conspirator than a boss, someone who provides inspiration and support - in many ways your archetypal bearded spiritual guide.

Getting Organised

I’m a little different to the other fellows, a little older, a little slower and I have a young family which means I opted not to be embedded in Al Jazeera, Doha and work remotely. So when I travel to Qatar to visit the newsroom in the desert, I like to make the most of it.

As fellows we are encouraged to see each other’s work environments and I managed to organise for Nicola Hughes — ex-fellow with The Guardian — to fly over at the same time and give a course on a subject close to her heart - Data Journalism. This was a big win for me. It showed that with a bit of persistence we can actually organise events thousands of miles away that will hopefully make a tangible and positive difference to a news organisation.

Data Journalism

Luckily my contacts in the newsroom did a great job in paving the way and it turns out Nicola is a natural teacher so it all went splendidly. We gave an overview in the AJE boardroom where people as diverse as Dick Cheney, Adriana Huffington, Kevin Rose and Senior Hamas officials have been known to congregate (not necessarily at the same time). It was encouraging to see a large cross-section of AJ English and Arabic in attendance. The main cut and thrust of the course was about how to find stories in data and to that end, how the journalist can scrape, refine, store, query and visualise data. It’s worth dwelling on this whole data journalism concept a little, it’s pretty amazing for several reasons:

1. There are big stories in big data, it’s just a matter of finding them.
2. Traditional investigative journalism is expensive. Data Journalism is investigative journalism done cheaply.
3. Increasingly more data is being made available by organisations and governments.
4. Data well presented can tell a story. Sometimes very few words or opinions are required.
5. Readers can do their own investigating if you give them the option. Readers like to share their perspective.

I’m interested in all aspects of data journalism but the bit that really inspires me is data visualisation - how we present that information to the reader in a meaningful and beautiful way. Yes beautiful - the aesthetic is important, so is the design of course, likewise the content - all fundamental parts of a good visualisation - that’s what makes it so much fun. Admittedly Edward Tufte has had his due influence on me, when I read his work something just clicks and when I meet another Tufte admirer, we click. It’s almost like there’s this secret underground pipe-smoking bohemian society that believes information can and indeed should be beautiful.

The Web is the Medium

And then there’s our old friend the World Wide Web - a medium that is fast maturing into a vibrant and interactive canvas for presenting data. We have libraries born from news organisations and the strong desire to visualise data in new ways. We have interactivity. We have transitions. We have transparency. We have community. We have a low cost method of getting shit out there. The web has always been the ideal platform to present data and now it’s becoming the ideal place to visualise data. This is an exciting place. I might want to hang around a bit.

Making a Difference

I was happy to not just bring Al Jazeera and Nicola together in this way, but also be part of the data journalism course itself, I talked a little about visualisations and gave a session on a library called D3.js. The good news is that Al Jazeera are now looking to run the course again and they have committed to investing in data journalism next year.

Mark Boas

December 20, 2012 02:29 PM

December 19, 2012 03:54 PM

November 30, 2012 11:31 PM

The Mozilla Festival

Only now getting some time to reflect on my third Mozilla Festival and all the fantastic events and interactions that took place at Ravensbourne College in London this year.

Once again I demoed Hyperaudio at the festival’s Science Fair and it was great to see the encouragement and enthusiasm surrounding that, but the main event for me was Saturday morning’s Second Screen Prototyping session. This post is intended to document that session.

What is the Second Screen?

The ‘second screen’ or the companion app refers to the additional screen you use whilst ‘watching’ the main screen. The main screen is usually the TV, but can also refer to the cinema screen or even the radio. The second screen can be a phone, tablet, game controller, a laptop or PC.

Who’s using it and who’s making it?

Research suggests that as many as 75% people use a second screen already but generally use everyday applications and services such as email, social networking and sites like wikipedia.

Realising the value and opportunity that the second screen presents - many companies are creating a new generation of applications that are tailored and intended exclusively to compliment main screen content.

What can we do with the Second Screen?

Many, many things. To mention a few :

• main screen control
• augment programme* information 
• provide social communication around a specific programme*
• facilitate user generated content
• vote, poll and quiz (participate)

*Note my spellchecker doesn’t approve of the word programme but in this instance I mean “A presentation that is broadcast on radio or television.” http://en.wiktionary.org/wiki/programme

What did we hope to achieve with the Session?

Since second-screen technology is relatively new, the whole concept is a rich vein to mine. I hoped to attract a group with diverse backgrounds that could come together as small teams and paper-prototype a few ideas specific to media genres such as film, sport and entertainment.

How did we get things Started?

More people turned up than I had expected. Thankfully I was helped out with the facilitating by the very able Ben Moskowitz. We started by polling the group on their knowledge of the subject and then talking a little bit about what we thought the second screen was. I then gave a very brief presentation on second-screen stats and some example usage.

It was great to have Mike Pennisi from Bocoup and Travis Daub from PBS’s Newshour join and present their second-screen companion app for the election. We were also privileged to be joined by Simon Klose who’s making an interactive documentary The Pirate Bay AFK using a second-screen app called Linkontrol and has been researching and implementing second-screen technology for some time now.

Getting Going

After the brief presentations we asked people about the sort of media to which the second screen could be applied, encouraged them to form small groups around those different themes and get some ideas down ready for the next stage of paper-prototyping.

The technology behind second-screen apps can actually get pretty complicated but I told everyone not to worry about that and just to assume everything was possible. I wanted to capture ideas.

The various groups were self-organised around the following categories:

• Movies x 2
• Sports
• News

At about the half-way point we asked folk to share their ideas with the other groups in the session get feedback and re-huddle for paper-prototyping.

So what came out?

Ben and I both moved from group to group and worked on some of the ideas and it was great to see people having fun and not applying a ‘no limits’ attitude to their brain-storming. We were pretty insistent about asking people to get their ideas down on an etherpad so that nothing would be lost and actually I’m pleased to say that some very cool ideas came out of it all.

(Italicised text below represents that pasted directly from the etherpad)

Movies - Part 1

(Ingrid Kopp, David Illsley, Ian Jempson, Nic Wisteich, María Yáñez)

” 1. App for bookmarking clips / time-frames while watching any content 

• Options for both unobtrusive bookmarking experience or deep dive into content and social experience while film is playing
• Clips can be shared out to social media
• Metadata surrounding clips is included eg. locations, music playing, actors
• Possible option for SoundCloud-like commenting system on the film’s timeline?

2. Airplane mode for movies… director can indicate moments when interruptions are particularly unwelcome - can delay delivery of sms/redirect to voicemail… or redirect to a voicemail saying you’re watching something and give the option to ‘break in’

Fan fiction on the second screen to extend the life of a feature film

• the file doesn’t change, but the context does

Two UX options:

• a waterfall / list of metadata linked to timecode scrolling (like a Twitter stream)
• a fixed width visualisation (like a Soundcloud player)

Some nice ideas here, certainly I hadn’t considered an Airplane mode but that could be a very convenient piece of functionality given that we could access the necessary telephony APIs.

There was another suggestion that I think could apply to many different forms of media.

3. Second screen for activism - get people when they’re fired up - donate/call your congressman via skype..? “

This is interesting because services like Shoutabout and Avaaz which facilitate news related activism already exist and the second-screen could be the perfect medium to call for action while people are watching related material.

Movies part 2 (Physical Effects)

(Sarah Wolozin, Robert Basden, Wesley Lindamood)

” Physical Effects is a second screen application that serves as a hub to control connected devices in your home.  By creating effects in your physical environment, it facilitates a more immersive film experience.

Using sensors on the tablet (camera, microphone) and physical sensors, heartbeat monitor , perspiration monitor (through apps like fitbit) “Physical Effects” will subtly and intelligently adjust light, sound, temperature, and other connected devices while the film is playing.

By sharing your experience with others using this application, ambient sound from their physical space can be incorporated into your home recreating a shared audience experience.

Lastly, to make the experience more tactile. 3-D printers, will be used to print out objects from the film that may be out of view on the first screen but provide more information and meaning to the film. “

This is great - a classic case of thinking without borders. It sounds a lot like something we’ve been working on with Ian Forrester’s team at BBC R&D under the umbrella term ‘Perceptive Media’. Which coincidentally was demoed later on in the festival.

Sports

(William Bailey, Mark Fullbrook, Martin Skelly)

“Digital Stadium 

A second screen app that celebrates the emotional rollercoaster of being in the crowd at a football stadium.It pulls in the digital stories created with social media around premier league football matches. 

* Principles *

• Celebrating the adversity of team sports fans 
• Wider digital conversations of football matches in real time 
• Design for passive and passionate fans 
• Optional log in 

* Criteria * 

• Tablet based (mobile options) 
• Simple log in and match choice system 

* Features * 

- Visualisation -

• tweets per minute line graphs 
• see tweets in real time 
• word clouds 
• sentiment and feeling scroller 

- Tweet Categorisation -

• correlation to match events on timeline 
• encourage game tweets - with themes tweets suggestions 
• method of segretating tweets based on hashtag 

- Automated -

• knows what games are scheduled 
• automated creation of virtual stadiums based on schedule api 

- Other -

• match build up - anticipated matches on schedule “

Lots of opportunities with sports programmes, especially live ones. Currently people tend to use their social networking service of choice as a general second-screen app but there’s plenty of scope to tie these services more closely to main screen content. 

News 

“Goals:

• Improve on the weaknesses of the first screen
• Give viewers more control
• Enable viewers to participate (or choose not to)
• Add a deeper layer of information and expertise. 
• Engage social users
• Increase the flexiblity of the media. 

Idea:

Build a second screen app that interfaces with the user’s DVR. The content provider provides a main televsion stream, but also alternate video or content streams that can be viewed and then return to the main program. Streams can be viewed on the second screen or on the first, depending on user discretion.  

The main program can extend or contact depending on the user’s interest. Portions they like may feature longer uncut interviews that seamlessly appear inside the program at the users’s discretion.  

User can create commentary using the camera on the second screen and add it to the timeline. Other users see popups that “User X” has created a video comment, watch it now…”  Programs grow and nurture communiites of experts to contribute to those programs. “

Again plenty of scope with news apps and this group came up with an interesting idea where it seems that an almost hypertext-like mechanism is applied to video. Allowing viewers to discover and explore there own route through the news.

Brain Food

This was an experiment in process as much as anything else. What happens if you put variety of mostly unrelated people in the same room and ask them to brain-storm ideas? How much do we need to guide the process in order to achieve something worthwhile? Well the answer is that as facilitators we kept our initial presentations to a minimum and moved as quickly as possible to inspire the larger group and provide ‘brain food’ (a phrase coined by Ben). 

We encouraged people to break into small groups and had a lot of fun jumping from group to group to see how things were coming along adding input if we felt it was needed but really just encouraging people to come up with their own ideas.

I think the idea to imagine that there was no restrictions was a good one - it allowed people to let their creativity flow and actually, I think all the ideas we got are pretty much feasible.

It took a while to write up but I definitely think it was a worthwhile session and I’d like to thank all involved. Hopefully these ideas will become food for thought for other people and lead to further brainstorming and innovation.

Further Reading

In preparation for this session I undertook a fair amount of research and the end of this post it seems like as good a place as any to share those resources.

1. Great Pinterested stuff on Second Screen and Social TV 
2. Bravo’s “Play Live” Adds Polls, Games to All Its Shows 
3. KIT digital takes fresh approach to second screen 
4. 24% of people use second screens while watching TV 
5. Synchronized second screen 
6. The Guardian’s Second Screen: your indispensable London 2012 companion 
7. Anatomy of an interactive: a look at the code behind our Second Screen 
8. THE RACE FOR THE SECOND SCREEN: 5 APPS THAT ARE SHAPING SOCIAL TV 
9. Multiscreen PatternsPatterns to help understand and define strategies for the multiscreen world.
10. Grab Magic: My hack for The Boat That Hacked #mipboathack 
11. RWW Recommends: The Best Channel-Surfing Companion App for TV Trivia 
12. SmartGlass, Xbox 360 and the battle for the connected living room 
13. Google Multiscreen World 
14. Making great TV even better: The BBC’s approach to companion 
15. Shazam Is TV, Not Music 
16. Use your phone as the controller for in-browser games. 
17. Is This the Second-Screen TV App That Finally Goes Mainstream? 
18. Network Service Discovery API support in Opera 
19. The Second Screen Hype Cycle 
20. “Second Screen” Ideas for Radio 
21. IBC 2012: Clash of Four Second-Screen Titans
22. Netflix sneaks in some Simple second screen functionality 
23. Companion screen services - one year on 

November 19, 2012 12:36 PM

Last weekend (9-11 November 2012) saw the second Mozilla Festival. Over a thousand people came through the doors of Ravensbourne College, packing 9 floors and hacking to their hearts content. Digital journalism super stars Aron Pilhofer, Brian Boyer, Scott Klein, Miranda Mulligan and the Guardian Interactive Team were in attendance.

I optimistically decided to run two learning labs. Both teaching non-coders to hack and hackers some media literacy hacktivities and interactive video with Popcorn.js. So there is something for everyone! These will be put into a hacktivity kit but until I find a home for making my workshops into online lessons I’ll be putting all the links right here.

First a word of note, if you’re attending my workshops or learning labs bring a laptop and not a tablet. I’m very hands-on!

HTML for Journalists

Using Mozilla’s new webmaker tool, Thimble, you can code and see what the browser sees. In this hacktivity you markup and style a news article, learning the bascis of HTML and CSS. For those of you already web savvy there is a media literacy game suggested. Try it out!

Location-based Storytelling Using Popcorn Maker and Popcorn.js

This was really fun to make and teach. Popcorn is the project that is going to get me to further hone my JavaScript skills. It is the most applicable webmaker project to journalism. Here non-coders use Popcorn Maker to replicated a BBC Interactive and those who code for the web can recreate it easily using Popcorn.js. Choose a track and try it for yourself. For non-coders I suggest you do the HTML course first, go through the web fundamentals and JavaScript fundamentals track on CodeCademy and then have a crack at the Popcorn.js assignment!

 

Besides running the above workshops, which was great fun and so many journalists attended, all the OpenNews 2012 fellows also gave a one minute presentation on something they got up to this year. Watch it here (25 minutes in). The 2013 fellows were also announced. It was great to meet almost all of them and I look forward to following their amazing journey next year. It’s mind-blowing to think that such talented and diverse people are offering their skills to newsrooms around the globe. Counter to what most people would think, I think now is an incredible time to be in journalism. People are telling their own stories, instantly and to the world. The fabric with which media people now work with is truly intricate and interwoven through the public sphere. Now we should be experimenting with new types of stories that can be told.

November 13, 2012 05:20 PM

This is the second of three posts about the state of development in journalism, where we’re at with the Knight-Mozilla OpenNews project, and where we’re going. It caps off on Thursday with the announcement of the 2013 Knight-Mozilla Fellows, an announcement that then launches us into the Mozilla Festival in London, starting Friday

With the Mozilla Festival approaching in just two days, and the announcemnet of our 2013 Fellows happening tomorrow, it’s a nice moment to reflect on how far the Knight-Mozilla OpenNews project has come in 2012 and where we’re going in 2013.

Writing this in the looming shadow of a trans-Atlantic flight to London for the Mozilla Festival, it’s actually pretty overwhelming just how far our project has transformed since I “thought out loud” about opportunites in the intersection of journalism and tech prior to last year’s Mozilla Festival. So it’s time for a little more thinking out loud, both about where we’ve been this year, and where we’re going next.

OpenNews 2012: there and back again

Back in February, we announced a new name and an “evolved” focus for the newly-christened Knight-Mozilla OpenNews project. The idea was to keep our Fellowship program intact, but to build out a much larger program dedicated to growing the community around coding and journalism. Here’s how we did:

Hack Days: We went into 2012 with a new initiative to sponsor, promote, and support hack days around the world that adopted journalistic themes. I firmly believe that if you want to grow the community around tech and journalism, you need to engage people in a way that demonstrates this is a place hackers, developers, and engineers want to play. Hack days are incredibly effective in doing that, and here as we approach the end of year, we will have helped sponsor more than 20 hack days around the world, with over 2000 participants.

Source: Throughout 2012, I’ve been incredibly lucky to work with the talented Erin Kissane and Ryan Pitts to create Source, a website designed to be a centerpoint for the journo-code community. Launched last month after being in a public beta since the Summer, we’ve been able to collect looks at how news devs reacted to Hurricane Sandy, disections of election-related visualzations, and much, much more. Source is just getting started, and it’s already become a go-to, well, source.

Code Sprints: I announced our Code Sprint initiatives from the stage at the Hacks/Hackers Media Party in Buenos Aires Argentina, and I’m excited to say that last night, the first project to recieve funding from the project successfully went live. A collaboration between the techs at WNYC in New York and KPCC in Los Angeles, they built an open-source parser and embeddible map for the XML election data stream coming out of the California Secretary of State’s office. Used by multiple news organizations big and small, it’s exactly the kind of creative, collaborative, reusable, and back-end focused project that we envisioned the Code Sprints for. That’s one down, nine to go—with a rolling application process, what’s stopping you?

Fellowships: And of course, there’s the core of the OpenNews program—our Knight-Mozilla Fellowships. We placed five incredible people into five of the best newsrooms in the world and gave them an open mandate to hack, remix, and recreate news for the open web. They’ve filled code repos, gotten bylines, attended dozens of hack events, and generally made the most of their fellowship year. As they conclude their fellowships a little later this year, I’ll be posting more about what each of them did.

OpenNews 2013: learning on the horizon

Much of our gameplan for 2013 is more, more more. We have plenty more Code Sprints and Hack Days to fund, we’re placing eight fellows—tomorrow, we’re naming names—in newsrooms around the globe, and Source will continue to crank out new content and document the vibrant community of journalism coders.

But if you look back at the original OpenNews 2012 announcement, you’ll note that there’s one promise not yet fufilled: Learning. Learning. Back in the summer, I wrote a quick post announcing a team of “Learning Avengers” made up of some of the best minds in the journo-code community. We’ve laid out some good plans together, but it wasn’t until I was watching the Avengers on a flight back from Argentina that I realized something was missing: You can’t assemble the Avengers and not have a Nick Fury helping guide the team (comics folks, if I’ve totally messed that up, apologies—I’m a DC guy).

And so I’m excited to announce that I found that Nick Fury in Kio Stark, a professor at NYU’s ITP program, and a woman who is literally writing the book on informal learning. She’s getting the Avengers together to launch OpenNews learning big in 2013. The gameplan, in short, is awesome: We’re going to educate developers about how journalists work. How they use and interact with raw data, how they use visualizations, how they use mapping, and what they need to make all three of those more efficient and more informative.

In addition to the developer-focused learning iniatives our Avengers are overseeing, we’re starting to pilot Webmaker learing projects oriented towards journalists who want to learn basic coding skills. We’ll be testing out a new “hacktivity kit” at the Mozilla Festival this weekend. If you’re there and want to learn the basics, join in.

OpenNews Learning will be another part of the ecosystem we’re building around journalism and code, and we’re all excited to kick it off starting in early 2013.

I’m excited about EVERYTHING in 2013, and can barely wait to tell you about the new Fellows we have starting—but that will have to wait until tomorrow, as I have a flight to catch.

See you in London!

November 07, 2012 10:36 PM

This is the first in a series of posts this week about the state of development in journalism, where we’re at with the Knight-Mozilla OpenNews project, and where we’re going. It caps off on Thursday with the announcement of the 2013 Knight-Mozilla Fellows, an announcement that then launches us into the Mozilla Festival in London, starting Friday.

Election season is the Super Bowl for news-application teams. There isn’t a single developer at a US- or global-oriented news organization that hasn’t been bringing their A-game these last months. And in just a few hours, as election day begins in the US, the big game is played.

The in-depth articles and weekly newsdev roundups we publish in Source have charted the projects news apps teams have built around the debates, around election returns, and looking at swing-state outcomes, and much more. Now, all that work comes to a head.

It’s going to be a fun night for people who want their election information visualized, streamed, plotted, mapped, or presented pretty much any other way you can imagine. These aren’t election night holograms, but instead are genuinely innovative ways to improve the user experience of news and help people to better understand the information flow in what might be a chaotic news night.

This is what news application developers do best: They take information and they make it easier to parse, navigate, and understand. Which is exactly what we’ve always turned to journalism for and is why the growing news-dev community is filled with luminaries like Jeremy Ashkenas, creator of Backbone.js and Mike Bostock, creator of the d3 visualization library. People want to set information free on the web, more and more, those people are drawn to journalism.

We’ve been able to watch that draw lead to real growth in the community of people actively building new tools in journalism over the course of the last year of the Knight-Mozilla OpenNews project. Helping to build and strengthen that community is job one at OpenNews, and that’s why we’re excited that this week, post-election, people from that community—people doing news dev from the New York Times, the Guardian, the BBC, Zeit Online, La Nacion, Al Jazeera, the Boston Globe, ProPublica, Bloomberg, and many, many others—will convene in London for the Mozilla Festival.

With the Festival happening so quickly after the election (and with all fingers and toes crossed that the election, you know, ends by then), it seemed like a great opportunity to bring the community together to compare notes on what we built this election season, and make big plans for building new tools. So guess what we’re going to do?

That’s right: At the Mozilla Festival, we’re going to bring together the incredible talent that’s descended on London for three hours of thinking about what we built for elections this year and—more importantly—how we build upon what we learned from it all. We’re going to bring people together to sketch, design, and hack together new ideas and tools for election coverage, from visualizations to the backend.

This is the very best of what we do at OpenNews: Get smart people in a room together and turn them loose on creating new things in journalism. So if you’re in London, be one of those people and join us at the Mozilla Festival for some election hacking. And if you’re not, follow along at #mozelect, and look for a compilation of how news orgs approached the election on Source next week.

Tomorrow we’ll continue this series with a look back at what we’ve accomplished with OpenNews this year. And Thursday: NEW FELLOWS ANNOUNCED!

November 06, 2012 03:44 AM

I made something awesome last week: Opened Captions.

At face value it just looks like a live feed of C-SPAN’s Closed Captions. This alone is actually pretty cool if you think about it, especially if you are a deaf political junkie who sits far away from the TV and can’t read the closed captions.

Of course there is more. The real excitement comes when you contemplate what’s happening to get those words to appear on your screen.

This system unlocks and syndicates a real-time dataset that used to be a pain in the ass to access. Now anyone can build applications and visualizations that update before those crafty politicians have even finished making their points. This post explains why Opened Captions is worth hacking with, what it takes to use it, and how it works.

What is it Good For?

The Internet is filled with real-time updates triggered by online activity, but it still feels like magic when we see automatic updates driven by the real world. Opened Captions makes it easy for programmers to use live TV transcripts as an input.

Note: version .001 only supports a single channel (and my server is pointed to C-SPAN). Eventually the protocol should expand to allow multiple channels.

Let’s consider C-SPAN. If a computer knows what is being said on C-SPAN this very second, it can do things like:

• Change the background of your email client to reflect the issues being debated right this moment on the senate floor.
• Generate modified, more amusing, transcripts by replacing key words and phrases with Tolkien lore (i.e. C-SPAN for Middle Earth)
• Search through lyrics and generate a C-SPAN medley for you to rock out to while voting.
• Send SMS messages 24/7 commanding you to “drink” when certain phrases are spoken on air.

There are also possibilities that aren’t ridiculous. For instance, you could make tools that…

• Improve the transcript by automatically adding contextual information, such as definitions and histories thefted from Wikipedia.
• Send emails with transcript snippets whenever a specific representative or state is being discussed on TV so you know what’s going on.
• Parse out paraphrases of known fact checks and insert a credibility layer over the transcript feed (real time fact-checking).
• Draw parallels between what is being said on TV and what is being said on Twitter.

I could go on and on and on. There is just so much potential!

The Backend

Behind the stream is a first stab at a distributed architecture for Closed Captioning live-feeds. Opened Captions servers can pull a CC stream over a serial port, or (more likely) they will connect to an existing Opened Captions server and pull the stream from there. What that means in de-jargon is that anybody can set up a server that does exactly what mine is doing, even if they don’t have access to hardware, software, or a live TV stream.

When I say exactly, I mean it — your new project runs the same code as mine, and will serve the feed too. People can connect their servers to yours in the same way you connected yours to mine. Practically speaking this architecture means a few things:

1. Once your amazing mashup gets popular it won’t break my server. Your application is syndicating the captions to your users. I serve the captions to you, you serve them to the world!
2. Your server creates a fork of my stream. Want to modify the text so the politicians sound drunk? Add extra layers of information to the message payload? Translate the captions to Klingon? Go for it. If your tweaks happen server side then others can build their apps from your stream to modify it further.
3. You don’t have to rely on anyone else for the Closed Captions. If you want to spend some extra time setting up your own scraper you can point your server to that source instead of a third party. You have total control.

Check ‘Em

Wondering if this is worth your time? Well, it doesn’t require much of it. The service takes about two minutes to set it up if you already have Node.js and Git installed on your computer. Here’s a video to prove it:

Installation instructions can be found in the readme and you can always get in contact with me through the blog or on twitter.

October 25, 2012 08:16 PM

Now is the time to experiment. The second decade of the second millennium is all about crises. The stock market, the Arab Spring and the EU. The characters of history will be politicians for sure, but the celebrities, the cultural icon for these harsh and unsure times is the billionaire entrepreneur.

Shaking things up, moving things around and exploring new territory is how we have survived as a species for so long and this crisis will stand to prove that those who adapt will survive. Nowhere is this more evident than the tech industry. Nowhere is this easier to do than the web. The web is a new ecosystem to explore. It is the new medium not just for monetary transactions but for social internactions. So why is the move from print to web proving so difficult for the news industry?

I have been fortunate in the last two years to get a glimpse at the greener side of the road. As part of the OpenNews programme developers are put into news organizations. I, on the other hand, was in news. I trained in news and worked at Channel4, BBC and CNN International. For me, the exciting new prospect was not just to work with developers to hone my data skills but to be part of Mozilla. I didn’t realize how much a part I would be until last month where I found myself at the All-Hands (AGM) with 50 other Mozillians.

It was a real eye-opener and a real privilege to be there. It was my chance to learn more about Mozilla and the projects they are undertaking. Before I tell you about those I want to list my observations on the way Mozilla is run which I think reflects the web-based tech industry as a whole. Mozilla are:

• Inclusive of every employee: everyone had a say, everyone could get involved and contribute to decision-making
• Promoters of creativity: success on the web is built on ideas and ideas can come from anywhere and anyone
• Hugely varied in their employees backgrounds and skills: from designers to teachers to musicians, there was a wide range of personalities
• Risk-takers: they see themselves not just as making fun things for the web but as taking on the likes of Facebook and Youtube
• Concerned about sustainability: even though the foundation has funding they are still looking at how they can support their work in perpetuity
• Mission-driven: every project must have a purpose and this must fit into the Mozilla mission statement of an open and inclusive web
• Moving to coherency: for their message to be loud and clear they are creating threads between all their projects

The modern newsroom is still faction-based between digital and editorial. There is still too much of a top-down approach to management. Your product is still your content, no matter what medium it is on. If that content is ripped from wire copy it is information when contained in a touch and take newspaper but has very little value on a web of social and global links. People need to touch a paper and people need to be touched by the web. There needs to be creativity (and humanity) in the creation and presentation of content. The medium is the message.

For the audience to feel a connect to the news product there needs to be a real connection in its creation. News cannot be produced in a factory and it cannot be produce in code (the aggregators will lose their audience to a truly digital news platform). There needs to be a passion for journalism, real journalism in the newsroom for a news organization to survive. In that sense, hack days should not just be for developers, they should be for journalists also; to hunt down the stories that have passed under their nose in the process of producing copy. Developers and journalists should have access to as wide a range of skills as possible to make these ideas reality.

Managers should take risks at news organizations. Now is the time to do things differently as your product has to do its job differently. Be agile and be lean, just make sure you have learning metrics so you know the results of the changes you make. Turn to your developers, turn to your journalists and turn to your supporters. You no longer have readers or an audience or even users. You have supporters. They can be supporting you financially and in other ways. Your product is not a chair, it serves no definitive function. It is ideological. Your supporters choose to support you because they want you to continue working to your ethos and principles. In that sense, you do not have a brand. You have a mission. Managers in newsrooms speak of brand far too much, and building products around said brand. No. You have a mission and you find ways for your supporters to connect with and further their support in your mission.

On the surface, I want news organizations to be more like tech organizations because I want 20% time, I want the high spec computers and great internet connection, I want the cereal bar and soft drink filled snack bar, I want the fun events and travel. But I think it goes deeper than this. I think they should adopt the ideology.

The Mozilla Festival 2012 will be in London 9-11 November. I will be running two workshops in the journalism track, Location-based Storytelling using Popcorn.js and Popcorn Maker (yet to be released but watch this video) and HTML for Journalists using Thimble. Do come and join me.

October 23, 2012 03:40 PM

I sent an e-mail from a hotel room in Berlin in September of last year, while completely blitzed out from jetlag. In it, I mentioned the idea of putting together a site that could serve as a center-point for a lot of the amazing code being written in the journalism community. The response I got from the couple people I ran the idea by was “Yeah, that sounds great, but who’s going to do it?”

Earlier this year I decided the answer was “us,” and I assembled a team to build it.

And today, after many months of building something from nothing, it’s launched: Announcing, Source

Through feature articles that dig into the specifics of the code and the motivations that behind it, through an index to open code repositories produced by the journo-code community, and an index to that community itself, Source connects the many lines of code that make up journalism today with the people that write them. We’ve built relationships between code, people, and organizations deep into the data models of Source because we know that code is always a reflection of the individuals that create it and that those individuals combine to create a thriving community.

Journalism is in a time of massive innovation and reinvention. From data journalism to building news applications, news organizations both big and small are trying things anew. Rethinking the way the world learns about itself is a huge, exciting, and inspiring task. At OpenNews, we’re assisting this lofty goal by helping to strengthen and grow the code and community that is working to build journalism’s future. We do this through our fellowship program, through our sponsorship of hack days, through our code sprint grants, and now through Source.

This wouldn’t have been anywhere near possible without the incredible work of the Source team: Erin Kissane, who I tricked into coming to a conference in Phoenix last December so I could talk her into running this project with me, and Ryan Pitts, whose coding skills and amazing insights didn’t just build a great site, but built *the right* site. Their work, along with the always-there kick-assery of Erika Owens and the steady server-side hand of Ross Bruniges, has been thrilling to be a part of.

It also wouldn’t have been possible without feedback and ideas from dozens of people in the journo-code community who, over the last ten months, saw this project as their own. We’re proud to be a part of that community and to bring something like this into it.

This blog post is already longer than it needs to be: Go view Source! Go follow us on Twitter! And, most importantly, spread the word!

October 16, 2012 03:24 PM

I’m nearing the end of my Knight-Mozilla Open News fellowship and I’m eager to go out with a bang. Initially I intended to travel over to Doha for the elections to capture the excitement of a news organisation operating at full tilt, but that trip has been postponed until the end of November. It works out though - one of my personal remits was to show that it was possible to work effectively while remote and with much help from my on-site colleague Mohammed Haddad we seem to be doing that.

We decided to do something with hyperaudio (a project I’ve been developing on the side) and the election debates. So for the last three weeks or so I’ve been working pretty hard on that. Work expanding to fit time - there’s nothing like an immovable deadline to make you work at maximum capacity and you’ll find plenty of absolute deadlines in the news business.

Real Deadlines

The great thing about the US election debates are that there are four of the blighters. We had already compressed about a months work into two-weeks for the first debate but now we get to bring out versions 2 - 4 over the next weeks. This is the fun bit - I’ve always envied developers who’d shipped version 1 and were onto the next stage of streamlining and improving for the next release. Usually though this process takes months or even years. I was going to get to do this in weeks.

So a bit about the process - to call it agile would be to do it a disservice, to be honest to call it a process would be doing it a disservice. There is no process - not a fixed one anyway. We’d been discussing what we’d like to see on and off over the last few months but we had no best laid plans. I’d been travelling recently and so I didn’t really have time to start thinking about this seriously until about a month ago - but I’m a great believer in leaving ideas and thoughts on that great perculator at the back of your mind. I had a tonne of ideas but Mohammed and his colleague did a great job of cutting these down to the minimum and viable and with that and a few rough sketches we were off to the races.

Nipples

Anybody at this point expecting to hear tales of exquistely crafted code, tightly packaged JS libraries, unit-tests and all the things you are supposed to do, better turn away now. In my other life I mostly make prototypes and demos and get other people to do the crafting (only half-joking). I appreciate that I am spoiled but I feel this quick-fire approach suits the transient world of journalism quite well, where it is all about shipping - there is no room for people of an obsessive-compulsive disposition here. Of course it has to be ‘good enough’ ™ but I guess the stats will be the judge of that.

My approach was not to start from scratch but as I like to have working code from the very beginning - especially when there’s a tight deadline - I took the hyperaudio pad, which contains a lot of the functionality I wanted and started hacking on that. I just did the minimum to remove pieces I didn’t need (display:none is your friend) but it meant I could get something demoable in a day or two, which then allowed Mohammed to feedback and allow us to continually adjust course. This was extreme repurposing and from a code-base perspective a little inefficient. In fact when we shipped we still had blocks of code, structure and styling that were never used, but then again by leaving everything in I would often discover functions that I could repurpose and was very glad about it. In a vague attempt to justify this approach to someone I used the analogy of the male nipple which has no conceivable purpose but in the end is ‘easier’ to ‘leave in’.

The Wolf

You know that bit in Pulp Fiction where things get just a little bit too messy and they have to call in The Wolf? Well luckily for me I have my very own Wolf. My colleague Mark Panaghiston is the author of jPlayer, and as such has the wonderful ability to identify things that can be improved when it comes to web-based audio or video. At about 24 hours to go we realised that although things were working, laptop fans were spinning and things were perhaps a little slow to load. Bringing Mark in also helped me psychologically - suddenly things got git-hubbed and there were two people under pressure coding and bouncing crazy ideas off each other. Hilarity usually ensues - it was fun and we went twice as fast and pretty much implemented every feature I’d hoped to and made for an enjoyable three-legged sprint!

Doing it Live

Of course there has to be a price to pay for all this slackness and due to the plan of releasing immediately after receiving the transcript we had a few problems getting things working initially and sure enough we didn’t test thoroughly enough and similarly to how you only notice your spelling mistakes after you send that important email, we hadn’t noticed a subtle but vital bug until after the interactive had actually been published. The good news was we fixed the bugs and thanks to the fact that we were using client-side only iFramed sandboxed code that lived on a server that Mohammed could easily deploy too, we managed to deploy any changes relatively quickly

Postmortem and Moving On

I mentioned that a large measure of success in the journalism sphere is how many people read/use your work and I was pretty disappointed by the reaction to the interactive. Many people praised it but not that many people used it. To date we’ve had 389 Facebook likes and 203 tweets.

At one point @ajenglish with its 1.3 million followers tweeted it and we only got 14 retweets off the back of that. One theory was that there was lack of interest possibly due to timing caused by the fact that we had to wait almost 18 hours to get the transcript and it was an hour or so before we got everything working as we should. Another theory was that performance was lacking.

You can judge for yourself http://www.aljazeera.com/indepth/interactive/2012/10/20121049528478583.html

But the good news was that the low level analytics I’d built into the app worked and we were able to analyse how people were using the interactive (all this without creating any back-end code). That information was fed-back into the piece as :

The top searched keywords: Economy, Big bird, Obama, Jobs, Iran, Tax, Health, War, Obamacare, Taxes

So we’ve been around the loop another time this week, but less frantically and as I wait for the transcript to come in and write this blog post (when I really should be testing things), my expectations are lower, especially as general interest in this debate seems more subdued, but whatever happens we’ve got two more shots at it after this one! Iterate or die! :)

October 12, 2012 11:25 AM

In an article on the Guardian Data Blog Claire Provost outlined how the recent furore over consultancy spending by the Department for International Development (Dfid) should not be about turning the aid tap off but about making aid work for the donor country. One way to promote development in donor countries is to untie aid, to allow companies and consultancies in developing countries to win contracts for work at home. In that way, grow local industry and promote local expertise.

To look at this angle I scraped the Dfid contracts from contracts finder and looked at which contracts were won by UK companies. The article had the data but ScraperWiki has the code for any of you interested in digging up contracts.

Firstly, you should scrape all the links to the individual contracts from the search result page. Here is the one for Dfid. Click on “Copy” to get your own and change the “search_page” variable to the URL of your search. To make sure you get all the URLs change the page size in the URL to make sure they are all on one page.

Next, go into each URL by attaching the data from your search results scraper into a new scraper which extracts the HTML and pulls out the necessary information. Here is the one for Dfid. I have used the HTML scraping library BeautifulSoup. You can find the documentation here.

So take a look at the source, take a look at the code and take a look at the documentation. Open data and open up the news.

September 24, 2012 03:17 PM

As ONA12 gets going in San Francisco, we’re hitting the last stretch with Source, the OpenNews community and index for news development. We even have a launch date, and it’s…really soon.

On our end, we’ll be spending the next couple of weeks dropping many (so many) new projects and updates into the code index, tweaking UI and content structure, and beginning formal Q&A. 

We need you

The craziest and most exciting thing about this project has been that from the very beginning, working news developers and data journalists have been so enthusiastic about it. All year, at hackathons and conferences—and a mud-filled tent at SXSW—some of the smartest people I’ve ever have said the same thing: “how can I help?”

And I’m really happy to finally be able to say: here’s how.

First, please hit the dev site and send bugs, comments, and questions as Git issues. Source is for you, and as we lock down a stable set of features for our first release, we want to make sure they’re as useful to you as they can be.

Second, SEND US YOUR STUFF. Got a project writeup you’ve been meaning to do and haven’t quite finished? We want it. Thinking about a quick walkthrough of a technique you just used on a newsdev project? Send it our way. We’ll be happy to talk structure, edit rough drafts, and interview you or your team members—whatever it takes to get the knowledge out of your head and into the world. We’ve all done enough share-your-work documentation to know how much time it takes, and we’re here to make it easier.

I’m erin@incisive.nu and @kissane on Twitter, and our community manager, Erika Owens, is at ONA, so stop her in the hallway or find her on Twitter at @erika_owens if you’d like to discuss any of this in person. 

September 20, 2012 02:12 PM

As ONA12 gets going in San Francisco, we’re hitting the last stretch with Source, the OpenNews community and index for news development. We even have a launch date, and it’s…really soon.

On our end, we’ll be spending the next couple of weeks dropping many (so many) new projects and updates into the code index, tweaking UI and content structure, and beginning formal Q&A. 

We need you

The craziest and most exciting thing about this project has been that from the very beginning, working news developers and data journalists have been so enthusiastic about it. All year, at hackathons and conferences—and a mud-filled tent at SXSW—some of the smartest people I’ve ever have said the same thing: “how can I help?”

And I’m really happy to finally be able to say: here’s how.

First, please hit the dev site and send bugs, comments, and questions as Git issues. Source is for you, and as we lock down a stable set of features for our first release, we want to make sure they’re as useful to you as they can be.

Second, SEND US YOUR STUFF. Got a project writeup you’ve been meaning to do and haven’t quite finished? We want it. Thinking about a quick walkthrough of a technique you just used on a newsdev project? Send it our way. We’ll be happy to talk structure, edit rough drafts, and interview you or your team members—whatever it takes to get the knowledge out of your head and into the world. We’ve all done enough share-your-work documentation to know how much time it takes, and we’re here to make it easier.

I’m erin@incisive.nu and @kissane on Twitter, and our community manager, Erika Owens, is at ONA, so stop her in the hallway or find her on Twitter at @erika_owens if you’d like to discuss any of this in person. 

September 20, 2012 02:12 PM

Being immersed in a passion, an upcoming field, a new area of exploration is of course exhilarating. But living in a bubble of code, data and journalism can lead you to adopt certain assumptions and to fall into naive paradigms. This is often geo-located. I, like most in the West, tend to look to the West for innovation and progress. Even though I am not completely Western it is easy to get enveloped in your immediate online community and not those occupying the next node on the web.

This became apparent to me on my recent trip to Argentina for HacksHackers Buenos Aires. Not only is the journalism-coding community thriving in this bustling South American city but the hub of activity is something every newsroom developer and webmaker journalist in the West should be envious of. They organized a star-studded lineup of new media composers for talks and workshops. A three day media party with over 700 attendees. If that wasn’t impressive enough, they had live translators so that both Spanish and English speakers could attend.

I was included in the keynote and workshop run my the members of the Interactive Team. What I said was as follows:

Journalists used to work with information in print. Now we work with data in the digital age. Now “we” means not just the journalist with pen, camera and microphone but anyone with a phone and access to the internet. Now we all aggregate, curate and cultivate in the age of big data in the hope that not one person can dictate.

Now philanthropical bodies like Knight and campaigning foundations like Mozilla are enabling news organisations to openly embrace data driven investigative journalism by funding projects, training and education centres. Even giving headstrong idealists the opportunity to work with the new multi-skilled teams being fostered in newsrooms around the globe.

Because what newsrooms typically have no longer works. It does not work for collaboration, for visualisation, or for big data. To be digital, news organisations have to now move at the speed of web. And with the advantages of legacy come the disadvantages of rigidity.

So how do we engage with data? How do we move forward in our understanding of the typical news story? How do we pitch a story without a headline until the majority of the work has been done? How do we decide how to tell the story before we have it? How come it is already happening all over the world simultaneously?

How do we strive for data integrity when the structure we know is the sentence and the paragraph? How can we ensure accuracy when we only have one source, the data? How can we interrogate data on a scale that cannot be consumed by a human being? And how can we find stories in data whilst upholding the cornerstones of impartiality, accuracy and fairness?

I don’t know the answers but I do know that to do all of this we need tools. We all need to collectively and openly share ideas, data and code. We are no longer news-makers on the web, we are news-makers of the web. And I have had the great fortune of seeing The Guardian team and indeed the news industry tackle these challenges.

This went down well with the crowd but what I failed to communicate was how impressed I was with them. Before the media party kicked off I had the great fortune of meeting up with the data team at La Nacion. Not only were they present and active participants but they brought along an entire university class whose students are taught on campus and in the newsroom. What a brilliant idea!

Another inspiring concept incubated by the data team is that of the digital journalist notebook. The government of Argentina publish reports of spending, expenses, contract awards, etc in paper bulletins which are available online as PDFs. Each region has it’s own take on the general layout. The team scrape all these PDFs and have a search for the contents of the documents.

In this way they have made all government reports into a digital library where, using their journalistic hat, they can connect who is who and who gets what. Hard-hitting investigative stories have already emerged. With data “more open” in the West, newsrooms there shold be taking a leaf from their digital notebook.

September 17, 2012 02:54 PM

2012 has been a pretty incredible year for the Knight-Mozilla OpenNews project. We recieved 165 applications for our 2013 Fellowships (we were expecting around 80) and are now actively interviewing semifinalists. By the end of the year, we will have sponsored at least 20 hack days around the world. Our website, Source, a destination for information about the code being written in journalism, is almost out of development (and being updated regularly while still in dev). By every measure, it has been a hell of a year.

And today we’re adding something new to the OpenNews project: Code Sprint Grants.

What’s that?

Well, right now, the OpenNews projects helps create code that gets made in very short bursts at hack days and in very long strides by our Knight-Mozilla Fellows. These are both incredibly useful endeavors, but there’s a lot of ground between the two that is ripe for great code to be written. Code Sprint Grants are designed to fit there.

I’ve been thinking about the gulf between hack days and fellowships for a while now. Back in November I wrote:

There are a myriad of projects that need more attention than a hack day might provide, but that a year is overkill. … This kind of mid-range project—something that requires a dedicated time commitment of only a few weeks or months—how do we support that? Because I think that may be the lynchpin for some really vital making.

Code Sprints are designed to fit in that mid-range. We see Code Sprint Grants as funding small-scale tools and utilities that are focused on solving real needs of news organizations. By collaborating with news orgs to define problems and help move toward solutions, Code Sprint Grants are a way to get code written that helps to solve specific, repeatable—and real—journalistic problems.

Collaboration with more news organizations is another goal. The news partners that are vital to the Knight-Mozilla Fellowships are big, well-established news companies with global reach and impact. Code Sprint Grants allow us to forge relationships with many more news organizations of different sizes, approaches, and reach. By broadening the organizations we collaborate with, we hope to produce a diverse range of tools.

Code Sprint Grants are for $10,000 and are designed to get going quickly (our application for interested news organizations is just six questions long) and, like all of our programs, are optimized for flexibility (we want to help in ways that will be most useful to you and the code). When the code is ready for release it will be well documented, open-sourced, and available to anyone to fork, modify, and implement.

We’re excited to announce that we’re accepting applications for news organizations interested in collaborating on Code Sprints today. There is plenty more detail, and a link to the application, available on the Code Sprint Grants page on the OpenNews site. Hope to hear from many of you soon.

PS. I’m writing this from Buenos Aires, Argentina, the home of our 2013 news partner La Nacion, and was beyond thrilled to be able to first announce our Code Sprint Grants during today’s keynote at the amazing Hacks Hackers BA Media Party, which has brought 400+ developers and journalists from across South America (and around the world) to share and build together. The whole event has been an inspiration and I’m thrilled to have been involved in it.

August 31, 2012 02:00 PM

Tell me if you’ve been in this situation: you’re chatting about online anonymity with your wife and the other Knight-Mozilla Fellows over a pizza in Florence. A quiet-spoken stranger who had been sitting across the room walks up to your table and says “are you all here for the Tor hackathon?” You respond “why yes, yes we are!”

He goes on to explain that he is a journalist writing about Tor. He also tells us that he bets that the CIA and the Italian Secret Service are going to have moles there. What he obviously meant to say was “I work for the CIA and I’ve been watching you now for quite some time.”

It’s possible that he didn’t actually work for the CIA. His name and photo checked out under the website he claimed to write for. It was probably just a one-time job. Even if this isn’t true, even if a network of government spies didn’t track my position across Europe just to meet us in a restaurant, his comment set the tone for my weekend in Florence.

Tor is serious business.

What the hell is Tor?

Tor is a program that makes you anonymous. This means that, for better or for worse, the big brothers, neighborhood hackers, and ad agencies of the world can’t tell what you are doing on the Internet without going through a lot of effort and expense.

Is that too abstract? Here are some illustrative statements. *Taps the microphone*

• A Tor user walks into a bar, the bartender says “who are you?”
• How many Tor users does it take to screw in a light bulb? Only a few, but you’ll never know who did it.
• I used Tor last night and now my wife says that she doesn’t even know who I am any more.

I’ll be here all night.

If you use Tor you become Spartacus. Tor takes everything you do, makes it look exactly like what everyone else is doing, and gets random computers on their network to do the talking for you. Ta-da! Now it is practically impossible to pin an action on you.

The Original Need

I bet you wouldn’t have guessed that this idea was invented by The U.S. Navy. You would have? Oh.

Put on your paper sailor hat and I’ll explain. Imagine you are the king of the Navy and you’re going to war with your fleet of a thousand brand new Navy cars (I don’t really know how the Navy works). Being king, you are in the most important car of all because you’re calling the shots. You don’t want the enemy to know which vehicle is yours. You also don’t want them to know who is receiving orders because that could give away your tactics.

“I know,” you say, “I’ll encrypt everything so that they can’t see the content. Then they won’t be able to tell that my broadcasts are more important than others.”

Unfortunately for you, the enemy has fancy technology. They can’t decrypt messages but they are able to track where everything comes from and where it is going. They can’t tell what you’re saying, but they have all they need.

After about 5 minutes you think you’re doing well. Half of the enemy cars are already on fire! Yours explodes. “How did they do that?” you say in the afterlife. “Easy,” responds god, “they were able to see that your car was sending out the most messages. They knew exactly where you were.” Then he slaps you with a piece of linguini and drifts away.

To prevent this from ever happening again the Navy decided to invent the concept of an “Onion Network” (not to be confused with The Onion Network). Now instead of having packets go directly from point A to point B, each one randomly hops around the fleet first. Because of encryption, the enemy can’t tell the difference between a new message and a “hop” message — they all look the same. It’s like running an invisible sprinkler in a thunderstorm.

Suddenly nobody but the sender and the recipient can figure out the end points of a message chain. Even the middle men (the ones doing the hops) don’t know the path. Each piece of the hop — each “layer” of the message — is encrypted with a different key, so the only thing a relay knows is who gave them the package and where it should go next.

Onions have layers too, that’s why this setup is called an Onion Network. Get it? It’s like Shrek!

What’s it Good For?

Tor has applications in the real world. You can buy drugs and guns, share illegal pictures, and hire assassins. Oh wait, I’m just describing Tor’s reputation (more on that later). Seriously, there are a lot of important situations where people have moral and compelling reasons to want anonymity.

Here are a few:

• Protecting witnesses and victims of domestic abuse. Anyone who wants to be able to access the internet without being discovered by a third party can use Tor to defend against their stalkers.
• If you don’t like being tracked by your government, Internet Service Providers, or search engines.
• Providing truly anonymous tips. There are times when people need or want to share information against the wishes of powerful and potentially dangerous forces (e.g. mafias, governments, or corporations).
• Safely bypassing censorship. If you live in Syria, China, or The United States of RIAA/MPAA, you might use Tor to access content from the outside world more safely.

These kinds of reasons explain why organizations with very good reputations, like the Knight Foundation, are devoting resources to Tor.

The Dark Web

What I’ve just described is a spin on the way people access normal information online. If you point Tor Browser to Google you will see the same old Google, it’s just that now Google doesn’t know who you are. That’s powerful enough, but there’s more: Tor also lets you see hidden content on the Internet.

Using Tor is like entering a cheat code into real life and playing the lost levels. It is the digital equivalent of platform 9 and 3/4. This secret section of the Internet is possible because Tor users can serve content anonymously too.

If you don’t know much about how the Internet works, believe me when I say that if a web site’s location is hidden it becomes essentially impossible to access. It would be like trying to visit someone’s house without knowing anything about where they live — not even the country. Tor gives you a blindfold and leads you there. You still don’t know where the house is, but at least you can visit.

Anonymous sites are accessed through something called an “onion address,” which is made up of a series of random letters and numbers. For instance, this is a “clean” version of Tor’s wikipedia: 3suaolltfj2xjksb.onion. Feel free to try clicking the link, it won’t work (Unless, of course, you are using the Tor browser).

Note: even if that link worked you wouldn’t see any terrible images. However, you need to use your brain before you start actually clicking around if you don’t want to get really upset.

That random looking string is used to find the server within the Tor network. Because the addresses don’t point to a real address on the Internet, there is no way to fully access this content without Tor. There are services you can use to get there without using Tor, but you lose all benefits of anonymity and content is often censored.

Onion addresses are the most fascinating part of Tor, albeit the most potentially disturbing. Rest assured that they don’t all lead to child porn, guns, and drugs. For example there is a secret version of Twitter, a bunch of blogs, a search engine, and an email service. There is even a secret version of 4chan (called Torchan), which I won’t link to because that one does lead to child porn and drugs.

These types of content networks—ones that are served on top of the normal web so that you need special programs to reach them—are known as the Dark Web. Not necessarily because the content is darker (it is), but because it is hidden from view and can’t really be searched and scraped as reliably.

Implications of The Dark Web

Most uses for Tor become more potent with onion addresses. Anonymous servers are just as protected from higher powers as anonymous users. If Amazon suddenly started selling illegal drugs they would get in trouble. If a Tor marketplace started selling illegal drugs, the law would have to figure out a way to find them first.

This power applies to legitimate uses as well. If a government official wanted to contact The Boston Globe with a corruption leak, he or she could use Tor to create a gmail account anonymously. The government could then subpoena Google, and Google might be willing to give away the information they have. They won’t know much, but now things like account access patterns and full email logs would be fair game.

If the official had used Tormail then even Google wouldn’t know what happened. The government would have no course of action because there would be no service provider to ask. Every journalist in the world should be able to agree that there is no good reason for a watchdog to trust the organizations they are watching. Why should you trust in corporations and governments to keep sources safe?

Tor has a reputation because it has a lot of criminal content, but the social good that it supports is just so important (criminals will always be criminals). I’m working on a game called Torwolf to simulate a few situations where Tor would be effective (if you have played Werewolf or Mafia, you can start to imagine what the game will be like). In the mean time, read up on Tor if you’re curious. Better yet, go try it out.

IMPORTANT EDIT: while Tor is much better than nothing, it is neither foolproof nor perfect. If maintaining anonymity could be a matter of life / death / imprisonment, then you need to know more than what I could fit into the scope of this overview. This paper is a good starting point, but seriously, spend some time researching on your own.

August 17, 2012 02:15 PM

This is a short follow up post to my previous article : New Ways to Consume Video in which I touch on the importance of the shared experience.

Olympics Schmolympics

I was fascinated not so much by the Olympics itself, but by how much fun people were having sharing the experience. During the live broadcast of the opening ceremony my tweet-stream took on a life of its own - providing more entertainment to me than the event itself and judging by the reactions of others, I wasn’t alone. The problem was that for many not watching the event - this frenzied Twitter activity was — I presume — a little bit annoying.

Lunatics and Asylum

Yesterday I got caught up with the events outside the Ecuadorean embassy where Julian Assange was currently holed-up and awaiting the decision from Quito on his asylum status. I have to admit to being somewhat distracted for an hour or two as I watched events unfurl live on the occupynewsnetwork channel.

A great addition to the experience was the comments coming in live, reactions and additional perspectives of people watching the same thing at the same time! I say great but there was a lot of crazy noise among the signal, but it still added to the experience more than it took away. It was also interesting to clock the ever-fluctuating number of users watching the stream.

The Second Screen is Already Here

So excitingly the potential seems to be there to create very compelling and engaging experience, sure it needs a little refinement and I’m confident changes are being considered to allow you to mute tweets with certain hashtags in Twitter and mute noisy people on UStream’s open chat. Actually UStream includes Twitter comments hashtagged appropriately. Although it remains to be seen how long that will continue to be possible.

For all intents and purposes we are already there. Twitter provides us with a nice decoupled way of using the second-screen but perhaps something like app.net is more suited. Crucially it appears to allow you to take full advantage of its API with few restrictions and subtly but perhaps importantly allows you slightly more characters. Note: app.net is a paid-for service.

We’re Only Human

There’s something about being human that means we love to share certain experiences. Cinemas and live music events are still very popular and I think that if we provide an opportunity for people to watch something together, they will, especially if you allow them to interact with each other. Good odds then that what Apple will do if they get the rights to stream cable TV content through their Apple TV is try and implement some live commenting system.

I’m wondering how we can activate peoples mics upon hearing laughter or applause and add that to sound, but that path is a little fraught.

Mark Boas

August 17, 2012 11:24 AM

I do the hard work so you don’t have to! In fact, I need to do it for me. If I am to find my scripts and organize them, I need to structure my project at the start and keep to a standard layout for all my projects.

So I’ve made a GitHub repository called skel (for skeleton) for all data driven journalism projects. Whenever I have a new project, I’ll make a zipped copy of the skel folder without git (see pdf instructions below), giving it the project name and build up my scripts and data in the various folders.

skel contains the vagrant I talked about in my last post so it has iPython, BeautifulSoup, MySQL and Elasticsearch. With every project iteration I will be adding to the vargrant (and possibly repackaging it) and refactoring my skel repository. So if you a journalist interested in data projects, get on GitHub, clone the repository and watch it for any updates. Below is my very simple guide to using GitHub for non-developers:

How to Use GitHub

August 16, 2012 03:23 PM

Over the past few months with my mentors at ZEIT ONLINE I have been doing a lot of thinking about how to better monitor news organizations performance in the social media space. To successfully monitor how stories propagate into the social world you are going to need access to a few things:

1. A feed of the home page ( or whatever list of articles/urls you want to monitor ).
2. A machine to collect and store what you find.
3. A way to find the metrics from the social media world.

The home page is a good source because it is always changing so it will provide you with a constant list of urls to collect that the news organization feels are important enough to show on their home page. This is usually provided in some sort of xml ( via rss ) or json format that is constantly updated and publicly available for you to access at any time. In other words you need a harvester to constantly pick the stories from the top of the pile.

Once you have access to those resources you can start to aggregate them together to create an access point so that yourself and others can use to make sense of the data. Below is a small diagram to explain visually exactly what I just mentioned followed by an explanation of exactly how I am harvesting ZEIT ONLINE.

 

Starting off in the top left, Amo provides me with all the share values for a given url. It is constantly being used to provide feedback to a story’s share counts. Google+ did not have an API for getting the number of +1s a url has received so I had to create one myself. Facebook returns their share data in XML. None of the services from the social world was available in JSON except for the Twitter API. Without Amo I would have to constantly poll Twitter, Facebook and Google separately and then merge them all together inside of my app code. Amo is just an abstraction of social share API’s all wrapped up into one nice “likeable” JSON object.

The next piece you see is the “Harvesting Layer”. It feeds the database everything that it needs to serve up the “API and Caching” layer. The following is how I update the database from the constantly polling feed of ZEIT ONLINE articles.

1. Grab a list of the articles from the home page and compare them to the recent articles I have collected. In my case grabbing the ZEIT home page was easy. They use xslt to drive their whole site, in other words their whole site is one open API for me to consume. For example, go to the ZEIT home page. Once you are there replace the “www” with “xml” ( xml.zeit.de/index ). What you will find is a massive XML structure that represents all of the data and meta data behind the page you see their. It is pretty much a playground for developers. If you are using node make sure to check out xml-simple, it is the best xml parser i know of. Inside the centerpage you will find the feed. If you want to harvest another source you will have to become familiar with their output structure in order to properly break it down in your app. You really only need three key ingredients: publish date, title and url.
2. Strip out any illegal characters that JSON won’t parse ( this can happen when dealing with non-english content, for example ). Make sure you set the encoding properly to deal with umlauts. Put the new article inside of the database with Mikeals’ request. Because I used CouchDB, I needed a unique identifier. I used a nice little node utility to create a UUID for me for every url. At this point you are done with the URL harvesting.
3. Set up a worker to handle all of the updating of the share objects. For this I created a tool to constantly poll the database to check for articles. Once I have a list of the articles I want to track I send them off to Amo to collect the share information. Once I have the share information I put everything back into the database and create a new revision. Another great reason to use CouchDB is for how easy it is to go back in time and review all of the past revisions. This allows me to really track a stories growth and see when it is on the rise.

At this point you will have a constant feed of articles being monitored and collected as well as their share counts being updated at the same time. The database is filling up and the next step is to set up a way to access this from the outside. This is where the API layer comes in. You are going to want to sanitize all of the data you collected in a way that any developer in the future who wants to use your stuff can understand. Creating an API means defining a layer of urls to your data that will remain consistent and will hide the complexity of a system like this for future developers who rely on your data.  I really like express because it is easy to create routes and from those routes you can return whatever you like. In my case the database returns me a filtered result set based on publish date. Luckily with couch this is really easy. I just added a stream proxy to my express route and passed the data parameters requested to the database and boom anyone in the ZEIT network now is able to filter ZEIT ONLINE articles by date.

 

August 14, 2012 11:35 AM

During my OpenNews Fellowship at The Guardian I am learning some very valuable lessons. Just as a trainee reporter learns the ropes, the beat and how to organise oneself for the steady flow of the news day so the budding data journalist needs to learn the same; only in terms of bits and bytes.

So what am I talking about? Frankly, I’m talking about the other day when I realised my Python setup tools were completely mangled. This is the danger of delving into the realm of code. You break the one thing you are reliant upon. This is what I refer to as a CARcrash. So how do you practice good driving?

First thing’s first: never deadline and drive. That is, never code frantically to a deadline. When you need to munge some data in a hurry it is very tempting to look for a new set of tools which promises to do it all in a few lines, install it and all its dependancies. Doing that leads to trouble; a computer that jumbles software versions and won’t install them correctly.

So how can you avoid a CARcrash? The answer is virtualization. That is, create a virtual machine in your machine that has all the software you need for the project. Run your code on the virtual machine. Export the data and kill the machine when your project is done. Sounds scary? Well it’s actually quite easy.

Download VirtualBox to create a virtual box. These are not easy to use so you want to get yourself a Vagrant. For the budding data journalist, I have a prepared vagrant for you (kindly made by a talented developer at The Guardian who swears by them), so all you need to do is download and install VirtualBox and Vagrant. Download this zip file, unzip it and move the contents into the top of the directory you have stored the files you want to work with (the vagrantfile has to be at the top of the tree as anything branching from the directory’s path will be made available to the vagrant on your virtual box).

This vagrant is an Ubuntu box with iPython, Elasticsearch, MySQL, and BeautifulSoup already installed. If you haven’t used any of those stay tuned and I should have some examples for you. Every time you want to use these, you put the contents of the zip folder at the top of the directory you are using, navigate into it in you terminal and use these simple commands:

 > vagrant up # creates your vagrant for the first time, you only need to do this once
> vagrant ssh # logs you on to your vagrant where you can run your files using MySQL, Elasticsearch, etc
> ls # lists the files on your vagrant, this will include all the files in the directory you are working in
> exit # logs you off your vagrant
> vagrant destroy # once you have run your programmes and copied all the files created onto your directory you can destroy your vagrant and free up space

I will be using this virtual box for all of my tutorials from now on (and teaching but more of that to come) so stay tuned and follow along! The content of the box may change. I’ll let you know if this happens.

August 13, 2012 03:24 PM

When I was a kid, I was obsessed with the “Choose Your Own Adventure” series of books. If you’re familiar with them, you’re already nodding your head enthusiastically, if you’re not, it’s pretty simple to explain:

The books were written non-linearly, had multiple endings, and every few pages the reader was faced with a choice like this one, from “The Mystery of Chimney Rock”:

If you run into the tomb to escape, turn to page 117.
If you scream for help, turn to page 118.

As a child, when your options in life are very controlled, the choices posed in the book’s pages were thrilling. Every decision, a new adventure.

As an adult, we certainly have more choices, but most of them lack the true adventure that lay inside the pages of those books. Except today.

Today, if you’re a developer, a hacker, a data geek, an open-web nerd, a technologist, a maker, or someone who defies easy categorization but builds amazing things on the web, you have exactly that kind of choice in front of you. Because today marks the final chance to apply to become a 2013 Knight-Mozilla Fellow. Saturday night August 11, at midnight Eastern Time, that chance ends.

As the current Knight-Mozilla Fellows explained last week, every day as a Fellow is an adventure. You are inside some of the best newsrooms on the planet when world-changing news breaks. You fly across oceans to hack new ideas with brilliant colleagues. You spend 10 months experimenting and creating open-source solutions to all sorts of problems. You make life-changing connections. You become a thought leader in the global conversations around the future of news.

You will have 10 months of adventures. But you have to make a choice right now, and that choice is to apply.

As I outlined earlier this week, by becoming a Knight-Mozilla Fellow, your adventure is covered by an incredible compensation package that doesn’t just give a stipend but helps with healthcare, childcare, housing, and even covers some equipment and all your travel.

You will have 10 months of opportunity. But you have to make a choice right now, and that choice is to apply.

As our news partners have been saying for the last few weeks, the opportunities to have a positive impact on journalism by bringing new ideas, new perspectives, and new solutions into the mix is massive. You may be embedded in New York City with the New York Times or ProPublica, in London with the BBC or the Guardian, in Germany at Zeit Online or Spiegel Online, in Boston with the Boston Globe, or in Buenos Aires with La Nacion. Wherever you land as a Knight-Mozilla Fellow, you’ll be working closely with some of the best journalism organizations in the world. But you’ll also stay connected to your fellowship cohort, building things together and making lasting bonds as you all work together to prepare journalism for a new era.

You will spend 10 months helping to shape the future of information. But you have to make a choice right now, and that choice is to apply.

So there it is: the choice is yours. You have until Saturday August 11th at midnight Eastern Time to make that choice.

It’s up to you to choose your own adventure:

If you decide to spend 10 incredible months traveling the world and hacking the news, turn to page 117.
If you scream for help, turn to page 118.

August 10, 2012 04:49 PM

This is it: We’re in the final countdown to apply to become a 2013 Knight-Mozilla fellow.

Our current Knight-Mozilla Fellows spent last week talking about how many incredible trips they’ve taken, how many people they’ve met, how many cool things they’ve built, and how much fun they’ve had. And some of our 2013 News Partners have explained why they’re involved in the program and what they’re hoping to build with their Fellows in 2013.

But today, I want to talk about something less exciting, but crucial: The Knight-Mozilla Fellowship benefits package. Your 10 months as a Knight-Mozilla Fellow offers good compensation as well as a comprehensive collection of supplimental benefits that range from help with housing costs to money for equipment and research purchases.

The full host of suppliments are detailed on our benefits page, but I wanted to outline some of them here as well:

• Housing and Moving: If you’re moving across the globe for a Fellowship, know that we have your back: We offer up to $4000 to help you move to the country your newsroom host is in, and suppliments to help cover your housing (ranging from $500 a month for a single fellow to $1000 a month for a fellow with three or more children).
• Healthcare: We don’t want you to get sick during your time as a Fellow, but if you do, we help to cover your health insurance costs, from $3500 for single fellows to $7000 for a family.
• Childcare: I’ve mentioned children and families in the last two items, and that’s because we believe that a Fellowship can be an amazing experience for everyone, and want to help defer the childcare costs that may arise, offering suppliments that range from $5600 to $7200, with additional suppliemnts for infants.
• Equipment: We want our Fellows to have up-to-date technology, and so we offer $3000 for equipment purchases (it can also be used for research costs).
• Travel: Starting with the Mozilla Festival in London, where we’ll announce the 2013 Fellows, we cover all costs for our fellows to travel to conferences, hack events, and opportunities to collaborate together. We even help you book your travel!

Add to all of these suppliments the $60,000 stipend each Fellow recieves for their 10 month Fellowship, and you’ll see that the Knight-Mozilla Fellowships package—like the experience itself—is pretty amazing.

There are only three days to apply to join the program in 2013. The application is quick, but if you wait just a few days you’ll miss out. So apply now.

August 08, 2012 04:25 PM

In seven short days your chance to apply to become a 2013 Knight-Mozilla Fellow will be over. Because at midnight on Saturday August 11th, the application window closes.

We are looking for the best developers, hackers, data geeks, engineers, analytics nerds, math whizzes, and technologists around the world to spend 10 incredible months hacking and experimenting in eight of the best newsrooms on the planet. You’ll also travel the globe attending conferences and hack days, to collaborate with some of the brightest minds out there, and to have an incredible amount of fun.

Don’t take my word for it. Ask our current five Knight-Mozilla Fellows, or ask some of the newsrooms that will be your host. The opportunity is incredible—don’t miss out.

The application is super short—just 450 words and some links. We want to make it easy for you to apply, so go do it—because in one week the opportunity will be over.

August 05, 2012 01:57 PM

August 03, 2012 08:36 PM

Mark Boas, Cole Gillespie, Nicola Hughes, Dan Schultz, and Laurian Gridinoc on the deck of the MIT Media Lab, June 2012

This week, each of the 2012 Knight-Mozilla Fellows told stories of what they’ve been up to during their time as Fellows. Each story captures both the unique experiences of each Fellow, but also captures their singular personality. And each story is a captivating reason for why you, with just a week to apply, should join their ranks as a 2013 Knight-Mozilla Fellow.

For Mark Boas, who has been working with Al Jazeera English, he writes that his time as a Knight-Mozilla Fellow has meant getting his work in front of new audiences and for leaning the discipline that comes with deadline-based development:

There is great opportunity to innovate and see your experiments incarnate on websites that get very many eye-balls and of course get all that lovely feedback. And when I say lovely I don’t mean complimentary I just mean that all feedback is lovely even when it is negative and the more you get – the better. In fact, I think one of the most important things you can do when publishing to a site like AlJazeera.com is measure the usage in as much detail as possible. Certainly for me it’s not often that I will be able to collect so many stats on things that I have had a hand in making.

The unpredictable and somewhat transient nature of current affairs also presents tremendous opportunities. One of the projects I’m working on is an interactive slide-show that displays a series of slowly zoomed images to a YouTube soundtrack. I had just got a rough proof of concept together when my colleague mentioned they had some fresh photos and an audio soundtrack from Syria and that they wanted to create an audio-slide show from it to go live the next day. Frantic hacking of code and content ensued but we got it out in time. I wrote in my last post that situations like these are an opportunity to hone your shipping skills and a good exercise in delivering the minimum viable product.

Nicola Hughes, who has been embedded with the Guardian’s Interactive News team, writes of the boundaries she’s pushed and the distance she’s come as a Knight-Mozilla Fellow:

So what have I got to say? A young woman of colour, trained in broadcast journalism, who had never used the command line until this year. From the very beginning I felt I had the least to offer the OpenNews programme. I never thought I would get it. I was enticed to participate by the various rounds in the competition. As a fledgling programmer, I loved hackdays. Being able to connect with those at the edge of digital journalism and those interested in the field was reward enough.

But I did win and here I am. So what have I done? I have advanced my skills beyond what I could have done on my own. I am more comfortable with the strategies of data digging and programming. I know what skills I want to add. But most of all I know I should be here and I deserve to be here. Not as Nicola Hughes or DataMinerUK but as an OpenNews fellow. And by ‘here’ I don’t mean The Guardian or the OpenNews programme. ‘Here’ is web-making, data-digging and story-building in the open.

A big part of this resolution to create, innovate and take news beyond the written word is my fellow fellows. I feel truly blessed to know such creative, talented and forward-thinking individuals. This has been a big benefit to me and one I will take beyond the fellowship.

Cole Gillespie, who moved to Berlin from North Carolina to be a fellow at Zeit Online, punctuates his reflection with photos from “the best year of my life,” as he writes:

A question we are all often asked when we meet new people is “What do you do?”. I used to find that question annoying unless I was talking to other technical people because it meant that I had explain to them the details of a highly technical field in order for them to get it. While I love telling people about hardware virtualization and all of the details of the work I did at IBM it seems that most people get lost in that conversation and immediately switch the topic. Now I revel in such opportunity to explain what exactly it is that I spend time doing on the day to day hack with Mozilla and Zeit Online. It gives me a chance to explain how exciting working in an open way for the news room can be. Of course it comes with a unique set of challenges just like any software situation these days but Zeit has done an great job at making it easy for a developer to get access to all of the proper tools necessary to get the job done.

For Laurian Gridinoc, who has been working with the BBC to break Flash’s stranglehold on their news interactives, it was working (and playing) with the other Fellows that he’ll remember the most:

Working with the other fellows was the most rewarding experience of this fellowship. While day-to-day work at BBC was challenging, it was limited in the aspect that it was serious work and not experimental bat-shit crazy stuff that may work only on one browser. With my fellow fellows, we played with arduino, scrapers, speech recognition and video transcripts, natural language processing, seriously.js, processing.js and all the other *.js cool toys at the hack days we attended together.

Finally, Dan Schultz, who has only been a Knight-Mozilla Fellow at the Boston Globe for two months, reflects on the unique position a Knight-Mozilla Fellow finds themselves in:

We are trying to publicly understand, question, observe, and create in the context of news. There are so many chances to do all four of those things. Not a day has gone by where I haven’t been exposed to something new — a new idea, a new problem, or a new opportunity.

You are being thrown into an organization that may have a vision for you to work with, or may expect you to invent a vision of your own from scratch. Either way your time is going to be your own and you will be expected to make great use of it. This kind of freedom is difficult to cope with, especially when people have high hopes for you. People will throw you questions to ponder, ideas to critique, and problems to solve and you will need to prove yourself.

In return you get to ask anyone anything. You will get to bend the rules and do things that other people around you might have to fight hard to accomplish. If you are interested in something, you will be able to work on it. If you have a question or concern you will be able to get an audience with the CTO or the chief editor. Nobody else at your organization has your title.

These five people had the extraordinary skills to become 2012 Knight-Mozilla Fellows. In 2013, we expand that opportunity to eight and expand our host news partners to include the New York Times, ProPublica, La Nacion in Argentina, and Spiegel Online in Germany.

This is an opportunity to see the world, to hack the news and to have the time of your life. but to become a 2013 Knight-Mozilla Fellow you need to apply: the application window closes on August 11th—just one week. Do not hesitate: Apply to become a 2013 Knight-Mozilla Fellow today.

August 03, 2012 05:27 PM

Uh oh… I need to write a post about the Knight-Mozilla Fellowship and Chris Marstall already masterfully captured all my points last week! Thank goodness there are still a million reasons why OpenNews is awesome.

By now you may have read four different accounts of my program, but if you make it through this one you will be rewarded with Internet gold. My backstory is fast. I started this process later than the others because I had to graduate first. I’m only two months in, which means I’m about 20% complete.

In that short amount of time I have already:

• Blown my tech stipend on the new hotness and become the envy of everyone around me.
• Moved to an entirely different state and bought a kitten.
• Toured Europe for three weeks with my wife (and it wasn’t considered vacation).
• Been monitored by the CIA at the Tor Hackathon in Florence.
• Broken the law in Berlin by infiltrating an old abandoned spy building under the leadership of Cole Gillespie.
• Watched In Bruges in Bruges.
• Raided The Guardian, The BBC, and Mozilla’s London HQ.
• Drank a Tactical Nuclear Penguin.
• Worked with the other fellows to dream up complete iterations on two home-brewed projects (Torwolf and Newsquest).
• Dreamed with my peers and started moving on dozens of ideas including Raspberry Pi crisis cams, time-lapse Internet radio, and magical data extraction APIs.
• Contributed to the infrastructure that supports The Boston Globe.
• Challenged and questioned policies and processes at The Globe.
• Learned first hand about the politics and hurdles within news organizations (I call this “the real world.”)
• Regularly met with VIPs at The Globe, The Guardian, The MIT Media Lab, and The New York Times, to absorb their insights about the state of journalism.
• Maintained strong ties with the people of my past by arranging a formal ongoing relationship with the Media Lab.
• Not been sued or fired yet.

You’ll notice that this list contains a mixture of productivity, fun, and life. This is because fellowships are not just paychecks, they are about personal growth as much as personal output. This one is no different, as shown by the fact that I’ve spent almost half of my time traveling around the world.

If you are saying to yourself “holy crap that’s sweet” you are absolutely correct. In fact, one point of this post is to help you understand why The Knight-Mozilla Fellowship is one of the most rewarding jobs on earth right now and that you should apply this week.

There are some things you should expect if you make it in.

1) You will understand why the news industry is struggling to survive, and why there is hope

I decided not to use this post to talk about my observations and insights about journalism. I already write about that subject on this blog. This item still gets top billing because it defines our mission. We are trying to publicly understand, question, observe, and create in the context of news. There are so many chances to do all four of those things. Not a day has gone by where I haven’t been exposed to something new — a new idea, a new problem, or a new opportunity.

2) You will become unemployable (in a good way)

This now makes two positions in a row where I’ve heard my colleagues say that our work has made us unemployable. This isn’t because employers won’t want to hire us, it’s that we are being spoiled. We’re getting so used to creative freedom, security, and special treatment that simply doesn’t usually come with a traditional job. It’s a good problem to have.

3) You will be challenged, and you will be special

You are being thrown into an organization that may have a vision for you to work with, or may expect you to invent a vision of your own from scratch. Either way your time is going to be your own and you will be expected to make great use of it. This kind of freedom is difficult to cope with, especially when people have high hopes for you. People will throw you questions to ponder, ideas to critique, and problems to solve and you will need to prove yourself.

In return you get to ask anyone anything. You will get to bend the rules and do things that other people around you might have to fight hard to accomplish. If you are interested in something, you will be able to work on it. If you have a question or concern you will be able to get an audience with the CTO or the chief editor. Nobody else at your organization has your title.

4) You will make friendships that last the rest of your life

It’s a cliche, but it’s for real: one of the most rewarding parts of this is the people. You won’t just be part of a community, you will be part of creating a community. It started last year during a week long hackathon in Berlin, where I met dozens of people who I still see all the time. That sense of comradery continues to dominate this experience, and of course it also includes the people in your newsroom.

For the next round of fellows there won’t be a Berlin event, but you will still get to be a part of an 8 person group for almost a year in addition to becoming immersed in a young community of passionate people. You will drink whiskey in foreign countries with friends, share stories of trials and tribulations in workplaces half a world away from you, and find yourself in areas you never would have entered alone.

And with that I think I promised you a reward… I present to you the Amazing Spinning Gridinoc!

ALSO: Please come talk to the folks in the OpenNews community by signing into on our chat room below. Just come in and say hello!

August 03, 2012 04:00 PM

In two months, my Knight-Mozilla OpenNews fellowship at BBC News Specials will end and everybody wonders what I’ll do next.

August 02, 2012 11:30 PM

The Guardian Interactive team talks about their experience with the Knight-Mozilla Fellowship program and makes their case for why you should apply for the 2013 Fellowship—deadline is August 11th!

August 01, 2012 04:43 PM

I am not Asian nor Caucasian although I have both in me. I am not religious nor agnostic although I have lived in countries of many different faiths. I have studied across the science and art disciplines although I practice neither. I am neither journalist nor developer. I am female although I work in male-dominated areas. I have no class, race or creed. I am, however, an OpenNews Fellow.

I have for a long time been DataMinerUK. Standing alone, trying to learn the ways of the web and mine data for stories. I have preached by my stumbles and failures, and an unwavering sense that doing something different differently is the way forward in journalism. Now I am not alone and stubbornly grappling in the dark world of object oriented programming. Now I am a fellow fellow

.

This week you will hear from all the fellows, Mark, Cole, Laurian and Dan. So what have I got to say? A young woman of colour, trained in broadcast journalism, who had never used the command line until this year. From the very beginning I felt I had the least to offer the OpenNews programme. I never thought I would get it. I was enticed to participate by the various rounds in the competition. As a fledgling programmer, I loved hackdays. Being able to connect with those at the edge of digital journalism and those interested in the field was reward enough.

But I did win and here I am. So what have I done? I have advanced my skills beyond what I could have done on my own. I am more comfortable with the strategies of data digging and programming. I know what skills I want to add. But most of all I know I should be here and I deserve to be here. Not as Nicola Hughes or DataMinerUK but as an OpenNews fellow. And by ‘here’ I don’t mean The Guardian or the OpenNews programme. ‘Here’ is web-making, data-digging and story-building in the open.

A big part of this resolution to create, innovate and take news beyond the written word is my fellow fellows. I feel truly blessed to know such creative, talented and forward-thinking individuals. This has been a big benefit to me and one I will take beyond the fellowship.

Each fellowship will be as unique as each fellow. What will be in store for next year’s fellows is very much dependent on the news partner and the background of the fellow. As with most things, it will be what you make of it. But one thing you can depend on is meeting with like-minded individuals who want to take news further into the digital age.

So this goes out to all the doubters, all the people who don’t think they have what it takes or the right fit to apply. This goes out to all the ladies, all the new adopters of code. All the journalists, freelancers and recent graduates. The Mozilla open-web ethos is not just for hard-core developers. It’s for all of us who use the web, who could, if we so choose, be makers of the web.

It’s the best thing that has ever happened to me, so if you’re in any doubt – Go for it!

 

July 31, 2012 10:47 AM

maboa: News and Technology - An Intoxicating Mix:

maboa:

So I’m trying to work it out in my head : How many months have I been a Knight-Mozilla OpenNews fellow and how many months have I got left? It’s a ten month gig and I started in February … and damn I’m over half-way already! Oh well, I guess it’s one of those glass-half-empty situations…

July 30, 2012 08:51 PM

So I’m trying to work it out in my head : How many months have I been a Knight-Mozilla OpenNews fellow and how many months have I got left? It’s a ten month gig and I started in February … and damn I’m over half-way already! Oh well, I guess it’s one of those glass-half-empty situations which I need to view as nearly half-full but it’s so hard when the contents are so deliciously intoxicating.

I’m talking about that cocktail of news and technology. It’s a mix alright and although I enjoyed both separately, it wasn’t until I started mixing that I realised how amazing it was.

I have various people to thank for getting me started on what I feel already will be a life long addiction. I’m not going to name them all but let’s just say that both the Knight Foundation and the Mozilla Foundation are two amazing organisations to work for and I feel privileged.

So what’s so special about the intersection of news and technology, specifically in my case on-line news and web-technology? What’s the big attraction?

The People

I’ve been lucky enough to be assigned to Al Jazeera English – but I assume this is broadly true of any of the news organisations taking part in the OpenNews initiative – and in my experience journalists are very interesting people. Their mission is to tell stories and they have some very interesting stories, not all of which have been published. Get to know them though and they will weave you a good yarn. One of my Al Jazeera colleagues told me how he met his future wife on Twitter in Palestine trying to figure out where close-by shells he heard were landing. It’s fascinating to talk to people that know so much about what is actually going on the world. It’s an education. Really.

But it doesn’t stop there. I love this idea of a fellowship, you’re not alone on this journey, there are other fellows doing the same sort of thing as you. Fellows seem more than just colleagues – they are co-conspirators and already we’ve become very close. Recently my fellow fellows visited me in Florence – well actually they came to attend a couple of hack-days on the Tor project. They ended up AirBnBing a cool flat right next to Piazza del Duomo and this became the centre of operations for a few days. We hacked on code, we talked (a lot), we ate together. Later on in the week I invited them to my home town and they met my family and told stories to my kids. It was all very relaxed and very very cool. In the end I feel I will end up learning as much from them as the news organisations. 

The Opportunities

There is great opportunity to innovate and see your experiments incarnate on websites that get very many eye-balls and of course get all that lovely feedback. And when I say lovely I don’t mean complimentary I just mean that all feedback is lovely even when it is negative and the more you get – the better. In fact, I think one of the most important things you can do when publishing to a site like AlJazeera.com is measure the usage in as much detail as possible. Certainly for me it’s not often that I will be able to collect so many stats on things that I have had a hand in making.

The unpredictable and somewhat transient nature of current affairs also presents tremendous opportunities. One of the projects I’m working on is an interactive slide-show that displays a series of slowly zoomed images to a YouTube soundtrack. I had just got a rough proof of concept together when my colleague mentioned they had some fresh photos and an audio soundtrack from Syria and that they wanted to create an audio-slide show from it to go live the next day. Frantic hacking of code and content ensued but we got it out in time. I wrote in my last post that situations like these are an opportunity to hone your shipping skills and a good exercise in delivering the minimum viable product.

I hadn’t anticipated that we would be encouraged to travel so much and take part in so many different events. On every single trip I met amazing people, people that were not only technologists, journalists — or some hybrid of the two — but also filmmakers, writers and artists. Believe me when I tell you that attending a technology conference after one of these events feels distinctly one dimensional. Technology to me is something that works best when applied and the more I mix with people outside my technological comfort-zone the more applications I discover.

A Unique Position

Being an OpenNews fellow means you are free to experiment and disrupt and generally cause a little controversy without fear of too much reprisal. It’s a temporary job and you’re being sponsored - what’s the worst that could happen? Actually I think newsrooms need and welcome a bit of disruptive thinking.

I feel I have the freedom to suggest or criticise anything – which is a beautiful freedom to have. More recently I’ve also found myself becoming something of an honest broker as I try and foster collaboration between Al Jazeera and third-parties, be it the Internet Archive or Universal Subtitles (now Amara). I didn’t expect to play matchmaker but it’s a role I thoroughly enjoy.

The biggest coup that I think all the fellows are working towards is to get all the OpenNews news partners working together in some mutually beneficial way. Honestly, I think we have a very good shot at it. We’re trusted by our newsrooms to be given a company email address, they evidently trust us to do what is in their best interests.

The Big Question

So, open, what is open in this context? The big question : What does ‘Open News’ mean – or at least what do I think it means?

I don’t remember ever being told and although we’ve touched upon it in conversation we’ve never really discussed it in great detail, although we may have done one of the times Laurian brought a bottle of whisky to a meet-up. It’s almost as if this is something we were meant to discover along the way and I suspect we all have slightly different interpretations of it. To me it’s many things – yes it’s bringing open source and the open source mentality into the newsroom, it’s also using and sharing open data, it’s opening up discussion, it’s also open license-free content, it’s working in the open, it’s even about being open about what you are doing but more, for me, it’s about opening up journalistic channels to everyone.

Living in Italy I am all too aware of the power that the media can hold and how that power can be abused when wielded by too few people, conversely I followed Al Jazeera and how they opened up and integrated the new channels of social communication during the Arab Spring. I’ve seen the results of both approaches and I’m not saying things are black and white here, but I’m definitely leaning towards the latter.

So here I am, trying to concentrate on the fullness of the glass, but reflecting on the bit that is already in me. What I love about all this, what I really love, is that as part of this whole process you are expected to take the initiative, find your own way, that’s some responsibility, yet as part of OpenNews and the fellowship and the people - above all the people – you know you will always find friendship and support along the way.

The good news — for other people — is that these are rotating positions, which means that if you’re a developer, technologist, engineer or programmer you can apply to be an OpenNews fellow too.

July 30, 2012 08:44 PM

Six months into being a Knight-Mozilla Fellow, in what has proven to be the best year of my life, 2012 continues to get better in the ZEIT ONLINE news room in Berlin. Over a year of planning leads up to today. We moved into a beautiful brand new office next to Anhalter Bahnof. The dust is still settling as the movers put the last pieces of furniture in place but I can already start to feel the energy of the news room buzzing throughout the place.

A question we are all often asked when we meet new people is “What do you do?”. I used to find that question annoying unless I was talking to other technical people because it meant that I had explain to them the details of a highly technical field in order for them to get it. While I love telling people about hardware virtualization and all of the details of the work I did at IBM it seems that most people get lost in that conversation and immediately switch the topic. Now I revel in such opportunity to explain what exactly it is that I spend time doing on the day to day hack with Mozilla and ZEIT ONLINE. It gives me a chance to explain how exciting working in an open way for the news room can be. Of course it comes with a unique set of challenges just like any software situation these days but ZEIT has done an great job at making it easy for a developer to get access to all of the proper tools necessary to get the job done.

On the other hand, Berlin does a really great job setting a creative vibe. With graffiti all over the city, endless open airs and a thriving music scene to explore it is almost as though the whole city is one big canvas for anything that has to do with the arts. This city is still being rebuilt and is starting to bud into one of the most creative cities in the world. Still being relatively cheap it is starting to attract a diverse set of makers and artists as well as a diverse group of start ups. Working as a fellow at ZEIT gives me a chance to be right in the middle of a transforming city. Mozilla will also be moving to the kiez so as a fellow you will have a space to show everyone all the awesome you are making!

Being a fellow gives you unique access to a very great resource, the other news fellows. One of the best parts about being an open news fellow is thinking outside of the box with your counterparts and getting to spend time with them and random cities all over the world. The first group of news fellows at this point has become one big family and when we have issues with things we are building we look to one another for advice. I find that to be a priceless aspect of being an open news fellow. The comradery formed between us is very unique given our situation. We are all place in some of the worlds most renowed news rooms acting almost as spies, sharing the info we find with one another in order to improve not only ourselves but the curent news model.

If there is one down side to being an open news fellow it would be the fact that you have one of the coolest jobs in the world and after your 10 months of amazing are over you are left wondering “What now? How can anything top this?!”. I still have four months left in this fellowship and every day I wake up I look forward to hopping on my bike and scurrying through the Berlin streets heading into the office and working with my friends at ZEIT ONLINE. There are few things I can think of that are better than the life of an open news fellow. Luckily for you, you get a chance to do it too!

 

July 30, 2012 12:19 PM

I may have graduated, but I still get very good advice from my mentors. The most recent came from Ethan Zuckerman: “Dan, please try not to get fired in your first month. That would be really embarrassing for everyone.” His delivery reflected a hint of genuine concern.

There are many reasons why he might have said this, but two stand out. For one thing I had just given a presentation about NewsJack, a media manipulation platform that I created from Mozilla’s Hackasaurus with Sasha Costanza Chock. When NewsJack was released it was immediately met with a Cease and Desist from the New York Times (note that The Times is the parent company of The Boston Globe).

It is also possible that he was inspired because I had just confessed on stage that one of my first thoughts when walking into The Globe’s headquarters was “I wonder what it would take to bring down this organization.” I’m betting it was the juxtaposition.

The Backstory

During my first few days at the globe I wanted to understand opportunities for innovation as quickly as possible, but to do that I needed to understand their resources and values. It occurred to me that if you want to identify an organization’s most valuable assets but you don’t know where to start, you should just pretend to be a super villain and plot their destruction.

Assuming you’re a competent villain, whatever you end up targeting should be important. Not only that, but the target will reflect your personal passions and expertise. Try the mental exercise yourself and share the results. I dare you.

For example, to take down a newspaper you could…

• Open up their paywall (if it exists), steal their content, and make it freely visible to the world without giving them any form of recognition or compensation.
• Eliminate their productivity, either by instigating a massive strike or by hiring away all of their employees.
• Scare away their advertisers so they lose a significant revenue stream and can no longer pay their bills.
• Destroy their infrastructure (printing presses, websites, etc), thus disabling their ability to ship product.
• Corrupt their editors and slowly replace key actors with your henchmen so that the paper becomes your mouthpiece.
• Buy sharks with laser beams attached to their heads.

A super villain’s master plan needs to be intricate enough to be interesting and difficult enough to be impressive. Blunt ideas like “take down their website” or “steal all their money” are a bit too obvious. It must also be simple enough for a diverse audience to understand. If nobody can figure out what you did, why it was sinister, or how it actually worked then it is hardly going to make headlines. Finally, it can’t be a series of bee stings; the evil needs to be condensed enough that it could fit in a tweet.

The Plan

My evil plan didn’t take long to imagine (given my recent work). If I were evil and wanted to destroy a newspaper I would ruin their brand’s credibility. This could be accomplished in many interesting and convoluted ways, but the “how” isn’t the point, the important question is “why?”

A media product will die miserable and alone unless it differentiates itself from the rest of the Internet. Luckily, newspapers have something that the chaff doesn’t: they have the capacity to create trustworthy information experiences. They are the ones with paid reporters asking the hard hitting questions, they have the editors and the internal fact-checkers, they don’t have an agenda and aren’t trying to manipulate me! right?

Well, maybe. As a reader I don’t know where content comes from or how much journalism went into it. All I have is faith in their brand. I trust that the sources I read are doing their jobs. That faith didn’t come from nowhere. I might have liked what they had to say in the past, or I saw my parents reading their paper, or their brand has a strong reputation. Regardless, I am now far more likely to trust what they have to say than I am to trust, for example, what my crazy friends like to read.

Just to drive this home: given the way content is presented today I could read the exact same article on the front page of the New York Times, Fox News, or the Huffington Post and my decision to trust it would be more strongly influenced by my opinions of the publisher than by the content itself.

To drive it home a different way: hijacking a newspaper’s credibility is as simple as imitating their brand.

Save the Day

The wheels are turning and it is already out of my control! IP lawyers are powerless compared to the forces of the anonymous web! But seriously, brand is a really fragile way to differentiate on the Internet. So what’s a newspaper to do?

Take a page from Apple and redefine the way people consume content. Train your readers to expect a certain experience not just from your website, but from every source of news. Make sure that experience is either expensive or impossible for alternative sources to replicate. Newspapers need to make their readers expect proof of everything. People should feel uncomfortable trusting information without explicit, functional credibility.

Newspapers have journalists doing research, checking facts, and taking names. They have multiple people and multiple systems touching every piece of content before it gets published, so why does the product usually end up being a bunch of words with prose-based evidence?

News organizations need to make the world hold information to their standards.

Like I said earlier, it makes sense that this particular plot and solution are coming from me. I dedicated my thesis to credibility layers — interfaces that lead to credible information experiences based on more than faith and trust. There are many paths to differentiation. Some are evil, some are entertaining, and some could even change the world.

* Drawing courtesy of Lyla Duey

July 25, 2012 04:26 PM

Last week was Discovery Week at The Guardian. Four of the five fellows (Dan, Laurian, Mark and I) were in attendance. Our project is NewsQuest and a host of other great ideas arose from knocking developer heads together and giving them the time and space to create.

The BBC will be hosting similar events and hopefully more digital organisations will try and cash in on the creativity within their own institutions by downing tools and fostering a startup attitude. For those who are planning on discovering here are my dos and dont’s from my experience and The Guardian’s first Discovery Week.

Do Invite People Outside Your Institution

Two heads are better than one and if you want people to break out of paradigms you need ideas from outside your organisation. OpenNews Fellows should be on your list! Reach out to people who share your goals and ethos and be open about the challenges your organisation faces.

Don’t Give Them Preference

A Discovery Week is not an opportunity to foster ties for commercial purposes. All ideas are equal and besides organising travel and accommodation for guests, they should work with the same setup as everyone within the institution. Everyone should be allowed to pitch an idea equally and have the same resources.

Do Get Audience Feedback

Pitching your idea to an audience early on is a great way to decide where to concentrate your resources. Game-changing ideas cannot be built in a week. Feedback from future users really helps shape your core idea and focus your efforts so that the end result has clout. Knowing that you will have to present your idea to the audience also helps focus your project towards the needs of the news organisation.

Don’t Micro-Manage Everything

With a lot of developers working on lots of ideas things can get chaotic. The whole point of a Discovery Week is to build for the future and that will require some organising. However, tying up people’s time with meetings, form-filling and emails will only stifle creativity. Teams need to be aware of each other so that mergers and collaborations can form but let there be open work spaces and  a call-out board for those who need it rather than making everyone report everyday.

Do Make Sure The Ideas Have Reach

This means getting more than the people working involved. An entire news organisation cannot all down tools. But you should make sure that journalists, editors, board members, etc all have a chance to see what is going on and find out what people are working on. Use Discovery Week to foster ties between the newsroom and developers.

Don’t Have One Lengthy Presentation

A Discovery Week never ends. It is meant to build momentum and for that the whole organisation needs to be on board for change. In that sense, presentations serve little purpose. Momentum needs to be built throughout the week and a science fair exhibition setup would best suit the purpose.

Crossing the digital divide between journalist and developer has many challenges but a gem in the data journalism crown is the hack day. I really wish editorial would adopt this and have a day where one person from each team works together on a story idea (preferable investigative). To create wonderful, powerful and imaginative stories and ways of telling them involves unshackling oneself from paradigms and the daily grind. I hope that discovering becomes the norm and not the privilege of organisations with many developers. Whatever is needed to make it happen, whatever form it has to take, it will be worth it.

 

July 24, 2012 12:09 PM

July 19, 2012 07:25 PM

The lead article in today’s International section of the Guardian is a data investigation into US Food Aid which I worked on as part of my OpenNews Fellowship.

Sadly, there is no GitHub repo for this particular data dig. Just under half of the data we looked at are in PDFs. However, there is some code on ScraperWiki.

For a write up of the overall process, source links and sample data, read the “How We Did It” post on The Guardian. To explore the condensed data check out the interactive. This was made using The Miso Project, a JavaScript open-source interactive library.

The code I have is not great, I’ll be the first to admit that. I really just wanted it to do the job. I’m slowly beginning to do things ‘developer-style’. My first step is to actually ensure the structure of the code is robust. So I split each scraper (the bulk Purchase Award Contracts and two types of packaged Purchase Award Contracts had to be scraped separately) into 3 – 4 steps.

For instance, looking at the packaged contracts, I :

1. Fetched the link to each text file for all the years
2. For each text file I fetch all the content into one long string (these two steps could easily be combined, but if the content changes/gets wiped I will always have the original content in a separate file. Keep all original records, it’s just the journalist in me)
3. The content of each text file was then parsed for the relevant information. Separating these two steps means ScraperWiki run time is kept to a minimum when making calls to a website. This time is limited so you would want to separate your parsing from your scraping.
4. Port allocation reports were structured differently to the Purchase Awards so they were parsed differently from the script in step 2.

I downloaded these as CSVs and put them into a SQL database. For the fiscal year we were interested in, I pulled the Excel files of the Cargo Availability Reports together by copy and paste. I put this into the database and joined Purchase Award Contract to Cargo Availability Report on contract number, port and quantity. The other fields like company and commodity were free text fields so those might not match even if they were supposed to be the same thing. I wanted to avoid duplicating entries on the join so these fields looked best to me.

The Excel files were not consistent in their heading structure so pulling them together  programmatically would probably involve more effort. Given time and better programming skills I would have liked to have done this and written a combinatorics programme to match the contracts to the cargo reports when the quantities were split over several shipments.

The biggest complication in getting the data together was finding out where everything was and how it was all related. Another obstacle that highlights how difficult it was to piece together the data, is the fact that the reference numbers for 4 different documents have the same number structure. For instance, the packaged PDFs on the dreadful web portal have a “Bid Invitation” numbers that look exactly like the “Purchase Requisition” number in the Cargo Availability Reports but in fact refers to the same structured number (2000000XXX) in the filename!

There is public data and data made for public use. These are not the same thing.

 

July 19, 2012 04:41 PM

I recently was asked to speak at the inaugural meetup of HacksHackers Canterbury. I wasn’t sure what proportion of developers  and journalists would turn up or what they would like to hear. So I decided to give my top ten tips for working on a data driven project and managing the relations between what the journalists need and what the developers need. These observations are based on my time at The Guardian but also rays of enlightenment from experience at the BBC and CNN as well as struggling with my own data digging projects at ScraperWiki. I introduced the topic by pointing out that developers love cats so put cats everywhere.

Naming Conventions:

For a team of developers working on a project; file, variable and method names have consistent formats and should be labelled logically. This is a practice they all know of even if they do not adhere to it strictly. Sharing, reusing and publishing code require certain standards and best practices. Yet, traditionally, journalism was not a team event and journalists were allowed to organise themselves any which way they liked. Notes were written and filed (hopefully). Now, all journalists have email, twitter, Google docs, spreadsheets, Evernote. A digital cacophony spread across platforms each with their own filing system. Dealing with an array of files filled with data at various stages of aggregation and cleaning, along with documents on what was done is a nightmare. Not only is it a pain that slows the process but it is a danger to the integrity of the story. The lack of clear organisation increases the probability of introducing human error in constructing the data. So no files called “temp part 3.4 final revised” should ever exist. For command line use, never put spaces in a name (it’s a pain). Choose a logical convention and stick with it. thisIsCamelCase and this_is_snake_case. Use one or if you have different structures, use both to make this apparent, for example use camel case for file names and match them to the tables, and snake case for the column headings in the table.

Understand The Structure:

A really easy tip for journalists is to open files in a text editor. These are free to download, any text editor will do. Often, what you see in your application (Excel, OpenOffice, Google docs, etc) is not what is actually seen by a programme. Even if the journalist can’t do anything about it the person dealing with the data needs to be aware so that they can communicate proper time frames to the journalist. Often, getting the data in the right structure is just as time intensive as building an interactive to display it. But there are simple things the journalist can do. When working with CSVs (spreadsheets) make sure there are no carriage returns in the cells. Identify the lowest common denominator in data dependencies and record those rather than the resultant data point, for example, if you have a quantity and a price per quantity that give you a total price, log the quantity and price per quantity as the total price can be calculated from those two. Always add an id column to your dataset so that if the data needs to be joined artefacts are more easily found.

Do The Journalism First:

This may seem very obvious but working with data requires another layer of questions before you pass it on. Besides looking to see if there are interesting stories in the data you need to ask yourself: Is this the full data? What calculations can I do to check? Are there columns I can join to add value to the data set? Do I know what all the columns mean and how they were generated? Has there been a change in the way a column has been measured/gathered? What results can be caused by artefacts? In that sense, don’t just hand over a download of the data to the data architect, give a link to where you found it.

Get Comfortable With Your Tools:

This applies to the budding data journalists. Ideally, you will have to use an array of tools. Know how they work and what best they work for. Know how they can be combine to reduce the time frame or the probability of introducing errors. Be comfortable with them but don’t feel you have to master them all. If you want to dedicate your time to mastery, pick the coding language as software tools always change or disappear entirely. This is the really difficulty with working with data. Each project will require an intensive use of only a subset of the tools you have in your tool box. Because projects can take 6 to 8 weeks, you learn to use those tools better. But it may be a good while before you use any particular one again and by then you’ve lost the mastery and have to jog your brain a bit. Developers do it all the time. So seek their help, comfort and advice.

Always Be Learning:

All developers have to keep learning to keep up with the web and journalism is going the same way. I really enjoy this part but I realise that I am in a unique situation. I would strongly urge editors and managers to give the journalists time to pick up new skills. If not on their own then with training courses or better yet, peer-to-peer programming and tutoring. Regardless, there are a lot of free online resources which are great for beginners including Codecademy, LearnCodeTheHardWay and a range of courses from Coursera and Udacity. The main lesson is never expect to know everything or have to do everything. The best practice is to build atop someone else’s work. Just adhere to best practices in terms of openness and accreditation.

The Command Line Is Your Friend:

One thing that was and still is quite hard for me (but really should have been the starting point) is using the command line. It is hugely beneficial to everyone to run the computer through the terminal. It will also impress the developers. For manipulating files it is much faster than any piece of software. What I found enlightening, this was passed to me by a developer at The Guardian, is to use the command line to pipe CSV files though Python parsing scripts rather than writing a script that looks at one file and prints to another. This means I can write individual scripts to change one things, keep those, file them and reuse them without having to write one long script.

Work Open Source:

If you’re going to take away one tip this would be it. This is the most useful. What I mean by work open source is work as if you’re in a team and are going to publish your workings and even as if you are going to make a tutorial. This means you comment your code, structure your code and your files and put it in a place where you can find it when you forgot what you did. It may seem needless and tedious at first but it pays off in spades in the long term. It can also attract attention and connect you to the type of people you need to know so share and share alike. You may sometimes fell like the cat here, horrified. But if people point out that you did something in a less than pretty way it means you will know the more elegant way to do things and are less likely to make the faux pas again. There is nothing to be gained from keeping your learning and your work closed.

Do Fun Things:

If you are thinking of learning to code to augment your journalism then go to hack days, conferences and socialize in the developer community. They are very good fun. Find a HacksHackers and get talking to people. Newsroom developers are lucky in that even news organisations have adopted the hackday ethos of creativity. Journalists have never had the opportunity to take a day or even a week (as is the case in The Guardian) to work on any idea they have. This is how investigative journalism should be brought back into the newsroom, by having days where every journalist can pitch an idea and get a team together to see if a feasible investigation can be made. Sadly, that is not the case. But developers have this amazing ‘right’ to hackdays. They are all about creativity, meeting new people and geting news perspectives (plus finding out what’s going on in different development circles). So take advantage of these when you can.

Catch Bad Guys:

Developers like creating things. They like making fun things or functional things. What they don’t like is having to tear things apart, spend days digging and constructing data. They don’t like having to call people up about it. Many developers can and will do all these things but the typical development community is centred around building software, libraries and websites. Data driven investigations is where the data journalism niche lies and this, to me, is the most exciting area of journalistic potential. This is also where the developers are most appreciative of your skills.

Think About Others:

At the talk I said I haven’t met a rude or egotistical programmer since I started which Martin Belam pointed out is owing to my mileage. I have met egotistical and rude programmers who are also programmers but what I meant by this (which may also be naive) is that I haven’t met a programmer who thinks he/she can make/do/build everything themselves (unlike some journalists I’ve met). They are always looking on stackoverflow and Github, always use libraries and ask others what the proper syntax for a language they haven’t used in ages is. When it comes to design and creativity, you get egos but just learning the nuts and bolts of code, I have found everyone quite humbling. That being said, I feel it very important that the journalist in the equation does not get an ego. I feel the roles as they stand now have the journalist throwing data or ideas to developers, telling them what they want (which is usually something they’re seen from a competitor’s site) and expecting the results in a timely fashion.

For the future of quality journalism through a connective, social, immediate medium, each person in the work flow stream has to understand and know a fair bit about everyone’s role.

July 06, 2012 03:13 PM

I’m now only moderately jet-lagged from a week spent in the US in which I crammed in two hackathons and a conference. Here’s the scoop.

I was excited to visit Boston and super excited to be invited to the Knight Civic Media Conference hosted at MIT Media Lab. Boston is a fine city with lots of tech-related activity and since my colleague and Boston Globe Open News fellow Dan Schultz had recently graduated from the MIT Media Lab I had an in! Jet-lagged but happy I got my first glimpse of the famous lab when I and another fellow Open News fellow Laurian Gridinoc were invited in to watch Inception with a Rifftrax soundtrack playing over the top it. It sounded like a kinda chilled out media hack and after food, beers and caffeine tablets I found myself in what seemed to be a very cool building with more inventions and gadgets (let’s just call them toys) that you could shake a memory stick at. As I saw more, it soon dawned on me that this was the place dreams were made of and I started to wonder if there were any adults about. As I watched Inception unfold with added Rifftrax sarcasm I started to wonder why I didn’t know more about what was – for me at least — a type of media and technology Mecca.

The next day - a Saturday - we started hacking. The hackathon itself (sponsored by Mozilla) started in the afternoon, but as a warm up exercise and just because it was there, I attended the Hacks/Hackers meet-up immediately before it, in the same building. As you might have guessed from the title, this was an event for journalists and developers. We were lucky enough to have Miranda Mulligan facilitating the event which touched on interesting and useful subjects such as responsive design for online news sites. I guess when you intersect two disciplines you get straight to the point of what is actually useful.

So on to the hackathon on another floor, the now familiar introductions around a circle and rules of engagement, more coffee and away! I didn’t suggest any projects this time as I wanted to work on somebody else’s for a change. The theme was The Story and the Algorithm and there were plenty of cool projects to chose from. I gravitated towards the folk planning to do something with web based video and after a bit of brainstorming we pretty much had a plan and started hacking. We had something working by the end of the day and the next day we finished it off and added the a distinctive front page and lo surfbored.tv was born - it provides a way of surfing YouTube channels passively. Warning - if you tune in to the Andy Carvin channel you may encounter graphic content. It was fantastic to work with a crack unit of developers, designers and journalists which made up our team James Burn, Jesse Shapins, Kara Oehler, Corey Ford and Brian Boyer.

The hackathon ran from 3pm until 4pm the next day (thankfully sleep and socialising were encouraged in-between) after which we were given a whistle-stop tour of the Media Lab by Dan Schultz who you got the impression was very sad to leave the place. But leave he must for now, as he made the error of graduating. Actually I think he has a year’s leeway so watch out for him at future Rifftrax events. After the tour in which I probably lost a boat-load of Twitter followers by compulsively photographing and tweeting EVERY SINGLE LAB we were ushered up to the totally different atmosphere of the conference centre on the top floor and the start of the conference proper. More canape’ than cyberpunk we were provided with refreshments, food, networking opportunities and a great view of the city. All very plush.

The conference took place over the next two days and it was inspiring to once again witness the fusion of journalism and technology. Best of all was the opportunity to liaise with the other four Open News fellows and Dan Sinker our inimitable leader (it’s not often we’re all in the same place). There were many good talks and discussions, I especially appreciated seeing people I knew or had followed from afar get up on stage and do their thing. A couple of the highlights that stood out for me was Erin Kissane’s short but beautifully told story on her relationship with books and how digital books and related services are replacing them. Being a great lover of the papery thing, I very much identified. Long and short - it’s hard to let go but there are just too many advantages to using electronic versions. I also loved Ben Moskowitz  talk on Drones and Journalism - a fascinating subject, somehow I have retained the love of remote control vehicles from my youth.

And so on to Washington DC and the second leg of the tour. Another city, another hackathon - this one was a Mozilla-ITVS-LivingDocs initiative dubbed Silverhacks by organisers Brett Gaylor and John Archer. I travelled down with Zeit Online fellow Cole Gillespie. It was all part of the Silver Docs Indie Documentary Festival  where Cole and I got to work with two local filmmakers - Brandon and Lance Kramer who had spent two years filming material for a documentary exploring “the lives of people employed by the modern-day DC Green Corps, an urban forestry job training program […]” They had some great material and our job as developers over the next couple of days was to help make an interactive piece with a selection of it. The result was a Popcorn.js / Google Map API combo and I think we were all fairly happy with the process and the result (the audio part after the initial video is where all the fun starts).

Overall the trip was extremely worthwhile - amazing people were met and caught up with and some valuable lessons were learned. Hackathons or just hack-days, to me, are a great way to work with other people and concentrate on getting the minimum viable product out in a very short space of time - invaluable practice in fact for working in a newsroom and I feel that although I’m now behind with all my other work I am much better off for the experience. The best part is that both of the things that we ended up making could be used in a news organisation and I hope to incorporate these ‘hacks’ in the near future.

Hack on!

June 27, 2012 04:26 PM

June 22, 2012 06:29 PM

I just finished the second week of my fellowship at The Boston Globe, so at this point I have a stronger sense of how things are done and where I might fit in. Having just come from an academic program like the MIT Media Lab there is a tinge of culture shock (for instance, everyone already knows everyone and they use actual development processes), but I’m finally starting to meet people and learn the ropes.

At least I’m not the weird new guy wearing dress shirts that don’t really fit any more; I’ve upgraded to button down T’s.

Update: Matt Stempeck has told me “it isn’t your fault, but nobody in the world can pull off the button down T.” This is why I usually let the Internet choose my wardrobe.

The Trilogy

Over the next ten months I’ll rotate between the three key technology groups at the Globe. I’ve given them battle-clad nicknames for the purposes of this post.

• Round 1: Front Lines. I’m starting on the product team, working under Mike. This is where people work on longer term product-level projects. This includes things like the two web sites (BostonGlobe.com and Boston.com), the Content Management System (what people actually use to write articles), mobile experiences, and comment systems.
• Round 2: Special Ops. My second rotation will be for the Interactive News team (NewsDev for short), working under Miranda Mulligan. They work on more targeted technology projects such as reporter tools or interactive interfaces.
• Round 3: R&D. I’ll end my run working in the Boston Globe’s Media Lab / R&D department under Chris Marstall. This is where the more “cutting edge” / “insane” / “impractical” / “highly creative” work gets done. I really like the idea of this being my capstone rotation since it feels the most in line with my darker past.

Each of these groups has a different set of responsibilities, constraints, and skills. It seems like the Chris’s group (R&D) has the most freedom but isn’t as likely to ship full blown features and systems that get used by millions. Mike’s team (Front Lines), on the other hand, pushes out code that will support the newsroom and the world. As a result they need to keep their eyes on reality — they can’t take as many risks.

Miranda’s (Special Ops) gets to be somewhere in the middle simply because of the nature of user-facing interactivity and the fact that many of their projects are “reusable one-offs.” For instance, if an interactive feature didn’t scale it would be very unfortunate, but at least the entire organization wouldn’t collapse in a fiery heap of paper. The same could not be said for the code that renders the front page of Boston.com.

Free Developer… like Free Beer?

I was jovially introduced to most people as the free developer (“You’re not free any more! Mwahahah”), although I’m hoping to do more than just hang out and crank out code. I’m coming in with some ideas in mind and a general mission, but narrowing that down into something actionable became a lot easier after seeing my fellow fellows this weekend at the Knight Civic Conference.

• Understand pain points, identify opportunities, and communicate. I heard battle stories from the other fellows and immediately understood that one purpose of the OpenNews experiment is to unearth those stories and put them into the open for discussions. The entire industry needs to know examples of what works and what doesn’t, even if there are a few worms under the rocks.
• Stay creative, pursue wild and crazy projects, and share that creative spirit with everyone around me. I’ve just come from one of the most forward thinking places in the world, but the Lab isn’t defined by skill sets and brains. What makes the lab great is the attitude, approach, and the ability to fail effectively. There is no reason why some of that spirt can’t be transported to the globe (in fact some of it is already here.)
• Stay grounded and write awesome code. At the end of the day I am also a developer! I want to learn some new technologies and create some great projects. The key for me here is that I want to make stuff that will have applications both inside and outside of The Boston Globe.

And so this is my hopeful breakdown of the next 10 months:

1. 30% on immediate needs of the Boston Globe, like new platforms and systems.
2. 30% on my own fleshed out ideas, like Truth Goggles, or The Meta Meta Project.
3. 30% on wacky ideas like Newsquest or ATTN-SPAN.
4. 10% on reflection, in blog posts like this.

I see a parallel between the three groups I’ll be working in and those three major chunks, but I’ll be aiming to do it all, all the time. If you were at last weekend’s OpenNews hackathon at MIT you know that I’m feeling multithreaded.

June 21, 2012 07:36 PM

This was my first week at my brand new job as a Knight-Mozilla Fellow and I can tell you already that it is going to be an awesome time. I’ve worked in a newsroom once before, as an intern in the Pittsburgh Post-Gazette back during my undergraduate years, but I have a lot to learn and things feel very different here (bigger, more developers, fewer Steelers jerseys, etc.).

I started the process by meeting with Mike Krolak. He is the man responsible for the technology related to product at The Boston Globe. This means web sites, but also things like printing presses, delivery trucks, and employee management systems. I’ll be starting my fellowship in one of his teams, which is the reason why I ended my first day with an ID badge, credentials, and access to most systems. Impressive, most impressive.

When we talked over lunch he was clearly curious to learn more about my purpose and personal goals, at which point I said “you and me both, bub” and stared awkwardly past his head. What I actually did was share my high level understanding of the Fellowship: it’s an experiment, but the goals are to help technologists learn what it is like to work in the journalism industry and to infiltrate and destroy newsrooms get newsrooms more comfortable with external collaboration and open development. Really we’re figuring out the details as we go along.

I got to learn a little more about Mike, what motivates him, and why he has spent more than a decade working for The Boston Globe. Here are three of my favorite points:

• There are so many simple, unsolved problems. Mike has a math background (I’m wary to tell people with actual math backgrounds that I was a math minor, but I was immediately able to relate), and one of the things that motivate math geeks is unsolved problems. In fact, the quest for the perfect proof can cause them to actually go insane! Mike sees the news industry as a great way to go mad, with tons of challenges ripe for the picking. A prime example: “did you know that there isn’t one major newspaper in the world that can tell you how many papers they distributed today?”
• Technology done wrong can get in the way of good journalism. I feel silly admitting this, but the idea that technology could actually hinder the way a newsroom produces content had never crossed my mind. For example many papers have publication processes that force articles through up to 50 different systems. Each transition leaves room for accidental content modification which results in more edits. What you get by the end is a professional version of whisper down the lane (or “telephone” for you non-Pennsylvanians).
• Newspapers are getting more comfortable with innovation. As you probably know already, when the Internet got popular the News industry sat around fat and happy and tried to use it in the same way they used any medium: to publish their content. In the past 10 years we have seen the advent of R&D departments and research spaces in newsrooms. There is a lot of catchup to do, but at least they’re trying.

For me these points frame the essence of the OpenNews initiative. Now that newspapers are innovating we want them to take a page from the mathematicians and make sure unsolved problems get solved in a way that everyone can learn from, expand upon, and contribute to (e.g. by publishing them to the world). As fellows we are also trying to understand what technology means for journalism and to share our lessons about where it can help and where it can hurt.

June 11, 2012 12:00 PM

With my thesis complete I’m about to embark on the next phase in life, and in true Millennial fashion it will last about 10 months because job security is for chumps. I’m going to be a Knight-Mozilla Fellow (feel free to call me “Mr. Fellow”) working in The Boston Globe as part of the Open News project led by the Knight Foundation and the Mozilla Foundation.

My first full day was yesterday and I’m just about finished a post about my first impressions and lessons learned. Before I hit publish on that, though, I want to give a nod to the option I decided not to pursue: a PhD at the MIT Media Lab.

Choosing a Path

I ultimately decided not to apply to the PhD program at the Media Lab for now. This wasn’t easy (and doesn’t mean I won’t come back and apply later), but it came down to a few points:

I’m not really a great academic. Don’t get me wrong, the motto of “Demo or Die” generally trumps “Publish or Perish” at the Media Lab, but this doesn’t change the fact that a PhD is ultimately an academic endeavor. When it comes to technology my natural inclination is to code and blog, not to design studies and write scientific papers.

I got a huge amount out of my time at the lab. Yes, the awesome ability of the Media Lab to jump start lives is actually part of why I’m moving on. In two years I feel empowered to create anything I want, I’ve redefined my understanding of the world, quadrupled my confidence, developed an amazing network of friends and connections, and become part of a wonderful community. I’m sure that four more years at the lab would continue improvement on all of those fronts, but I doubt that it would be nearly as saturated.

Four years is much longer than 10 months. Call me an opportunist but I’m married, and who knows where Lyla and I will want to be over the next four years. Actually it turns out we do know (Lyla got into RISD and starts her graduate program in September), but that isn’t the point. Locking myself into something for four years seems dangerous at a time when there are so many problems to solve and ideas to pursue. What would have happened if Luke just chilled out with Yoda for four years?

All of that being said, if I ever commit to a PhD it will be at the Media Lab. I already miss the place! Luckily I’ll stay an official part of it for at least another year as a “zero cost visiting researcher.” This means I don’t get paid, but I get to keep access to resources and have a good explanation as to why I’m still skulking around the hallways at night.

This leaves me here, living the Tolkien dream and beginning a brand new fellowship.

June 05, 2012 09:10 PM

Source is a website dedicated to documenting and supporting the increasingly vibrant world of journalism code through project walk-thoughs, source-code docs, tutorials, interviews, and more.

The alpha of Source is coming soon, and we’re ready to start collecting information on news apps and other newsdev projects. We’re using Wufoo to gather the first batch of information, and we’ve put up a few different forms to make the info-entry process as quick and easy as possible:

• Tell us about a project
• Link to a code repo without doing a write-up
• Make a quick how-to/tutorial for a process
• Or something else

Holler if you have any trouble or questions.

June 04, 2012 04:03 PM

In progress.

June 01, 2012 10:20 PM

There has been some noise on the web in regards to learning to code. Jeff Atwood declared: “Please Don’t Learn To Code” in response to the New York City Mayor‘s tweet that he was joining in Codecademy’s Code Year. My favourite web-based programming mentor, Zed Shaw, retorted with “Please Don’t Become Anything, Especially Not A Programmer” saying:

… learn to code for your own reasons, then you’ll be just fine and get out of it what you put into it. If you attach your identity to being a programmer, then changes like “everyone can code” will lead to resentment because you are no longer unique

Even my favourite data journalist, Dan Nguyen, joined in the fray with “Fashionistas (and bureaucrats and journalists): Please learn to code“.

I am learning to code, but not just how to write a for loop, or scrape a site, or set up a database. I have said from the very beginning “I don’t want to be a developer”. I am learning to code to burrow into data not to build web-based tools, so my journey will be very different to that of a web developer. I have a clear goal in mind even if the course material is unknown to me. I can see where both parties are coming from. This is because code is worthless.

This is exactly what a developer of two decades told me recently. And he’s right. Software companies made the mistake of thinking their code was their product and they kept it under lock and key. Then the web made a new playing field and open source projects burgeoned into new innovative territories where the code was the medium and knowledge the product. The code for Facebook is worthless. The code for Twitter is useless. The understanding of users’ wants, needs and evolving doxa is the true treasure.

Similarly, news organisations made the mistake of thinking their content was their product. Now that everyone can generate their own content it is the organisation who understands the power of fostering awareness, engagement and community that will survive the digital cull of what was formerly known as ‘news’. The words which make up a story are worthless.

Every piece of code I write and which you all will see is worthless. Just as building a resume or website with Codecademy will be worthless to Mayor Bloomberg. It’s what you learn doing it that has value. The knowledge you gain whilst trying something new is the only thing worthwhile.

Knowing what to look for when deciding whether or not to scrape a site is valuable to a data journalist. Finding the story by querying the data is valuable to a data journalist. Pitching the headline to an editor is valuable to a data journalist. Knowing how best to tell the story is valuable to a data journalist. You cannot get any of these from the scraper code, so it is worthless.

And this is the crux of the open source and open news movement. The words, the code have no worth to its creator. It’s the experience and knowledge gained by weaving these fibres into something that can tell you more of what you need to know that is worthwhile.

May 31, 2012 02:28 PM

Sometimes I have trouble shipping, I think this is a common problem in the software world. Nothing is ever finished is it?

A great advantage of working in a newsroom is that people are constantly shipping. The product changes from hour to hour, so you need to get on with it and get things out. There’s no choice. I’ve sat with people as they rushed to put together a breaking story, the excitement is quite palpable – very similar to that feeling you get as a developer when you’re about to deploy a new version of your app. That shipping feeling.

There is usually of course a great disparity in time between shipping a story and shipping a software product.

However it is the atmosphere of shipping that interests me and it’s something that I’m really appreciating while working with Al Jazeera English. It has been both motivational and inspirational to work with the people there and to take on-board the culture of ‘getting-things-out-there’.

As a software developer a news website is a perfect place to get your projects seen, viewers will generally appreciate that news is a transient medium and naturally give you a bit of leeway. After all, you’re not creating an app that they have to use every day, maybe just some interactive video thing that they watch one programme on, or an audio slide-show they let play for 5 minutes, a quick visualisation perhaps.

Recently I have been working on an “interactive video thing”. Admittedly it took a few iterations to get right and we took a very reactive approach as we worked through those iterations, varying both content and function. Many features were tried, adjusted or simply removed. It took a while to get right – but as part of that process we demoed and sought feedback from the wider developer community, we even soft-launched a version as suitable content came along. Finally we got somewhere near to where we wanted to be and we launched.

The “Internet Indians: In Contextual Video Player” aims to add a bit of interactivity to the standard video experience. The idea being that while the video is playing additional info is displayed to the viewer. The viewer can then interact with this info using social and playback controls. Social control in this context means that they can easily tweet an excerpt of that additional info and link that to the part of the video it relates to. The player was also built with the consideration that media can be consumed in different forms and on different devices. To this end and with the idea that the viewer should determine the level of interactivity, we baked in a more passive full-screen mode. We also made sure that the app could be ‘detached’ from the page and be used on various mobile devices.

Technology-wise we opted for Popcorn.js which gives us good control over time generated events and as our player framework we used jPlayer which was skinned and extended to work with Popcorn. I can’t help feeling that having an application in a constant state of flux is a good thing. We were used to changing things about completely and managed to incorporate two new building blocks as they came out. The first of these was an IE8shim for Popcorn that together with jPlayer allowed us to target the vast majority of browsers. The second was the excellent Tabletop.js which helps you grab data from a Google Docs Spreadsheet and incorporate it into your application, live. This gave us a rough, ready and entirely usable content management system allowing the editors to quickly and easily add and view content.

We learned a lot developing and a few things deploying, especially about iFrames and cross domain restrictions. There were a few glitches initially, but we soon got them ironed out. Our code was client-side and sand-boxed in an iFrame and so it was very easy to deploy without having to go through the official security procedures that exist for server-side apps.

Agility aside, the biggest lesson learned was not to fear shipping. Whatever you will do will never be perfect and it’s transient anyway.

Now that we have these versions out, we have measured, taken on-board feedback and taken stock, we are very ready to move on, take things to the next level and I’m raring to go – I want that shipping feeling back.

Mark Boas

May 22, 2012 10:07 AM

The German blogosphere came together last week at re:publica to kick start Berlin web week. A few weeks before this Kai Bennerman approached me with an idea about tracking this conference as it was being talked about in real time via twitter. He had a few ideas about some different key words that he wanted to follow that were being tagged with the #re12 hash. My approach to building this widget first started with the streaming API that twitter provides the only downfall of this is that using the streaming api would require some server pieces to work properly and the assets were not available to build such a service in the time frame that we had to complete this project. I decided to use twitters public search api instead as my platform to build a jQuery plugin on top of.

The first idea was simple. Filter tweets that contain the following words: ’session’,'track’, ‘vortrag’, ‘talk’, ‘ and panel’ that were included in tweets that contained the rp12 hashtag. At first you would think that you would just include these words inside of the query string that we send to the twitter api but if you do that the results seem to get over filtered and raise the complexity of the query string which increases your chance of getting rate limited by the search api. Another problem with adding the items to the query string meant that twitter would try and return these tweets to you ONLY if all words were matched, I need an OR scenario not an AND.

My way around this was to query the search for only the hashtag #rp12 and then use javascript to search for the requested strings myself vs having twitter do it. As I searched through all the tweets I would keep track of how many I found that met the proper search criteria and once I had enough items I would cut off the query to twitter. While I am doing this checking as soon as I found a match I would create the DOM element and append it right away so it would be able let it scroll on the page and be visible to the user.

You can find the code on the openNews github.

May 09, 2012 02:55 PM

As you can probably gather from the title, part of my remit as Knight-Mozilla OpenNews Fellow is to investigate and help implement open solutions. Solutions that will not only benefit Al Jazeera English but also other enlightened news organisations. Luckily AJE is already somewhere down the path to openness having released much of their content under a Creative Commons license. This license allows others to use this content without having to worry about royalties - this is of particular interest to me, planning as I am to build tools that will allow anybody to create new content by mashing up existing stuff.

A related approach currently being taken by AJE is to use the open source Drupal Content Management System to power the blogs section of their website. They’ve also had the vision to employ one of the key contributors to the Drupal codebase and community Dick Olsson. Significant customisation of the new CMS will be required in order to meet the requirements of a newsroom and so bringing in someone already active within the Drupal community makes good sense. I spoke to Olsson and his colleague Alaa Batayneh about how they saw this all panning out. We discussed architecture, testing and crucially the release cycle. It seems they already have an agile methodology in place and seeming like two very capable developers I imagine they will be able to push custom code pretty quickly. This is of course essential - as needs of the newsrooms evolve, so must the supporting framework. There will be no quicker evolution than at the very start but no system of this kind is ever finished. This will be real reactive user driven development.

The real beauty of using a popular open source library like Drupal (now on its 7th major iteration) is not only the fact that you start from a solid base but importantly that any modifications you make can be released back out into the ‘wild’ for others to test, enhance and benefit from. The modular approach is key - it means you don’t need to modify the base of the framework and so in theory your modules can be applied to new versions of the base as it is released. Further, by plugging together modules to make solutions that work in a certain environments — in this case news — you can create distributions that address those specific areas of industry. So although the modules are not necessarily news-centric, the distribution they make up is and it is this that AJE and Olsson are keen to release.

Assuming then that Drupal is properly modular and can be easily and comprehensively customised, I would expect an open strategy to yield very real benefits. This is an opportunity to take advantage of an eager and vibrant community of Drupal volunteers working on the continual improvement of the core of the product while initial customisation takes place in-house. Actually I am thinking of creating a couple of modules myself just for the fun of it. When something is both cool and open it draws curious developers to it like bees to honey.

Sometimes for large and established organisations to adopt new ways forward it just needs someone to lead the way, so here’s hoping Al Jazeera will do with systems what they did with content - that way we all win.

Mark Boas

April 25, 2012 08:33 AM