Staffan Malmgrens blogg

Niklas Lundblad directs me to a couple of interesting propositions about pending laws regarding computer crime (one of them actually uses the phrase “crimes in cyberspace” — very 1995′ish retro). Both Ds 2005:5 and Ds 2005:6 are intended to be the first steps in implementing recent EC legislation (particularly “Convention on Cybercrime ETS no.:185“) in swedish law.

Unfortunately, both documents are very thin when it comes to defining what should be regarded as “illegal access to information systems”. Swedish law has not been well defined in this area before, and I was hoping that maybe the lawmakers would take the opportunity to clarify this.

The issue I’m mostly concerned with is screen scraping. I like screen scraping. I think screen scraping is cool. I wrote my first screen scraping program the same week I got my first job, almost ten years ago (it was a simple script to automatically download the latest Dilbert cartoon and email it as an attachment to myself).

Now, it’s easy to see why a content provider would be opposed to screen scraping. When I got my Dilbert strip in my inbox, there was no advertising attached to it, thus I was depriving Unitedmedia of ad revenue.

Years later, I was involved in the XMLTV project as I was playing around with a homebrew HTPC. For Swedish TV listings, there was a simple program that would fetch TV listings from dagenstv.com and re-format them into the XMLTV format. One day, their service started to serve up a very hostile-worded text file when I ran this program. Basically, dagenstv.com had changed their web server configuration so that requests from a certain User-agent (our screen scraper program), would get a stern warning that we were doing illegal things and that our IP adress had been logged, or something like that.

Now, were we doing something illegal? Keep in mind that each user would run this program on his or hers individual computer; we never redistributed the content. Wheter or not we could have done that legally in Sweden is another question, one that maybe could be answered by pondering URL 49 § and related materials. It’s an interesting question in it’s own right, just not the subject of today’s blog post.

Was the mere act of accessing the site with a different tool than the site owner intended, thus gaining access to digital data in a non-approved way, illegal? For me as a programmer, this feels like an absurd question. I’m only sending humble GET requests, if the site owner doesn’t want me to have the information, then don’t send it! But with my legal student-glasses on, this could be considered as computer infringement, as per the wording in BrB 4:9 c: “Den som […] olovligen bereder sig tillgÃ¥ng till upptagning för automatisk databehandling […]döms för dataintrÃ¥ng till böter eller fängelse i högst tvÃ¥ Ã¥r.”. (A rough translation would be “Someone who gets hold of a recording for automatic computer handling without permission is to be sentenced for computer infringement to fine or prison for no more than two years”).

Dagenstv.com could be said to have given users with normal web browsers implicit permission to access the data, but probably not to us with our screen scraper. If we had asked the site owners, they would very likely had said “no”, and therefore, they could well argue that we were getting hold of a “recording” witout permission.

(As an aside: the use of “recording” (”upptagning” in swedish) in the quoted law text is interesting in it’s own right — the legislation was originally written with telephone wiretapping, opening of letters, and similar things in mind, then “adapted” (used in the loosest of senses) into the digital age.)

I would prefer that questions like these were solved by technical, not legal, means. Dagenstv.com used one such mean (the User-agent discrimination) to block our screen scraper. We could have changed our program to masquerade as a normal Internet Explorer browser, but that would only escalate into a pointless arms race. Someone wrote a different script that fetched the data from another site instead, and that was the end of it. Furthermore, if we had bypassed dagenstv.com’s User-agent check, we would have essentially said “Even though we’ve been told in no uncertain terms that what we’re doing is not permitted by the site owners, we’re choosing to ignore that and circumvent the access control” — if we had done that, dagenstv.com would really be right in saying we were getting hold of data without permission.

But there’s a lot to be said for screen scraping. lagen.nu could not exist without screen scraping. A lot of really cool web services over the years have been made possible by screen scraping. It has enabled loose coupling years before anyone had talked about web services. It’s the basis for a lot of interesting research and data mining. And sometimes it just enables plain cool stuff.

Furthermore, it would be wrong to assume that all content providers are opposed to screen scraping. For example, what is the one thing that distinguishes forward-thinking web companies? They provide API’s to their services (Amazon, Livejournal, Yahoo, Google, Flickr), enabling anyone to build cool applications on top of their data, just like we wanted to build a cool HTPC application using data from dagenstv.com. By providing API’s, smart web sites remove the need for actual screen scraping (which, in all fairness, is a messy and seldom very interesting technological challenge, and furthermore only a means to an end), but enable and encourage the same kinds of applications. These API’s (and the XML-RPC/SOAP-based underpinnings) did not emerge from a vacuum. People have been screen-scraping Amazon.com for their own little needs since it was launched. Smart service providers realise that it’s better to work with all this creativity than against it.

If web site providers choose to do what dagenstv.com did, then fine. They’ve stated their intent, it’s their service, their rules, they’re entitled to take their ball and go home. But before a site owner puts such a block in place (which could also be done through a robots.txt file), screen scraping should in no way be considered unlawful computer infringement.

It turns out that the EC convention that this new legislation is to implement provides for these kinds of distinctions, under article 2 (my emphasis):

Article 2 - Illegal access

Each Party shall adopt such legislative and other measures as may be necessary to establish as criminal offences under its domestic law, when committed intentionally, the access to the whole or any part of a computer system without right. A Party may require that the offence be committed by infringing security measures, with the intent of obtaining computer data or other dishonest intent, or in relation to a computer system that is connected to another computer system.

Knowingly circumventing a access control system by, for example, changing the User-agent string, might be considered infringing security measures (weak as they are), but an unassuming GET request could, with this definition, never be considered illegal access. I hope that Sweden takes this opportunity to better define what should be considered illegal access.

Another aside: Since lot of my current activities, and thus my blog writing, revolve around swedish law, it’s sometimes difficult to write in English, as there are a lot of precise Swedish legal terms that I’m not comfortable translating. For anyone versed in Swedish law, posts about it in English is probably way harder to read. Furthermore, most of these posts are probably of limited interest to non-swedes.

Therefore, I’m considering switching the language of this blog to Swedish. If you don’t understand Swedish, but would like to continue reading this blog, please say so in the comments. Thank you.

Tags: dataintrång, upphovsrätt, webscraping

This entry was posted on tisdag, mars 29th, 2005 at 16:33 and is filed under law, programming. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.