Drupal/Freebase Integration Notes

Freebase and Drupal connectedFreebase is an open database (CC-BY) of world information with a web service interface. Think Wikipedia with an API. I've been playing around Drupal/Freebase integration for a community project we have brewing, Semantic Web Community Barn Raising in Vancouver. This post contains some research results, in the form of PHP code, on integrating Freebase and Drupal.

This code is intended as a proof-of-concept example and jumping off point for other developers. It is not intended as an example of production-ready code.

If you have feedback or a better method, please leave a comment!

Freebase Background

Those of you familiar with Freebase can skip this part. If not, read on!

Freebase uses the Metaweb database. The Metaweb database is not a row/column style relational database, data is organized as an object graph. Documentation on Metaweb architecture is available at the Freebase reference guide: Chapter 2. Metaweb Architecture.

The language used to query Freebase is the Metaweb Query Language, MQL. MQL queries are written in JSON, and are nothing like SQL. The Metaweb documentation page, Freebase MQL Reference Guide, has details and examples of creating JSON MQL queries for reading and writing data.

Demonstration Environment

The code samples shown were run in the scripting template I discussed in Drupal Command Line Script Template. For brevity, only Freebase query code is shown. The full script files are available for download at the end of this post.

Freebase maintains a sandbox server for testing potentially destructive code. The write sample runs against the sandbox.

Reading from Freebase

The Freebase read services API documentation is available at Chapter 4. Metaweb Read Services.

Reading is straightforward:

  1. The query is created either as a string, or a PHP array. PHP 5 provides JSON/PHP-array encode and decode functions.
  2. The query is placed in a query envelope
  3. The query (envelope) is sent to the Freebase read API via a HTTP GET
  4. The query results are viewed

Code

<?php
 
/**
   * Definitions
   */
 
$read_endpoint = 'http://freebase.com/api/service/mqlread';

 
/**
   * Create the query
   */
 
$json_query_string = '{
    "id":"/topic/en/philip_k_dick",
    "/film/writer/film":[]
  }'
;
 
$query_array = json_decode($json_query_string);
 
// Or Create Array Directly:
  // $query_array = array(
  //   'id' => '/topic/en/philip_k_dick',
  //   '/film/writer/film' => array()
  // );
 
$query_envelope_array = array('q1'=>array('query'=>$query_array));
 
$query_envelope_json = json_encode($query_envelope_array);

   
/**
   * Run the query
   */
 
$request_url = $read_endpoint . '?queries=' . $query_envelope_json;
 
$freebase_query = drupal_http_request($request_url);

 
$query_result = json_decode($freebase_query->data, true);
 
print_r($query_result);
?>

Output

>php freebase-read.php
Freebase Read Example
Array
(
    [code] => /api/status/ok
    [q1] => Array
        (
            [code] => /api/status/ok
            [result] => Array
                (
                    [/film/writer/film] => Array
                        (
                            [0] => Blade Runner
                            [1] => Impostor
                            [2] => Minority Report
                            [3] => Paycheck
                            [4] => Next
                        )

                    [id] => /topic/en/philip_k_dick
                )

        )

    [status] => 200 OK
    [transaction_id] => cache;cache04.p01.sjc1:8101;2009-04-28T00:06:54Z;0008
)

The code example shows the query as both a JSON string and in a commented-out PHP array. Programmatically, you'd probably assemble the query as an array. For playing around, I found it more convenient working directly with JSON. Also, if you're browsing Freebase databases it's possible to view the JSON query used for page display. Copying the JSON examples and pasting them directly into the script is a useful learning tool.

Freebase Query Viewer
Screen Shot of Freebase Query Viewer

Writing to Freebase

The Freebase write services API documentation is available at Chapter 6. Metaweb Write Services.

Writing is almost as straightforward as reading.

  1. Username and password are sent to the login API via a HTTP POST
  2. The cookies containing the login credentials are extracted from the received result
  3. The header for the HTTP post is assembled
  4. The MQL write statement is created
  5. The MQL write statement is placed in a query envelope
  6. The query (envelope) is sent to Freebase via a HTTP POST
  7. The results are viewed

In this example, I've already created a type named "foo" under my Freebase account using the web interface. The code creates an object in foo.

Code

<?php
 
/**
   * Definitions
   */
 
$fbase_user         = 'username';
 
$fbase_pass         = 'password';
 
$login_api_endpoint = 'https://sandbox.freebase.com/api/account/login';
 
$write_api_endpoint = 'http://sandbox.freebase.com/api/service/mqlwrite';

 
/**
   * Login
   */
 
$header = array();
 
$header['Content-Type'] = 'application/x-www-form-urlencoded';
 
$body = 'username=' . urlencode($fbase_user) . '&password=' . urlencode($fbase_pass);
 
$login_request = drupal_http_request($login_api_endpoint, $header, 'POST', $body);
 
$login_result = json_decode($login_request->data);
  if (
$login_result->messages[0]->code != '/api/status/ok/account/login') {
   
print_r($login_result);
    exit;
  }

 
/**
   * Extract cookies (containing login tokens)
   */
 
$cookies = array();
 
$header_cookie_data = explode(',', $login_request->headers['Set-Cookie']);
  foreach (
$header_cookie_data as $cookie_entry) {
    list(
$cookies[]) = explode(';', $cookie_entry,1);
  }

 
/**
   * Write
   */
 
$header = array();
 
$header['Content-Type']       = 'application/x-www-form-urlencoded';
 
$header['X-Metaweb-Request']  = 'True'
 
$header['Cookie']             = implode('; ', $cookies);
 
 
$json_create_string = '{
    "create":"unless_exists",
    "type":"/user/dalem/default_domain/foo",
    "name":"Dale was here",
    "id":null
  }'
;
 
$creation_array = json_decode($json_create_string);
 
$envelope_array = array('query' => $creation_array);
 
$envelope_json = json_encode($envelope_array);
 
$body = 'query=' . urlencode($envelope_json);

 
$write_request = drupal_http_request($write_api_endpoint, $header, 'POST', $body);

 
print_r($write_request->data);
?>

Output

>php freebase-write.php
Freebase Write Example
{
  "code": "/api/status/ok",
  "result": {
    "create": "created",
    "id": "/guid/9202a8c04000641f800000000ba145f9",
    "name": "Dale was here",
    "type": "/user/dalem/default_domain/foo"
  },
  "status": "200 OK",
  "transaction_id": "cache;cache01.sandbox.sjc1:8101;2009-04-27T23:59:07Z;0002"
}

As mentioned before, but worth repeating. This code runs against the Freebase sandbox server. Which ever server, regular or sandbox, you work on, make sure you create the "foo" object on the same server the login and write api endpoints ($login_api_endpoint and $write_api_endpoint) are defined for.

Acknowledgments

My thanks to Vancouver developer Jim Pick for his pointers getting started, and to Freebase user jamie for his post, Creating Freebase applications with PHP5.

AttachmentSize
Plain text icon freebase-read.php.txt1.44 KB
Plain text icon freebase-write.php.txt2.3 KB

Comments

What a fantastic post, Dale! We'll have a Freebase module in no time :P

At the Semantic Web meet-up in San Francisco last night. The room was almost entirely Drupalers.

The only new technology I wasn't familiar with was Freebase. In true Drupal fashion, I learn about the latest coolest thing, go to bed thinking about it.

When I wake up, someone in the community has already started working on it.

Great job Dale!

Cheers,
Kieran

I've been meaning to dig into freebase and do some cool things. I'll reference this article if and when I get around to it. Great stuff!

Good compilation. Thanks. - Mike

Thanks for shedding light on Freebase. This post has aroused me to learn Freebase. Tips on 'writing to Freebase' is very useful. Heading on to learn Freebase right away! - Simon