Re: [tlug] Questions about skydb

[Edward Middleton <edward.middleton@example.com>]

On 01/26/2013 11:53 PM, Edward Middleton wrote:
> Ben Johnson the creator of skydb posted answers to a few of the
> questions people asked during the meeting.  His twitter is @benbjohnson
> and if you have further questions the skydb mailing list is hosted on
> sky@example.com

I also forgot to show the code for the loader, it is structured as a
yaml file given bellow.


file_type: :tsv
headers:
- app_id
- platform
- dt
- tm
- event_name
- txn_id
- v_tracker
- v_collector
- v_etl
- user_id
- user_ipaddress
- visit_id
- page_url
- page_title
- page_referrer
- mkt_source
- mkt_medium
- mkt_term
- mkt_content
- mkt_campaign
- ev_category
- ev_action
- ev_label
- ev_property
- ev_value
- tr_orderid
- tr_affiliation
- tr_total
- tr_tax
- tr_shipping
- tr_city
- tr_state
- tr_country
- ti_orderid
- ti_sku
- ti_name
- ti_category
- ti_price
- ti_quantity
- br_name
- br_family
- br_version
- br_type
- br_renderengine
- br_lang
- br_features_pdf
- br_features_flash
- br_features_java
- br_features_director
- br_features_quicktime
- br_features_realplayer
- br_features_windowsmedia
- br_features_gears
- br_features_silverlight
- br_cookies
- os_name
- os_family
- os_manufacturer
- dvce_type
- dvce_ismobile
- dvce_screenwidth
- dvce_screenheight
fields:
  object_id: object_id:int
  timestamp: timestamp:date
  action: {}
  data: {}
translate: |
  output['object_id']

  bdb = TokyoCabinet::BDB::new
  if !bdb.open(".objects.tcb", TokyoCabinet::BDB::OWRITER |
TokyoCabinet::BDB::OCREAT)
    ecode = bdb.ecode
    STDERR.printf("open error: %s\n", bdb.errmsg(ecode))
  end
  if bdb.has_key? input['user_id']
    output['object_id'] = bdb.get(input['user_id']).to_i
  else
    id = bdb.rnum.to_i + 1
    output['object_id'] = id
    bdb.put(input['user_id'],id)
  end
  if !bdb.close
    ecode = bdb.ecode
    STDERR.printf("close error: %s\n", bdb.errmsg(ecode))
  end

  year,month,day     =* input['dt'].split(/-/)
  hour,minute,second =* input['tm'].split(/-/)
  output['timestamp'] = Time.new(year,month,day,hour,minute,second)

  output['action'] ||= {}
  case
  when input['ev_category'] == '\N'
    uri = URI.parse(input['page_url'])
    output['action']['name']       = 'page_view' + "[" + uri.path + "]"
    output['action']['protocol']   = uri.scheme
    output['action']['host']       = uri.host
    output['action']['port']       = uri.port
    output['action']['path']       = uri.path.gsub(/^\/(en|ja|tw|cn)/,'')
    output['action']['args']       = uri.query unless uri.query.nil?
    output['action']['referrer']   = input['page_referrer'] unless
input['page_referrer'] == '\N'
    output['action']['locale']     =
uri.path.gsub(/^\/(en|ja|tw|cn).*/,'\1') if uri.path =~ /^\/(en|ja|tw|cn).*/
  when input['ev_category'].downcase == 'adobeflash' &&
input['ev_action'].downcase == 'pageload'
    output['action']['name']       = 'custom.adobeflash.version'
    output['data'] ||= {}
    output['data']['flashversion'] = input['ev_value'] unless
['\N','undefined'].include?(input['ev_value'])
  when input['ev_action'].downcase == 'login'
    output['action']['name']       = 'custom.session.login'
    output['data'] ||= {}
    output['data']['logged_in']    = true
    output['data']['guid']         = input['ev_property'] unless
['\N','undefined'].include?(input['ev_property'])
    output['data']['login']        = input['ev_label']    unless
['\N','undefined'].include?(input['ev_label'])
  when input['ev_action'].downcase == 'logout'
    output['action']['name']       = 'custom.session.logout'
    output['data'] ||= {}
    output['data']['logged_in']    = true
  else
    output['action']['name']       =
"custom.#{input['ev_category']}.#{input['ev_action']}".downcase
    output['action']['page']       = input['page_url']
    ['label','property','value'].each do |atr|
      ev_key = "ev_#{atr}"
      output['action'][atr] = input[ev_key] unless
['\N','undefined'].include?(input[ev_key])
    end

  end
require: [uri,tokyocabinet]