Cloning Internet Applications with Ruby
We stand on the shoulders of giants. This has been true since the time of Newton (and even before) and it is certainly true now. Much of what we know and learn of programming, we learnt from the pioneering programmers before us and what we leave behind to future generations of programmers is our hard-earned experience and precious knowledge. This book is all about being the scaffolding upon which the next generation of programmers stands when they build the next Sistine Chapel of software.
There are many ways that we can build this scaffolding but one of the best ways is simply to copy from what works. Many programming books attempt to teach with code samples that the readers can reproduce and try it out themselves. This book goes beyond code samples. The reader doesn’t only copy snippets of code or build simple applications but have a chance to take a peek at how a few of the most popular Internet applications today can possibly be built. We explore how these applications are coded and also the rationale behind the way they are designed. The aim is to guide the programmer through the steps of building clones of the various popular Internet applications.
What This Book Covers
Chapter 1, Cloning Internet Applications gives a brief description of the purpose of the book, the target readers of the book, and a list of the four popular Internet applications we will be cloning in the subsequent chapters. The bulk of this chapter gives a brief rundown
on the various technologies we will be using to build those clones.
Chapter 2, URL Shorteners – Cloning TinyURL explains about the first popular Internet application that we investigate and clone in the book, which is TinyURL. This chapter describes how to create a TinyURL clone, its basic principles, and algorithms used
Chapter 3, Microblogs – Cloning Twitter. The clone in this chapter emulates one of the hottest and most popular Internet web applications now—Twitter. It describes the basic principles of a microblogging application and explains how to recreate a feature-complete
Twitter clone. Chapter 4, Photo -sharing – Cloning Flickr. Flickr is one of the most popular and enduring photo-sharing applications on the Internet. This chapter describes how the reader can re-create a feature complete photo-sharing application the simplest way possible, following the interface and style in Flickr.
Chapter 5, Social Networking Services – Cloning Facebook 1. The final two chapters describe the various aspects of Internet social networking services, focusing on one of the most popular out there now—Facebook. These two chapters also describe the minimal features of a social networking service and show the reader how to implement these features in a complete step-by-step guide. The first part is described in this chapter, which sets the groundwork for the clone and proceeds to describe the data model used in the clone.
Chapter 6, Social Networking Services – Cloning Facebook 2. The final chapter is part two in describing how to create a Facebook clone. This chapter follows on the previous chapter and describes the application flow of the Facebook clone we started earlier.
Social Networking Services – Cloning Facebook 1
One of the most dominant Internet services today is the social networking service. According to a report by the Nielsen Company, in January 2010, the amount of time an average person spent on Facebook is more than seven hours per month, which amounts to more than 14 minutes per day. If you lump together the time spent on Google, Yahoo!, YouTube, Bing, Wikipedia, and Amazon, it still doesn’t beat Facebook! By March 2010, Facebook accounted for more than seven percent of all Internet traffic in the United States, surpassing visits to Google. Social networking services have risen in the past few years to be more than just a passing fad, to be an important communications tool as well as a part of daily life.
We will be building our last and most complex clone based on Facebook, the most popular social networking service as of date. The clone we will build here will be described over this and the next chapter. In this chapter we will cover basic information about social networking services, main features of the clone that we will build, as well as the description of the data model we will be using for the clone.
All about social networking services
A social networking service is an Internet service that models social relationships among people. Essentially it consists of a user profile, his or her social links, and a variety of additional services. Most social networking services are web-based and provide various ways for users to interact over the Internet, including sharing content and communications.
Early social networking websites started in the form of generalized online communities such as The WELL (1985), theglobe.com (1994), GeoCities (1994), and Tripod (1995). These early communities focused on communications through chat rooms, and sharing personal information and topics via simple publishing tools. Other communities took the approach of simply having people link to each other via e-mail addresses. These sites in cluded Classmates (1995), focusing on ties with former schoolmates, and SixDegrees (1997), focusing on indirect ties.
SixDegrees.com in a way was the first to bring together the first few defining features of a social networking service. The basic features of the first online social networking services include user profiles, adding friends to a friends list, and sending private messages. Unfortunately, SixDegrees was too advanced for its time and eventually closed in 2001.
Interestingly the most popular social networking service in Korea, CyWorld, was started around this time in 1999. The original intention for CyWorld was to develop an online dating service similar to Match and provide an open public meeting place for users to meet online. In 2001, CyWorld launched the minihompy service, a feature that allows each user to create a virtual homepage. This was highly successful as celebrities and politicians took to this platform to reach out to their fans and audience. CyWorld also eventually included a virtual currency called “dottori” in 2002 and a mobile version in 2004. Up to 2008, CyWorld had more than one third of Korea’s entire population as members with a strong penetration of ninety percent in the young adults market.
Between 2002 and 2004, a few social networking services became highly popular. Friendster, started by Jon Abraham in 2002 to compete with Match .com , was highly successful initially. However due to platform and scalability issues, its popularity plummeted as newer social networking services were launched. MySpace , launched in 2003, was started as a Friendster alternative and became popular with independent rock bands from Los Angeles as promoters used the platform to advertise VIP passes for popular clubs. Subsequently, MySpac e facilitated a two-way conversation between bands and their fans, and music became the growth engine of MySpace. MySpace also introduced the concept of allowing users to personalize their pages and to generate unique layouts and backgrounds. Eventually MySpace became the most dominant social networking service in U.S. until Facebook took over in 2009.
Mixiis the largest online social networking service in Japan with a total of 20 million users to date and over ninety percent of users being Japanese. Launched in February 2004 by founder KenjiKasahara, the focus of Mixiis to enable users to meet new people who share common interests. An interesting feature of Mixi(counterintuitive) is that it’s an invitation by friend social network, which means that a new user can only join Mixithrough an invitation by an existing user. This feature is only found in niche and private social networks such as http://www.asmallworld.net, a successful social networking service that caters to celebrities and high net worth individuals. This invitation-based model holds the user responsible for who they invite, and thus reduces unwanted behavior within the network, refl ecting Japanese culture itself.
Social networking began to emerge as a part of business Internet strategy at around 2005 when Yahoo! launched Yahoo! 360 , its first attempt at a social networking service. In July 2005 News Corporation bought MySpace. It was around this time as well that the first mainland Chinese social networks started. The three most notable examples in chronological order are 51.com (2005), Xiaonei(2005), and Kaixin001 (2008). 51.com drew its inspiration from CyWorld, and later MySpace and QQ. On the other hand, Xiaoneihas a user interface that follows Facebook, though it also offers the user flexibility to change the look and feel, similar to MySpace. Kaixin001, the latest social networking platform in China with the fastest growing number of users, started in 2008 and the platform and user interface are remarkably similar to Facebook.
It was also around this time that more niche social networking services focusing on specific demographics sprang up, with the most successful example being LinkedIn, which focused on business professionals. At the same time media content sharing sites began slowly incorporated social networking service features and became social networking services themselves. Examples include QQ (instant messaging), Flickr (photo-sharing), YouTube (video-sharing), and Last.FM (music sharing).
As mentioned earlier, as of early 2010 social networking services are the dominant service and purpose for many users on the Internet, with Internet traffic in US surpassing the previous giant of the Internet.
Facebook is the most dominant social networking service till date, with 400 million active users, 5 billion pieces of content shared each week, and more than 100 million active users concurrently accessing Facebook through their mobile devices. It is also the most widespread, with 70 percent of its users from outside of US, its home market.
Mark Zuckerberg and some of his Harvard college roommates launched Facebook in February 2004. Initially intended as an online directory for college students (the initial membership was limited to Harvard College students) it was later expanded to include other colleges, then high schools, and finally anyone around the world who is 13 years old and above.
Facebook features are typically that of many social networks that were created around that time. The more prominent ones are the Wall (a space on every user’s profile five friends to post messages on), pokes (which allows users to virtually poke each other, that is to notify a user that they have been poked), photo uploading, sharing, and status updates, which allow users to inform their friends of their whereabouts and what they were doing. Over time, Facebook included features to form virtual groups, to blog, to start events, chat with instant messaging, and even send virtual gifts to friends.
Facebook launched Facebook Platform in May 2007, providing a framework for software developers to create applications that interact with Facebook. It soon became wildly popular, and within a year 400,000 developers have registered for the platform, and built 33,000 applications. As of writing date there are more than 500,000 active applications in Facebook, developed by more than 1 million developers and there are more than 250 applications with more than 1 million monthly active users!
In this chapter we will be cloning Facebook and creating an application called Colony, which has the basic but essential features of Facebook.
Main features
Online social networking services are complex applications with a large number of features. However, these features can be roughly grouped into a few common categories:
- User
- Community
- Content-sharing
- Developer
User features are features that relate directly to and with the user. For example, the ability to create and share their own profiles, and the ability to share status and activities are user features. Community features are features that connect users with each other. An example of this is the friends list feature, which shows the number of friends a user has connected with in the social network.
Content sharing features are quite easy to understand. These are features that allow a user to share his self-created content with other users, for example photo sharing or blogging. Social bookmarking features are those features that allow users to share content they have discovered with other users, such as sharing links and tagging items with labels. Finally, developer features are features that allow external developers to access the services and data in the social networks.
While the social networking services out in the market often try to differentiate themselves from each other in order to gain an edge over their competition, in this chapter we will be building a stereotypical online social networking service. We will be choosing only a few of the more common features in each category, except for developer features, which for practical reasons will not be implemented here.
Let’s look at these features we will implement in Colony, by category.
User
User features are features that relate directly to users:
- Users’ activities on the system are broadcast to friends as an activity feed.
- Users can post brief status updates to all users.
- Users can add or remove friends by inviting them to link up. Friendship in
both ways need to be approved by the recipient of the invitation.
Community
Community features connect users with each other:
Content sharing
Content sharing features allow users to share content, either self-generated or discovered, with other users:
- Users can create albums and upload photos to them
- Users can create standalone pages belonging to them or attached pages belonging to events and groups
You might notice that some of the features in the previous chapters are similar to those here. This should not be surprising. Online social networking services grew from existing communications and community services, often evolving and incorporating features and capabilities from those services. The approach adopted in this book is no different. We will be using some of the features we have built in the previous chapter and adapt them accordingly for Colony.
For the observant reader you might notice that the previous chapters have clones that end with clone. The original name of this clone during writing was Faceclone, but apparently Facebook has trademarked Face for many of its applications. In order to avoid any potential trademark issues, ichose Colony instead.
Designing the clone
Now that we have the list of features that we want to implement for Colony, let’s start designing the clone. The design and implementation of this clone will be described over this and the next chapter. We will start with the data model in this chapter and move on to describing the application flow and deployment with the next chapter.
Authentication, access control, and user management
Authentication, access control, and user management are handled much the same as in previous chapters. As with the other clones, authentication is done through RPX, which means we delegate authentication to a third party provider such as Google, Yahoo!, or Facebook. Access control however is still done by Colony, while user management functions are shared between the a uthentication provider and Colony.
Access control in Colony is done on all data, which prevents user from accessing data that they are not allowed to. This is done through control of the user account, to which all other data for a user belongs. In most cases a user is not allowed access to any data that does not belong to him/her (that is not shared to everyone). In some cases though access is implicit; for example, an event is accessible to be viewed only if you are the organizer of the event. Note that unlike Photoclone, which has public pages, there are no public pages in Colony.
As before, user management is a shared responsibility between the third party provider and the clone. The provider handles password management and general security while Colony stores a simple set of profile information for the user.
Status updates
Allowing you to send status updates about yourself is a major feature of all social networking services. This feature allows the user, a member of the social networking service, to announce and define his presence as well as state of mind to his network. If you have gone through the Twitter clone chapter, you might notice that this feature is almost the same as the one in Tweetclone.
The major difference in the features is in who can read the statuses, which can be quite subtle yet obvious to someone who has read the previous chapters. In Tweetclone, the user’s followers can read the statuses (which is really just anyone who chooses to follow him or her) while in Colony, only the user’s friends can read the statuses. Remember that a user’s friend is someone validated and approved by the user and not just anyone off the street who happens to follow that user.
Status updates belong to a single user but are viewable to all friends as a part of the user’s activity feed.
User activity feeds and news feeds
Activity feeds, activity streams, or life streams are continuous streams of information on a user’s activities. Activity feeds go beyond just status updates; they are a digital trace of a user’s activity in the social network, which includes his status updates. This include public actions like posting to a wall, uploading photos, and commenting on content, but not private actions like sending messages to individuals. The user’s activity feed is visible to all users who visit his user page.
Activity feeds are a subset of news feeds that is an aggregate of activity feeds of the user and his network. News feeds give an insight into the user’s activities as well as the activities of his network. In the design of our clone, the user’s activity feed is what you see when you visit the user page, for example http://colony.saush.com/user/sausheong, while the news feed is what you see when you first log in to Colony, that’s the landing page. This design is quite common to many social networking services.
Friends list and inviting users to join
One of the reasons why social networking services are so wildly successful is the ability to reach out to old friends or colleagues, and also to see friends of your friends. To clone this feature we provide a standard friends list and an option to search for friends. Searching for friends allows you to find other users in the system by their nicknames or their full names. By viewing a user’s page, we are able to see his friends and therefore see his friend’s user pages as well.
Another critical feature in social networking services is the ability to invite friends and spread the word around. In Colony we tap on the capabilities of Facebook and invite friends who are already on Facebook to use Colony. While there is a certain amount of irony (using another social networking service to implement a feature of your social networking service), it makes a lot of practical sense, as Facebook is already one of the most popular social networking services on the planet. To implement this, we will use Facebook Connect. However, this means if the user wants to reach out and get others to join him in Colony he will need to log into Facebook to do so.
As with most features, the implementation can be done in many ways and Facebook Connect (or any other type of third-party integration for that matter) is only one of them. Another popular strategy is to use web mail clients such as Yahoo! Mail or Gmail, and extract user contacts with the permission of the user. The e-mails extracted this way can be used as a mailing list to send to potential users. This is in fact a strategy used by Facebook.
Posting to the wall
A wall is a place where users can post messages. Walls are meant to be publicly read by all visitors. In a way it is like a virtual cork bulletin board that users can pin their messages on to be read by anyone. Wall posts are meant to be short public messages. The Messages feature can be used to send private messages.
A wall can belong to a user, an event, or a group and each of these owning entities can have only one wall. This means any post sent to a user, event, or group is automatically placed on its one and only wall.
A message on a wall is called a post, which in Colony is just a text message (Facebook’s original implementation was text only but later extended to other types of media). Posts can be remarked on and are not threaded. Posts are placed on the wall in a reverse chronological order in a way that the latest post remains at the top of the wall.
Sending messages
The messaging feature of Colony is a private messaging mechanism. Messages are sent by senders and received by recipients. Messages that are received by a user are placed into an inbox while messages that the user sent are placed into a sent box. For Colony we will not be implementing folders so these are the only two message folders that every user has.
Messages sent to and received from users are threaded and ordered by time. We thread the messages in order to group different messages sent back and forth as part of an ongoing conversation. Threaded messages are sorted in chronological order, where the last received message is at the bottom of the message thread.
Attending events
Events can be thought of as locations in time where people can come together for an activity. Social networking services often act as a nexus for a community so organizing and attending events is a natural extension of the features of a social networking service. Events have a wall, venue, date, and time where the event is happening, and can have event-specific pages that allow users to customize and market their event.
In Colony we categorize users who attend events by their attendance status. Confirmed users are users who have confirmed their attendance. Pending users are users who haven’t yet decided to attend the event. Declined users are users who have declined to attend the event after they have been invited. Declinations are explicit; there is an invisible group of users who are in none of the above three types.
Attracting users to events or simply keeping them informed is a critical part of making this or any feature successful. To do so, we suggest events to users and display the suggested events in the user’s landing page. The suggestion algorithm is simple, we just go through each of the user’s friends and see which other events they have confirmed attending, and then suggest that event to the user.
Besides suggestions, the other means of discovering events are through the activity feeds (whenever an event is created, it is logged as an activity and published on the activity feed) and through user pages, where the list of a user’s pages are also displayed. All events are public, as with content created within events like wall posts and pages.
Forming groups
Social networking services are made of people and people have a tendency to form groups or categories based on common characteristics or interests. The idea of groups in Colony is to facilitate such grouping of people with a simple set of features. Conceptually groups and events are very similar to each other, except that groups are not time-based like events, and don’t have a concept of attendance. Groups have members, a wall, and can have specific pages created by the group.
Colony’s capabilities to attract users to groups are slightly weaker than in events. Colony only suggests groups in the groups page rather than the landing page. However, groups also allow discovery through activity feeds and through user pages. Colony has only public groups and no restriction on who can join these public groups.
Commenting on and liking content
Two popular and common features in many consumer focused web applications are reviews and ratings. Reviews and ratings allow users to provide reviews (or comments) or ratings to editorial or user-generated content. The stereotypical review and ratings feature is Amazon.com’s book review and rating, which allows users to provide book reviews as well as rate the book from one to five stars.
Colony’s review feature is called comments. Comments are applicable to all usergenerated content such as status updates, wall posts, photos, and pages. Comments provide a means for users to review the content and give critique or encouragement to the content creator.
Colony’s rating feature is simple and follows Facebook’s popular rating feature, called likes. While many rating features provide a range of one to five stars for the users to choose, Colony (and Facebook) asks the user to indicate if he likes the content. There is no dislike though, so the fewer number of likes a piece of content, the less popular it is.
Colony’s comments and liking feature is applicable to all user-generated content such as statuses, photos, wall posts, activities, and pages.
Sharing photos
Photos are one of the most popular types of user-generated content shared online, with users uploading 3 billion photos a month on Facebook; it’s an important feature to include in Colony. The photo-sharing feature in Colony is similar to the one in Photoclone.
The basic concept of photo sharing in Colony is that each user can have one or more albums and each album can have one or more photos. Photos can be commented, liked, and annotated. Unlike in Photoclone, photos in Colony cannot be edited.
Blogging with pages
Colony’s pages are a means of allowing users to create their own full-page content, and attach it to their own accounts, a page, or a group. A user, event, or group can own one or more pages. Pages are meant to be user-generated content so the entire content of the page is written by the user. However in order to keep the look and feel consistent throughout the site, the page will be styled according to Colony’s look
and feel. To do this we only allow users to enter Markdown, a lightweight markup language that takes many cues from existing conventions for marking up plain text in e-mail. Markdown converts its marked-up text input to valid, well-formed XHTML. We use it here in Colony to let users write content easily without worrying about layout or creating a consistent look and feel.
Technologies and platforms used
We use a number of technologies in this chapter, mainly revolving around the Ruby programming language and its various libraries. Most of them have been described in Chapter 1. In addition to Ruby and its libraries we also use mashups, which are described next.
Mashups
As with previous chapters, while the main features in the applications are all implemented within the chapters itself, sometimes we still depend on other services provided by other providers. In this chapter we use four such external services—RPX for user web authentication, Gravatar for avatar services, Amazon Web Services S3 for photo storage, and Facebook Connect for reaching out to users on Facebook. RPX, Gravatar, and AWS S3 have been explained in previous chapters.
Facebook Connect
Facebook has a number of technologies and APIs used to interact and integrate with their platform, and Facebook Connect is one of them. Facebook Connect is a set of APIs that let users bring their identity and information into the application itself. We use Facebook Connect in this chapter to send out requests to a user’s friends, inviting them to join our social network. The steps to integrate with Facebook Connect are detailed in Chapter 6, Social Networking Services — Cloning Facebook 2.
Note that for the user invitation feature, once a user has logged in through Facebook with RPX, he is considered to have logged into Facebook Connect and therefore can send invitations immediately without logging in again.
Building the clone
This is the largest clone built in the book and has many components. Unlike the previous chapters where all the source code are listed in the chapter itself, some of the less interesting parts of the code are not listed or described here. To get access to the full source code please go to http://github.com/sausheong/Colony
Configuring the clone
We use a few external APIs in Colony so we need to configure our access to these APIs. In a Colony all these APikeys and settings are stored in a Ruby file called config.rb , shown as below:
S3_CONFIG = {}
S3_CONFIG['AWS_ACCESS_KEY'] = '<AWS ACCESS KEY>'
S3_CONFIG['AWS_SECRET_KEY'] = '<AWS SECRET KEY>'
RPX_API_KEY = '<RPX APiKEY>'
Modeling the data
This is the chapter with the largest number of classes and relationships. A few major classes you see here are similar but not exactly the same as the ones in the previous chapters, so if you have gone through those chapters you would roughly know how it works.
The following diagram shows how the clone is modeled:
User
As before the first class we look at is the User class. If you have followed the previous chapters you will have realized that that this class is very similar to the ones before. However, the main differences are that there are more relationships with other classes and the relationship with other users follows that of a friends model rather than a followers model.
class User
include DataMapper::Resource
property :id, Serial
property :email, String, :length => 255
property :nickname, String, :length => 255
property :formatted_name, String, :length => 255
property :sex, String, :length => 6
property :relationship_status, String
property :provider, String, :length => 255
property :identifier, String, :length => 255
property :photo_url, String, :length => 255
property :location, String, :length => 255
property :description, String, :length => 255
property :interests, Text
property :education, Text
has n, :relationships
has n, :followers, :through => :relationships, :class_name =>
'User', :child_key => [:user_id]
has n, :follows, :through => :relationships, :class_name => 'User',
:remote_name => :user, :child_key => [:follower_id]
has n, :statuses
belongs_to :wall
has n, :group s, :through => Resource
has n, :sent_messages, :class_name => 'Message', :child_key =>
[:user_id]
has n, :received_messages, :class_name => 'Message', :child_key =>
[:recipient_id]
has n, :confirms
has n, :confirmed_events, :through => :confirms, :class_name =>
'Event', :child_key => [:user_id], :date.gte => Date.today
has n, :pendings
has n, :pending_events, :through => :pendings, :class_name =>
'Event', :child_key => [:user_id], :date.gte => Date.today
has n, :requests
has n, :albums
has n, :photos, :through => :albums
has n, :comments
has n, :activities
has n, :pages
validates_is_unique :nickname, :message => "Someone else has taken
up this nickn ame, try something else!"
after : create, :create_s3_bucket
after :create, :create_wall
def add_friend(user)
Relationship.create(:user => user, :follo wer => self)
end
def friends
(followers + follows).uniq
end
def self.find(identifier)
u = first(:identifier => identifier)
u = new(:identifier => identifier) if u.n il?
return u
end
def feed
feed = [] + activities
friends.each do |friend|
feed += friend.activities
end
return feed.sort {|x,y| y.created_at <=> x.created_at}
end
def possessive_pronoun
sex.downcase == 'male' ? 'his' : 'her'
end
def pronoun
sex.downcase == ' male' ? 'he' : 'she'
end
def create_s3_bucket
S3.create_ bucket("fc.#{id}")
end
def create_wall
self.wall = Wall.create
self.save
end
def all_events
confirmed_eve nts + pending_events
end
def friend_events
events = []
friends.each do |friend|
events += friend.confirmed_events
end
return events.sort {|x,y| y.time <=> x.time}
end
def friend_groups
groups = []
friends.each do |friend|
groups += friend.groups
end
groups – self.groups
end
end
As mentioned in the design section above, the data used in Colony is user-centric. All data in Colony eventually links up to a user. A user has the following relationships with other models:
- A user has none, one, or more status updates
- A user is associated with a wall
- A user belongs to none, one, or more groups
- A user has none, one, or more sent and received messages
- A user has none, one, or more confirmed and pending attendances at events
- A user has none, one, or more user invitations
- A user has none, one, or more albums and in each album there are none, one, or more photos
- A user makes none, one, or more comments
- A user has none, one, or more pages
- A user has none, one, or more activities
- Finally of course, a user has one or more friends.
Once a user is created, there are two actions we need to take. Firstly, we need to create an Amazon S3 bucket for this user, to store his photos.
after :create, :create_s3_bucket
def create_s3_bucket
S3.create_bucket("fc.#{id}")
end
We also need to create a wall for the user where he or his friends can post to.
after :create, :create_wall def create_wall self.wall = Wall.create self.save end
Adding a friend means creating a relationship between the user and the friend.
def add_friend(user) Relationship.create(:user => user, :follower => self) end
You might realize that this was a follows relationship in the previous chapters, so you might ask how could it go both ways? The answer to this question will be clearer once the discussion turns to sending a request to connect. In short, Colony treats both following and follows relationships as going both ways—they are both considered as a friends relationship. The only difference here is who will initiate the request to join. This is why when we ask the User object to give us its friends, it will add both followers and follows together and return a unique array representing all the user’s friends.
def friends (followers + follows).uniq end
The Relationship class is almost the same as the one used in the other chapters, except that each time a new relationship is created, an Activity object is also created to indicate that both users are now friends.
class Relationship
include DataMapper::Resource
property :user_id, Integer, :key => true
property :follower_id, Integer, :key => true
belongs_to :user, :child_key => [:user_id]
belongs_to :follower, :class_name => 'User', :child_key =>
[:follower_id]
after :save, :add_activity
def add_activity
Activity.create(:user => user, :activity_type => 'relationship',
:text => "<a href='/user/#{user.nickname}'>#{user.formatted_name}</a>
and <a href='/user/#{follower.nickname}'>#{follower.formatted_name}</
a> are now friends.")
end
end
Finally we get the user’s news feed by taking the user’s activities and going through each of the user’s friends, their activities as well.
def feed
feed = [] + activities
friends.each do |friend|
feed += friend.activities
end
return feed.sort {|x,y| y.created_at <=> x.created_at}
end
Request
We use a simple mechanism for users to invite other users to be their friends. The mechanism goes like this:
- Alice identifies another Bob whom she wants to befriend and sends him an invitation.
- This creates a Request class which is then attached to Bob.
- When Bob approves the request to be a friend, Alice is added as a friend (which is essentially making Alice follow Bob, since the definition of a friend in Colony is someone who is either a follower or follows another user).
class Request include DataMapper::Resource property :id, Serial property :text, Text property :created_at, DateTime belongs_to :from, :class_name => User, :child_key => [ :from_id] belongs_to :user def approve self.user.add_friend(self.from) end end
Message
Messages in Colony are private messages that are sent between users of Colony. As a result, messages sent or received are not tracked as activities in the user’s activity feed.
class Message include DataMapper::Resource property :id, Serial property :subject, String property :text, Text property :created_at, DateTime property :read, Boolean, :default => false property :thread, Integer belongs_to :sender, :class_name => 'User', :child_key => [:user_id] belongs_to :recipient, :class_name => 'User', :child_key => [:recipient_id] end
A message must have a sender and a recipient, both of which are users.
has n, :sent_messages, :class_name => 'Message', :child_key => [:user_ id] has n, :received_messages, :class_name => 'Message', :child_key => [:recipient_id]
The read property tells us if the message has been read by the recipient, while the thread property tells us how to group messages together for display.
Album
The photo sharing capabilities of Colony is transplanted from Photoclone in the previous chapter and therefore the various models involved in photo sharing are almost the same as the one in Photoclone. The main difference is that each time an album is created, an activity is logged.
class Album
include DataMapper::Resource
property :id, Serial
property :name, String, :length => 255
property :description, Text
property :created_at, DateTime
belongs_to :user
has n, :photos
belongs_to :cover_photo, :class_name => 'Photo', :child_key =>
[:cover_photo_id]
after :save, :add_activity
def add_activity
Activity.create(:user => user, :activity_type => 'album', :text =>
"<a href='/user/#{user.nickname}'>#{user.formatted_name}</a> created a
new album <a href='/album/#{self.id}'>#{self.name}</a>")
end
end
Photo
The Photo class is the main class in the photo-sharing feature of Colony. Just like the Album class, this is very similar to the one in Photoclone, except for some minor differences.
class Photo
include DataMapper::Resource
include Commentable
attr_writer :tmpfile
property :id, Serial
property :title, String, :length => 255
property :caption, String, :length => 255
property :privacy, String, :default => 'public'
property :format, String
property :created_at, DateTime
belongs_to :album
has n, :annotations
has n, :comments
has n, :likes
after :save, :save_image_s3
after :create, :add_activity
after :destroy, :destroy_image_s3
def filename_display; "#{id}.disp"; end
def filename_thumbnail; "#{id}.thmb"; end
def s3_url_thumbnail; S3.get_link(s3_bucket, filename_thumbnail,
Time.now.to_i+ (24*60*60)); end
def s3_url_display; S3.get_link(s3_bucket, filename_display, Time.
now.to_i+ (24*60*60)); end
def url_thumbnail
s3_url_thumbnail
end
def url_display
s3_url_display
end
def previous_in_album
photos = album.photos
index = photos.index self
return nil unless index
photos[index - 1] if index > 0
end
def next_in_album
photos = album.photos
index = photos.index self
return nil unless index
photos[index + 1] if index < album.photos.length
end
def save_image_s3
return unless @tmpfile
img = Magick::Image.from_blob(@tmpfile.open.read).first
display = img.resize_to_fit(500, 500)
S3.put(s3_bucket, filename_display, display.to_blob)
t = img.resize_to_fit(150, 150)
length = t.rows > t.columns ? t.columns : t.rows
thumbnail = t.crop(CenterGravity, length, length)
S3.put(s3_bucket, filename_thumbnail, thumbnail.to_blob)
end
def destroy_image_s3
S3.delete s3_bucket, filename_display
S3.delete s3_bucket, filename_thumbnail
end
def s3_bucket
"fc.#{album.user.id}"
end
def add_activity
Activity.create(:user => album.user, :activity_type => 'photo',
:text => "<a href='/user/#{album.user.nickname}'>#{album.user.
formatted_name}</a> added a new photo - <a href='/photo/#{self.
id}'><img class='span-1' src='#{self.url_thumbnail}'/></a>")
end
end
First of all, we removed the feature of storing temporary file caches on the filesystem of the server. The main reason is that of economy—we want to be able to deliver everything from Amazon S3 deploy on the Heroku cloud platform (which does not serve files). Of course this can be changed easily if you’re planning to customize Colony.
Next, and related to the first difference, is that we no longer store the original photo. Instead, we only keep a reduced-size display photo and a thumbnail of the original photo. The rationale for this is the same as with Facebook. Colony is not a full-fl edged photo-sharing site for photographers and is meant to share photos with friends only. Therefore storing large original files is unnecessary.
Photos can be commented on so it includes the Commentable module (explained later). Also each photo has none, one, or more comments and likes.
Finally as with many of the classes in Colony, creating a Photo is considered an activity and is logged for streaming on the activity feed. Note that we don’t log an activity after a save, but only after we create an object photo. This is because save will be called each time the photo object is edited, annotated, or has its caption or description modified. This is not an activity we want to log in to the activity stream.
class Annotation
include DataMapper::Resource
property :id, Serial
property :description,Text
property :x, Integer
property :y, Integer
property :height, Integer
property :width, Integer
property :created_at, DateTime
belongs_to :photo
after :create, :add_activity
def add_activity
Activity.create(:user => self.photo.album.user, :activity_type
=> 'annotation', :text => "<a href='/user/#{self.photo.album.user.
nickname}'>#{self.photo.album.user.formatted_name}</a> annotated
a photo - <a href='/photo/#{self.photo.id}'><img class='span-1'
src='#{self.photo.url_thumbnail}'/></a> with '#{self.description}'")
end
end
Annotation is another class, part of the photo-sharing feature that is transplanted from Photoclone, with activity logging added in. We will not go into this, if you want a refresher please read Chapter 4.
Status
Just as the Album, Photo, and Annotation classes are transplanted from Photoclone, the Status and Mention classes are derived from Tweetclone.
class Status
include DataMapper::Resource
include Commentable
property :id, Serial
property :text, String, :length => 160
property :created_at, DateTime
belongs_to :recipient, :class_name => "User", :child_key =>
[:recipient_id]
belongs_to :user
has n, :mentions
has n, :mentioned_users, :through => :mentions, :class_name =>
'User', :child_key => [:user_id]
has n, :comments
has n, :likes
before :save do
@mentions = []
process
end
after :save do
unless @mentions.nil?
@mentions.each {|m|
m.status = self
m.save
}
end
Activity.create(:user => user, :activity_type => 'status', :text
=> self.text )
end
# general scrubbing
def process
# process url
urls = self.text.scan(URL_REGEXP)
urls.each { |url|
tiny_url = RestClient.get "http://tinyurl.com/api-create.
php?url=#{url[0]}"
self.text.sub!(url[0], "<a href='#{tiny_url}'>#{tiny_url}</a>")
}
# process @
ats = self.text.scan(AT_REGEXP)
ats.each { |at|
user = User.first(:nickname => at[1,at.length])
if user
self.text.sub!(at, "<a href='/#{user.nickname}'>#{at}</a>")
@mentions << Mention.new(:user => user, :status => self)
end
}
end
def starts_with?(prefix)
prefix = prefix.to_s
self.text[0, prefix.length] == prefix
end
def to_json(*a)
{'id' => id, 'text' => text, 'created_at' => created_at, 'user' =>
user.nickname}.to_json(*a)
end
end
As before, each time a user updates his status, an activity will be logged. Statuses can be commented upon and also liked. The Mention class is unchanged from Tweetclone. For an in-depth description of this class please refer to Chapter 3.
class Mention
include DataMapper::Resource
property :id, Serial
belongs_to :user
belongs_to :status
end
URL_REGEXP = Regexp.new('\b ((https?|telnet|gopher|file|wais|ftp) :
[\w/#~:.?+=&%@!\-] +?) (?=[.:?\-] * (?: [^\w/#~:.?+=&%@!\-]| $ ))',
Regexp::EXTENDED)
AT_REGEXP = Regexp.new('@[\w.@_-]+', Regexp::EXTENDED)
Group
Each user can belong to none, one, or more groups. Each group that is created also belongs to a user and it’s this user that the activity is logged to. Each group has a set of features:
- A group can have none, one, or more pages.
- A group has a wall where other users can post to. This wall is created right after the group is created.
class Group
include DataMapper::Resource
property :id, Serial
property :name, String
property :description, String
has n, :pages
has n, :members, :class_name => 'User', :through => Resource
belongs_to :user
belongs_to :wall
after :create, :create_wall
def create_wall
self.wall = Wall.create
self.save
end
after :create, :add_activity
def add_activity
Activity.create(:user => self.user, :activity_type => 'event',
:text => "<a href='/user/#{self.user.nickname}'>#{self.user.
formatted_name}</a> created a new group - <a href='/group/#{self.
id}'>#{self.name}</a>.")
end
end
Note that the User-Group relationship is a many-to-many relationship, and we use the DataMapper::Resource class as an anonymous class to represent the relationship. For convenience we also provide a method in the User object to retrieve all groups a user’s friends belong to. This becomes useful for us later when suggesting groups for users to join.
def friend_groups groups = [] friends.each do |friend| groups += friend.groups end groups - self.groups end
Event
Events are quite similar to Groups but with a twist. As before we log it as an activity each time the event is created. Each event has an administrative user who is the person who created the event.
class Event
include DataMapper::Resource
property :id, Serial
property :name, String
property :description, String
property :venue, String
property :date, DateTime
property :time, Time
belongs_to :user
has n, :pages
has n, :confirms
has n, :confirmed_users, :through => :confirms, :class_name =>
'User', :child_key => [:event_id], :mutable => true
has n, :pendings
has n, :pending_users, :through => :pendings, :class_name => 'User',
:child_key => [:event_id], :mutable => true
has n, :declines
has n, :declined_users, :through => :declines, :class_name =>
'User', :child_key => [:event_id], :mutable => true
belongs_to :wall
after :create, :create_wall
def create_wall
self.wall = Wall.create
self.save
end
after :create, :add_activity
def add_activity
Activity.create(:user => self.user, :activity_type => 'event',
:text => "<a href='/user/#{self.user.nickname}'>#{self.user.formatted_
name}</a> created a new event - <a href='/event/#{self.id}'>#{self.
name}</a>.")
end
end
In addition, each event has three types of members depending on their current attendance status:
- Users confirmed to attend the event
- Users who are still undecided on attending the event
- Users who have declined to attend the event
For this implementation we use a separate class for each type of user, that is we have a Confirm class for confirmed users, a Pending class to indicate users who are undecided, and a Decline class to indicate users who have declined to attend the event.
class Pending include DataMapper::Resource property :id, Serial belongs_to :pending_user, :class_name => 'User', :child_key => [:user_id] belongs_to :pending_event, :class_name => 'Event', :child_key => [:event_id] end class Decline include DataMapper::Resource property :id, Serial belongs_to :declined_user, :class_name => 'User', :child_key => [:user_id] belongs_to :declined_event, :class_name => 'Event', :child_key => [:event_id] end class Confirm include DataMapper::Resource property :id, Serial belongs_to :confirmed_user, :class_name => 'User', :child_key => [:user_id] belongs_to :confirmed_event, :class_name => 'Event', :child_key => [:event_id] end
As with Group, we have a convenient method in the User class to help us find the events the user’s friends are attending. We only retrieve confirmed events for this list, which is then sorted according to ascending chronological order.
def friend_events
events = []
friends.each do |friend|
events += friend.confirmed_events
end
return events.sort {|x,y| y.time <=> x.time}
end
Page
Pages are a simple means for users to publish their own web pages. A Page can be owned directly by a user, through a group, or through an event.
class Page
include DataMapper::Resource
include Commentable
property :id, Serial
property :title, String
property :body, Text property :created_at, DateTime
has n, :comments
has n, :likes
belongs_to :user
belongs_to :event
belongs_to :group
after :create, :add_activity
def add_activity
if self.event
Activity.create(:user => self.user, :activity_type => 'event
page', :text => "<a href='/user/#{self.user.nickname}'>#{self.user.
formatted_name}</a> created a page - <a href='/event/page/#{self.
id}'>#{self.title}</a> for the event <a href='/event/#{self.event.
id}'>#{self.event.name}</a>.")
elsif self.group
Activity.create(:user => self.user, :activity_type => 'group
page', :text => "<a href='/user/#{self.user.nickname}'>#{self.user.
formatted_name}</a> created a page - <a href='/group/page/#{self.
id}'>#{self.title}</a> for the group <a href='/group/#{self.group.
id}'>#{self.group.name}</a>.")
else
Activity.create(:user => self.user, :activity_type => 'page',
:text => "<a href='/user/#{self.user.nickname}'>#{self.user.formatted_
name}</a> created a page - <a href='/page/#{self.id}'>#{self.title}</
a>.")
end
end
end
Page also logs activities according to whichever object that owns it.
Wall
A wall is a place where users can place their posts. A wall can belong to a user, event, or group. In fact each time a user, event, or group is created, we will automatically create a wall on its behalf.
class Wall include DataMapper::Resource property :id, Serial has n, :posts end
The implementation of a wall by itself has no definite properties other than being a container for posts. A post is the actual content that a user will submit to a wall and it is something that can be commented and liked. A post on a wall can come from any user, so a post is also associated with the user who created the post.
class Post include DataMapper::Resource include Commentable property :id, Serial property :text, Text property :created_at, DateTime belongs_to :user belongs_to :wall has n, :comments has n, :likes end
Activity
An activity is a log of a user’s action in Colony that is streamed to the user’s activity feed. Not all actions are logged as activities, for example messages are considered private and are therefore not logged.
class Activity include DataMapper::Resource include Commentable property :id, Serial property :activity_type, String property :text, Text property :created_at, DateTime has n, :comments has n, :likes belongs_to :user end
Activities are commented and can be liked by other users.
Comment
Comments in Colony are stored and managed through the Comment class. All usergenerated content including pages, posts, photo, and statuses can be commented by users in Colony. Activities can also be commented on.
class Comment include DataMapper::Resource property :id, Serial property :text, Text property :created_at, DateTime belongs_to :user belongs_to :page belongs_to :post belongs_to :photo belongs_to :activity belongs_to :status end
Like
Like and Comment classes are very similar. The main difference between them is that the Like mechanism is binary (either you like the content or you don’t) whereas you need to provide some content to comment.
class Like include DataMapper::Resource property :id, Serial belongs_to :user belongs_to :page belongs_to :post belongs_to :photo belongs_to :activity belongs_to :status end
The implementation of the Like mechanism in Colony requires each class of objects that can be liked or commented on to include the Commentable module.
module Commentable
def people_who_likes
self.likes.collect { |l| "<a href='/user/#{l.user.nickname}'>#{l.
user.formatted_name}</a>" }
end
end
This allows you to retrieve an array of people of who likes the content, which are then formatted as HTML links for easy display.
This wraps up the data models that we will be using in Colony. In the next chapter we will cover Colony’s application flow and deployment.
Summary
This is the second last chapter in this book and also the first one in a series of two chapters describing how we can clone a social networking service like Facebook. Social networking services are the next step in the evolution of Internet applications and Facebook is currently the most successful incarnation of this service. Cloning Facebook is not difficult though, as can be attested in this chapter and also in the many Facebook ‘clones’ out there on the Internet. Let’s look at what we have covered in this chapter.
First, we went through a whirlwind tour of social networking services and their history, before discussing the most dominant service, Facebook. Next, we described some of its more essential features and we categorized the features into User, Community, and Content sharing features. After that, we went into a high level discussion on these various features and how we implement them in our Facebook clone, Colony. After that, we went briefl y into the various technologies used in the clone.
After the technology discussion, we jumped straight into the implementation, starting with a detailed discussion of the data models used in this chapter. Our next chapter is the last chapter in this book. We will finish what we have started in this chapter with a detailed step-by-step description of Colony’s application flow, followed with the deployment of Colony on the Heroku cloud platform.
