I just finished phase one of the biggest projects I’ve ever done in my life. It’s called Heywire Guild & Gallery. While it’s still pretty new (as of this writing it’s four days old) it’s gaining steam rapidly. I wanted to document my experience while it’s all still fresh in my head.
A little over a month ago, my old friend Steve King came to me with a crazy project. Steve gets a lot of crazy projects, but this one was especially crazy. Essentially, the client wanted to build “Facebook meets eBay, but for art.”
Being a guy that builds social networking sites for a living, I usually tune out right around the time someone starts a sentence with “It’s Facebook meets…” This is because most of the time, the person saying it feels they are the only person that ever thought of harnessing the unspeakable power of “social networks” and “the Internet” for whatever their niche interest happens to be. They also usually don’t have any money, or worse, want you to work for questionable equity. (read: free) This is because they are true visionaries. Isn’t the prospect of basking in their utter awesomeness payment enough?
This, however, was different. Steve’s client (who needs to remain nameless) is a smart business man who made a lot of money in several other ventures. He has a lot of good ideas and the resources to make them happen. There were also no shady equity terms. He wanted it built and was prepared to pay for it. All good so far. Then came the kicker: The site needed to launch December 6th. I found out about it on October 19th.
Normally, I avoid projects that I feel are impossible, no matter how much I want to do them or how much I need the money. It’s better to decline work you can’t do than try to do it, fail and not get paid. I very seriously considered passing on this job considering the time constraints. However, I had just come off building a few other large sites that had a lot of similar elements, so I wouldn’t need to start from scratch. In fact, for going on a year now, I’ve slowly been building up a community application based on top of the Zend Framework. Each time I do a new site, I add to and refine the the underlying app. This seemed like a great opportunity to not only reuse earlier work, but further the platform. So I took the gig.
Step One: Tear it Down
So what I had when I started Heywire was the code from three different apps: MashupChallenge, LazyLibrary and BonAbode. I was pretty proud of how these came together, but they were all pretty site specific. MC and BA share a ton of code, but they’re still distinct apps. The first thing I wanted to do with Heywire was make the application utterly generic, so that building a site on top of it would mean writing a series of modules, plug-ins and a theme. This way, I could eventually port BA and MC to it, so I’d be maintaining one core app instead of several. I wanted to make the Wordpress of community sites, in so much as that when you customize Wordpress, you do it via plug-ins and themes, not by hacking the code base.
To me, there’s a pretty small list of things that are core to absolutely every app I build. I needed to get down to these core things and rip out anything site specific.
First, I needed an MVC framework. Zend does a nice job of this. You can easily break your app into modules, each with it’s own controllers and models. I really wanted to use Smarty for my templating solution instead of phtml, as I find the syntax pretty friendly and it has a nice precomiling and precaching system, say nothing of its rad plugin system. By implementing Zend_View_Interface, Smarty can easily and transparently supplant the view system supplied with the framework. That’s just how they roll over at Zend. Zend_DB is a really flexible way to handle the model layer, as you can be talking to a generic database table in about three lines of code. So now we’ve got the controller, view and model stuff taken care of.
A note on the model thing. I use MySQL 5 for two important reasons. Stored procedures and views. Zend_DB_Table can talk to a view as easily as it can talk to a table. (It just can’t insert or update, obviously.) I can tell you right now that there is not a single bit of in-line SQL in this application, and it’s pretty complex. Any time I have a query that makes for an overly complex where clause or needs a join, I make a new view and a new model object to go with it. This saves you the overhead of creating a huge, ridiculous query at run time and makes your code a lot more manageable. It also saves you from a lot of SQL-injection-hacky-type-stuff.
Core app functionality consists of a few important things. There are users, roles and resources. Users (”jimmy” and “pam”) have roles (”user” and “admin”). Roles can access resources. (”post blog entry” and “delete user”) Zend_Auth and Zend_Acl in combination give you user authentication and access control. It was really important to me to keep access control out of the controller logic. That way, changing what each role can do is a configuration change, not a code change.
I decided to control access at the action level. This has so far proved enough for my purposes. If I need something more granular than I can do with one action, I simple make another. To control access to these, I needed to dynamically get a list of what they are for each controller class. First I tried using the introspection API, but that proved annoying as I had to actually load each class to introspect it. I ended up going with a much quicker and dirtier solution. Knowing that Zend_Controller looks for actions based on their function name being “somethingAction”, I just regexed the text of the class files for “function [something]Action”.
I also have my own extended version of Zend_Registry that stores all its values in a db table. Whenever I need a configurable value in the application, I call App_Registry::get(’some_value_you_can_set’,'default value for this’) - this checks the registry for said key, and if it doesn’t find it, it adds it and uses the default value provided. I have a form for editing these values in the application. This is cool, because it means that when you add a new feature that requires a setting, it doesn’t break the app if the db doesn’t have it. The value may not be correct for that instance of the app, but it’s easily set in the db. My bootstrap loads the config from the db and sticks it in the registry. Why not use config files for this? Easy. There is one thing a config file should have in it, in my mind: the location of the database. EVERYTHING ELSE GOES IN THE DATABASE, WARDEN. Seriously. I have two development copies of this thing and three people working on it, I do not need to be copying instance specific config data all over the place. As far as versioning, I do not store config files in subversion. I store a template of a config file, which gets copied to the real version on install. This prevents config values getting overwritten by svn updates.
So now I’ve got my user, role, config and resource model objects and views to create and edit them. I’ve also got actions and views for registration, account editing and basic administration.
Step Two: We Can Rebuild Him
From here, I added things that probably aren’t going to get used in every single app, but would probably be used in most. Things like comments and media. As far as implementation, I made these generic, so they could be attached to users as well as anything else in the system.
At this point, we’ve got the basic elements of a community site. Users, profiles, comments, media and administration. This comprises the core app. We’ve also got a theme, which is a neatly organized set of Smarty templates in a directory tree outside of the application code directory. This way, we can swap themes out as needed without messing the core app up. It It’s also useful when you’re working with a UX person, like I was in this case, because they can make relatively easy subversion commits and updates independent of the core application.
Step Three: Add Modules, Rinse, Repeat
To expand the app from being Generic Social Network to being Heywire Guild & Gallery, I needed to start building out additional modules. I store these next to the default module and maintain them in a completely separate subversion repository for easy separation. (More on that next time.) By assuming a directory structure, I can automatically add new modules to my include path in my bootstrap. This is cool, because it means that in the future, add on modules will be easily integrated and removed.
So in some cases, modules are completely standalone, like the blog. I won’t be managing the content here, so it’s important I have an interface for the client to do it. Wordpress is easy to use and well known. For this reason, the blog is nothing more than a series of XML-RPC calls to a hidden copy of Wordpress, because, frankly, I’d rather do my own work than repeat what Matt Mullenweg did just for the sake of having it all under my own roof. I call this process “sidecarring” and I do it wherever it makes sense. In fact, the whole “CMS” in this thing just uses the pages functionality of Wordpress. I hate duplicating the work of others, especially when said work is such high quality.
In most cases, though, modules need to tie back into the default module. The “usercontent” module (my generic term for what is called “artwork” in heywire) needs to know about users. For example, when I delete a user, all of their content should also get deleted. I could of course hack the user module, but then I’d be violating my “keep it generic” rule. To overcome this, I borrow a concept from Wordpress and many other well-written apps - callback functions. In each of my controllers and model objects, whenever there’s something that could usefully be extended or filtered, I register a “hook” with in my registry. These are all accessible via a plugin object that extension code can instantiate and use. Extensions can register “actions” or “filters”. Both specify a class and a function to be called by a hook. The actions must specify a callback function that accepts an array called “params”. Filters work the same way, except that they must also return an array with the same number of keys. So for our earlier example, the “user” model object has a hook called “post_delete”, which passes passes a param array of user records that were just deleted. Usercontent has a plugin that adds an action to that hook to delete any usercontent objects belonging to that user. Using this method, I can add pretty much anything to the core app and keep it completely distinct from it at the same time.
Things I Have Learned From This
Normally I’m busy learning from my mistakes. This one time, I’m going to learn from my success, and I hope you can too.
First, if you’re rolling your own app from scratch these days, you are dumb. There’s just no other way to say it. There are no shortage of excellent application frameworks these days suited to a wide variety of different sites. What we did here simply would not have been possible without a robust, extensible application framework to do the tedious stuff, like talking to the database and such. I like Zend because it makes sense to me, but this isn’t like…Zend über alles or anything. I looked at CakePHP and Symfony before deciding on the relatively new Zend Framework. All three are very solid. I went with Zend because they have a very take it or leave it approach to everything. If you want to use the framework and nothing but the framework, it’s pretty complete. If you want to skip some stuff and add other stuff in, they make that really easy. It’s a tool, not a way of life.
Second, always build your apps generically. Wherever possible, create plugins and extend existing classes rather than hacking things for the app you’re working on. Yes, it requires more thinking, but once you get into that mindset, it will save you tons of work and headaches.
Lastly, and perhaps most important: specialize and make it easy for other specialists to work with you. Unless you’re explaining what you do to your grandmother, there’s no such thing as “an Internet guy” any more. Since around 1997, there are several different disciplines under the umbrella of web site production. If you purport you can do all of them, you’re good at one and lackluster at the rest. Know what you’re good at and find people whose work you admire to do the parts you’re not good at. I am a developer. A good one, maybe. I am a below-average HTML/CSS guy. If you are a developer, do not be fooled by HTML’s apparent simplicity. Cross-browser compatibility and accessibility issues are enough to break a man. Also, engineering types tend to make the worst interface decisions imaginable. Web interfaces are an area of specialized knowledge these days, if it’s not your core competency, find a ninja. They’re out there. Lucky for me, I know one of the best UX/HTML/CSS guys in the industry, that being Cory Duncan. Cory brings a level of polish to everything he does that makes him indispensable. Hire Cory if you can. Tell him I sent you.
Where I’m Going From Here
You may have noticed that I haven’t shown any code samples from the generic application I’ve been talking about here. There’s a good reason for that. Pretty soon, you’re going to be able to look at the whole thing yourself. That’s right. I’m in the process of turning it into an open source project, under a BSD-style license. It’s going to be called communit.as, and I’ll be setting up a site for it shortly. I’ve benefited so much from the open source software that made the thing possible it seems wrong not to share it. Stay tuned.
If you have any questions about the stuff I’ve talked about here or thoughts on how I’m doing things right/wrong/stupid, feel free to leave a comment or email me.
