Monday, December 13, 2010

Git SSD Speed comparison

SSD's provide significant improvement in disk IO. How well does that translate over when using Git? A lot.

After watching prices drop all year long during black friday I picked up a OCZ-Vertex2 120G SSD. I installed it in my current MacBook Pro (2.66 GHz i7 with and 4GB of ram) using a MCE OptiBay to replacing the DVD drive so I can have both a HD and a SSD installed at the same time.

All tests were run in Snow Leopard using the Git 1.7.3.2 pre-built binary package. The HD is a 500G Toshiba MK5055GSXF with a Negotiated Link Speed of only 1.5 Gigabit.

Test 1
Creating a new repo, populating it with the contents of linux-2.6.36.1, add commit, create a branch and run git status.



The git add took around ten minutes to run on the hard drive compared to the 40 seconds for the ssd. Committing was also 1/3 of the time taking only 20 seconds v.s. over a minute.

The ten minutes to perform the git add operation on the hd made me think something must be broke and it just seems beyond slow. Turning on/off spotlight and everything I can think of in OS X makes no real difference. To further eliminate variables I mounted an old MacBook HD (Toshiba MK8034GSX 80GB 5400 RPM) I had lying around using a USB connection to my mac and found the following results:
git add - 6m30.422s
git commit - 1m47.226s

In conclusion on OS X slow 5400rpm drives produce slow results and ssd are really that much faster. Moving on...

Zooming in on the graph we can see the improvements to the other commands. The biggest win for me is that git status, a command I run all of the time (I have aliased to gs) runs in half of the time.


Test 2
Cloning a large repository from a local network was the next test. I was hoping to see improved performance on verification and it was faster, but compared to everything run on the server and transfer speeds the seven seconds saved wasn't that much compared to the seven minutes the server (a slow arm based box) was preparing the repo.


Test 3

Removing the network here is a clone of the Qt repository from the hard drive to the ssd and hard drive to hard drive. Followed by some simple walking.


Walking was faster on the SSD, but as soon as the data got into ram it was equally fast for them both.

Summary
As expected Git runs faster when its data is on an SSD. For operations like git branch which just create a file with a sha in it the performance isn't much different, but when running an io intensive operation like git add you can see that it is extremely worth it. Git status was probably the big winner by getting a speed improvement of around 50%.

Averaged data:
Creating a new repo and populating it with the contents of linux-2.6.36.1

         HD          SSD
init      0m0.011s    0m0.010s
add       11m2.903s   0m40.041s
status    0m2.323s    0m1.138s
commit    1m4.097s    0m20.410s
status    0m2.027s    0m1.532s
branch    0m0.039s    0m0.009s
checkout  0m1.465s    0m1.473s
status    0m2.414s    0m1.073s

clone qt-snapshots from local network
clone2    7m21.066s   7m13.310s
rev-lis   0m0.082s    0m0.077s

cloning qt (32611 commits) from hd with --no-hardlinks
clone     0m36.770s   0m13.050s
log       0m1.042s    0m0.977s
rev-list  0m1.697s    0m0.767s


For those on Linux checkout some linux ssd tweaks.

Thursday, October 21, 2010

The future of the MacBook is the MacBook Air

The new MacBook Air is not something to compare to netbooks, but a roadmap of where the MacBook is going with its improved performance and reduced size and parts.

Obsolescence
Back in 1998 Apple removed the floppy from the iMac G3. There is no argument that it was obsolete technology. Small, easily damaged with better replacements like the zip drive, cd-r, usb sticks, and networking. For those who still had floppy disks you could get a usb drive. The iMac G3 led the way and the rest of Apple products followed suit.

One of the primary data replacements of floppy was the "cd". The cd drive, cd burner, dvd drive, and finally dvd burner while useful is less useful than it used to be. Most backups don't fit on dvd drive. Software and games are usually downloaded and not bought in a package these days. CD's are no longer bought and ripped, but usually obtained also online. Even movies are transitioning to streaming services. Even creating home dvd movies has been depreciated in favor of posting online. The big final reason was the OS install disk, but that is now just a usb stick. Just how back in 1998 we still had a pack of floppies disks on the shelf we have dvd's lying around. But we can get one external drive to share among the computers as we transition. I myself plan on pulling out mine from my MacBook Pro and putting it into an external case with a optibay. Replaced with the internet and usb sticks disk drives are dead.

As for ethernet, 802.11n is around 130Mbps (real world) which is decent for most everything. Other than transferring GB's of data WiFi is good enough. Backups are the biggest thing most users will transfer, but even that is done overnight (over WiFi!) and in the background. After the disk drive the ethernet port is pretty tall so removing it allows for thiner laptops. At the end of the day, like the floppy if you really need an ethernet port you can get a usb version and just like the Firewire port it will get the axe on MacBook models.

IO
The last ten years the speed of the cpu on the computer has had less and less of an effect on the performance than one would think. First was the race to 4GB of memory, followed by graphics and now io speed.

The most dramatic improvement someone can make to their computer these days is to switch to a solid state hard drive. As the size of flash has continuously increased while the price has dropped it becomes more and more attractive for everyone from desktops to server farms. There is a lot of research and experimentation in this area with leaps in performance coming out all of the time. The next several years will be very interesting to watch as flash grows. While it seems mature flash is still new enough to warrant whole blog posts by users who just upgraded extending the virtues. From the lower power requirements, the lack of any noise, fitting in different shapes, and of course mind boggling speed, ssd drives are the future. Apple has offered flash as an option for a while now on some of its computers, but with the new Air it takes the option away and makes it the default. As prices continue to drop and sizes rise ssd will become less and less the option and more the default (or only) option.

tl;dr
It would be no surprise if in the next refresh or two the macbook moves to ssd and drops the super drive and ethernet port. The MacBook will be thiner, have a longer battery and a faster user experience.

As the tagline says:
The new MacBook Air. The next generation of MacBooks

Saturday, June 19, 2010

Managing project, user, and global git hooks

I recently put up a new tool on GitHub called git-hooks. This is a tool to manage project, user and global Git hooks for multiple git repositories.

Hooks can be very powerful and useful (as I have previously written about in my git hooks blog post). Some common hooks that I always want running no matter what git repository I am commiting to include:

- Spell check the commit message.
- Verify that any new files contain a copyright with the current year in it.

Every project also has specific hooks that I run such as:

- Verify that the project still builds
- Verify that autotests matching the modified files still pass with no errors.
- Pre-populate the commit message with the 'standard' format.
- Verify that any new code follows the 'standard' coding style.

And I have some hooks that are very specific to just me such as:

- Don't allow a push to a remote repository after 1am in case I break something and will be asleep.
- Don't let me commit between 9-5 for projects in ~/personal/ as I shouldn't be working on them during work hours.

In the past I have just been copying these from one git repository to another, making symlinks or other nasty hacks. I finally sat down and wrote a tool to manage these different hooks.

After running 'git hooks --install' in a repository rather then using what is in .git/hooks the following locations will be checked for hooks:

- User hooks that are installed in ~/.git_hooks/
- Project hooks that are installed in the git_hooks directory inside of each project.
- Global hooks that are specified with the hooks.global configuration.

If you have many hooks you don't have to write a wrapper to call them all, they all simply go inside of a directory (such as git_hooks/pre-commit/bsd.sh).

git-hooks/contrib also includes various simple hooks I are useful that you might like to use. git-hooks is released under the BSD license.

Monday, April 26, 2010

What makes a fun Mario level?

This years 2010 Mario AI Championship includes a level generation contest. Having been interested in creating a platform game for several years I have written up my notes on Mario levels for those who are working on an entry. Disclaimer: I have never worked on a shipping platform game so this is simply what I might try to achieve if I was making a level generator.

Basic Rules:


While objects could be places randomly, good levels that are generated should at the bare minimum follow the basic Mario universe rules such as:
  • A pipe flower must come out a pipe, not the ground
  • Fish are only found in the water and other bad guy should only appear in their respective level themes. i.e. Bowser shouldn't appear in level 1
  • Mario must be able to reach rewards such as coins or boxes without dying.
  • All platforms graphics have left, middle and right images. They should all be used.

These might seem overly obvious, but I have messed around with enough home grown Mario games to see that it needs to be stated.

Average Rules


On any given level you will find the majority of the following to be true:
  • A maximum of seven bad guys on the screen at once by default, lower is better. If you play Mario 1 you will find that there are often only one bad guy on the screen at at time. The more bad guys, the more harder it is and the less fun. Two or three seems to be the magical amount.
  • Mario must be able to reach all platforms. A platform that mario doesn't think he can reach should only be there to lead to a secret and can in fact be reached via a hidden block, flying or other means. Everything should have a purpose.
  • There are between 50 and 100 coins in a level. Play a level twice and get an extra life.
  • A Mushroom box at the start of the level.
  • One save point midway through the level.
  • A Mushroom box after the save point.
  • The final bit of the level is always the same within a variation. In many Mario games you had one bad guy, a hill/tower/blocks. In Mario 3 you had one bad guy and the box to give you stuff.
  • A level is only X wide or high. While rooted in the technical limitation back in the day this size is what players have come to expect so while truly massive levels could be created the average level should fall into the expected size.

Overall level theme design


  • The level should choose a graphics theme such as desert, forest, castle and stick with the graphics and bad guys from that theme. Having a water level with bats doesn't make much sense.
  • At the start or end of a level there can be transitions from one theme to another such as leaving the desert and entering the forest.
  • Location: water/ground/air. Within each theme you can have water, ground or air levels. Walking on top of the castle in the air, water under a forest etc.
  • Does the level scroll vertically or horizontal? The generator should be able to create both types.

What is fun?


Getting to the meat of the contest is making levels that are fun to play. There are many different things that are fun and often times it is important to know what is not fun to know what is fun. Here are just a few:
  • Completing the level is fun. Unless it is world eight no level should be hard or take more then a few lives to get through. In fact many levels you have never seen before you should be able to get through on your first or second try. Dying is not fun. It is temping to make really hard or confusing levels, but don't. Let players run through the level and entice them back with the many ways levels can be replayed.
  • When you almost die (but don't) it is fun. Walking along, a block falls from the ceiling right in front of you making you jump. Blocks falling from the ceiling that you can miss if you just run is fun. The longer you can run without stopping is fun. Being scared, but not in danger is fun.
  • Finding a secret area is fun. Newer Mario games have about three in a level. The secrets shouldn't be completely hidden, there should be a hint such as a block in the middle of nowhere. Running along you could easily miss it, but if you went slowly you would notice it and after a little exploring you would find it. Finding secrets is a lot of fun because you want to show them off.
  • Choosing the right path is fun. When there are multiple platforms you could run across, one being fast and another being slow with lots of bad guys it is fun knowing that you choose the right path.
  • New bad guys are introduced in a learning progression. The first time you see a new bad guy he is all alone and you can approach it slowly and figure out how to kill it, only later having the bad guys come several at a time or unexpectedly.
  • Collecting is fun. The most basic collecting in Mario is the coins, other Mario games have you collect five big coins per level, MB3 had power ups that you could get at the end of the level.
  • Hidden rewards. Coin blocks, power ups, blue coins, one-ups. Nothing beats telling someone where the one-up is on 1-1.
  • It is fun to beating the level quickly. Maybe through a pipe warp or through a trick like running on the ceiling on Mario1 1-2. Players love finding ways around actually playing the game and love telling their friends.
  • Many levels tell a story along with having you run through the level. Ship levels have you arrive at the front of the ship and you progress past the masts and to the back. Level design can include nods to the story.
  • Many levels have a story with the bad guys. On a vertical level there could be a slow but constant stream of the bad guys. Near the end there can be two pipes with the bad guys coming out of them implying they all came from those pipes. Same can be said for many castle levels were you have fireballs coming from bowser who is far off. Having a story makes the level unique and interesting.
  • Highlight one game engine or sprite mechanic. While each level could simply combine the random elements it is much better to have each level be primarily about one thing. A sky level where you have platforms that move on rails. There might be a stationary platform or two, but mostly it is about the moving platforms. Ghost houses are another example where the levels are designed to explore the ghost bad guys and the different ways they can be presented to the player. Choose one aspect to highlight per level. Don't just create another bland Forest 4-2.

Procedurally world and game generation


The last section highlights what I think might be the Achilles heel of the contest. Even if I could generate a pretty good forest level playing 32 of them wouldn't be very fun as the same tricks and level designs would probably be repeated. It is a lot more fun to have several forest levels, several ghost houses, a coin ship, castles, ships, water levels, waterfalls, sand traps all spread out. Much more interesting then level generators would be the task of generating a world, such as the following task:
I just came from the forest world and am entering the desert world. Generate eight levels, two castles, one ghost house, one or two warps and include the following bad guys ....

Given those parameters a generator could introduce the bad-guys one by one knowing that in later levels you can easily beat them. It would be able to tie the levels together such as having one or two desert bad guys in the castle and have odd levels such as the coin ship which while not very fun by itself makes for a fun addition to a collection of levels for a world. World 1-1 should be very easy unlike world 8-4. A stand alone level generator doesn't know where it fits into the overall game or what levels came before or after it. It might just generate ten forest worlds in a row.

There is a random Mario game that you can play online. It generates a world of random levels for you to play, but I only played a few level because it became obvious that all of the levels were about the same and there was nothing new to play. Games that show you every trick in the book in one level are not very much fun later on.

"Fun" is a funny thing. Several years ago I created a random Quake level generator. Me and several friends knew all the Quake levels to well so one weekend I wrote a tool that would automatically generate a map, populate it with items, lights etc, compile it and distribute it to our boxes. Every ten minutes we got a new level no one had ever played before. Most levels were interesting and fun, but one day it generated a level that was a massive room with one wall in the middle. After spawning we immediately realized what we were in, grabbed a weapon and proceeded to run around this one wall. We spent several minutes chasing each other around this one wall laughing the whole time. While definitely the simplest and stupidest level it ever created it was a ton of fun, not something I would have ever set out to design. Following a set of rules for making a fun Quake level I would have never made what it had with my random rules.

I look forward to seeing the entires that are submitted. Writing up this has me itching to go back and play various Mario games to discover more fun aspects and to try out various ideas on my own level generator.

Edit: More information about platform design can be found in this paper: A Framework for Analysis of 2D Platformer Levels

Image by KER

Monday, April 19, 2010

WebKit Meeting

Last week I went to Apple for the WebKit Contributors Meeting. It was really great meeting so many people that I have only ever known through IRC and email. I had countless great discussions and even got some hacking done on some WebKit2 code. The WebKit community is very vibrant and I look forward to what will be added to WebKit in the future.

tagged names on flickr

Monday, March 15, 2010

Git Achievements

This weekend I created a little application called git achievements. Similar to the XBox360 Achievements you can unlock all sorts of Achievements while you use git. It brings a bit of fun (and bragging) to those who use or who are learning git.
Almost all of the git commands have achievements of different levels (depending on number of times a command is used). For example you can become a "Apprentice Author" after only two commits, "Author" after sixteen and finally a "Master Author" after 128 commits. Various other Achievements can be unlocked just be performing an action such as adding a .gitignore file or setting your user name in git config. When you unlock an achievement you will get a message on your console showing the new achievement like so:
******************************************************************
Git Achievement Unlocked!

Caretaker
Added a .gitignore file to a repository.
******************************************************************
Having collected all of these achievements you can set a git config flag and have it automatically publish your achievements to your github pages. The published page includes a list of all of your unlocked achievements, total points as well as a list of achievements that are still locked. I have my git achievements published so you can checkout the full list of achievements.

Installation is a snap, after forking and cloning the repository just add git-achievments to your path and alias git=git-achievements.

Creating this was a lot of fun. It was simple and a bit silly. After getting the idea I had something working in no time at all. It was fun coming up with the different names for the Achievements and as I was working on the project unlocking them. Originally I had only planned to create the output to the console, but it only took a few minutes to realize just how easy it was to publish to GitHub Pages which brought the achievements to a whole fun new level. Achievements in video games often causes the user to explore parts of the game that they might not have and often cause them to learning actions that they might otherwise ignore. I am not sure if git achievements is very useful as a learning tool, but it is fun to have installed and popup with a new achievement now and then. If there is a trick in Git that you think other people don't know about or should learn let me know (or send me a merge request) and i'll make a new achievement for it.

Photo by terren in Virginia under the cc license.

Tuesday, February 16, 2010

Anigma, a game made using only CSS3 Animations and Transitions

This past week I have been playing around with the new css3 features. I wanted to learn more about the new animations and transitions API. The end results is a fun little puzzle game called Anigma.

Anigma is a simple game. To solve each level you must remove all the jewels. Jewels disappear from the board when they are touching another jewel of the same color. The jewels are stacked on the board so if there is nothing under them they will fall down. This can make the order in which you remove the jewels important on some levels. There are over 500 included levels of various difficulty. If you make it to the end there is a nice little end credits animation (nothing amazing).

All throughout the game you will find animations. From the jewels moving to the clock counting down your time left it is all handled by WebKit and not through a 3rd party javascript library. Using the HTML5 audio tag there are also a dozen 8bit style music tracks to enjoy. Absolutely zero flash is required to play this game, but you will need a WebKit browser (Arora, Chrome and Safari4 are all good). (Updated: Got it somewhat working with the nightly build of Firefox or the latest Opera beta both of which have initial support for transitions.)

I had a lot of fun making the game, but having spent a week on it I decided to wrap it up and release it. A few of the ideas that I have thought about adding to the engine to make interesting levels out of:
  • Elevators (I got this partially working in the js source, but ran out of time)
  • Teleporters
  • Ice blocks
  • Trap doors that would come down after you go under them.
  • Buttons that must have a block in it to activate something such as opening a door
  • Dumb waiter type elevator with two platforms and whichever is 'heavier' will go down and the other up.
  • Loose ceiling piece that falls and can crush a jewel
  • A better 'you finished the level animation'
  • Don't do anything and it will show a level being solved i.e. demo mode
  • Public High score board
  • Use JSON for the level format or something better then the text files.
  • Import the rest of the extra vexed levels
  • Generate a random level
  • 'I give up button' that automatically solves the current level.
  • A snapshot state button so you don't have to restart all the way from the beginning.
  • Sort the levels based upon server logs for difficulty.
While it would be fun to implement many of those features it was really about showing some features of CSS3/HTML5 so having accomplished that I decided to stop. I have posted the source code up on github so you can feel free to fork the code and improve the game.

Hopefully you get a kick out of the game and get some neat ideas for what you can do with the new CSS properties. If you enjoy the game hit the 'e' key to bring up a level editor. If you create something good email it to me and i'll include it.

Tuesday, January 26, 2010

Founders At Work

For my Birthday I got Jessica Livingston's "Founders At Work: Stories of Startups' Early Days". It was an enjoyable book and over the past two weeks I quickly went through it. The book is broken up into thirty-two different interviews with company founders. Most of the stories are about web companies, but there are a few hardware stories mixed in.

Covering so many different companies and individuals one coundn't help but compare them. You have companies like Apple that started with the brains behind it to make a really fantastic product and on the flip side you have Yahoo which looks like it was pure dumb luck. Yahoo was the biggest surprise for me. Started as a way to keep track of links for a dissertation the founders spent the first eight months manually entering in urls people would send them, sometimes eight hours a day the book says. They were in the right place at the right time and that was it. I found it extremely hard to read very deeply into any advice given from companies like Yahoo or HotOrNot which (another random luck story). But to match there are plenty of stories of companies who put in several years of hard work and it paid off. Luck is always a factor in a startup, sometimes it is tiny and sometimes it is a metric ton dragging you forward whether you know it or not.

The story of TiVo mentioned another company called ReplayTV which is no longer around. ReplayTV was TiVo's big competition when they were first growing. All of the stories in the book are mostly successful in the end (the company might have died, but the founder got to cash out). Discussing ReplayTV made me take note of the fact that there was no stories of failed companies. This is probably due to the fact that there are plenty of good stories without having to discuss failures and it is easier to sell a book about companies people have heard of, but no doubt the companies that failed had plenty of good lessons that they learned the hard way.

One story that resinated with me was the story of FireFox. While I lived through looking back I hadn't realized just how much they had going for them at the time. I.E. development team had been discontinued, Netscape only 5% of the users and a horrible interface and no change in sight. Out comes this little Firefox... Comparing that to Arora. Between new versions of I.E. that Microsoft is pushing, new versions of FireFox, Safari and then Chrome comes out. I am glad that I didn't try to really push Arora and kept it as just a QtWebKit browser that utilizes what Qt has to offer.

The book has a large quantity of material and at times I felt a few stories were just repeating earlier ones, but it was a fun read, I picked up a few things and while it doesn't get into a lot of technical detail there are lessons that you can pickup from the book such as: You will probably need to change your startup idea, it will be stressful, and patience and persistence pays off.

Sunday, January 03, 2010

Hay Maze


A week before Holloween Jen and I walked to Parlee's farm, a local farm just a few miles away where they had setup a seven acre hay maze. Going through a life size maze is one of those things I always wanted to do as a kid so when we discovered one nearby we went over on the next sunny weekend to try it out.



Walking through the maze took us almost an hour to get through. Even though the hay was only about four feet tall (at the highest) I couldn't see very far and wasn't able to follow were the trail went with my eyes because the hay would blend together. There are two bridges in the second half of the maze that you get to walk on top of and look out over the maze. Amusingly though when you got to the bridge and looked out you really couldn't solve the maze just by looking because the after twenty feet or so it is all just meshes together and you can't tell where the paths are.


This was very nice as I feared that I would just stand still look at the maze and then walk the correct path. This meant that I got to enjoy solving the maze with only the paths I have walked (I wasn't even really sure of the dimensions of the farm/hay fields). When you are solving randomly generated mazes on paper as a kid the only real rule was 'always take a right and you can't go wrong'. But with a life size hay maze you get all sorts of social clues that I quickly started to notice.

In a random maze generated by a computer one of the paths from the very first junction could lead you to the exit. When it is maze number 8 of 500 in a book that is ok, but for a hay maze that makes it not very fun. They want you to have fun and this knowledge can be exploited. Fun is spending a while in the maze, getting to go over each bridge and not wasting too much time on the wrong path. This particular maze was subdivided into sub sections. Each section was its own little maze with one correct path out to the next section. This way if you are lucky on one of the smaller mazes you might not be so lucky on the next (thus having more fun) and you can't spent too much time going the wrong way. Of course once you realize this you can exploit it by taking paths that would lead to where you think the next area would be and knowing you are going the right way if you have not reached a dead end in a while.


Another thing I noticed was that the creators of the maze made several sections that were designed for a human. In one junction that you enter you have four paths in front of you, but if you turn around you will find another path just to the right of where you entered the junction. Visually hidden from the entrance this path and separate from the other four it was clearly the correct path because it would be fun to discover it after walking down one of the wrong paths and returning.

Lastly given that there are people in the maze not only can you see them exploring areas ahead of you and behind you, but you can hear their joy and anguish. This quickly provides hints about what to do in the future. From how big a section is to which side has the right/wrong path. At times I couldn't help but laugh at the family who was stuck in the section behind us and the kids found dead end after dead end.

I went to the maze not really sure what I would expect. I thought it would be similar to every randomly created maze I have dealt with, but found something different and much more interesting. While actually getting through the maze was fun everything else about the maze was much more exhilarating. From ponding how one might make (and exploit) a "fun maze", being outside, people watching, and having fun with Jen made it a very memorable experience. If you ever get the opportunity to go through a life size maze I very much recommend it.

Popular Posts