The iPod for Adults

Quoting my brother, Lars:

"The iPhone is the iPod for adults."

Think about it. Makes good sense, doesn't it? comments...

Companies and Software

There are two kinds of companies in the world. The old kind that buys software and the new kind that doesn't.

Old-school thinking: We're building a large-scale application supporting 250,000 new users per day. We can't afford to trust such a critical infrastructure on some non-profit organization's web application suite or database engine. Who would we call if something went wrong?

Facebook's thinking: We're building a large-scale application supporting 250,000 new users per day. We use Apache / PHP / MySQL.

(figures from an interview with Jonathan Heiliger)

Another way to say this is if something goes wrong with a product an old-school company relies on, they want to hold another company accountable. In contrast, a modern company prefers to rely on the community instead. While an old-school company feels their risk is minimized by having all of their eggs in one large company's basket, a modern company sees that more as a liability and views the community as a place where risk is more distributed. comments...

Tutorial: SIP and DNS SRV Records

SRV records are a nameserver record type that returns the port as well as weight and priority information for a request. For example, SRV records are commonly used by SIP clients to discover the IP address and port of the SIP server it should use.

Lets say a SIP client wants to find out what server to use given a call to 2125551212@example.com. The server part of that address, example.com, is easily recognized but lets say the answer should include three public servers on various IPs and ports. You could just set up three standard "A" records for example.com but the client wouldn't know what answer is more preferred over another nor what port to use for each of those servers. (not to mention you probably already have "A" records that point to web servers) So instead we use an SRV record which contains this information.

In this example, our SIP client uses UDP as opposed to TCP as a transport so by convention it will append _sip._udp. to the front of the domain name. So the task is to look up the SRV records for _sip._udp.example.com. (we'll use the command-line tool "host" in this example because of its brevity though you could use anything else such as "dig".)

~> host -t srv _sip._udp.example.com
_sip._udp.example.com has SRV record 100 30 5060 sip.nyc.example.com.
_sip._udp.example.com has SRV record 100 20 5070 sip.lax.example.com.
_sip._udp.example.com has SRV record 100 10 5080 sip.dfw.example.com.
~>

We get three records back, presumably for servers in New York, Los Angeles and Dallas / Fort Worth. But which one do we actually go to? The next two numbers on each line describe priority and weight. Lower priority numbers are picked first but all three responses have the same priority of 100. So falling back to weight, it looks like sip.nyc.bandwidth.com has the highest weight so we should try that server first. Of course if for some reason that server wasn't reachable, we would move on to sip.lax.example.com and then lastly sip.dfw.example.com.

The last number we have in the SRV record is the port that we should be using for the service. In this case, sip.nyc.example.com wants to be contacted on port 5060. This happens to be the standard SIP port though as you can see, other ports are used for the other servers should we need to fail-over to them.

The last step is for us to to a standard "A" record lookup on sip.nyc.example.com to find out what IP to go to.

~> host sip.nyc.example.com
sip.nyc.example.com has address 1.2.3.4
~>

So to complete this call, we are going to send our SIP INVITE for the number 2125551212 to port 5060 on the server 1.2.3.4 using UDP.

I use djbdns fairly extensively for nameservice but SRV records aren't natively supported by the djbdns package. While there is a patch that adds an SRV record type to djbdns, you can also create SRV records with an unpatched version of djbdns by using the generic record format.

Coming up with the generic record syntax for an SRV record isn't very straightforward. To help with this, I have written a djbdns Record Builder for SRV records that lets you fill out a form to get generic SRV records. While these records aren't the easiest to administrate in their encoded form, it doesn't require patching djbdns.

It is easy to see how SRV records can be used to load balance traffic across geographic locations. You would imagine that you could get a somewhat even spread if all three answers in the example above had the same weight and priority. It would be up to the SIP client to pick one of the three at random. Of course because these records are returned dynamically and you can have a fairly low TTL (Time To Live) value on each of these records, you could drain off traffic to a particular server over time by removing its SRV record and letting the others take the load.

I usually set my TTLs to 10 minutes so I can stop new INVITES fairly quickly. Of course there is a penalty in the form of excessive DNS queries if you set the TTL too low so you should see what works best for your situation. Also, stopping initial INVITES coming in doesn't necessarily mean all traffic has stopped on a server. Calls in session would still be there. So to completely drain the average server, you would have to remove the SRV record, wait for the TTL timeout period and then wait for the average call duration. You might get lucky with all calls naturally hanging up quickly but you could be caught by one call that hangs on an extra hour or two more than expected as well!

Note: SRV records are described in RFC 2782. comments...

Asterisk vs. FreeSWITCH

Over the past several months I have been heavily involved in implementing FreeSWITCH (which just celebrated its first 1.0 release) as an Asterisk replacement. I come from a solid Asterisk production background with several hundred implementations in the field so I would classify my knowledge of Asterisk as fairly good. I must say though that I am very impressed with FreeSWITCH, particularly in the area of performance and flexibility. Given the strengths, I wouldn't be surprised to see FreeSWITCH migration announcements coming out of major Asterisk based PBX vendors within the next year.

Running Asterisk in production with standard hardware, I'm not able to reliably handle over 250 concurrent calls. But with the same hardware, I have tested to 1,000 concurrent calls using FreeSWITCH without any issues. Evidence from others on the FreeSWITCH mailing list claim 3,000 concurrent calls without issues which would seem to approach wire speed! Anecdotally, the first limit you reach with FreeSWITCH might not be with the CPU!

So is FreeSWITCH a drop-in replacement for Asterisk? Not quite, but it is headed somewhat in that direction. Let's look at what it can do versus Asterisk and how it is configured.

The first thing you are likely to notice when configuring FreeSWITCH is the decision to do all configuration in XML files. Personally I think the project would be much better off using standard text configuration files instead because XML files are too free-form. This leads to confusion like "Where should I configure such-and-such?" questions. Usually the answer to this question is "anywhere" but people want to configure things in the "right place" and move on. The free-form nature of XML makes things ambiguous at best because one setting can be contradicted somewhere else in some other XML file. Most people configure things with text editors anyway so XML just ends up being more typing while settings become less clear, even if code coloring is turned on. An XML editor might help here but who has an XML editor on a text mode system? For performance reasons, none of the machines I have ever deployed have had a windowing system so frankly an XML editor isn't realistic. However, I would say nearly every Unix based system in the world has a text editor.

FreeSWITCH succeeds on so many other levels that I am willing to forgive this minor blemish. For example, just the idea that you can use regular expressions in the dialplan is freeing. And you aren't limited to matching only on the destination number. You can take other things into account such as the source extension so for example you could send someone to listen to their voicemail if they call their own number while sending everyone else to the front door to leave a message. FreeSWITCH makes good use of SQL for storing things like session data and registration data. This can be migrated off-box via UnixODBC so operating a cluster of FreeSWITCH instances on one central set of data becomes and exciting prospect. This also leaves the door wide open for writing interesting web front-ends and exposing call state to various widgets.

But probably the most interesting feature of FreeSWITCH is the fact that you can jump out of the confines of the traditional dialplan / module jail and take over call control using a variety of languages. I was able to write a full featured database driven IVR system (with schema) in a matter of days with this. (including the learning curve!) Rather than reinvent the wheel, FreeSWITCH just leveraged existing languages with all of their diverse libraries by adding in a few call control methods. This simple move breaks open the VoIP application programming landscape giving a tool set to developers outside of the normal C programming confines. Suddenly a web developer versed in JavaScript can write a VoIP application!

For example, picking up a call and playing a message with JavaScript becomes as easy as:

session.answer( );
session.streamFile( "message.wav" );

So why does FreeSWITCH perform so much better than Asterisk? It largely comes down to the way FreeSWITCH remembers what is up with each channel at any given time. Because FreeSWITCH uses one thread per channel to maintain channel state, the complexities of locking and unlocking a linked-list of open channels in the Asterisk model are avoided. This more segmented design leads to fewer deadlocks and virtually eliminates channel state corruption issues. It is hard to imagine that this design alone garners a 10x performance boost but it looks like that just might be the case.

FreeSWITCH leverages a number of already existing libraries to support a wide variety of technologies. For example, FreeSWITCH uses the Sofia SIP stack instead of reinventing a new version of SIP. This saves a great deal of time and complexity for the FreeSWITCH developers while opening up the capabilities of the project. FreeSWITCH has been extended in this way to also communicate with H.323, IAX2 and XMPP (GoogleTalk) services among others.

FreeSWITCH is compelling, but compared to a very mature Asterisk, it obviously still has some distance to cover. The question is whether or not the remaining differences matter enough to be a reason not to switch. It is early in the game, but I would say for most projects the issues probably aren't that significant. The upside of a far more configurable and efficient platform is usually too much to ignore. For very specific applications like conferencing and media serving, FreeSWITCH is the clear winner. If you need a business PBX with a pretty web based front-end, you still might use Asterisk because no option exists yet for FreeSWITCH, but that should change in time.

If you have already made the switch from Asterisk to FreeSWITCH, leave a comment with your experiences. If you are considering making the jump but haven't yet, what are your concerns? Is something significant holding you back? comments...

Peer to Peer Lending Comes of Age

Over the past year and a half I have been experimenting as a lender on several peer to peer lending websites. The idea is simple. Rather than have a bank lend money to an individual, a group of people lend the money instead. Lenders bid on a slice of the loan at whatever minimum rate they are willing to offer. After enough lenders have bid, the loan becomes fully financed and additional bids start knocking the loan's interest rate down. (think eBay) If your lending rate as a loaner is less than or equal to the current rate, your bid stands. When bidding finishes, a disbursement is made to the borrower and they begin to pay back the loan. Lenders get to spread the risk across a number of loans while borrowers benefit from lower rates as lenders compete.

I think the explosion of peer to peer lending like this comes from the confluence of several factors: the many to many nature of the Internet, people's willingness to make financial transactions online and the credit squeeze as traditional capital markets tighten up. Of course this kind of investing still bares "head for the hills" signs. Peer to peer loans are usually not insured and are generally uncollateralized to boot! But if you can stomach the risk, peer to peer lending can be significantly rewarding for lenders in the game for the long haul.

I started out with Kiva which offers a philanthropic wrinkle to the peer to peer lending idea. Kiva streamlines micro-loans to entrepreneurs in second and third world nations, a cause near to my heart. The loans are interest free for the borrower under the idea that giving someone a fishing pole and teaching them to fish is more sustainable than just donating fish. As a lender, you typically bid on a slice of a loan totaling less $1,000 for something as simple as a peanut grinder or a sewing machine. The local community elects and vouches for the borrower in hopes that they too someday might get a loan. If the loan is paid back in full, the local community gets to elect another member for a loan. It is literally stunning to see how much of an impact a few hundred dollars can make in a developing community over time. Last Christmas I gave out Kiva gift certificates as presents.

Eventually I began to investigate the for-profit peer to peer lending market as well which includes Prosper.com among others. As these for-profit peer to peer loans carry interest rates anywhere from the low single digits to upwards of 35%, there is ample opportunity to significantly out-pace the return you would get from your average bank. If you think about it, a traditional consumer bank thrives on the spread between the interest they pay on the financial instruments they offer such as CDs (let's say 3%) and the loans they make to people and small businesses on things like credit cards. (lets say 14%) That leaves 11% to cover operational costs, marketing, risk and profit. A peer to peer lending company such as Prosper.com can run with a significantly lower overhead (typically charging about 1%) letting you as a lender enjoy a much more significant portion of the spread. It brings economies of scale to the traditional banking model with the added benefit of distributing risk amongst many borrowers rather than one bank.

However, as mentioned before, it isn't all roses. For an idea of how Prosper.com has been working out for me, I decided to post some real-world numbers. Now one could easily make the claim that I am fairly risk tolerant at this point (I'm still fairly young) so don't think of this as your average portfolio. Also keep in mind that these are results from ~1.5 years of usage, hardly a statistically relevant sample.



Of 281 loans, only 250 are current. The number of loans that are less than 15 days past due fluctuates quite a bit. It is usually at the top of the single digits in my case but maybe the broader economic trends are catching up to me here! If a loan reaches the 2 months late bucket, especially with the first payment, I probably won't see that money again. You do sign up with a collections agency when you start up so some of that money might be recovered though you can't count on it. The good thing for me is the average interest rate in my case is somewhere in the neighborhood of 28% across all of my loans. It remains to be seen how the numbers will pan out though. I'll do a post in another two years when a significant number of loans have come due.

So what has your experience been with peer to peer lending? Anyone else care to divulge some numbers for comparison? Do you see peer to peer lending as a fad or is the market really moving in that direction? What other peer to peer lending banks have you tried? Leave a comment! comments...

JustHumans Financial Models

A number of users have asked about reliability guarantees with JustHumans, a free hosted "form spam reduction" solution for webmasters. If JustHumans were to go down, users would loose the ability to get forms posted for the duration of the outage. This is obviously a problem that needs to be addressed.

JustHumans, while in beta at the moment, is still under active development as we iron out all of the remaining minor kinks. The idea with the project has always been to concentrate on making a 100% functional hosted service that solves real problems for users before thinking about what to do with the service. But users are asking about reliability and I don't believe that problem gets fixed without a financial component.

So we started thinking about financial models that would support JustHumans. After kicking some ideas around for a little bit, I have decided to post them here for public comment. The important thing to remember is these are ideas only. All of this is open to discussion and nothing will be implemented anytime soon. If we were to decide on a strategy and plan to roll something out, nothing would happen without a number of emails to all JustHumans users and at least 6 months of continued free service. The trust of the JustHumans community is far more important to maintain than short term financial gains.

To fix the reliability problem, JustHumans would need to have a financial structure around it that grows with the service. We see three options for this: "good", "better" and "best" for lack of better terms.

The "good" option would be to keep the free system going by having it advertiser supported. Users would complete some simple advertising awareness objective like "Click the RED Skittle" instead of the generic or uploaded pictures the system currently uses. This works well as an idea because users who are filling out forms have a vested interest in getting their form submitted and advertisers have a vested interest in capturing the user's attention. If we were to marry the two by making the user solve a simple puzzle that an advertiser has created, both the user and advertiser get something out of it.

Some webmasters won't like peddling a random advertiser's message, particularly if they are a for-pay site. (and possibly even a direct competitor of some of the advertisers!) For them, the "better" option of the JustHumans service would be a pre-paid account (say $10 to start) that would be charged 2 cents per legitimate post through the system. This way, heavy users would pay more for the service than light users yet the cost would still be small enough for the average not-for-profit to cover. In the event that a user's balance in this pre-paid system were to go to zero, they would automatically switch back to the advertiser supported option above. Emails would go out before the account got in danger of being converted and the user could always recharge their account at any time.

The "best" option would be to purchase the JustHumans code so it can be made to run directly on customer's websites. (ie: Make it no longer a hosted service) This is the only 100% failsafe option in terms of JustHumans reliability. We don't have the code packaged up for sale yet but we were thinking a $500 price tag.

To be clear, we are a long way from actually doing any of the above options. Nothing is decided and everything is open to discussion. We will send an email to all of the JustHumans users about this if we actually decide to put a financial model in effect and give at least 6 months of free service past when we make such an announcement.

We would love to hear from you, our users, about these ideas. Please leave your comments and let us know what you think about them. Would you continue to use the service if it were run in this way? What plan would you likely go with? Do you have a better idea on how to make JustHumans reliable? How could we make this better? comments...

DVDs on AppleTV with Handbrake

The AppleTV is great at playing rented and downloaded movies but leaves you out in the cold with your existing DVD collection. I can rip my Audio CDs into iTunes and play them on my AppleTV but DVDs have no such support.

To remedy this, some say the AppleTV should have had a DVD drive but it is a forward thinking move on Apple's part not to have one. As I mentioned before, physical formats will become a thing of the past. Having a DVD drive in an AppleTV would have sidetracked the whole point of the AppleTV by making it a glorified DVD player. By making the AppleTV unlike anything else in the livingroom, Apple kept the focus on creating a bridge with the computing world with its concept of downloaded files. Five years from now when the DVD collection will be considered "so 2000", Apple's ideas about breaking free of the physical formats will seem commonplace.

But we still need to unlock the those DVD collections and turn them into files that the AppleTV can play. The AppleTV is fairly conservative about what it will play, opting for a maximum of 1280 x 720 MPEG 4 video encoded at no more than at 5 Mbps but this resolution is still far better than the best DVDs out there. So all we need is a way to take those DVDs with their MPEG 2 video files and transcode them into MPEG 4 within the above constraints.

Thankfully, the free program "Handbrake" exists with a built in preset for the AppleTV that takes all the complexity out of doing this. Over the past few months I have been ripping all of my DVDs into files that can be loaded into iTunes and played on the AppleTV. The thought is to rip and eventually throw away all of my DVDs just like I did with my Audio CDs back in 1997.

The process is simple. Start Handbrake and drop in a DVD. When the DVD is recognized, Handbrake will automatically select the largest video which is usually the main feature on the DVD. Pick the "AppleTV" preset from the drawer on the right and then click "Picture Settings..." to check if you need to de-interlace the picture. (Just set it to "fast" if you see frames with blurred lines. The picture will clean up if de-interlacing helped.) Then you click the big "Start" button near the top and wait a while for the process to finish. The result will be a video file that you can drag into iTunes and sync to the AppleTV.


(Screenshot)

This works well for DVDs but what about other video files like the ones you make with your video camera? Shouldn't those be viewable on your AppleTV as well? Thankfully there is another free (but Mac only) program with handy presets for the AppleTV for that as well called VisualHub. Similarly, you drop video clips in just about any format into the main window and select iTunes on the tabs and AppleTV from the pulldown and click "Start". I usually check "H.264 Encoding" and set the quality to "High" as well. The resulting files can be dragged into iTunes and synced to an AppleTV.


(Screenshot)

Now that I have a little system figured out for all of my video media files, I have been on an encoding binge to get everything converted. The problem is that the AppleTV can't store everything I have converted. There is a way to hack your AppleTV to add external USB storage but that is the subject of another post. Additionally, the AppleTV's menu system isn't the most convenient for navigating what will become hundreds of video files. I'm afraid I'm going to have to wait for Apple to figure that one out as they add more and more space to the AppleTV units of the future. I suspect we won't see significantly increased AppleTV space until the menu system for local files becomes more navigable.

The sooner we convert to a world without physical format limitations, the better. You know that whole HD-DVD vs. BlueRay debate? That was so 2000's... comments...

Simulating Weightlessness

Since as early as I can remember, I have always wanted to go into space. Unfortunately, that's about as far as I thought at the time and the whole thing remained just a dream. It wasn't until I got older that I realized that I could have actually done something about it in my life. Of course, had I been smart, I would have joined the military with the goal of becoming an astronaut and had them pay for everything. But apparently I'm not smart. Instead, I'm going to rely on the burgeoning private space industry.

While I can't afford the ~22 million it would take to catch a lift with the Russians to the Space Station, there are other things I can do to prepare for my eventual trip. So this weekend I went to NASA's Kennedy Space Center in Florida and boarded a Boeing 727 with a largely empty interior. We took off of the runway the Space Shuttle lands on and flew out over the Atlantic Ocean into a Military Operating Area some 10 by 100 miles in size. Aside from the 50 or 60 feet of leg room I had, this would have been a normal flight up to this point. But this airplane is certified to do parabolas which mimic varying levels of gravity. Most interestingly, it can give you 30 seconds of weightlessness!

We started out lying on our backs on the floor while the jet pitched into a parabola. We waited through about 45 seconds of 1.8 Gs which felt to me as if I weighed about 320 pounds rather than my regular 180. As we started to crest the first parabola, the jet mimicked the gravitational force of the planet Mars, or 1/3 Earth's gravity. I was roughly 60 pounds at this point so I did a few push-ups - using one hand - and I was able to get well up into the air!

The call came to lie down on the floor again for the troth of the parabola. I was 320 pounds again and stuck to the floor. Once we pulled out, the jet simulated the gravitational force you would feel on the moon, or 1/6 Earth's gravity. I was 30 pounds and so tried my push ups again. This time all I needed was a fingertip or two of one hand and on the last one, I pushed all the way up to a standing position! I tried walking around a little and almost hit my head on the ceiling more than once. Remember those movies of the guys jumping around on the moon?

After another simulation of Moon gravity, we did our first zero gravity simulation. This, of course, was what I had come for. The jet started to crest the parabola and I could feel my weight against the floor slowly roll off to zero. Your tenancy is to push off of the floor as you get up so everyone pretty much inevitably ends up at the ceiling. I was no exception, but the experience was nothing short of spectacular! It was almost exactly as I had imagined it would be. That whole "equal and opposite reaction" stuff that I had thought through for years was brilliantly apparent. Thankfully there were a number of surfaces and hand holds that we could use to arrest any unwanted drift and everyone was clearly elated!



(that's me on the right)

Over the next 11 parabolas we did, all simulating zero gravity, we played with water floating around like globules in front of us, M&Ms disbursed in the air and we even played catch with one another. I was the ball. With the water and the M&Ms, the trick was to get them fairly still floating in front of you and then figure out how to do a Homer Simpson and fly through them eating as you go. I think I only got 2 or 3 M&Ms, mostly because I wasn't using a hand hold.

All in all we had some 6 minutes of weightlessness and one and a half minutes of diminished gravity across 15 parabolas. The jet flew from the bottom of the parabolas at 24,000 feet to the top at 32,000 feet, a difference of roughly 1.5 miles. I estimate that from the "release" point, the jet threw us a little more than half a mile up into the air and let us fall back that half mile before "catching" us.

After romping around in our playground over the Atlantic, we headed back to the Kennedy Space Center, landing again at the Shuttle landing facility runway. At roughly 3 miles long (15,000 feet) it is one of the longest runways in the world though unfortunately there wasn't a window near me. At 3 miles, I don't think you need to use much to stop the airplane though it seemed like a normal landing to me.

I would say of the trip, most significantly I learned that I don't have a "sea sickness" problem with zero gravity. I'm just as happy to be upside down floating around as I am to be stuck to the floor flying straight and level. I had refused the medication before the flight which turns out to have been a great idea. This will come in handy when I eventually make it into space though I probably need to try out how 6 to 8 Gs feels to see how I do with that first.

If you are interested in trying this out, Zero Gravity is the only company in the United States certified to do parabolic flight. You can also do it in Russia though I haven't tried that program. As you would imagine, it isn't cheap but you do get to keep your flight suit at the end. OK, so it is an extremely expensive flight suit! Go try it and leave a comment. comments...

My Cat on Google Maps Street View

Like many cat owners, my cat and I have an intense unspoken relationship. Who knows what's going on in that little pea brain of his? Well I was about to find out.



(Screenshot)

In yet another move to one-up me, I find that "Hobbes" has managed to snagg the limelight of Google Maps Street View fame. Even his classic "and what are you looking at?" pose is in there!

I'd like to take this opportunity to publicly bow out of the competition. He's obviously won. No sense in fighting it. My endorsement is currently up for grabs... comments...

Linux Tutorial: Deploying OpenSer Under Linux-HA - Heartbeat v2.0

heartbeat (or more formally, Linux-HA) provides application monitoring with the ability to restart or migrate a service (like OpenSER) and dependent resources (like IP addresses) to other machines in the event of a failure. Typically a monitoring process returns the status of a resource. (can be as simple as a ping or as complex as a full fledged application level test) In the event of a failure, a tree of services (typically the IP alias and the service that runs on top of it) are restarted or migrated to a new, more desirable node.

The Linux-HA project started as a simple process monitoring and failover application that didn't take service hierarchy into account among other things. Version 2 of Linux-HA was major rewrite of the application which added hierarchically defined services and used the industry standard OCF definition to describe service monitoring tools and dependency trees. read more...