alternative to click and wait for audio video content

In a previous post, we identified that our userbase wants to download content such as audio and video of various miqats and of Aqa Maula (TUS) to their local machine. It’s only community sites such as Sautul Iman who want to stream their content due to their need to protect their IP and their revenue stream. Whether they do streaming the right way or not is a discussion for another time.

So, you dear reader come across our site and start clicking on our audio, video links to initiate the download and playback. As you must realize, this data is fairly large in size and there are limits even when you send cache friendly headers since few proxy caches will cache large files. Thus the question arises, what’s a good technique to share lots of large files amongst a community.

The answer is Bittorrent. Bittorrent a peer-to-peer file sharing mechanism has been generating a lot of buzz for its ability to share large files with a much reduced bandwidth strain to the hoster and it scales better as more and more people look for the same content.

However the user is still waiting to click on the torrent link so what’s the solution to this.

Let’s step back a bit and look at something we view everday, yes it’s the device known as the idiot box. What it provides is the ability to tune to channel and periodic download of content. Is there such an equivalent in the computer world.

The answer is RSS 2.0 enclosures. As the previous link states

First, RSS and BitTorrent complement each other naturally. RSS was designed to report freshly available content, which is exactly where BitTorrent shines. RSS 2.0 enclosures were designed to automate the download process that BitTorrent optimizes.

Think of the possibility, as a user one would just subscribe to the Ashara news feed and the content would be on the desktop regularly. Since there would be a lot of interest in our community for this feed, the Bittorrent swarm would be buzzing and downloads will occur very rapidly.

Similary, one could subscribe to audio feeds categorised by naats,marthiya or zakereen groups.

An excellent Bittorrent client which has good support for RSS enclosures is utorrent. Interested community members are encouraged to play around with the above idea and demonstrate a prototype.

getting community audio cd information into online databases

Our community the Dawoodi Bohras regularly publishes CD’s whether they are from Sautul Iman or CD’s published by various zakereens or even audio CD’s containing nasihaats such as “Allah Taala na Hamd’, ‘Biraadar tu Nasihaat sun’.

What we lack is information about the tracks on the CD’s being published to sites such as Gracenote and FreeDB which would allow listeners from our community to download the track information and other metadata into their players.

Players such as iTunes make it very easy to enter the information into the Gracenote database and there are tons of tools which allow for entering the information into the FreeDB database.

This is a simple project for someone or a group of people who have the original CD’s to help provide accessible metadata to a wider audience. If your zakereen group has compiled a CD containing your group renditions of marthiya, you owe it to the mumineen you might be selling or gifting the CD that the track information is in both Gracenote and FreeDB.

streaming downloading and all that jazz

Our community, the Dawoodi Bohras have a rich and varied oral tradition which is expressed in the forms of qasidas, marthiyas, naats, munajaats and the recital of Quran-e-Majeed. Over the years, we have tried to aggregate examples of our oral tradition and made them available at our audio site.

On our audio site, at present we request samples to be made available in WAV format. We would really prefer any loseless encoding instead of WAV whether its FLAC, WavPack, Monkeys Audio and many others.

Please understand that formats like MP3, AAC (Advanced Audio Codec, used in iPods) and Windows Media Audio are known as lossy codecs in the sense that they lose some information from the original CD recording in order to be able to compress the file to approx 10% of the original size. They use lots of fancy algorithms to create the perception that the sound is as close to the original as possible.

But nobody in their right mind would use a lossy encoding as their master source, hence the requirement of loseless codec which as their name suggests do not lose any information yet are able to compress the original file size albeit to only about 50% of the original size.

In terms of delivering our content, we provide at times two links one named ‘Download’, the other named ‘Listen’. Frankly, from a user experience point of view this is not right. The intent is to ‘listen’ always so what does ‘download’ indicate. It’s primarily historical. The content team wanted to provide what in the industry is known as ‘streaming’. That’s what the ‘Listen’ link intention is. Users would click on the link and their player would launch and play the file whereas with the ‘Download’ link would just download the file on the users desktop.

With todays proliferation of freely available media players and their tight integration with web browsers the ‘Download’ link really doesn’t work since it too launches the player and plays the file.

True streaming refers to the ability to play audio as the data is coming in over the network so one doesn’t have to wait for the entire download. We really don’t do true streaming, since if you observe carefully we generate a .RAM file which is a real-audio metafile that points to the .rm file.

We also serve this file via the HTTP protocol. Delivering Real Audio files over HTTP has quite a few limitations including one that requires the entire content be downloaded before it can be played and if someone has the illusion that it prevents the content to be stored on the end user machine then that person should not complain when their DRM intentions are bypassed

In order to do true streaming, we would need to setup a streaming server such as the Helix Server, Quicktime Streaming Server or Windows Media Server.

Users would like the ability to store the audio/video content we provide locally so they can listen to it without needing the ability to be connected to our server. The ‘Listen’ link we currently provide is an anachronism.

In a future post, we’ll discuss some interesting ideas on delivering large content rapidly to a userbase.

Waiting for some hardware

We are for some hardware so that we can run our mailing list engine on a separate box so as not to cause sysadmin grief when running multiple MTA’s on the same OS instance. It’s not that it’s not possible to run such a setup but such things are best left to people who run mail systems for a living. Also separating our mailing list from our website in terms of boxes gives us some form of resilience in case one box croaks. Some of you may have observed that our existing site including jamaat mailing lists were down for a fair bit, this was due to our existing Pentium II box not being available. Hopefully the new hardware can be provisioned quickly. The PII box is being pushed to its edge and our userbase can be inconvenienced if the box is down again.

Whilst we are waiting for the hardware, let’s take the opportunity to talk about some of the services we provide today, discuss what’s good/bad about them. Throw some new ideas into the mix and see where this leads towards. Remember that cannot always be expected to come up with an end2end implementation. The community must step up to the plate.

Impact of latest Sober virus

Extract of mail traffic summary

Per-Day Traffic Summary                                                         
 date       received  delivered   deferred    bounced   rejected
  Nov 20 2005       83         22         0        0         420           
  Nov 21 2005      398        181         0        0         690           
  Nov 22 2005      150         59         0        0        5169           
  Nov 23 2005      433        231         0        0        7517      

delivered email includes verp bounces. but realistically the main list only receives 2-3 messages a day, jamaat lists a few more. So assume that only around 10 “real” messages flow through our server/day.

So what happened, Sober struck again and one of our main mechanisms to survive was our plain-text only policy.

Now would be a good time to tell your friends and family to

Upgrade to Firefox 1.5!