The Changing Face of File-Sharing

05/15/11 | by Charlie [mail] | Categories: Technology

As the technology and the internet (and the regulations governing it) continue to evolve, so too do the means by which people abuse it. Where there is an easy way to get something for free, people will do it. Such is the nature of file-sharing. I got on the internet relatively early (which is to say in the 90's), and in the past decade, there's have been hundreds of preferred or *best* ways to illegally access files. As some died, others arose, this is their tale.

The Golden Age
Right around the turn of the millennium was the high point in digital file-sharing. The technology was ripe: computers were just gaining the storage space necessary to accommodate large amounts of digital media, people were writing programs which showed the vast potential of the internet, and connection speeds were finally rising from 56k dialup modems.

For reference, I used to have dialup. When I tried to download a song, I basically just queued up a couple and left. Downloading a game was a multi-day endeavor. Based on the 3MB, low-quality song size that was typical back then, it took me about 12 minutes to download a single song over dial-up.

You downloaded songs!? Yeah, I did, but I was a clever little 10-year-old. I stayed away from the hard-drugs (Napster), cause my mom would have been mad. Instead, I simple used AltaVista Music Search. Yup, I literally used a search engine to download music.

At this point in time there were dozens of option for downloading files. As I mentioned, search engines were a great starting point. Of course, there were also the numerous file-sharing programs and networks which all popped up right around 1999. Napster, Kazaa, E-Mule, Gnutella, Morpheous, and Azureus were the mainstream programs. Now while those were the main programs, they operated on a smaller number of networks (which still exist) such as Ed2K, mp2p, and Gnutella. In addition, there were tons of off-brand programs which interacted with the others. It's fun to note that nearly every single program on that list has been sued and now tries to operate as a legitimate music program.

The Dark Ages
As more and more programs started coming under fire from the RIAA, however, it became more risky to use those programs, or at least it seemed that way. A relatively small number of people were actually sued of the millions of people who inhabited those networks.

In any case, simply making all your files available to peers (or the government) was no longer a good idea. Luckily, for the consumer level file-sharer, there was an alternative. Internet Relay Chat (IRC) has been around since the early 80's and is what most people are talking about when they mention a chat-room. When I played video games, my group of friends had a chat room we hung out in, when I worked on an open-source software project, we offered support through a chat room, and of course, people with common interests often found their way to a chat room.

But IRC wasn't just for talk. If you knew where to look, you could find a special search engine which monitored all these rooms and compiled a list of (ro)bots. These bots would sit in specific channels/rooms and broadcast a message every 5 minutes or so listing the files which they hosted. These were typically big-ticket files like albums, games and movies. You then had to open a personal chat to the bot, and type specific commands which informed it which file you wanted. The good ones only transmitted to a handful of people at a time (to increase speeds), so you'd have to wait for quite some time before you actually got the file.

Of course, with the rise of DSL and cable internet, these were an important stepping stone to modern file sharing. They provided relatively high transfer rates on a variety of files. This also provided a high amount of safety from detection. For instance, in Canada it's only illegal to upload illegal content, not to download it. The US law is different, but in any case, it's usually the uploading part that causes trouble.

The Renaissance
What's better than downloading from one server? Downloading from a million! Enter torrents. So far the transfers have all been peer-to-peer or server-to-peer, but now it's going to get a bit more complicated. Torrents are peers-to-peers. There's actually no server involved (other than a coordinating server).

It works like this. Someone creates a very small torrent file (which is a listing of files it contains and servers and some other special stuff). They then announce this torrent on a server. People search on that server, see the torrent file and load it.

Anyone who has a full copy of ALL the data is considered a seed, while anyone with a partial copy is a peer. Seeds will send peers a portion of the file, then peers can send each other parts of the files at the same time. So a seed could send 4 people a quarter of a file apiece, and at the same time, the peers can transfer their respective parts to each of the other 3. Thus in the amount of time the server would have transferred one file, four copies have been distributed.

That was a terrible explanation, so you should probably just go stare at the wikipedia page and animation for a bit.

In any case, it should be apparent that there is no true server in this model. At the same time, every downloader is also a server. This leads to potentially very fast transfer speeds with the caveat that if everyone with a full copy of the file logs off, the torrent could potentially die.

Torrents are actually an incredibly powerful tool for large, legal file transfers (although you internet service provider hates them). For instance when I downloaded Linux, I used a torrent. It saves the host servers bandwidth and provides high upload speeds to everyone else.

Unfortunately, this method also draws considerable scrutiny from your ISP. They actually inspect what you're transferring and sometimes can detect illegal files. See Net Neutrality on Wikipedia for more on that topic.

The Modern Age (and future!)
Today, people use a variety of the above methods. If you know what you're doing, you can even download from Google. With cloud-storage and file-transfer sites (think YouSendIt!) becoming more popular, those are the natural progression of file sharing. They provide the perfect cover of anonymity and distribution potential. There's literally hundreds of services like that. Simply upload your file and send random people the link.

Honestly, I have no idea where file-sharing will go in the future. If ISPs ever loosen their strangle-hold on upload bandwidths, I really do think BitTorrent is an ideal model. It embodies cloud-computing with its distributed-server model. I almost used it the other day for a project I'm working on this summer, BlueFusion (You'll get some blog posts about that later). We needed to download a huge amount of data (~15GB), and it was available as a torrent.

You can always share files across local networks (college dorm rooms) or by sharing external hard drives. So no matter what, friendly file-swaps will always exist. The future of internet sharing depends a lot more on how the FCC Net Neutrality story unfolds. Lemmie know if you have any other methods I forgot to mention!

Charging for Student Shows at Yale

10/22/10 | by Charlie [mail] | Categories: College


A short piece I wrote for the YDN opinion page. Check it out!

Observations (Pt. 3)

10/17/10 | by Charlie [mail] | Categories: The Life, Making Money, Travel

This is a tad belated, but I still wanted to share it with everyone. After working in Germany for 3 months this summer I noticed a lot of differences between my previous positions in America.

Shaking Hands
I was surprised about how often people shook hands. It wasn't so much about the hand-shaking itself, but rather the daily greetings. Everyday when you arrive in the office for the first time, etiquette dictates that you walk around to every co-worker in the department and greet them. Eventually I got more comfortable with the idea. Only a few people actually shook hands with everyone, lots of others did a mix of handshakes and greetings, so I settled into a comfortable mix.

This whole process was often repeated on the way to/from lunch with the Bavarian greeting "Mahlzeit". There's no direct translation, but it's basically a fit-all greeting which is usable around any meal. Awesome. Sidebar, two favorite Bavarian treats: Kaiserschmarrn (Basically mixed pancakes) and Dampfnudel. They eat those as main dishes!!!

Expletives (expect some)
I was a little surprised at the level of profanity in the workplace. There was a lot of shit going down. Sometimes a few other things were slipped in, but seemed like s*** was very commonplace especially in referring to anything that wasn't going terribly well. My feeling from the American workplace was that language like that would generally be restricted to 1-on-1 conversations with very intimate co-workers. Although perhaps I didn't get a good read on the context abroad.

Just to underscore the cultural differences, I felt like I had to spice up my slang with a few choice words there. In contrast, I had a visitor from Germany during high school and was surprised at how much they attempted to swear in English. It got to the point where I couldn't tell if they were actually angry or just trying to accentuate the slang with my friends and I. Swearing, like comedy, just seems to be one of those very nuanced cultural things. I suppose I'm lucky I was on the cautious side!

Work Ethic
Not just ethic, but environment as a whole. I walked in to my office before starting work, and wanted to clarify what the dress code was. They told me there wasn't one. Naturally, I was 90% sure I hadn't understood, but after further questioning, I was sure. OK, so there was no explicit dress code, but I figured I should still wear slacks and a polo. Nope. People were there in shorts, jeans, t-shirts, all kinds of things. Naturally that really sucked, because I had already lugged my entire dress wardrobe across the ocean.

Work hours were also totally lax. I did a pretty standard 8:30 - 5:30. But some people showed up around lunch, and other left around 2pm. There were a couple Fridays where I was literally the only person in the office from 2pm - 6pm. I had a fixed 35 hour work week, but with flex hours. So I often worked late so that I could skip days and go on 5-day trips to other countries.

I never counted anyone else's hours, and of course they get actual vacation days, but I swear only half of the office was there on any given day. This isn't just because of time off though. A large percentage of the people there were either part-time, or work students. I think there were about 6 work-study students, and usually only 2 came in per day (They are restricted to 20 hour weeks during school).

Unfortunately, work-study students made a good deal more cash than I did...which was weird. They also had hourly rates instead of a fixed-hour week, which would have been nice for flexibility. So know that if you go to do an "Internship" in Germany, that means "Praktikum" should check and see if they can hire you as a work student!

Anyways, just some office observation I made. Sorry it's been so long since my last post. School has been incredibly busy, but I'm hoping to at least sneak a few posts in every now and then. Happy Autumn!

The Open-Source Experience (Pt. 4)

08/23/10 | by Charlie [mail] | Categories: Web, Technology

My Involvement

Thus far I?ve only talked about my work with CKEditor, however back in high school I was involved with another project called ALP (Autonomous LAN Party). I found it when I was searching for LAN party software and became interested in it. When I discovered it, a first version had been developed but had since fallen into disrepair. I made a few modifications and found another guy who had made a few modifications and together we took over the project and developed it for awhile. To this day, it still (more or less) works, although the last version was released about 4 years ago.

We got to the point where we had outgrown the original slopped together version and needed to do a total rewrite. Unfortunately, the prospect of so much work and the lack of people to do it caused the project to fall apart in the planning phase. Nevertheless, all our work is publicly available and maybe someday I or someone else will pick it up and keep going.

CKEditor, just finished a rewrite and is much better off for it. Although this means it?s not totally mature yet and has a few bugs still lurking around. I encountered some of these and they were easy enough problems that I could hop right in and fix them. Thus it was quite easy for me to get involved in the project.

The hard part, however, was running the support forums (I just tried to answer a lot of posts). I ended up fielding about 7 posts a day. Some of the problems were legitimate or valid debugging things. I helped people work out their configurations and do things I had been through when I first setup the editor. On the other hand, some were face palmingly stupid questions which could have been solved by 2 minutes of effort or simply using google. I often had very little patience in these topics.

My feeling is that you should at least be able to understand the software you are using and how it works before asking questions. There were people who simply needed to find a configuration option (Which is all given on a single page on the site?s support area). They just hadn?t looked around before asking. While those cases are stressful, it?s also nice to see the people who say ?oh thanks!? when they solve their problem. Those are the people who have actually worked through their problem before asking.

In any case, working on this project has given me a better understanding of the software, a better understanding of javascript, and made me feel like I?ve contributed to the project. Sure in a week I probably will never touch the project again, but that?s how it goes. Next month some new developer will have to use the software and will begin stomping bugs right where I left off.

The whole process is really an art though. There?s open-source software to manage open-source software, I.E. Trac. It?s the program almost everyone uses to ?trac? bugs and manage source code. Most interestingly though is the fact that everything has to be so well thought out. Since none of the developers ever really see each other, there?s a remote code review process, standards have to be employed, and the whole project has to be well documented so that at a moments notice someone new can step in and start contributing.

It?s a fantastic process and one that I really support. If anything, it?s made me want to assemble a team to redevelop ALP. In summary I guess the open-source ideal, to me, seems to define a certain type of social norm. The tragedy of the commons doesn?t apply because even by being selfish, people will still help the project; It's impossible to use up all the open-source resources. Rather than not supporting the project, people will use the software for free and, when they themselves find bugs or make changes, will send their changes back into the commons helping future users.

Nevertheless, there are things that cost money in these projects. There is also the matter of incentive to develop. So next time you find one of these projects useful, consider sending $10 to help support it's development. After all isn't that better than having to pay hundreds of dollars for some commercial software? Support community development, choose open-source!

The Open-Source Experience (Pt. 3)

08/20/10 | by Charlie [mail] | Categories: Web, Technology


In case you haven?t caught on to how open-source software works, we?re in the middle of a lengthy discussion about how it all works. Next stop, pros and cons. Let?s get the biggest pro out of the way. IT?S FREE. Horray. But there are, of course, some downsides too.

With commercial software, support and follow-through must be exemplary in order to maintain your customer base for future releases. However with open-source, because the software is free, the developers have very little incentive to provide technical support. The support for CKEditor was a little lackluster (non-existent), it?s also quite poorly documented. This made it very hard to figure out what I was doing wrong without digging around and debugging the code. A lot of people will opt for commercial software because it?s already been extensively tested and will (usually) function very well out-of-the-box.

CKEditor might be an exception in this regard however, since normally open source projects will document every single function. Why spend all that time writing wikis and APIs? Because the hope is that if the program is well documented, it will be easier for people to alter the software on their own and thus more contributions will come back to the program. Still, support is often the blind leading the blind.

The upside to glitches in open-source software is that they are usually fixed quickly and the software tends to have more releases than its commercial counterparts. In addition, customers usually have the option of three different types of releases. Stable, Beta, and Nightly. Stable is the equivalent of a commercial release. It's supposed to be extremely stable and polished. If, however, you want to have the newest features and bug-fixes sooner, you can opt for a beta version. These versions tend to have some kinks that still need working out, but generally function quite well. Finally, there's nightly. This is basically a copy of the source code (From the SVN, remember?) which is built automatically every night.

Even though nightly builds are supposed to be quite unstable, CKEditor usually has very good nightly packages. This is because all code changes are reviewed before they are committed, thus changes tend to work with 95% certainty as soon as they are in the code base.

Features can be another huge factor in choosing. Again though, it?s a two part problem. While commercial software might have more features initally, it?s usually easier to request new features in open-source software and to have those changes made. (Try asking Microsoft to add 16-bit support to 64-bit OS?s, I dare you).

Here?s the sticky part though. Remember way back when we talked about licensing? Well that means that any part of an open-source program can be reused (Under certain licenses). Thus Microsoft could, in theory, use some part of CKEditor for it?s own nefarious purposes, however the same is not true in reverse. Commercial companies can use these things called patents to exclude anyone else from using the software. Despite the short 10-year lifespan, this is an eternity in computer terms and effectively means that commercial applications can have features that no one else does.

In fact, this problem is most present in video codecs right now. Remember when I said that linux was 100% open-source? Well that is except for one video codec. The problem is that it?s commercial and thus doesn?t have to include its source code. This is also a problem for HTML5 video tags (New way of playing videos in browsers more or less), people cannot decide on a standard codec to use because one is commercial (and a bit better?) but one is open source. Thus, due to asymmetric information?this round goes to commercial. Nevertheless, more and more companies are starting to give back to the open-source community in various ways. So keep an eye on this tussle in the future.

I think now you can see why not everyone is using open-source software. It sometimes isn?t available, especially in niche areas where there wouldn?t be enough developers and sometimes it just doesn?t get the job done as well as commercial products. I still HIGHLY encourage you to give it a try! Take the OpenOffice challenge and try to use the free program for 6 months instead of Microsoft Office. You'll see that OpenOffice is really really good, but probably be so used to Microsoft that it will be hard to use.

Anyways, next time, we?ll get into my involvement and experiences with open-source projects.

Pages: << 1 2 3 4 5 6 7 8 9 10 11 ... 29 >>

A collection of musings from my time at Yale along with some thoughts about my "Freshman year of life" in San Francisco.


XML Feeds

blog software