Podcast Downloading on FreeNAS (Followup)
I've been following the referrals to my blog lately and I noticed the FreeNAS Podcast Downloader post was getting a lot of traffic but that the post wasn't descriptive enough in it's actual use. So I've got some instructions how to actually USE the project.
First off this will involve installing a package that isn't default for FreeNAS, if this worries you then skip down a step or two to the link to download all the project files manually. All this involves is installing subversion for checking out the code and all necessary files for the project to run. SSH into your FreeNAS server and execute this command:
1 | pkg_add -r subversion |
This will take a while to run so don't worry if it looks like it froze. You should reboot your server after this is finished for all the new settings to take effect. This is just so you can download the latest copy of the project.
Next step is to check out the project which can be done using the package you just installed. Choose a folder on your data partition for this to go because in an embedded install any other location would be overwritten on reboot. I made a directory in /mnt/Main/Content/.db/ (This is where I've chosen for all the databases for UPnP and DAAP to be stored along with the scripts for my server that I've written):
1 | svn checkout http://svn.xp-dev.com/svn/bemasher-FreeNASPodder/ FreeNASPodder |
This will check out the latest copy of the code to a new folder FreeNASPodder in /mnt/Main/Content/.db/
If you're uncomfortable with installing new packages to your server and just want a copy of the project you can simply browse to http://svn.xp-dev.com/svn/bemasher-FreeNASPodder and download each individual file manually. The benefit of using subversion is that getting the latest version of the file requires only browsing to the folder it resides in and executing:
1 | svn update |
Once this is all done you've checked out a working copy of the code. There are a few minor things you'll need to change in the script and in the configuration file. First in bashpodder.sh you'll need to change podcast_dir to the directory that you'd like your podcasts to be downloaded to, do the same in select_podcasts.sh. Make sure this folder exists as the script won't create it for you, it will only create folders for individual podcast feeds inside this folder. Be sure to escape spaces with a backslash.
The next change you'll need to make is in feeds.list. This will have a list of the url to each podcast feed you'd like to download episodes from. Be sure to keep one blank line at the very end of the file it will skip the last feed if you don't.
If you don't want to download all the current episodes all at once run select_podcasts.sh from the shell using:
1 | sh select_podcasts.sh |
This will create the basic folder structure that all of your podcasts will go into and compile episode.txt files for each podcast feed in their respective folders wherever you specified podcast_dir to be. To download a particular episode just remove the url for the episode you'd like to download from the episode.txt of the cooresponding podcast. Then simply run:
1 | sh bashpodder.sh |
If you'd like to execute this using cron in the web interface go to: System -> Advanced -> Cron. Click the add button and use the following command to check for and download new scripts however often you'd like:
1 | sh /path/to/FreeNASPodder/bashpodder.sh |
Followup:
Good news! I've moved all of my subversion repositories over to GitHub, the latest code for FreeNASPodder can be found at http://github.com/bemasher/FreeNASPodder. And even better news: you no longer have to install svn or download each final manually, GitHub has the option to download an archive in tgz or zip format of the latest source files.
Socks FTW!
Well, not the fabric-y variety of SOCKS, the proxy variety. My friend Pete mentioned at some point a few weeks ago that I was "doing it the wrong" when I was doing some SSH tunneling to my FreeNAS server.
The original setup involved tunneling port 80 and a few other random ports for things like the web interfaces of the DAAP and UPnP servers. Which was a bit of a pain to add each time I wanted to tunnel the ports on a new computer whether it was my laptop, desktop or workstation. I could save the profile and just use that though it was a pain to be thorough enough to include all the ports I'd need. At the time Pete mentioned something about a SOCKS proxy which I didn't really understand and originally thought it would involve installing new software somewhere which I didn't really want to do at the time.
After a little bit of research later I found that one of the most common SSH servers OpenSSH has a built in SOCKS proxy. This is great because it meant installing no new software or even any new configurations for that matter. In Putty I discovered it was as simple as adding a dynamic port of my choice (1080 is probably best here since it's the default SOCKS proxy port). Once I did this I installed FoxyProxy plugin for firefox and was on my way to a quick and simple proxy to any service I desired on my FreeNAS server.
To do this from a command line is just as simple:
1 | ssh -D1080 user@example.com |
A quick way to test this is to go to a "what is my IP" site of some variety then connect to the server with the SOCKS proxy enabled and setup your browser with that proxy and reload the page, if your external IP changes to the external IP of the server you just ssh'd into then you've done it.
Power Set Generator
Recently I had a bout of programming withdrawal so I set out to write a power-set generator.
So a little background on the power set. The power set of a given set A is the set containing all subsets of A. Suppose that we have a set:
The power set of A would be:
Now looking more carefully at the power set of A you'll notice that it contains 2 to the power of the cardinality of A subsets, always containing the empty set.
The code I came up with for this is short and more or less simple:
1 2 3 4 5 6 7 8 9 10 | def PowerSet(base): power_set = [] b = len(base) map(lambda g: power_set.append(map(lambda x: base[x], filter(lambda x: g[x], range(0, b)))), map(lambda value: map(lambda x: (value >> x) & 1, range(b - 1, -1, -1)), map(lambda value: value ^ (value / 2), range(0, 2**b)))) return power_set print PowerSet([1,2,3]) |
I figured out shortly after I wrote this that I had the right general idea but in this particular case, since I'm not actually using set types... the graycode I'm generating is useless for this sort of thing.
The point of generating graycode for this is that graycode is used for binary counting such that only one digit is changed from one consecutive value to the next. It was originally designed so that mechanical switches in early computers wouldn't cause a race condition while counting fast enough.
In this particular solution using graycode is useful for only having to add one new element to a set at a time which if I were actually using sets, would be faster, but since I'm not, it isn't. I'll probably rewrite it later to make it play nicer with graycode.
The basic procedure here is that given a set A we're going to count from 0 to 2^|A| - 1 and in binary graycode, the 1's determine the elements of the base set that will be added as a new set to the power set and 0's indicate that that element of the base set will be ignored in the current subset.
Free Subversion Hosting
I've been in a very coding-prone mood lately. I've been working on developing a Python ETR (Employee Time Record) script for a friend and his club here at the University of Arizona. The project has grown significantly since I started it and this is one of the first projects in a while that I've developed for someone to use other than myself and I've been wanting a way to manage my code better.
I did some searching for free subversion hosting. I've seen google code hosting before and looked at it's feature set, which is quite complete. In terms of project management google code hosting is probably the best for my needs. Though after reading through more of their help//support section I discovered that there's a maximum project creation limit of 10. Supposedly you can email support at the google code hosting service and work out a deal to get more than 10 projects but that's really a hassle. So I started looking elsewhere. Currently though the PythonETR script is hosted at google code.
I stumbled upon ProjectLocker which looked really promising. It turned out to be a very well put together system you get 300MB of storage and unlimited subversion repositories along with Trac instances for each repository. There's just a big HOWEVER in the middle of what seems to be an awesome service. The however is that there is NO public anonymous subversion access. If you want your projects to be available there's no way for you to allow the public to check out a read-only copy of your project. They also only allow a maximum of 2 user accounts and you count as one of them.
Once I discovered all the limitations of ProjectLocker I kept on searching. The next promising service I found was XP-Dev. XP-Dev pretty much one-up's ProjectLocker on just about everything except a few crucial parts. There's no Trac, they provide their own "project tracking" tools like: stories, blogs, wikis, bugs. There's no real way to associate projects with subversion repositories on this service. You get 1.5GB of storage and as many subversion repositories//projects you want to fill that up with. If all you're looking for is free subversion hosting with no project tracking then this is probably the service for you. Though if you're even the slightest bit paranoid about crypto this service definitely isn't for you. Most of the defaults are for non-SSL connections and the services that allow SSL use a self-signed certificate by XP-Dev. I did discover that they do allow public read-only access to subversion repositories but only if you choose to enable this feature so for private projects you don't have to make them publicly available.
Podcast Downloading on FreeNAS.
I was thinking earlier this week about how files move too and from my file-server. I discovered that it's more convenient to have the server "pull" files to itself. This actually made me think a little bit more about podcast shows I watch on a pretty regular basis. I thought to myself "Wouldn't it be great if my server would get new episodes for me?". Once all that's done I can easily pull up the Podcasts directory on my friend's PS3 (which this will broadcast to using UPnP DLNA) and watch any new episodes that happen to be there.
I then set out almost immediately to figure out the simplest way to do this. I stumbled upon a script called BashPodder written by Linc. So I figured I'd just give it a try in it's original state. That turned out badly since I discovered that this was meant specifically for linux-based systems. My file-server runs FreeNAS a FreeBSD based OS. FreeBSD has an equivalent but different set of basic system tools. Fetch instead of wget, xml instead of xsltproc and so on.
After about 8 hours of learning my way around the basic set of FreeBSD tools and just generally refreshing myself on shell scripting I had heavily modified the BashPodder script to work on FreeNAS.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | #!/bin/bash cd "`dirname "$0"`" script_dir=`pwd` podcast_dir=/mnt/Main/Content/TV\ Shows/Podcasts/ while read podcast do cd "$script_dir" file=$(fetch -q -o - $podcast | xml tr parse_enclosure.xsl) cd "$podcast_dir" data_dir=$(fetch -q -o - $podcast | xml sel -T -t -v "//channel/title" | sed "s/ (.*)//") echo \"$data_dir\" mkdir -p "$data_dir" chown ftp:wheel "$data_dir" cd "$data_dir" for url in $file do filename=`echo $url | sed "s/^\(.*\)*\///g"` echo $filename if [ -z "`grep $url "episodes.txt"`" ]; then fetch -m -q $url echo $url >> "episodes.txt" if [ -n "`echo $filename | grep -v ".m4v"`" ]; then mv $filename `basename $filename m4v`mp4 chown ftp:wheel `basename $filename m4v`mp4 else chown ftp:wheel $filename fi fi done done < feeds.list /etc/rc.d/fuppes updatedb |
I think the only thing I didn't end up changing at all was the xml stylesheet Linc had written since it works perfectly. The basic flow of this program is thus:
- Move into the directory that the script exists in.
- For each line in feeds.list do the following.
- Move into the scripts directory (there are files we need to parse the feed here).
- Get the rss feed.
- Parse out the title of the feed.
- Parse the urls for each episode in the feed.
- Move into the podcast directory.
- Make a directory of the same name as the title if necessary.
- Move into the directory we just made (or already exists).
- For each url we parsed do the following.
- Determine the filename from the url.
- If the episode is not in episodes.txt do the following.
- Download the episode.
- Add the episode to episodes.txt
- Change the owner to ftp. (We're running this as root).
- Update the Fuppes database.
After all that was done and over with I just added a new file extension to the Fuppes (UPnP DLNA server) configuration for telling the PS3 that m4v files are really just mp4 files.
Also I wrote a slightly modified version of the above script for just getting lists of episodes. This is useful because if I don't want to download every single episode in each feed (for when I add new feeds to the feeds list) all I have to do is remove the urls from the episodes.txt file that is generated and run the main script. The modification is done in the for loop that handles episodes of each feed:
1 2 3 | filename=`echo $url | sed "s/^\(.*\)*\///g"` echo $filename grep -q $url "episodes.txt" || echo $url >> "episodes.txt" |
So instead of downloading each file it just makes sure it's in the episodes.txt file. If you've got any suggestions for optimization of the script let me know!
If you'd like to check out a read-only copy of the script and necessary configurations:
1 | svn checkout http://svn.xp-dev.com/svn/bemasher-FreeNASPodder/ FreeNASPodder |
Keep in mind that the subversion is the most up-to-date copy I'll have available and it may often contain broken code or errors though I'll try to keep that down to a minimum.
Python ETR and Google Code Hosting
Earlier tonight I was talking to Nick a friend of mine who I've been working with recently on a ETR script. He's one of the webmasters for the University of Arizona Baja Racing Club.
I had already written my own ETR script for easy entry of time records from my google calendar to the ETR form website at my job. So I figured that i could just as easily modify it to work for him.
The main structure for his particular needs will be a main google account which the script will authenticate with. All of the users that he wants to keep time records for will simply create and share a calendar with that account. Then when the script is run it will simply authenticate with the gdata api and retrieve a list of events and their descriptions for each shared calendar on the account. All of this data is then written to a csv file of the same name of the calendar along with the work week it was done in.
The script is more or less in proof-of-concept stage and still needs a lot of polishing but in the middle of doing this I found myself wanting to have a way to organize changes and revisions. I've briefly used subversion before this and never really made a habit of using it even though I should have. I suddenly remembered that Google Code Hosting provides subversion access to open source projects. So I went ahead and made a project of the name pythonetr.
It took about 15 minutes for me to download and install TortoiseSVN, a windows shell integrated gui for SVN. I then imported the first revision of the project to the subversion repository of the project on google code. After that I sent Nick a link to the page so that if there were any feature requests, bugs anything he wanted me to know about for the project he could just submit them as issues there.
Shortly after that I made some modifications to the code since the gdata api was only returning events between specified dates only if their cooresponding UTC time was within the date range. This breaks things if you expect only events between the date ranges that are returned are within your timezone. So I made all the changes and committed the new revision.
It's really satisfying to have a personal record of all changes made to code and I really should do this more often. I'd upload all my other projects but I believe there's a lifetime limit of 10 projects total for google code hosting.
If you're interested in checking out a copy of the project, you can do so with the following command:
1 | svn checkout http://pythonetr.googlecode.com/svn/trunk/ pythonetr-read-only |
It’s about time.
Something interesting is finally here. Looks like we may have started on melting the wicked witch of the west. With the economy sucking hardcore and all the major piracy lawsuits going. The RIAA is losing a lot of employees to layoffs AND on top of that North Carolina is suing them for among other things legal abuse.
QR Code
Blogroll
Comics
Info
Archives
Recommended Reading
- Spooky Men's Chorale - "Deep"
- Spooky Men's Chorale at the Holywell Music Rooms
- Simple Swedish Folk Song - The Spooky Men's Chorale
- Flying YouTube Packets Proves Visualization Can Make Anything Pretty [Visualization]
- uTorrent Linux Server Released, Client Coming Soon
- Full Metal Disney
- Neal Stephenson Unveils His Digital Novel Platform
- Panasonic's New 8GB and 16GB SDHC Cards Transfer 2x Faster [Storage]
- SweetSearch Finds Credible Research Sources for Students [Research]
- Test-Firing of 2000 Volt Washer Launcher a Success [Inventions]